Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pref_csv_separator is ignored with save statement for csv files #3817

Closed
Yapper689 opened this issue May 25, 2023 · 5 comments
Closed

pref_csv_separator is ignored with save statement for csv files #3817

Yapper689 opened this issue May 25, 2023 · 5 comments
Assignees
Labels
About Data Data files, data exportation, databases 🤗 Enhancement This is a request for enhancement 👍 Fix to be tested
Milestone

Comments

@Yapper689
Copy link

Describe the bug
List is not saved correctly in the CSV file, with each element written to a separate cell instead of all elements in one cell.

To Reproduce
GAMA model

model bug

global {
	
	list<int> closed_roads <- [1,2,3,4,5];
	int max_aqi <- 1;
	int mean_aqi <- 1;
	
	reflex save_results when: ( time > 0 ) {
		save [closed_roads, max_aqi, mean_aqi] to: "bug.csv" format: "csv" rewrite: false header: true  ;
	}
	 
}
	
experiment 'exp' {}
@lesquoyb lesquoyb added the 😱 Bug The issue reveals a bug in GAMA label May 26, 2023
@lesquoyb
Copy link
Collaborator

To be clearer here is an example of the files produced by this model:

closed_roads,max_aqi,mean_aqi
[1,2,3,4,5],1,1
[1,2,3,4,5],1,1

The problem is that the delimiter for the array and the columns are the same ( , ), and to my knowledge there's no way of setting the delimiter when writing csv.
The file is still understandable and could be parsed by an external script, but it can't be loaded correctly by gama operators, for example this:

file f <- csv_file("bug.csv", true);
		loop el over: f {
            write el;
        }

Raises an outofboundsexception.
And not even the csv viewer of gama can read it correctly:
image

@AlexisDrogoul
Copy link
Member

I have to say that there is no such thing as "saving a list inside a list" in the CSV format, at least no shared protocol and format to do so (contrary to other formats like JSON, which are based on nested data structures). This means (1) what you obtain is not a bug, just the normal result of saving a list inside a list; (2) you have to come up with your own way of encoding/decoding this list so that you can read it again. If the list is of fixed length, it is super easy, otherwise it might be a bit more complex (but not more complicated than what you would have to do in other languages).

@AlexisDrogoul AlexisDrogoul added About Data Data files, data exportation, databases ❌ Not an issue Wrongly opened as an issue or a bug and removed 😱 Bug The issue reveals a bug in GAMA labels May 31, 2023
@AlexisDrogoul AlexisDrogoul added this to the GAMA 1.9.2 milestone May 31, 2023
@AlexisDrogoul AlexisDrogoul self-assigned this May 31, 2023
@lesquoyb
Copy link
Collaborator

lesquoyb commented Jun 6, 2023

Yes the resulting file is not bugged, it's just very inconvenient and there's no way to produce a different csv with the current save statement.
Can't we add a delimiter facet so we can open the csv file directly in gaml using the csv_file operator instead of having to write a complete parser ?

@lesquoyb lesquoyb added the 🤗 Enhancement This is a request for enhancement label Jun 6, 2023
@lesquoyb
Copy link
Collaborator

lesquoyb commented Jun 6, 2023

I tried a workaround suggested by @AlexisDrogoul and found out that the csv_parameter in gama preferences is ignored:

model bug

global {
	
	list<int> closed_roads <- [1,2,3,4,5];
	int max_aqi <- 1;
	int mean_aqi <- 1;
	
	 
}
	
experiment 'exp' {
	
	init {
		gama.pref_csv_separator <- "*";
	}
	
	reflex save_results  {
		save [closed_roads, max_aqi, mean_aqi] to: "bug.csv" format: "csv" rewrite: false header: true  ;
		file f <- csv_file("bug.csv","*" ,true);
		loop el over: f {
            write el;
        }
	}
	
}

The generated file still uses , as a separator. I tried different separator values without any difference. I also try to change it from the preference window with the same results.

@lesquoyb lesquoyb changed the title Issue saving a list in CSV file pref_csv_separator is ignored with save statement for csv files Aug 7, 2023
@lesquoyb lesquoyb removed the ❌ Not an issue Wrongly opened as an issue or a bug label Aug 7, 2023
lesquoyb added a commit that referenced this issue Aug 17, 2023
please check I didn't break something
@lesquoyb
Copy link
Collaborator

Fixed with commit bfa98eb0a0a10c599d76ed8724f9e99cf6dda7b4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
About Data Data files, data exportation, databases 🤗 Enhancement This is a request for enhancement 👍 Fix to be tested
Projects
No open projects
Status: Done
Development

No branches or pull requests

3 participants