Configuration of export CSV is inconsistent with the docs

[szarnyasg]

Using this single node as the sample dataset:

CREATE (:SomePerson {id: 1, name: 'John'})

Exporting to CSV does not work as specified in the docs (Table 8 in Export/Import):

CALL apoc.export.csv.query('MATCH (x:SomePerson) RETURN x.id, x.name', 'file.csv', {delim: ';'})

This should give a CSV with semicolons and without quotes. Instead, it gives:

"x.id","x.name"
"1","John"

After diving into the code a bit, I realized that I should look around the ExportConfig class:

neo4j-apoc-procedures/src/main/java/apoc/export/util/ExportConfig.java

Lines 59 to 70 in 37034ba

	public ExportConfig(Map<String,Object> config) {
	config = config != null ? config : Collections.emptyMap();
	this.silent = toBoolean(config.getOrDefault("silent",false));
	this.batchSize = ((Number)config.getOrDefault("batchSize", DEFAULT_BATCH_SIZE)).intValue();
	this.delim = delim(config.getOrDefault("d", String.valueOf(DEFAULT_DELIM)).toString());
	this.quotes = toBoolean(config.get("quotes"));
	this.useTypes = toBoolean(config.get("useTypes"));
	this.nodesOfRelationships = toBoolean(config.get("nodesOfRelationships"));
	this.format = ExportFormat.fromString((String) config.getOrDefault("format", "neo4j-shell"));
	this.cypherFormat = CypherFormat.fromString((String) config.getOrDefault("cypherFormat", "create"));
	this.config = config;
	}

This suggests that the delimiter is defined by d and indeed it is:

CALL apoc.export.csv.query('MATCH (x:SomePerson) RETURN x.id, x.name', 'file.csv', {d: ';'})

This results is:

"x.id";"x.name"
"1";"John"

Quotes are supposed to be defined by the quotes boolean library, but I could not turn them off with quotes: false.

[jexp]

@szarnyasg Would you mind fixing both (support d and delim) and fix the quotes config
if you are looking at the code already?

[szarnyasg]

@jexp I'm looking into it this morning. It seems there are multiple issues: the useTypes field does not work either:

CALL apoc.export.csv.query(
  "MATCH (u:User) return u.age, u.name, u.male, u.kids, labels(u)",
  "query.csv",
  {useTypes: true, d: '|', quotes: false}
)

"u.age"|"u.name"|"u.male"|"u.kids"|"labels(u)"
"42"|"foo"|"true"|"[""a"",""b"",""c""]"|"[""User""]"
"42"|"bar"|""|""|"[""User""]"
"12"|""|""|""|"[""User""]"

[szarnyasg]

I looked into it. The delimiter issue was trivial to fix. However, without the quotes, the opencsv 2.3 library does not work correctly when quotes are turned off: the separator character is not escaped.

The opencsv 2.3 library was released in 2011, so I was suspicious that this issue has been fixed since then. To demonstrate this, I created an example project that has both opencsv 2.3 (net.sf.opencsv) and 4.0 (com.opencsv) as a dependency: https://github.com/szarnyasg/opencsv-test/tree/master/src/test/java

The results show the difference.

==> /tmp/my23.csv <==
Foo Bar,Bar, F.,123

==> /tmp/my40.csv <==
Foo Bar,Bar\, F.,123

@jexp do you think it would be possible to update the opencsv dependency in APOC?

[szarnyasg]

By the way, I checked the dependency tree: https://gist.github.com/szarnyasg/79f4e612169dccd86232ffd793107375

opencsv is a transitive dependency of org.neo4j:neo4j-cypher, defined here: https://github.com/neo4j/neo4j/blob/3128665316c75295d156511098ae11b5634700de/pom.xml#L1030 and here: https://github.com/neo4j/neo4j/blob/3128665316c75295d156511098ae11b5634700de/community/cypher/cypher/pom.xml#L341

[szarnyasg]

See PR #879

[szarnyasg]

Eventually, PR neo4j/neo4j#12009 was closed as wontfix, so this issue will require another round of thinking.

[jexp]

@mneedham we need to expose the config options of export csv in the docs (see above).
Esp. quoting / escaping seems important.