Support an arbitrary CSV delimiter #2263

zhkvia · 2018-04-22T14:35:32Z

Added a format_csv_delimiter setting for specifying an arbitrary CSV delimiter.
How it can be used:

As a client argument:

$ clickhouse-client --format_csv_delimiter=";" --query="INSERT INTO table FORMAT CSV"

As a session setting:

:) SET format_csv_delimiter=';'                                                                                                        │

SET format_csv_delimiter = ';'

Ok.

:) SELECT * FROM table FORMAT CSV

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

alexey-milovidov · 2018-04-23T00:44:17Z

dbms/src/Interpreters/SettingsCommon.h

+private:
+    void checkStringIsACharacter(const String & x) const {
+        if (x.size() != 1)
+            throw Exception(std::string("A setting's value string has to be an exactly one character long"));


It's not necessary to construct std::string explicitly.

alexey-milovidov · 2018-04-23T00:44:29Z

dbms/src/Interpreters/SettingsCommon.h

+private:
+    void checkStringIsACharacter(const String & x) const {
+        if (x.size() != 1)
+            throw Exception(std::string("A setting's value string has to be an exactly one character long"));


Missing ErrorCodes.

Style. Braces {} should be in a separate new line.

alexey-milovidov · 2018-04-23T00:45:41Z

dbms/src/Interpreters/SettingsCommon.h

+
+    void set(const Field & x)
+    {
+        String s = safeGet<const String &>(x);


We can use reference here to avoid copying.

alexey-milovidov · 2018-04-23T00:50:14Z

docs/en/formats/csv.md


-When parsing, all values can be parsed either with or without quotes. Both double and single quotes are supported. Rows can also be arranged without quotes. In this case, they are parsed up to a comma or line feed (CR or LF). In violation of the RFC, when parsing rows without quotes, the leading and trailing spaces and tabs are ignored. For the line feed, Unix (LF), Windows (CR LF) and Mac OS Classic (CR LF) are all supported.
+&ast;By default — `,`. See a [format_csv_delimiter](/docs/en/operations/settings/settings/#format_csv_delimiter) setting for additional info.


Are you sure that Markdown supports HTML entities?

It seems logical since markdown is compiled into HTML. Also, mkdocs in the docs/ directory compiles it correctly.

alexey-milovidov · 2018-04-23T00:53:20Z

Almost everything is Ok, but tests are missing.
You may use simple functional tests (look at dbms/tests/queries directory).
Both input and output should be tested.
Please add test cases when unquoted string use comma and delimiter is a semicolon or something like this:
abc,def;hello

amosbird · 2018-04-24T09:52:02Z

Hmm, can we still call that a CSV format without comma? I assume a new format with some delimiter argument would be a better choice. And there are some use cases of multi-bytes delimiters such as data exported from Netezza.

alexey-milovidov · 2018-04-27T20:17:40Z

Hmm, can we still call that a CSV format without comma? I assume a new format with some delimiter argument would be a better choice.

Yes, it is controversial. In fact, many "CSV readers" are configurable in this way.
We may add a format with name 'DSV', but CSV is also Ok.

And there are some use cases of multi-bytes delimiters such as data exported from Netezza.

Do they use multibyte delimiter by default?

amosbird · 2018-04-28T02:26:00Z

Do they use multibyte delimiter by default?

Nope. The default is |.

hereTac · 2018-06-22T01:54:07Z

does --format_csv_delimiter= support in v1.1.54385-stable?
Errors:
Bad arguments: unrecognised option '--format_csv_delimiter= '

AntonSaykovsky · 2018-07-17T13:15:43Z

does --format_csv_delimiter= support in v1.1.54385-stable?
Errors:
Bad arguments: unrecognised option '--format_csv_delimiter= '

1.1.54390 version has the same error. =(((

zhkvia added 7 commits April 22, 2018 01:20

Add a CSV delimiter setting

1fc6bb6

Add delimiter support in CSVRowInputStream

91cb03b

Add CSV delimter support in CSVRowOutputStream

a138ab2

Add a SettingChar type

e84263e

Unify string checks with a new type

e402ff3

Update docs about a CSV format

4db476b

Propagate a CSV delimiter to non-basic DataTypes

8cb4539

zhkvia force-pushed the arbitrary-csv-delimiter branch from 23c6017 to 8cb4539 Compare April 22, 2018 17:30

alexey-milovidov reviewed Apr 23, 2018

View reviewed changes

zhkvia and others added 3 commits April 26, 2018 20:41

Code review

4094e21

Add a test for CSV input and output formatting

a9653e5

Update SettingsCommon.h

96e2dfa

alexey-milovidov merged commit 093c054 into ClickHouse:master Apr 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support an arbitrary CSV delimiter #2263

Support an arbitrary CSV delimiter #2263

zhkvia commented Apr 22, 2018

alexey-milovidov Apr 23, 2018

alexey-milovidov Apr 23, 2018

ludv1x Apr 23, 2018

alexey-milovidov Apr 23, 2018

alexey-milovidov Apr 23, 2018

zhkvia Apr 23, 2018

alexey-milovidov commented Apr 23, 2018

amosbird commented Apr 24, 2018 •

edited

Loading

alexey-milovidov commented Apr 27, 2018

amosbird commented Apr 28, 2018 •

edited

Loading

hereTac commented Jun 22, 2018

AntonSaykovsky commented Jul 17, 2018


		When parsing, all values can be parsed either with or without quotes. Both double and single quotes are supported. Rows can also be arranged without quotes. In this case, they are parsed up to a comma or line feed (CR or LF). In violation of the RFC, when parsing rows without quotes, the leading and trailing spaces and tabs are ignored. For the line feed, Unix (LF), Windows (CR LF) and Mac OS Classic (CR LF) are all supported.
		&ast;By default — `,`. See a [format_csv_delimiter](/docs/en/operations/settings/settings/#format_csv_delimiter) setting for additional info.

Support an arbitrary CSV delimiter #2263

Support an arbitrary CSV delimiter #2263

Conversation

zhkvia commented Apr 22, 2018

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

ludv1x Apr 23, 2018

Choose a reason for hiding this comment

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

zhkvia Apr 23, 2018

Choose a reason for hiding this comment

alexey-milovidov commented Apr 23, 2018

amosbird commented Apr 24, 2018 • edited Loading

alexey-milovidov commented Apr 27, 2018

amosbird commented Apr 28, 2018 • edited Loading

hereTac commented Jun 22, 2018

AntonSaykovsky commented Jul 17, 2018

amosbird commented Apr 24, 2018 •

edited

Loading

amosbird commented Apr 28, 2018 •

edited

Loading