Support an arbitrary CSV delimiter by zhkvia · Pull Request #2263 · ClickHouse/ClickHouse

zhkvia · 2018-04-22T14:35:32Z

Added a format_csv_delimiter setting for specifying an arbitrary CSV delimiter.
How it can be used:

As a client argument:

$ clickhouse-client --format_csv_delimiter=";" --query="INSERT INTO table FORMAT CSV"

As a session setting:

:) SET format_csv_delimiter=';'                                                                                                        │

SET format_csv_delimiter = ';'

Ok.

:) SELECT * FROM table FORMAT CSV

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

alexey-milovidov · 2018-04-23T00:44:17Z

+private:
+    void checkStringIsACharacter(const String & x) const {
+        if (x.size() != 1)
+            throw Exception(std::string("A setting's value string has to be an exactly one character long"));


It's not necessary to construct std::string explicitly.

alexey-milovidov · 2018-04-23T00:44:29Z

+private:
+    void checkStringIsACharacter(const String & x) const {
+        if (x.size() != 1)
+            throw Exception(std::string("A setting's value string has to be an exactly one character long"));


Missing ErrorCodes.

Style. Braces {} should be in a separate new line.

alexey-milovidov · 2018-04-23T00:45:41Z

+
+    void set(const Field & x)
+    {
+        String s = safeGet<const String &>(x);


We can use reference here to avoid copying.

alexey-milovidov · 2018-04-23T00:50:14Z

+When formatting, rows are enclosed in double quotes. A double quote inside a string is output as two double quotes in a row. There are no other rules for escaping characters. Date and date-time are enclosed in double quotes. Numbers are output without quotes. Values are separated by a delimiter&ast;. Rows are separated using the Unix line feed (LF). Arrays are serialized in CSV as follows: first the array is serialized to a string as in TabSeparated format, and then the resulting string is output to CSV in double quotes. Tuples in CSV format are serialized as separate columns (that is, their nesting in the tuple is lost).

-When parsing, all values can be parsed either with or without quotes. Both double and single quotes are supported. Rows can also be arranged without quotes. In this case, they are parsed up to a comma or line feed (CR or LF). In violation of the RFC, when parsing rows without quotes, the leading and trailing spaces and tabs are ignored. For the line feed, Unix (LF), Windows (CR LF) and Mac OS Classic (CR LF) are all supported.
+&ast;By default — `,`. See a [format_csv_delimiter](/docs/en/operations/settings/settings/#format_csv_delimiter) setting for additional info.


Are you sure that Markdown supports HTML entities?

It seems logical since markdown is compiled into HTML. Also, mkdocs in the docs/ directory compiles it correctly.

alexey-milovidov · 2018-04-23T00:53:20Z

Almost everything is Ok, but tests are missing.
You may use simple functional tests (look at dbms/tests/queries directory).
Both input and output should be tested.
Please add test cases when unquoted string use comma and delimiter is a semicolon or something like this:
abc,def;hello

amosbird · 2018-04-24T09:52:02Z

Hmm, can we still call that a CSV format without comma? I assume a new format with some delimiter argument would be a better choice. And there are some use cases of multi-bytes delimiters such as data exported from Netezza.

alexey-milovidov · 2018-04-27T20:17:40Z

Hmm, can we still call that a CSV format without comma? I assume a new format with some delimiter argument would be a better choice.

Yes, it is controversial. In fact, many "CSV readers" are configurable in this way.
We may add a format with name 'DSV', but CSV is also Ok.

And there are some use cases of multi-bytes delimiters such as data exported from Netezza.

Do they use multibyte delimiter by default?

amosbird · 2018-04-28T02:26:00Z

Do they use multibyte delimiter by default?

Nope. The default is |.

hereTac · 2018-06-22T01:54:07Z

does --format_csv_delimiter= support in v1.1.54385-stable?
Errors:
Bad arguments: unrecognised option '--format_csv_delimiter= '

AntonSaykovsky · 2018-07-17T13:15:43Z

does --format_csv_delimiter= support in v1.1.54385-stable?
Errors:
Bad arguments: unrecognised option '--format_csv_delimiter= '

1.1.54390 version has the same error. =(((

zhkvia added 7 commits April 22, 2018 01:20

Add a CSV delimiter setting

1fc6bb6

Add delimiter support in CSVRowInputStream

91cb03b

Add CSV delimter support in CSVRowOutputStream

a138ab2

Add a SettingChar type

e84263e

Unify string checks with a new type

e402ff3

Update docs about a CSV format

4db476b

Propagate a CSV delimiter to non-basic DataTypes

8cb4539

zhkvia force-pushed the arbitrary-csv-delimiter branch from 23c6017 to 8cb4539 Compare April 22, 2018 17:30

alexey-milovidov reviewed Apr 23, 2018

View reviewed changes

zhkvia and others added 3 commits April 26, 2018 20:41

Code review

4094e21

Add a test for CSV input and output formatting

a9653e5

Update SettingsCommon.h

96e2dfa

alexey-milovidov merged commit 093c054 into ClickHouse:master Apr 27, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support an arbitrary CSV delimiter#2263

Support an arbitrary CSV delimiter#2263
alexey-milovidov merged 10 commits intoClickHouse:masterfrom
zhkvia:arbitrary-csv-delimiter

zhkvia commented Apr 22, 2018

Uh oh!

alexey-milovidov Apr 23, 2018

Uh oh!

alexey-milovidov Apr 23, 2018

Uh oh!

ludv1x Apr 23, 2018

Uh oh!

alexey-milovidov Apr 23, 2018

Uh oh!

alexey-milovidov Apr 23, 2018

Uh oh!

zhkvia Apr 23, 2018

Uh oh!

alexey-milovidov commented Apr 23, 2018

Uh oh!

amosbird commented Apr 24, 2018 •

edited

Loading

Uh oh!

alexey-milovidov commented Apr 27, 2018

Uh oh!

amosbird commented Apr 28, 2018 •

edited

Loading

Uh oh!

hereTac commented Jun 22, 2018

Uh oh!

AntonSaykovsky commented Jul 17, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

zhkvia commented Apr 22, 2018

Uh oh!

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

Uh oh!

ludv1x Apr 23, 2018

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Apr 23, 2018

Choose a reason for hiding this comment

Uh oh!

zhkvia Apr 23, 2018

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov commented Apr 23, 2018

Uh oh!

amosbird commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Apr 27, 2018

Uh oh!

amosbird commented Apr 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hereTac commented Jun 22, 2018

Uh oh!

AntonSaykovsky commented Jul 17, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

amosbird commented Apr 24, 2018 •

edited

Loading

amosbird commented Apr 28, 2018 •

edited

Loading