Importer for 0.8.9 data via the CLI

[corylanou]

This PR allows you to import an exported file from 0.8.9 (#3477).

Caveats

For the export/import to work, all requisites have to be met. For export, all series names in 0.8 should be in the following format:

<tagName>.<tagValue>.<tagName>.<tagValue>.<measurement>

for example:

az.us-west-1.host.serverA.cpu

or any number of tags

building.2.temperature

Additionally, the fields need to have a consistent type (all float64, int64, etc) for every write in 0.8. Otherwise they have the potential to fail writes in the import. See below for more information.

Running the import command

To import via the cli, you can specify the following command:

influx -import -path=metrics-default.gz -compressed

If the file is not compressed you can issue it without the -compressed flag:

influx -import -path=metrics-default

To redirect failed import lines to another file, run this command:

influx -import -path=metrics-default.gz -compressed > failures

It will import using the line protocol in batches of 5,000 lines per batch.

Understanding the results of the import

The batch will give some basic stats when finished:

2015/07/29 23:15:20 Processed 2 commands
2015/07/29 23:15:20 Processed 70207923 inserts
2015/07/29 23:15:20 Failed 29785000 inserts

Most inserts fail due to the following types of error:

2015/07/29 22:18:28 error writing batch:  write failed: field type conflict: input field "value" on measurement "metric" is type float64, already exists as type integer

This is due to the fact that in 0.8 a field could get created and saved as int or float types for independent writes. In 0.9 the field has to have a consistent type.

[beckettsean]

@corylanou how are errors reported? STDOUT?

[corylanou]

@beckettsean no, STDERR... get it? errors ... :-)

[beckettsean]

but of course...

:dunce_hat: should be an emoji, I need one.

[dgnorton]

Could the path just require http://... or https://... and then no need for ssl bool?

[corylanou]

It could, but then you have to parse and validate, and if they don't give you either, you aren't sure what they wanted (you could default to http, but that might not be right). Making them explicitly choose makes it their responsibility to get right. Otherwise, if they don't provide either, it's now on us to try to make the right decision without enough information.

[dgnorton]

I was thinking that http or https would be required but no strong feeling here.

[pauldix]

Looks mostly good. The one other thing I think would be good to add is to output any of the lines that fail to parse into a new file. That way the developer can write their own custom script to handle just those data points.

And if there are write timeouts that just can't be overcome, to output all the timed out data to a different file.

[benbjohnson]

Does the DDL and DML need to be processed in order? Right now it sends it to goroutines that are running concurrently and it seems like they could be run out-of-order.

[corylanou]

Yeah, I might of over engineered that a little. I might refactor that to straight method calls instead after seeing the final product.

[corylanou]

@pauldix none of the lines can fail to parse as they were written out by the same line protocol code we use to ingest. I could check for it, but imo that would be the equivalent of a corrupt input file at that point.

Do you want them to specify a file for "failed writes" that we write to? Otherwise I can write failed writes to STDOUT and all other info to STDERR and they can just pipe failed writes to any file they want.

[pauldix]

@corylanou oh right, never mind what I said about the parse thing. For failed writes I think let them specify a file to output to? That way they can just try again and point it at that file.

[benbjohnson]

You can drop the else since the if block will exit.

[corylanou]

@benbjohnson ok, I think I have your comments addressed. Please take another look when you get a chance. Thanks!

[corylanou]

@pauldix this now writes all failed lines to STDOUT, and messages to STDERR so the end users can do what they want depending on what they care about.

[pauldix]

+1, just give @beckettsean the invocation example that outputs the failed writes to another file.

[beckettsean]

@corylanou is there any way to throttle or back off on the import? Most users will be running the import against a live 0.9.3 server.