Fix appending to a sqlite table from a CSV with a header #77
Conversation
… copy of the file
|
|
||
| statement = """ | ||
| (echo '.mode csv'; echo '.import {abspath} {tblname}';) | sqlite3 {dbpath} | ||
| pipe=$(mktemp -t pipe) && rm -f $pipe && mkfifo -m 600 $pipe && (tail +{n} {abspath} > $pipe &) && (echo '.separator {delim}'; echo ".import $pipe {tblname}";) | sqlite3 {dbpath} |
There was a problem hiding this comment.
The first statement is probably better if there isn't a header. Also we'll need to be sensitive to windows.
There was a problem hiding this comment.
We raise an MDNotImplementedError() if os.name == 'nt', in the execution function
There was a problem hiding this comment.
Do we still get fast loads on windows in the no-header case?
There was a problem hiding this comment.
no, that would cause us to search for a new path, which would likely be through Iterator -> sa.Table.insert()
There was a problem hiding this comment.
i think keeping the first statement around is just going to be a bigger maintenance burden in the future. any particular reason you want to keep it?
If the user is on windows and if there is no header then it is useful and fast. Could code be something like the following? if not has_header:
statement = ...
elif not os.name = 'nt':
statement = ...
else:
raise NotImplementedError() |
|
no the commands with be different because batch commands are different than shell commands. echoing newlines in particular is very different |
|
i'll work out a windows solution at least for things without headers |
|
well it turns out that sqlite takes arguments so that's good. we may be able to remove some code |
|
i think is ready to go, we're passing on windows, i've xfailed the tests for csvs that have a header |
|
+1 |
Fix appending to a sqlite table from a CSV with a header
No description provided.