Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import data misses first character of the first field #3520

Open
fdmenendez opened this issue Apr 18, 2019 · 6 comments

Comments

@fdmenendez
Copy link

commented Apr 18, 2019

Details

When importing CSV data, the fist character of the first column gets lost for all records
For example a date will be imported like 019-04-18
it does not affect the header

Steps to reproduce

Import data from CSV default parameters

Operating system

Windows 10

SQLiteStudio version

sqlite-tools-win32-x86-3270200

@mrgchr

This comment has been minimized.

Copy link

commented Apr 19, 2019

IMHO, sqlite-tools's behavior is not a issue of this app but I tried to reproduce with sqlite-tools-win32-x86-3280000.
As a reuslut, I cannot reproduce an issue you mentioned in my local env(SQLiteStudio v3.2.1).

I've tried following 4 encoding csv files(zipped file attached):

foobar_csv.zip

  1. UTF-8 with BOM/CRLF
  2. UTF-8 with BOM/LF
  3. UTF-8 without BOM/CRLF
  4. UTF-8 without BOM/LF
sqlite> .open Sample.sqlite
sqlite> .mode csv
sqlite> .import foobar_CRLFwBOM.csv foobar_CRLFwBOM
sqlite> .import foobar_LFwBOM.csv foobar_LFwBOM
sqlite> .import foobar_CRLF.csv foobar_CRLF
sqlite> .import foobar_LF.csv foobar_LF
sqlite> .quit

the import sqlite3 database looks fine in SQLiteStudio(v3.2.1).
image

Would you provide more detailed info such as file encoding, new line character and sqlite script with executed in sqlite-tool ?

@mrgchr

This comment has been minimized.

Copy link

commented Apr 19, 2019

Now, I think I reproduce the issue you mentioned with SQLiteStudio Import Wizard(not sqlite-tools).

When CSV file with LF new line character, first character will be dropped.

image

image

This issue might be avoided by using csv file with CRLF new line.

@fdmenendez

This comment has been minimized.

Copy link
Author

commented Apr 19, 2019

hi
Yes, you are right, that seems to be the problem, my files end with LF. Thanks for finding it out!

btw, is there a way to solve it? i can go ahead and change my files but it would be nice to be able to import regardless the new line encoding.

Let me know if i shall close the issue

@mrgchr

This comment has been minimized.

Copy link

commented Apr 19, 2019

Hi @pawelsalawa , @fdmenendez

How do you see this behavior?
IMPO, some options, like "new line character" combo box in "Data source option" in wizard might be preferable.

@pawelsalawa

This comment has been minimized.

Copy link
Owner

commented Apr 19, 2019

As far as I remember this should be handled automatically (any kind of new line separator should be supported - Windows/Linux/Mac). It looks like a bug and need to investigate where it happens.

@davecampbell

This comment has been minimized.

Copy link

commented Jun 14, 2019

i just ran across this bug / condition - glad it is formally recognized here.
my use-case:
made a 'report' from a monitoring tool (logicmonitor), specifying 'csv' as output. no other parameters were offered (such as newline character).
when i used this file as the import source - it dropped the first character of the first field in every record.
can confirm that the line-endings are LF, not CRLF.

$ head device-inventory-all-details-20190614181741.csv | cat -e
Device,hardwaremodel,ips,model,hostname,categories,sysinfo,groups,displayname$
1st Floor MDF,,"169.254.17.12,169.254.17.13,169.254.17.11,10.10.0.138,10.255.4.227",,10.10.0.138,"snmp,snmpTCPUDP",Ruckus Wireless Inc (C) 2006 ,appriss-net/Wireless,1st Floor MDF$

ls-idf1-1-18,,"10.10.0.246,169.254.17.12,169.254.17.13,169.254.17.11,10.255.4.216",,10.10.0.246,"snmp,snmpTCPUDP",Ruckus Wireless Inc (C) 2006 ,appriss-net/Wireless,ls-idf1-1-18$

ls-idf1-1-20,,"10.255.4.221,10.10.0.247,169.254.17.12,169.254.17.13,169.254.17.11",,10.10.0.247,"snmp,snmpTCPUDP",Ruckus Wireless Inc (C) 2006 ,appriss-net/Wireless,ls-idf1-1-20$

can confirm that re-importing after running 'unix2dos' on the file did work CORRECTLY.

i also vote that the import should handle both newline sequences identically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.