Only 512 CSV columns supported

[Veltas]

Tried loading a JIRA CSV export, failed with this exception:

java.lang.ArrayIndexOutOfBoundsException
Caused: com.univocity.parsers.common.TextParsingException: java.lang.ArrayIndexOutOfBoundsException - null
Parser Configuration: CsvParserSettings:
	Auto configuration enabled=true
	Auto-closing enabled=true
	Autodetect column delimiter=false
	Autodetect quotes=false
	Column reordering enabled=true
	Delimiters for detection=[]
	Empty value=
	Escape unquoted values=false
	Header extraction enabled=false
	Headers=null
	Ignore leading whitespaces=true
	Ignore leading whitespaces in quotes=false
	Ignore trailing whitespaces=true
	Ignore trailing whitespaces in quotes=false
	Input buffer size=1048576
	Input reading on separate thread=true
	Keep escape sequences=false
	Keep quotes=false
	Length of content displayed on error=6
	Line separator detection enabled=false
	Maximum number of characters per column=16777216
	Maximum number of columns=512
	Normalize escaped line separators=true
	Null value=
	Number of records to read=all
	Processor=none
	Restricting data in exceptions=false
	RowProcessor error handler=com.ultorg.importer.spreadsheets.d$$Lambda$825/0x0000000800a231a0@558332ec
	Selected fields=none
	Skip bits as whitespace=true
	Skip empty lines=false
	Unescaped quote handling=STOP_AT_DELIMITERFormat configuration:
	CsvFormat:
		Comment character=#
		Field delimiter=,
		Line separator (normalized)=\n
		Line separator sequence=\n
		Quote character=�
		Quote escape character=�
		Quote escape escape character=null
Internal state when error was thrown: line=0, column=513, record=0, charIndex=16249
	at com.univocity.parsers.common.AbstractParser.handleException(AbstractParser.java:402)
	at com.univocity.parsers.common.AbstractParser.parseNext(AbstractParser.java:623)
	at com.ultorg.importer.spreadsheets.d.handleData(SourceFile:211)
Caused: com.ultorg.database.ImportException: Text parsing problem on line 1 (last content read was "")
	at com.ultorg.importer.spreadsheets.d.handleData(SourceFile:252)
	at com.ultorg.importer.spreadsheets.SpreadsheetDataSourceAccessor.populatePrimaryViewTemplate(SourceFile:161)
	at com.ultorg.importer.spreadsheets.c.populatePrimaryViewTemplate(SourceFile:81)
	at com.ultorg.database.DataSourceAccessor.createPrimaryViewTemplate(SourceFile:114)
[catch] at com.ultorg.project.c.a(SourceFile:406)
	at com.ultorg.project.c.a(SourceFile:257)
	at com.ultorg.project.i.a(SourceFile:503)
	at com.ultorg.project.i$a.b(SourceFile:1127)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1418)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:45)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:278)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)

If I understand correctly it seems the limit of CSV columns is capped at 512. I know good databases don't tend to have so many columns, but JIRA is JIRA! You can reduce the number of columns in JIRA output when exporting to just what's 'visible' while searching but it would have been nice to work with the full data.

[Veltas]

Also input buffer size looks around 1MB, if that's for a line that's okay. But the whole file is like 3MB.

[eirikbakke]

Thanks for the report, @Veltas! The error looks related to the 512-column limit, yes, but when I tried to create some sample data with 1024 columns myself, the CSV parsing library just quietly ignored any columns beyond the limit.

There seems to be some other error, which was about to be reported, but which triggered a separate bug in the CSV parsing library's error handler (due to the large number of columns).

Were you able to see a preview of the data in the "Connect to Text File" dialog, before the error happened?

If possible, would you be able to email me an excerpt of the file, perhaps just the first line with the table headers? (ebakke@ultorg.com)

Also input buffer size looks around 1MB, if that's for a line that's okay. But the whole file is like 3MB.

I think that's just a performance-related thing. The text file importer should work with gigabytes of data (and I have various 100-500MB CSV files that I've tested it with).

[eirikbakke]

Oh, never mind, managed to reproduce it...

[eirikbakke]

OK, ignore my earlier comment. It's just the 512-column limit, and error handling that could be improved. Since Ultorg uses PostgreSQL as an underlying data store, and PostgreSQL has a 1600-column limit, I'll set the new limit at 1500 columns (to leave a bit of margin for metadata columns etc.).

[eirikbakke]

Hmm, reading up on PostgreSQL's column limits, I'm nervous about increasing the maximum number of columns too much beyond 512. In fact, 450 might be a more realistic limit for text values.

How many columns are there actually in the JIRA export?

In any case, I have improved the error message a bit:

[eirikbakke]

I ended up with a different solution, which should allow up to 2048 columns in CSV/TSV files so long as at most 250 are actually visible in a perspective at any time. This will be in the next Ultorg version.

[eirikbakke]

This issue is now fixed in Ultorg 1.0.8, which was just released. Existing beta users can find the new Ultorg version at the old download link.