fixed width saver truncates data if columns are not fully expanded

[daviewales]

Small description

If you save or syscopy a table as fixed format, and the visible column width is less than the width of the data in the column, the data is truncated. Also, the resulting file is not a valid fixed width format file as the columns are not aligned with the headers.

Expected result

Saving or copying a table when the columns are not fully expanded should yield the same result as when the columns are expanded.

Actual result with screenshot

Steps to reproduce with sample data and a .vd
test-vd-fixed.zip

Additional context
saul.pw/VisiData v2.11

[daviewales]

Hmm... Looking at the code, it looks like the truncation behaviour is intentional?

visidata/visidata/loaders/fixed_width.py

Line 87 in d40fc93

widths[col] = col.width or sheet.options.default_width or col.getMaxWidth(sheet.rows)

I wouldn't normally expect a display mode to change the underlying data.

Also, I suspect there's something funny going on here, which is leaving us with a missing space in the truncated version:

visidata/visidata/loaders/fixed_width.py

Line 95 in d40fc93

fp.write(('{0:%s%s.%s} ' % ('>' if vd.isNumeric(col) else '<', widths[col], widths[col])).format(val))

[saulpw]

Yes, this behavior is (somewhat) intentional; the widths of the columns indicate the fixed widths of the fields. This is so you can set them precisely, in case the fixed file format has an exact specification. I agree though that it's undesirable to have the data truncated otherwise. What's the better thing to do here? Find the max width and use it instead of the column width, if it's larger? and warn or ask for confirmation if this is the case?

[anjakefala]

@daviewales, do you have a preference wrt:

What's the better thing to do here? Find the max width and use it instead of the column width, if it's larger? and warn or ask for confirmation if this is the case?

[daviewales]

My preference would be to always use the true max width by default, so data is never truncated. Truncation could be a separate feature which you explicitly opt into. (Save view vs save data?)

Admittedly, my use case for this is non-standard.
I use the fixed width saver to paste tables into emails and into MS Teams within code blocks. (gY fixed)
They look beautiful, but it's a surprise when the data is truncated.

[anjakefala]

Okay, I pushed that change to develop @daviewales! Try it out, and let me know how it feels. =)

[daviewales]

Thanks @anjakefala. I've had a play, but it's not quite what I had in mind.
I've pushed a PR which better captures my intent: #1890
Try it out with test-vd-fixed.tsv in the attached zip from the first comment.

Both this:

and this:

save to the exact same output:

[geekscrapy]

Hey @daviewales ! Can you draw in ascii what you are intending to happen? With the example lines above?

[daviewales]

Hi @geekscrapy,

In ASCII:

# The original data
very_long_column_name                  another_long_column_name                    
very long data wider than the column   more very long data wider than the column   

# The view in visidata when columns are displayed narrower than the width of the data:
very_long_column_n…   another_long_colum…        
very long data wide   more very long data

# How visidata used to save this as .fixed
very_long_column_name another_long_column_name 
very long data wider more very long data  

# How anjakefala's patch saves it as .fixed:
very_long_column_name another_long_column_name 
very long data wider more very long data     

# How my patch saves it as .fixed
very_long_column_name                  another_long_column_name                    
very long data wider than the column   more very long data wider than the column   

Perhaps I have misunderstood how @anjakefala's patch works?

[daviewales]

Looking at the code in 77ae017, I can see that if I set the sheet default_width to 0 using zShift+O, it will set the save width to the max width of the data rather than the width of the columns.

However, this has other side-effects. It's not something that you'd want to set in your ~/.visidatarc.
(Columns resize to hidden, when all columns are hidden self.cursorRowIndex becomes None, which causes errors which prevent you from unhiding the columns from the columns sheet)

[saulpw]

Columns resize to hidden

This is a bug; if default_width is 0 it should act as you suggest (always use maxwidth).

[daviewales]

I guess my issue is that I like having default_width as 20. I just don't like my viewing choices to change how the data is saved unless I explicitly opt in to that.