Drop blank sheets when converting to spreadsheet

[timgdavies]

When I use convert to spreadsheet for OCDS files, I often have to search through lots of tabs that don't have any data in them to find the sheet I want.

Could we have an option to ignore blank sheets when exporting spreadsheet? This would potentially give a big usability boost to use of the spreadsheet conversion for data checking.

[Bjwebb]

This should be fairly straightforward. We may event want to change the default.

[duncandewhurst]

+1 (million)

why didn't I think of asking for this???

[duncandewhurst]

Several users at IODC flagged this an issue, so it would be good if we can get this on the non-sprint priority list.

Could we also drop blank columns from the spreadsheet export?

[robredpath]

@Bjwebb this is the issue I was looking for earlier!

[Bjwebb]

So, I've been looking at this today, and I'm fairly sure that I'll get this done tomorrow.

This behaviour was added deliberately, as we thought it might be useful for spreadsheet users to be able to see how "complete" their data is, and fill in extra standard fields in the spreadsheet. Is it useful to retain this functionality behind a commandline flag in flatten-tool? This is fairly easy to do, but no point if its not actually useful. Having multiple options in OCDS CoVE is more work, and the big question would be how to put it in the interface.

Similarly this will by default affect 360 and IATI CoVEs. @stevieflow do we want this same behaviour for those (dropping blank sheets and blank columns)? Technically its quite easy to have different options for different CoVEs if there's a commandline flag. Could be confusing for people though.

[timgdavies]

This behaviour was added deliberately, as we thought it might be useful for spreadsheet users to be able to see how "complete" their data is, and fill in extra standard fields in the spreadsheet.

I do think this is still useful: but that the default use-case of analysing the data benefits from dropping the columns that are not filled.

I would suggest:

• Retain the option with a command line flag;
• Switch the CoVE default to 'filled columns only';
• @duncandewhurst announces the change to the standard-discuss mailing list when it is in testing, and before it is in live - and asks anyone who is relying on having all the columns in their output to get in touch;
• Assuming no-one gets in touch to say having all columns in output is essential, push the change live;
• If there is interest in having all the columns in output, undertake some brief user research to identify how the UX should work for that - and decide whether or not waiting for that should be a blocker to going live on default = filled columns only.

[Bjwebb]

Here's a pull request for flatten-tool - OpenDataServices/flatten-tool#241
The cli also defaults to dropping blank columns and sheets. There is a comandline option for the old behaviour.

Here's a dev CoVE instance that uses the relevant flatten-tool branch:
http://1019-no-blank-sheets.dev.cove.opendataservices.coop/

[Bjwebb]

Thanks @duncandewhurst for posting this to standard-discuss https://groups.google.com/a/open-contracting.org/forum/?utm_medium=email&utm_source=footer#!msg/standard-discuss/Sz6HmrnWVJc/pu5B7f3IBAAJ. Now we wait 2 weeks to see if anyone gets in touch.

[odscjames]

A new --remove-empty-schema-columns feature in flatten tool will be in the next release, 0.4.0. After discussion with @kindly, we changed how it works. By default, it works as it did so nothing will change. So this will need to be explicitly enabled per site for each site that wants it.

[Bjwebb]

So, this is now an option in flatten-tool. @duncandewhurst Do we also want to enable this in OCDS CoVE? (Looks like it already is for IATI).

[duncandewhurst]

It's already enabled as the default behaviour in OCDS CoVE too :-)