Separator for multi-choice fields in CSV registration export

[jouvin]

At LAL, we have a (critical) application relying on Indico to manage subscription to working groups. The basic architecture is that a dummy event is created and people use the registration form to select (with a "mltiple choice" filed) the WGs they want to participate too. The list of participants is then exported as a CSV and processed by a Python script to produce various informations like email list membership, web pages... This application was developed with Indico 1.2 and has been working very well for several months.

After the upgrade to 1.9.11, it appears that this application is broken. Apart from the column order which is different (and was easy to fix), I found one major issue with export of multiple choices. Préviously, when a user was selecting several choices in the same fied, they were exported separated by a semi-colone (;). In v2, they are now separated by a comma (,) . I don't want to discuss which one is the best in principle but in our case, it happens that the values of the possible choices contain commas (but not semi-colon), meaning there is no way to unambiguously split the different values.

I am not sure what the proper fix to this poblem could be: a per-event option to select the choice value separator that you want? In the meantime, I'd appreciate any workaround or hack suggestion to restore the semi-colon as a separator as this application is very critical for us presently.

[ThiefMaster]

The exports of the registrants is mainly targetting processing by humans (after importing into Excel etc.), so I don't think we can really take such special use cases into account. I think the ideal solution for you would be writing a small indico plugin that gets the data straight from the database - like this you can actually get it as a Python list without having to deal with strings and separators.

Eventually we also want to provide an API to export registration data. This would also give you a list with the options instead of just a single string.

However, as a short-term fix you could edit this line to change the separator: https://github.com/indico/indico/blob/master/indico/modules/events/registration/fields/choices.py#L220

[jouvin]

@ThiefMaster thanks for the pointer to choice.py. I can probably leave with this for the time being.

I understand that export of registrants is not targeted at the type of usage I mentioned. But in fact Indico is pretty good at addressing this use case too! CSV module in Python is rather good and can basically auto-discover columns from the header line which makes it pretty robust, even if the order of the columns is changing (as it was the case in v2). I think that if this multiple choice separator could be addressed (clearly there is no perfect one), this would remain a pretty good option until there is a proper API to get the data from the application and probably simpler than writing a plugin.

[jouvin]

Back on this issue, I was wondering if the separator change from , to ; was the result of a request or just a decision of the Indico team... I have the feeling that even for humans, this is not a very good change as there is a greater chance of having a , in one of the choice value than a ;...

[ThiefMaster]

I don't think there was any specific reason for it... Actually. I somehow thought that we are using ; and you are asking for .. I think using ; to separate the items might actually make more sense. @pferreir what do you think?

[jouvin]

It was the case in 1.2!

[jouvin]

Any news on this issue? Any agreement to restore 1.2 behaviour, that means ; as a separator?

[ThiefMaster]

Sorry, forgot about this issue. The change will be included in 2.0a2.