Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow importing registrations/contributions from CSV files #3144

Merged
merged 7 commits into from Jan 26, 2018

Conversation

@pferreir
Copy link
Member

@pferreir pferreir commented Nov 8, 2017

For now it only supports personal data fields (registration) and single speakers (contributions).

class ImportRegistrationsForm(IndicoForm):
source_file = FileField(_("Source File"), accepted_file_types='.csv')
skip_moderation = BooleanField(_("Skip Moderation"), widget=SwitchWidget(), default=True,
description=_("If enabled, the registration will me immediately accepted"))

This comment has been minimized.

@mvidalgarcia

mvidalgarcia Nov 8, 2017
Member

... will be immediately ...

@@ -40,6 +40,7 @@
from indico.web.forms.base import IndicoForm, generated_data
from indico.web.forms.fields import (EmailListField, IndicoDateTimeField, IndicoEnumSelectField, JSONField,
PrincipalListField)
from indico.web.forms.fields.files import FileField

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

can be imported from indico.web.forms.fields in the statement right above

description=_("If enabled, the registration will me immediately accepted"))
notify_users = BooleanField(_("E-mail users"), widget=SwitchWidget(),
description=_("Whether the imported users should receive an e-mail notification"))
create_pending = BooleanField(_("Match missing users"), widget=SwitchWidget(), default=True,

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

Is there any good reason to create pending users here? If a user creates an indico account later, they are associated with the registration anyway, and in the user search we never show pending users. So I'd rather offer to create an EventPerson so they can find him when searching to add e.g. a speaker - but never a pending indico User

This comment has been minimized.

@pferreir

pferreir Nov 8, 2017
Author Member

And how would we phrase that option (EventPerson)?

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

there's a reason why i didn't suggest a possible description for this :P

This comment has been minimized.

@pferreir

pferreir Nov 8, 2017
Author Member

Then I'll just remove the option and set it to False by default. Anyway, the unification of Event Persons with registrations in version 2.1 should make this irrelevant.

{%- endtrans %}
{% endcall %}

{{ form_header(form, multipart=True, action=url_for('.registrations_import', regform)) }}

This comment has been minimized.

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

the url is the same as the one the dialog is loaded from so omitting it should work. also, I think you can use simple_form() here

This comment has been minimized.

@pferreir

pferreir Nov 8, 2017
Author Member

Dropzone complains If I specify no URL.

{{ form_rows(form) }}

{% call form_footer(form) %}
<input type="submit" class="i-button big highlight" value="{% trans %}Submit{% endtrans %}" disabled>

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

why disabled? shouldn't this rather be data-disabled-until-change?

@@ -0,0 +1,30 @@
{% extends 'layout/base.html' %}

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

It's a dialog so this and the block shouldn't be needed

<li>Affiliation</li>
<li>Position</li>
<li>Phone Number</li>
<li>E-mail</li>

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

Wouldn't it make more sense to put First/Last/Email first since those are probably the most common fields?

This comment has been minimized.

@pferreir

pferreir Nov 8, 2017
Author Member

I put 'e-mail' at the end because some of the other fields may be empty (e.g. position), and the resulting CSV file may end up having < 6 columns.

from indico.web.forms.base import IndicoForm
from indico.web.forms.widgets import SwitchWidget


class CSVImportException(Exception):

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 8, 2017
Member

inherit from UserValueError so it's not reportable?

@pferreir pferreir force-pushed the pferreir:wip/registration-import branch from 543b7bd to 78eb55d Nov 8, 2017
@mic4ael
mic4ael approved these changes Nov 9, 2017

class ImportRegistrationsForm(IndicoForm):
source_file = FileField(_("Source File"), accepted_file_types='.csv')
skip_moderation = BooleanField(_("Skip Moderation"), widget=SwitchWidget(), default=True,

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 9, 2017
Member

I'd remove this if moderation is disabled, like we do in InvitationFormBase

try:
first_name, last_name, affiliation, position, phone, email = [to_unicode(value).strip() for value in row]
except ValueError:
raise CSVImportException('Malformed CSV data - please check that the number of columns is correct')

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 9, 2017
Member

_(...)?

with db.session.no_autoflush:
registrations.append(create_registration(regform, {
'email': email,
'first_name': first_name.title(),

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 9, 2017
Member

http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/ ;)

Some of the things there are probably not super relevant, but I have the feeling titlecasing the name will screw e.g. many dutch names ("Foo van Bar" etc.) that have lowercase words. Maybe better to only titlecase if the original value is lowercase-only?

This comment has been minimized.

@pferreir

pferreir Nov 17, 2017
Author Member

The problem is actually UPPERCASE NAMES.

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 17, 2017
Member

lowercase-only or uppercase-only then ;)

This comment has been minimized.

@pferreir

pferreir Nov 20, 2017
Author Member

Well, the question is whether we'd rather have people input all kinds of silly stuff (CAPITALS!) or miss a few ones here and there. I can't see any better solution, unfortunately.

@@ -492,3 +499,25 @@ def generate_ticket(registration):

def get_ticket_attachments(registration):
return [('Ticket.pdf', generate_ticket(registration).getvalue())]


def import_registrations_from_csv(regform, f, skip_moderation=True, notify_users=False):

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 9, 2017
Member

s/f/fileobj/?

@ThiefMaster
Copy link
Member

@ThiefMaster ThiefMaster commented Nov 16, 2017

Probably good to rebase to v2.1-dev and add a changelog entry

@pferreir
Copy link
Member Author

@pferreir pferreir commented Nov 20, 2017

I'll actually add some tests first.

@pferreir pferreir force-pushed the pferreir:wip/registration-import branch 4 times, most recently from e36e2f7 to ad0900f Nov 20, 2017
@pferreir pferreir changed the title Allow importing registrations from a CSV file Allow importing registrations/contributions from CSV files Nov 20, 2017
@pferreir pferreir changed the base branch from master to v2.1-dev Nov 20, 2017
@pferreir pferreir force-pushed the pferreir:wip/registration-import branch 2 times, most recently from bab2646 to 387e190 Nov 20, 2017
</ul>
<div class="group">
<a class="i-button js-enable-if-checked arrow disabled"
title="{% trans %}Export data{% endtrans %}"

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

wrong indentation

<ul class="dropdown">
<li>
<a href="#" class="icon-file-pdf js-submit-form js-enable-if-checked disabled"
data-href="{{ url_for('.contributions_pdf_export', event) }}">

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

wrong indentation (also a few more times below)

following order:
<ul>
<li>Start date/time in <a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601</a> format</li>
<ul>

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

an <ul> right inside an <ul> looks wrong

{% from 'message_box.html' import message_box %}

{% call message_box('highlight', large_icon=true) %}
{% trans -%}

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

is it really a good idea to have this all in a single trans block? seems like it could be split easily (one per list item, one for the intro sentence, one for the sentence at the end) without having something that might be messy in other languages

@@ -16,22 +16,29 @@

from __future__ import unicode_literals

import csv
import dateutil.parser

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

not stdlib


{% call message_box('highlight', large_icon=true) %}
{% trans num_columns=6 -%}
You should upload a CSV (comma-separated values) file with exactly {{num_columns}} columns in the

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

space around num_columns

{% from 'message_box.html' import message_box %}

{% call message_box('highlight', large_icon=true) %}
{% trans num_columns=6 -%}

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

I think this big trans block could also be split into small ones

'affiliation': affiliation,
'phone': phone,
'position': position
}, notify_user=notify_users, management=skip_moderation))

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

Using the management kwarg also affects e.g. the log realm. Maybe better to add a new skip_moderation kwarg?

assert 'phone' not in data


@pytest.mark.usefixtures('request_context')

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

I think we should rather use session.user if session else None (__bool__ of LocalProxy catches the RuntimeError and returns False in that case) when logging the registration instead of using/requiring a request context. It doesn't make much sense that something as general as create_registration only works with an active request context...



class ImportContributionsForm(IndicoForm):
source_file = FileField(_("Source File"), accepted_file_types='.csv')

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

in some places we specify a mime type for accepted_file_types, in others a file extension... not sure if there's any advantage of using one or the other.

This comment has been minimized.

@pferreir

pferreir Nov 21, 2017
Author Member

The MIME type wasn't working fine on my browser. I'm not sure OSs set always the same for CSV files.

@pferreir pferreir force-pushed the pferreir:wip/registration-import branch 2 times, most recently from 4b024fc to 9c1ec83 Nov 21, 2017
@ThiefMaster ThiefMaster added this to the v2.1 milestone Nov 21, 2017
@pferreir pferreir force-pushed the pferreir:wip/registration-import branch 5 times, most recently from c568fc3 to 3c36016 Nov 21, 2017
@pferreir
Copy link
Member Author

@pferreir pferreir commented Nov 21, 2017

OK, I believe everything has been fixed. Awaiting final QA.

if changes:
flash(_("Event dates/times adjusted due to imported data."), 'warning')
return jsonify_data(flash=False, redirect=url_for('.manage_contributions', self.event),
redirect_no_loading=True)

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

I don't think redirect_no_loading is useful here. I added it for actions that trigger a download prompt where you usually remain on the same page and thus don't want to block it with the "Loading" dialog

<li>
{%- trans link='<a href="https://en.wikipedia.org/wiki/ISO_8601">'|safe, endlink='</a>'|safe,
subitem='<ul><li><em>'|safe, endsubitem='</em></li></ul>'|safe -%}
Start date/time in {{link}}ISO 8601{{endlink}} format

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

missing space inside {{ ... }}

<ul>
<li>
{%- trans link='<a href="https://en.wikipedia.org/wiki/ISO_8601">'|safe, endlink='</a>'|safe,
subitem='<ul><li><em>'|safe, endsubitem='</em></li></ul>'|safe -%}

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

the indentation seems weird

reader = csv.reader(f)
contributions = []
all_changes = defaultdict(list)
for n_row, row in enumerate(reader, 1):

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 21, 2017
Member

row_num?

This comment has been minimized.

@pferreir

pferreir Nov 21, 2017
Author Member

😠

@pferreir pferreir force-pushed the pferreir:wip/registration-import branch 2 times, most recently from 72ff4bb to 40bcee8 Nov 21, 2017
@@ -62,6 +62,7 @@ def _ensure_consistency(contrib):


def create_contribution(event, contrib_data, custom_fields_data=None, session_block=None, extend_parent=False):
user = session.user if session else None

This comment has been minimized.

@mic4ael

mic4ael Nov 23, 2017
Member

is it only for making tests run?

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 23, 2017
Member

It also lets you use this operation from the Indico shell without creating a request context.

</li>
</ul>
{% trans -%}
Only the field "Title" is mandatory. Users will be matched with existing

This comment has been minimized.

@mic4ael

mic4ael Nov 23, 2017
Member

what about E-mail address? isn't it mandatory as well?

This comment has been minimized.

@ThiefMaster

ThiefMaster Nov 23, 2017
Member

No, you may have speakers without an email address.

This comment has been minimized.

@pferreir

pferreir Dec 11, 2017
Author Member

And no speakers at all 😉

@pferreir pferreir force-pushed the pferreir:wip/registration-import branch 2 times, most recently from e1f8229 to 39c97be Dec 11, 2017
@ThiefMaster ThiefMaster changed the base branch from v2.1-dev to master Jan 12, 2018
@ThiefMaster ThiefMaster force-pushed the pferreir:wip/registration-import branch from 39c97be to 77f4a15 Jan 26, 2018
Wrapping avoids the dialog being wider than other dialogs
@ThiefMaster ThiefMaster merged commit 1edf0f5 into indico:master Jan 26, 2018
1 check was pending
1 check was pending
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@pferreir pferreir deleted the pferreir:wip/registration-import branch Feb 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.