Unable to handle multi-line text

[ts2095]

Super useful tool. unfortunately, it does not appear to be able to handle multi-line text in the CSV.

Example data

Text,Label
"Line1
Line2",1

This renders correctly in Excel and pandas also reads it as expected. However, in SMART, this raises the following error:

Environment:


Request Method: POST
Request URL: http://xxxxxxxxxxx:xxx/projects/add/?X-Progress-ID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx

Django Version: 1.11.4
Python Version: 3.5.3
Installed Applications:
['core.apps.CoreConfig',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django_extensions',
 'django_celery_results',
 'rest_framework',
 'rest_framework.authtoken',
 'rest_auth',
 'django.contrib.sites',
 'allauth',
 'allauth.account',
 'postgres_stats',
 'formtools',
 'rest_auth.registration',
 'rest_framework_swagger',
 'webpack_loader',
 'progressbarupload']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/contrib/auth/mixins.py" in dispatch
  56.         return super(LoginRequiredMixin, self).dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/formtools/wizard/views.py" in dispatch
  248.         response = super(WizardView, self).dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/formtools/wizard/views.py" in post
  301.         if form.is_valid():

File "/usr/local/lib/python3.5/site-packages/django/forms/forms.py" in is_valid
  183.         return self.is_bound and not self.errors

File "/usr/local/lib/python3.5/site-packages/django/forms/forms.py" in errors
  175.             self.full_clean()

File "/usr/local/lib/python3.5/site-packages/django/forms/forms.py" in full_clean
  384.         self._clean_fields()

File "/usr/local/lib/python3.5/site-packages/django/forms/forms.py" in _clean_fields
  405.                     value = getattr(self, 'clean_%s' % name)()

File "/code/core/forms.py" in clean_data
  254.         return clean_data_helper(data, labels)

File "/code/core/forms.py" in clean_data_helper
  70.             "but step 2 was given {1}".format(', '.join(labels_in_data), ', '.join(supplied_labels))

Exception Type: TypeError at /projects/add/
Exception Value: sequence item 0: expected str instance, numpy.int64 found

[wengerm50]

Hi,

i've investigated and came up with a few things.

• First the stacktrace that you posted is actually due to a ValidationError raised if the labels you specify when creating a project do not match the labels in the file you uploaded. See this commit for more info 14b6796

• Next the actual issue with newlines is due to how we load the data into the database. We are using Postgres COPY and its expects every line from a csv to be a newline (which is causing issues when there are new lines in the text). I've replaced new lines with spaces. The UI would not render new lines anyway, so hopefully this is okay for now.

If you want to try it out and see if this works for you the changes are in #10

[wengerm50]

We've reviewed and merged the pull request. I'm going to go ahead and close the issue now. If you are still having this issue with the most recent version feel free to comment again.

[ts2095]

Hey,

sorry for not replying earlier.
We'd actually really like to have multiple lines as we're displaying emails that get really hard to read without linebreaks. I don't honestly understand enough about how the frontend works. Is there a chance to use something like <br> if \n breaks Postgres?

Thanks!