Can't re-import new Screendoor write-ins

[mayatmiller]

Hey all,

I'm working on the "Debthospitalslrn" project in Collaborate, and was trying to re-import new responses from Screendoor during Brandon's latest fix.

Now, when I try to re-import Screendoor responses I get directed to an error page. Here's a screenshot of the top of that page:

Feel free to reach me at maya.miller@propublica.org if you want to talk through further, or if I can send any additional information to make troubleshooting easier.

Thanks,

Maya

[rachelgli]

For reference, this project lives in our internal Collaborate.

I got a different error when I tried to import it into our external Collaborate.

[brandonrobertz]

These look like two different version of Collaborate being ran.

@mayatmiller Do you have the lastest code? Or did you change the settings to DEBUG=True?

I'm going to be working through this issue in the next day or two. If anyone has error logs from the failure, it would be helpful.

[mayatmiller]

I don't have the latest code and haven't been in the backend of the website at all (so haven't changed the settings in anyway).

The re-import option is still not working, unfortunately, and am receiving the same error.

I'm copying the entire code that appears in the error message in case it's helpful:

Request Method:	POST
http://collaborate.tools.propublica.net/db-config/refine-and-import/17/
2.2.7
InvalidDimensions
/usr/local/lib/python3.7/site-packages/tablib/core.py in _validate, line 287
/usr/local/bin/python
3.7.5
['/srv/app', '/usr/local/bin', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages', '/usr/local/lib/python3.7/site-packages/odf', '/usr/local/lib/python3.7/site-packages/odf', '/usr/local/lib/python3.7/site-packages/odf', '/usr/local/lib/python3.7/site-packages/odf', '/usr/local/lib/python3.7/site-packages/odf', '/usr/local/lib/python3.7/site-packages/odf', '/usr/local/lib/python3.7/site-packages/odf']
Mon, 23 Dec 2019 16:16:36 +0000

Let me know if there's anything else I can do to help!

[brandonrobertz]

I just tested through screendoor and I couldn't find any exceptions when I ran imports. I think it's something specific to the form. Things to check:

• Are you inputting the correct project ID? (just double check this)
• Does your project have multiple forms?
• If so, are you inputting a form ID? By default, the SD import will always select the first form if blank.

That's all I can think of with the information I have. An error log from the web server would be really helpful.

[BluClare]

Hi @brandonrobertz
We're having issues connecting Screendoor and Collaborate in the new instance we have set up at the ABC.
We have checked the API and project ID and tried repeatedly. Do you have any ideas what we might need to do to troubleshoot this?

Here's the error message:

InvalidDimensions at /db-config/begin/ by Django

Environment:

Request Method: POST
Request URL: http://localhost:8082/db-config/begin/

Django Version: 2.2.7
Python Version: 3.7.6
Installed Applications:
['dal',
'dal_select2',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.forms',
'social_django',
'import_export',
'taggit',
'django_models_from_csv',
'collaborative']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']

Traceback:

File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)

File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
115. response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
113. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
21. return view_func(request, *args, **kwargs)

File "/srv/app/django_models_from_csv/views/configuration.py" in begin
178. raise e

File "/srv/app/django_models_from_csv/views/configuration.py" in begin
163. form_id=int(sd_form_id) if sd_form_id else None

File "/srv/app/django_models_from_csv/utils/dynmodel.py" in unique_name_wrapper
59. return f(*args, **kwargs)

File "/srv/app/django_models_from_csv/utils/dynmodel.py" in from_screendoor
183. csv = importer.build_csv(project_id, form_id=form_id)

File "/srv/app/django_models_from_csv/utils/screendoor.py" in build_csv
146. return self.build_csv_from_data(form_data, response_data)

File "/srv/app/django_models_from_csv/utils/screendoor.py" in build_csv_from_data
130. data.append(row)

File "/usr/local/lib/python3.7/site-packages/tablib/core.py" in append
702. self.rpush(row, tags)

File "/usr/local/lib/python3.7/site-packages/tablib/core.py" in rpush
686. self.insert(self.height, row=row, tags=tags)

File "/usr/local/lib/python3.7/site-packages/tablib/core.py" in insert
677. self._validate(row)

File "/usr/local/lib/python3.7/site-packages/tablib/core.py" in _validate
287. raise InvalidDimensions

Exception Type: InvalidDimensions at /db-config/begin/
Exception Value:

[brandonrobertz]

@BluClare Thanks for the error log. I have a few ideas as to what it might be.

Just so you know, this is the generic error related to imported columns not matching what Collaborate initially detected. Some things this could be:

1. Do you have any duplicate fields?
2. Does your form have fields with the following names: name, email, responder.

One thing you can try is reverting back to 03d22c0 and doing the import. If it fails, then it's unlikely to be related to 2.

Let me know about those and we can go from there.

[BluClare]

Hi @brandonrobertz I think you are spot on with point 1. I've looked and there are many duplicate fields. I'll sort this out and then try again.
Thank you :)

[brandonrobertz]

Okay, if you find a specific duplicate that is causing this let me know. In the meantime I'll try and reproduce it in some unit tests.

[brandonrobertz]

@BluClare I pushed something that might fix the issue. Let me know if it does or not. Latest master branch: 17161fe

[brandonrobertz]

Also note that you may need to remove and re-import the data source if this master branch change allows you to import new Screendoor forms but not re-import existing ones. This is all kind of a shot in the dark since I don't have access to anyones' data. So thanks for bearing with me.

[brandonrobertz]

Closing this as latest master 17161fe appears to fix this issue (confirmed by @BluClare).