Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix dig import 2021 #66

Merged
merged 5 commits into from
Jun 25, 2021
Merged

Fix dig import 2021 #66

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b18d1f9
Importer doesn't use ckan logic and therefore uses exceptions not cov…
May 12, 2021
f3b00da
Added better DIG error messages for undefined excel cell range
May 12, 2021
3d6db15
Instead of erroring on missing DIG fields let the schema check the in…
May 12, 2021
a19c459
Clean up Exception handling and removing cruft per review
Jun 17, 2021
7ad85a5
Added comment on changing error processing from DIG import code to CK…
Jun 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ckanext/cfpb_extrafields/controllers/digimport.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@ def upload(self):
dig = request.POST["file"].file
group = request.POST["group"]
rec, errors = make_rec(dig)
if errors:
if False: #errors:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment explaining why we're skipping this, for now?

redirect_to("import_page", errors=json.dumps(errors), group=group)
else:
rec["owner_org"] = group
rec["name"] = make_name(request.POST.get("name") or rec["title"])
rec["notes"] = rec["notes"] or "Record automatcially created from DIG file"
rec["name"] = make_name(request.POST.get("name") or rec.get("title",""))
rec["notes"] = rec.get("notes","") or "Record automatcially created from DIG file"
try:
upload_rec(rec)
except ValidationError as err:
Expand Down
12 changes: 5 additions & 7 deletions ckanext/cfpb_extrafields/digutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,6 @@
import re

from openpyxl import load_workbook
try:
from ckan.plugins.toolkit import Invalid
except ImportError: # pragma: no cover
# If the custom exception can't be imported, use a more generic exception
# This happens when ckan is not installed locally, like when running unit tests on travis.
Invalid = Exception

from ckanext.cfpb_extrafields import validators as v

Expand Down Expand Up @@ -194,9 +188,13 @@ def make_rec_from_sheet(ws, fields):
for field in fields:
try:
result[field] = get_field(ws, field, fields)
except Invalid as err:
except NotFound as err:
errors.append(field + ": Unable to extract field from workbook - Check for duplicate or undefined cell ranges")
except (Exception, StopIteration) as err:
# Invalid or Not Found respectively
errors.append(field + ": " + getattr(err, "error", getattr(err, "message", "UKNOWN_ERROR")))
return result, errors

def make_rec(excel_file):
wb = load_workbook(excel_file, read_only=True)
version = get_schema_version(wb)
Expand Down