Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import errors and other datastore dashboard improvements #3723

Merged
merged 26 commits into from
Dec 7, 2021
Merged

Import errors and other datastore dashboard improvements #3723

merged 26 commits into from
Dec 7, 2021

Conversation

dafeder
Copy link
Member

@dafeder dafeder commented Dec 7, 2021 •
edited
Loading

Improvements to the datastore import status page and underlying services:

  • Displays datastore import errors on the import dashboard.
  • Improves readability of the overall table
  • Fixes bug in pager

QA Steps

  • Enable datastore_mysql_import
  • Upload a non-UTF8 CSV file as a resource.
  • Run cron
  • Visit /admin/dkan/dataset-status
  • Import status should show an error for the non-UTF8 resource
  • Confirm table looks good
  • Confirm pager works as expected

Known issues

There are still issues with underlying reporting from the fetcher and importer. A file that DKAN refuses to import, such as a non-CSV file, just shows up as "waiting" rather than giving useful feedback. Additionally, imports from the mysql importer ("LOAD INFILE") show as "0%" even though they are done.

dafeder and others added 26 commits November 22, 2021 14:25
@dafeder
Add error message to dataset info, API
0ea2553
@dafeder
Avoid error if no schema
e6bc676
@dafeder
Remove redundant assignments
887bdb9
Prevent status from being overwritten if error is not set
838ac28
@dafeder
Fix status/error in DatasetInfo
0c57c25
@dafeder
Add a test for importInfo
9523ff0
@dafeder
Clean up use statements
d1c5aef
@dafeder
Bytes processed
35a28e6
@dafeder
Merge remote-tracking branch 'origin/2.x' into dashboard-rows
08a3c3c
@dafeder
Fix pager bug
3a73b87
@dafeder
Full datetime in %modified
f95b2d7
@dafeder
Add source path to DatasetInfo
48f7879
@dafeder
Add templates
f3edcb7
@dafeder
Redo table
1b938d6
@dafeder
Pager weight
d1ad5bc
@dafeder
Template improvements
b70d973
@dafeder
Unused use statements
4ba4d43
@dafeder
Start fixing tests
c1435ae
@dafeder
Finish fixing tests; accomodate empty dist
2af1c10
@dafeder
Remove unused methods
856b5c5
@dafeder
Test improvements
e8010d6
@dafeder
CodeClimate fixes
245ba90
@dafeder
Clean up status cell code
1bc3dc8
@dafeder
Tighten up getResourcesInfo
1129427
@dafeder
Method cleanup
c1b8eb7
@dafeder
Consolidate
7ca4c68
@dafeder dafeder marked this pull request as ready for review December 7, 2021 19:28
clayliddell
clayliddell approved these changes Dec 7, 2021
View reviewed changes
Copy link
Contributor

@clayliddell clayliddell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; passed QA; automated tests are also passing. Approved!

@clayliddell clayliddell merged commit 72d67b5 into 2.x Dec 7, 2021
@clayliddell clayliddell deleted the dashboard-rows branch December 7, 2021 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clayliddell clayliddell clayliddell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dafeder @clayliddell