QA 2.0 (to go with Archiver 2.0) #24

davidread · 2015-11-05T17:38:43Z

This takes substantial improvements from the data.gov.uk team. It is intended to work with the Archiver 2.0 ckan/ckanext-archiver#15

Key changes are similar to the archiver:

results of the QA are now stored in a dedicated QA table, rather than in the TaskStatus objects (which were awful to query) and ResourceExtras (which makes it look like your dataset changed every time you archive, and if you're not careful this change triggers another archival and you get an infinite loop)
the Openness report is done using the neat ckanext-report infrastructure, generated nightly, rather than generated on the fly in ckanext-qa templates. This avoids the big back-end load when you click on a report. Now the report is available to not just admins but the public too (let's be open about broken links - it helps to add pressure to getting them fixed)
lots more tests
rather than the data.gov.uk team working on their own fork, they contribute back to the whole. Whilst some teams have been using the root fork of ckanext-archiver, no-one has taken much responsibility for it and it is not well maintained.

QA specific:

file format detection (rather than just looking at resource.format)

To do:

Merge changes from okfn master
Change report HTML from Genshi to Jinja

…ts back live on DGU. Queries now execute fast enough so that Varnish does not time out before they return and templates are better formatted. * Set filename of the al broken links CSV file to all_broken_links.csv (it was set to the built-in function id() before) * Changed certain SQLAlchemy queries followed by multiple tsak_status queries into single SQL statements * All reports only return data for active packages and resources * Capitalized the openness score descriptions * Tweaked all the main templates so that long links don't break column widths * Made the "License not open" check come after the other checks, because we prefer to know if links are genuinely broken, rather than that information being supressed by the license. Only working, non-openly licences links now get the "License not open" message * Noticed that requests library sometimes fails to get head requests after a 301 redirect, reporting bad port. Can't fix this but have added approprate log messages

…o not have every page filtered for insertion of qa.css etc. Much improved logging in tasks.py.

… tha failure count in this case.

…st time link worked. Excluded "Bad content type" from broken links report. Improved "reason" field to include more detail of why the score was given, instead of a description of the score which is useless. Added many data types, based on DGU data formats. Fixed up test_tasks.py

…calls(!). See #2942.

…e we have a better list of format that we are interested in. Removing unfinished attempt to correlate mime-types with extension. Promote mime-type as most important thing - holds publishers to account more. Broken datasets report - fixed rows getting jumbled, updated dates being wrong, hours and minutes mixed up.

…riter. Tidied table presentation.

…CKAN now alert bulk/priority queue as appropriate. Can now specify group (instead of package) in update command.

…ow defaults to correct queue when you specify a group. Cope with groups with deleted datasets in (!). Broken link reports now cope when no results. File sniffer now copes with unicode in filenames. Sniffer has more logging when it succeeds. Tests fixed up.

…mplete.

…e this.

…ensible there. Fixed some tests.

… score, since solr was getting overloaded.

…mmand. Avoid duplicate search index. Fix bug with archival_timestamp as datetime.

QA 2.0 (to go with Archiver 2.0)

QOL-8544 add more BDD tests

threeaims and others added 30 commits August 8, 2012 14:49

Don't treat working links that aren't openly licensed as broken

904b6f1

Option to use internal site_url - needed for Bytemark. Added option t…

0a019c3

…o not have every page filtered for insertion of qa.css etc. Much improved logging in tasks.py.

More places where internal site url required.

4c96f2a

Correct handling of new resources - expect an "error" when try to get…

fd997cf

… tha failure count in this case.

QA report updated to use CKAN groups, rather than published_by field.

d20ff90

Update requests version to match ckanext-archiver.

684d279

Fix funny header in new version of requests, that prevented CKAN API …

d1beb30

…calls(!). See #2942.

CSV writer for lists of dicts. Made report dict ordered to help csv w…

b99f963

…riter. Tidied table presentation.

Adding basic format sniffing

157c501

sniff_format tests now all pass.

4951d7e

Format sniffer vastly improved, using real files. Notifications from …

fafcab0

…CKAN now alert bulk/priority queue as appropriate. Can now specify group (instead of package) in update command.

Give one star for downloaded but unknown format. Reports still not co…

ef83abb

…mplete.

Revamp reports of broken links. Add paster commands: view & clean.

19b1182

Include number of packages and resources in broken results.

ad8ab15

Fix bad search-replace.

a913b3b

Added WMS to the sniffed formats. It is a service, but works fine lik…

1f6be7d

…e this.

check_link API call now uses the DGU formats as an option.

e9d171d

Recognise human readable format names in the format field.

cdd6f35

Moved get_cached_resource_filepath to ckanext-archiver - seems more s…

c0521f0

…ensible there. Fixed some tests.

I cannot see the point of this task_status update.

a8b6d20

Fix bug with python 2.6 and opening zipfile.

cdc1388

Forgot to add in this test file.

0774440

Be more efficient about reindexing (search) on giving a package a new…

839030b

… score, since solr was getting overloaded.

Forgot logic file.

22988a5

Test config added.

c7c67ed

Update in alphabetical order. Refactorer update chunking.

72db1ea

David Read added 26 commits November 19, 2015 21:51

Missed requirements file

110c13b

Improve tests.

e70b62e

Fix bug that meant rendering debug info was v slow.

91c502b

Add travis

fc00e8e

Travis fix

67f3637

create_qa_update_package_task moved to lib - since its shared with co…

c2154d6

…mmand. Avoid duplicate search index. Fix bug with archival_timestamp as datetime.

Report now in jinja

fe5ddaa

Travis fix

3bb435b

Move tests into familiar place - ckanext/qa

5a22da0

Merge branch 'master' into for-2.0-release

64460dd

Fix tests

4a6e22a

No controllers to test

e5e08b0

Travis fix

89f849d

Travis

fc33910

Travis fix

7e49be4

Travis fix

e5ab879

Add travis

fa9cb27

Travis fix

23e7dbf

Travis research

9ec3cc5

Try just installing libmagic

3f565b7

Travis

48ddaa0

Travis

171697f

Travis - more logging.

ec98451

Travis

a0359a2

Hide the remarks

d81a866

Add pics to readme

c06fbb8

davidread pushed a commit that referenced this pull request Nov 20, 2015

Merge pull request #24 from datagovuk/for-2.0-release

c1c3d6c

QA 2.0 (to go with Archiver 2.0)

davidread merged commit c1c3d6c into ckan:master Nov 20, 2015

davidread deleted the for-2.0-release branch November 20, 2015 18:34

HristijanVilos pushed a commit to keitaroinc/ckanext-qa that referenced this pull request Jul 18, 2022

Merge pull request ckan#24 from qld-gov-au/QOL-8544-ckan-2.9

b2dc89b

QOL-8544 add more BDD tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QA 2.0 (to go with Archiver 2.0) #24

QA 2.0 (to go with Archiver 2.0) #24

davidread commented Nov 5, 2015

QA 2.0 (to go with Archiver 2.0) #24

QA 2.0 (to go with Archiver 2.0) #24

Conversation

davidread commented Nov 5, 2015