Make datastore more robust, #171 #172
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
ThrawnCA
commented
Sep 7, 2022
ThrawnCA
commented
- Bypass URL caching when downloading uploaded resources to parse
- Ensure that the 'datastore_active' flag matches the actual datastore state
Develop to master
QOL-6190 add compatibility mode flag
Qol 6190 refactor to make nosetests and flake8 pass
Develop to master
Merge 0.5.0 changes from upstream
Bumps [unidecode](https://github.com/kmike/text-unidecode) from 1.0.22 to 1.1.1. - [Release notes](https://github.com/kmike/text-unidecode/releases) - [Commits](https://github.com/kmike/text-unidecode/commits) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Bump unidecode from 1.0.22 to 1.1.1
… is dropped and stored. i.e. +10 means that we get dates in the future which shows up as -1 days when the job ran
- otherwise they'll be displayed as if they were GMT, but they were probably not generated that way
Store dates as utcnow instead of server now as it seems timezone info…
* ci build
* correct psycopg2 for later generations of db, ignore masters for now
* change os for 2.7 build
* handle jetty8 and jetty9
* add xlsx tests, tweak for bionic builds libmagic works differently for xls files
* skip xls tests due to libmagic1 changes in os
libmagic1 that works out xls is not as forgiving as it once was, the two tests
were checking to see if messytables was used or an error thrown on partial load
since xls format stopped in 2004, and xlsx is the defacto format for excel
this might be ok to ignore now as it still works on Trusty LTS
Bionic
libmagic1:
Installed: 1:5.32-2ubuntu0.4
Candidate: 1:5.32-2ubuntu0.4
Version table:
*** 1:5.32-2ubuntu0.4 500
500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
100 /var/lib/dpkg/status
Trusty
libmagic1:
Installed: 1:5.14-2ubuntu3.3
Candidate: 1:5.14-2ubuntu3.3
Version table:
*** 1:5.14-2ubuntu3.3 500
500 http://security.ubuntu.com/ubuntu trusty-security/main amd64 Packages
100 /var/lib/dpkg/status
* ckan 2.9+ compatibilities
paster is replaced with ckan command from 2.9+
unicode, basestring is now just str in python3
deal with version and python differences in the cicd build
* python3 flake
ur not supported in ptyon3, use br"".decode('raw_unicode_escape')
utc fix, cicd pipeline
Bumps [responses](https://github.com/getsentry/responses) from 0.8.1 to 0.12.0. - [Release notes](https://github.com/getsentry/responses/releases) - [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES) - [Commits](getsentry/responses@0.8.1...0.12.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Bump responses from 0.8.1 to 0.12.0
[QOL-6491] enable file downloads to go through a proxy if needed
[QOL-7596] apply changes from upstream to prepare for CKAN 2.9
[QOL-7596] use 'six' instead of assuming features will exist
* Initial setup of github actions based on ckanext-scheming style * full text search needs to be set before tests can be run * update full text function as database owner instead of datastore user * allow the datastore user to overwrite the full text function for the purposes of unit testing - this is the account that the tests use * let the datastore account own the full text function for unit testing purposes * don't force ownership of the full text function to ckan_default in the script when it might not be the right account * [QOL-7596] fix import of six.moves.zip * install dev requirements before running tests Co-authored-by: william dutton <will.dutt@gmail.com> Co-authored-by: ThrawnCA <shell_layer-github@yahoo.com.au>
Develop to master
… into test_invalid_byte_sequence
- standardise whitespace, trim unused imports/variables
…ov-au/ckanext-xloader into QOL-7596-python-3
Add back the stefina test
Qol 7596 python 3
- use parameterised logging - use six and close resources for better robustness
QOL-7596 Merge 0.9.0 from upstream
… QOL-7596-ckan-2.9
QOL-7596 ckan 2.9
QOL-7596 merge upstream bugfix
- drop unnecessary code - use toolkit API more consistently - drop obsolete pip-requirements.txt, just use requirements.txt - handle CKAN 2.10 style API tokens - use double quotes for strings instead of single, this lets us use apostrophes
… QOL-9159-upstream-sync
QOL-9159 sync from upstream
- Add a constantly changing query string parameter to our download URLs for uploaded resources, to ensure that XLoader doesn't retrieve an old cached copy.
…an#171, QOL-9327 - On resource update, compare the flag to the actual state of the datastore, and update the flag if they don't match
Since GitHub Actions workflows are triggered on every push, triggering on pull request is usually redundant. The exception is when pull requests are opened across repositories. However, it should be sufficient to trigger a build on pull request if the target branch is 'master'. This will ensure that changes can't be merged into the master branch without being tested first, but will not always trigger extra builds for pull requests internal to a repo.
…rces QOL-9327 Make datastore loading more robust, ckan#171
…rces QOL-9327 fix syntax for new query string parameter
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.