backend: make crawlconfigs mutable! (#656) #662
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tw4l
reviewed
Mar 2, 2023
tw4l
force-pushed
the
mutable-crawlconfig
branch
3 times, most recently
from
130195e to
84f8c26
Compare
|
I tested the migration manually by creating some crawls and crawl configs on version |
tw4l
force-pushed
the
mutable-crawlconfig
branch
from
e946aa5 to
1974d70
Compare
|
@ikreymer I've now tested migrations from an empty slate and from existing 0002 data and all is looking good in terms of being able to re-run configs after migration and create new configs/crawls. API returns what I'd expect as well. |
tw4l
force-pushed
the
mutable-crawlconfig
branch
from
1318b33 to
85fb4e0
Compare
4 tasks
ikreymer
commented
Mar 6, 2023
ikreymer
commented
Mar 6, 2023
ikreymer
commented
Mar 6, 2023
SuaYoo
force-pushed
the
mutable-crawlconfig
branch
from
61caf2f to
42d64d5
Compare
SuaYoo
force-pushed
the
mutable-crawlconfig
branch
2 times, most recently
from
b02ee68 to
5603d41
Compare
- crawlconfig PATCH /{id} can now receive a new JSON config to replace the old one (in addition to scale, schedule, tags)
- exclusions: add / remove APIs mutate the current crawlconfig, do not result in a new crawlconfig created
- k8s: crawlconfig json is updated along with scale
- k8s: stateful set is restarted by updating annotation, instead of changing template
- crawl object: now has 'config', as well as 'profileid', 'schedule', and 'crawlTimeout' properties to ensure anything that is changeable
is stored on the crawl
- crawlconfig object: remove 'oldId', 'newId', disallow deactivating/deleting while crawl is running
- store each revision separate mongo db collection
- revisions accessible via /crawlconfigs/{cid}/revs
- update 'userid' on each crawlconfig modification, use 'modified' for modification date, 'useridCreated' for initial creation user
- store 'rev' int in crawlconfig and in crawljob
- backwards compatibility: add_crawl_config calls update_crawl_config and returns same id if oldId provided
…tadata (tags, name) tests: add tests for updating config, revision history api
…ed/removed, unify add/remove exclusions on crawl remove unused 'completions' field
…nal (just in case)
SuaYoo
force-pushed
the
mutable-crawlconfig
branch
from
b543d3c to
d3178b5
Compare
- update profiles: update PROFILE_FILENAME in configmap is profileid provided, clear if "" - return 'settings_changed' and 'metadata_changed' if either crawl settings or metadata changed - return if neither is true - update tests to check settigns/metadata changed return values
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Still do:
This change will require a DB migration, to remove crawlconfig props, and populate new props on the crawl