Datasets periodic refresh and centralised Jobs/Datasets memory store management. Fixes #375 #395
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
luipir
commented
Apr 29, 2021
| return wrapper | ||
|
|
||
|
|
||
| def json_serial(obj): |
There was a problem hiding this comment.
FYI moved here to be used in other context
luipir
commented
Apr 29, 2021
| job_dict['script'] = {"name": script_name, "slug": script_slug} | ||
|
|
||
| # do import here to avoid circular import | ||
| from LDMP.jobs import Jobs |
There was a problem hiding this comment.
import here to avoid circular import
luipir
commented
Apr 29, 2021
|
|
||
| class tr_jobs(object): | ||
| def tr(message): | ||
| return QCoreApplication.translate("tr_jobs", message) | ||
|
|
||
|
|
||
| def json_serial(obj): |
There was a problem hiding this comment.
moved to init to be used also by Datasets.py
luipir
commented
Apr 29, 2021
|
|
||
|
|
||
| @singleton | ||
| class Jobs(QObject): |
There was a problem hiding this comment.
DataStore containing all downloaded Jobs responses
luipir
commented
Apr 29, 2021
| def __init__(self, datasets: Optional[List[Dataset]] = None): | ||
|
|
||
| @singleton | ||
| class Datasets(QObject): |
There was a problem hiding this comment.
centralised DataStore with all Datasets availabel in disk or generated by Jobs
luipir
commented
Apr 29, 2021
| self.updated.emit() | ||
|
|
||
|
|
||
| class DatasetSchema(Schema): |
There was a problem hiding this comment.
schema is located here and not in shema.py because schema.py belogs to a nested project and I do not want/permission to touch it
ricardogsilva
approved these changes
May 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Datasets are scanned from:
cached jobs
eventually any other josn present in base_data_directly (jumping jobs folder)
actually polling is set to 60000ms get form config: "trends_earth/advanced/refresh_polling_time"
next step are:
check for local generated Jobs/Datasets
download remote datasets
NOTE. There are some architecturally critical aspects:
A) Originally there is no difference between Job/Dataset => I hope I coded to avoid misalignments
B) Jobs=Dataset and Job list are only a API image of that available remotely => Jobs and Dataset are naturally aligned
C) Job Dataset are download in a user defined location. e.g. there is no constraint where to save datasets or job descriptors => this can be a source of misalignment.
To approach has these critical aspects:
Jobs and Datasets are sets of Job and Dataset and are singletons (kind of memory store with an image of it's components as json file in base_data_directory
Any Jobs or Dataset know where save itself (using dump method)
=> still need a procedure and criteria to clean Jobs/Datasets