-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Oracle-ES consistency #240
Commits on Apr 3, 2019
-
Add consistency query to stage 009.
The first step in ensuring consistency between Oracle and ES is to obtain a very basic set of task data - id and timestamp - from Oracle. Add a query for doing so.
Configuration menu - View commit details
-
Copy full SHA for 9903938 - Browse repository at this point
Copy the full SHA 9903938View commit details -
Configuration menu - View commit details
-
Copy full SHA for 42d55b3 - Browse repository at this point
Copy the full SHA 42d55b3View commit details -
Configuration menu - View commit details
-
Copy full SHA for f27f278 - Browse repository at this point
Copy the full SHA f27f278View commit details -
Check for index' existence before working.
es.get() raises NotFoundError in both cases - when index does not exist and when document does not exist. Also, it's more reasonable to check index once since it's the same for all messages.
Configuration menu - View commit details
-
Copy full SHA for af9f212 - Browse repository at this point
Copy the full SHA af9f212View commit details -
- Add/update functions and their parameters' descriptions. - Update the script's description. - Add consistency' description into README.
Configuration menu - View commit details
-
Copy full SHA for 4c560a6 - Browse repository at this point
Copy the full SHA 4c560a6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2296f6d - Browse repository at this point
Copy the full SHA 2296f6dView commit details
Commits on Apr 5, 2019
-
Check that all fields supplied in input data are present in ES and their values are matching the input data, instead of working only with tasks and their timestamps. This will allow checking tasks' other fields as well as different types of documents such as datasets. Add stage 016 into consistency chain because it adds the fields required for getting documents of given type from ES.
Configuration menu - View commit details
-
Copy full SHA for b8a2ab1 - Browse repository at this point
Copy the full SHA b8a2ab1View commit details -
Save and display the info about different tasks.
Prepare the script for further development, where incosistent tasks will be automatically reloaded into ES.
Configuration menu - View commit details
-
Copy full SHA for acfe45a - Browse repository at this point
Copy the full SHA acfe45aView commit details -
Configuration menu - View commit details
-
Copy full SHA for e39efe2 - Browse repository at this point
Copy the full SHA e39efe2View commit details
Commits on Apr 17, 2019
-
Move certain shell functions to library.
These functions are either used by several scripts or will be in the future. Move them to library to uphold DRY principle.
Configuration menu - View commit details
-
Copy full SHA for 8a71791 - Browse repository at this point
Copy the full SHA 8a71791View commit details -
DEBUG mode in data4es-start exists to check the workflow without uploading anything to ES. Consistency check writes nothing, so DEBUG is unnecessary here. Do not redirect the stages' stderrs, leave them as-is.
Configuration menu - View commit details
-
Copy full SHA for 90380a9 - Browse repository at this point
Copy the full SHA 90380a9View commit details -
Move ES consistency script into a separate stage.
While the script is the stage 069's counterpart in data4es-consistency-check, they share no functionality.
Configuration menu - View commit details
-
Copy full SHA for 7bac202 - Browse repository at this point
Copy the full SHA 7bac202View commit details
Commits on Apr 18, 2019
-
- State what is retrieved by the query. - Remove unnecessary information.
Configuration menu - View commit details
-
Copy full SHA for 12dd86e - Browse repository at this point
Copy the full SHA 12dd86eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 944b5a2 - Browse repository at this point
Copy the full SHA 944b5a2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 26a1dfe - Browse repository at this point
Copy the full SHA 26a1dfeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 20875b1 - Browse repository at this point
Copy the full SHA 20875b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 46cf0af - Browse repository at this point
Copy the full SHA 46cf0afView commit details -
Update ES parameters handling.
- Show an error message and exit if no host, port, or index is specified. - Remove default values of the parameters.
Configuration menu - View commit details
-
Copy full SHA for 72d85a9 - Browse repository at this point
Copy the full SHA 72d85a9View commit details -
Remove batching of inconsistent records.
Printing all discovered inconsistent records to stdout as a batch contradicts with various things, such as pyDKB's file mode and the possibility of controlling the workflow with Apache Kafka. Create an output message with _id and _type for each inconsistent record. Still exit with code 1 if at least one inconsistent record was found, 0 otherwise.
Configuration menu - View commit details
-
Copy full SHA for cacba11 - Browse repository at this point
Copy the full SHA cacba11View commit details
Commits on Apr 19, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 4d8eb83 - Browse repository at this point
Copy the full SHA 4d8eb83View commit details -
- Add output samples for stages 009 and 016. Stage 071 has no sample because all the tasks in 016's sample are up to date in ES. - Update READMEs. - Add symbolic link for stage 071's input.
Configuration menu - View commit details
-
Copy full SHA for 181fb14 - Browse repository at this point
Copy the full SHA 181fb14View commit details -
Add information about the new stage and dataflow to the README.
Configuration menu - View commit details
-
Copy full SHA for 7117242 - Browse repository at this point
Copy the full SHA 7117242View commit details
Commits on May 21, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 165c5d2 - Browse repository at this point
Copy the full SHA 165c5d2View commit details
Commits on May 28, 2019
-
Type of _id is unknown - it can be str or int for task, and str for dataset.
Configuration menu - View commit details
-
Copy full SHA for 8f84d22 - Browse repository at this point
Copy the full SHA 8f84d22View commit details -
The field is required to get child documents such as output datasets.
Configuration menu - View commit details
-
Copy full SHA for d195650 - Browse repository at this point
Copy the full SHA d195650View commit details -
Remove service fields before checking.
Service fields are different from data fields and shouldn't be checked.
Configuration menu - View commit details
-
Copy full SHA for 612bf52 - Browse repository at this point
Copy the full SHA 612bf52View commit details -
Remove interpreter directives from lib files.
These are unnecessary because library files are not supposed to be executed.
Configuration menu - View commit details
-
Copy full SHA for 01ae258 - Browse repository at this point
Copy the full SHA 01ae258View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f86ddd - Browse repository at this point
Copy the full SHA 8f86dddView commit details