Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Oracle-ES consistency #240

Merged
merged 28 commits into from
May 28, 2019
Merged

Oracle-ES consistency #240

merged 28 commits into from
May 28, 2019

Commits on Apr 3, 2019

  1. Add consistency query to stage 009.

    The first step in ensuring consistency between Oracle and ES is to obtain
    a very basic set of task data - id and timestamp - from Oracle. Add a query
    for doing so.
    Evildoor committed Apr 3, 2019
    Configuration menu
    Copy the full SHA
    9903938 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    42d55b3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f27f278 View commit details
    Browse the repository at this point in the history
  4. Check for index' existence before working.

    es.get() raises NotFoundError in both cases - when index does not exist and
    when document does not exist. Also, it's more reasonable to check index once
    since it's the same for all messages.
    Evildoor committed Apr 3, 2019
    Configuration menu
    Copy the full SHA
    af9f212 View commit details
    Browse the repository at this point in the history
  5. Update documentation.

    - Add/update functions and their parameters' descriptions.
    - Update the script's description.
    - Add consistency' description into README.
    Evildoor committed Apr 3, 2019
    Configuration menu
    Copy the full SHA
    4c560a6 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    2296f6d View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2019

  1. Generalize 069-consistency.

    Check that all fields supplied in input data are present in ES and their
    values are matching the input data, instead of working only with tasks and
    their timestamps. This will allow checking tasks' other fields as well as
    different types of documents such as datasets.
    
    Add stage 016 into consistency chain because it adds the fields required
    for getting documents of given type from ES.
    Evildoor committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    b8a2ab1 View commit details
    Browse the repository at this point in the history
  2. Save and display the info about different tasks.

    Prepare the script for further development, where incosistent tasks will be
    automatically reloaded into ES.
    Evildoor committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    acfe45a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e39efe2 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2019

  1. Move certain shell functions to library.

    These functions are either used by several scripts or will be in the future.
    Move them to library to uphold DRY principle.
    Evildoor committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    8a71791 View commit details
    Browse the repository at this point in the history
  2. Remove DEBUG mode.

    DEBUG mode in data4es-start exists to check the workflow without uploading
    anything to ES. Consistency check writes nothing, so DEBUG is unnecessary
    here.
    Do not redirect the stages' stderrs, leave them as-is.
    Evildoor committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    90380a9 View commit details
    Browse the repository at this point in the history
  3. Move ES consistency script into a separate stage.

    While the script is the stage 069's counterpart in data4es-consistency-check,
    they share no functionality.
    Evildoor committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    7bac202 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2019

  1. Update a query description.

    - State what is retrieved by the query.
    - Remove unnecessary information.
    Evildoor committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    12dd86e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    944b5a2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    26a1dfe View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    20875b1 View commit details
    Browse the repository at this point in the history
  5. Fix pop() results handling.

    Evildoor committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    46cf0af View commit details
    Browse the repository at this point in the history
  6. Update ES parameters handling.

    - Show an error message and exit if no host, port, or index is specified.
    - Remove default values of the parameters.
    Evildoor committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    72d85a9 View commit details
    Browse the repository at this point in the history
  7. Remove batching of inconsistent records.

    Printing all discovered inconsistent records to stdout as a batch contradicts
    with various things, such as pyDKB's file mode and the possibility of
    controlling the workflow with Apache Kafka.
    
    Create an output message with _id and _type for each inconsistent record.
    Still exit with code 1 if at least one inconsistent record was found,
    0 otherwise.
    Evildoor committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    cacba11 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2019

  1. Configuration menu
    Copy the full SHA
    4d8eb83 View commit details
    Browse the repository at this point in the history
  2. Add consistency data samples.

    - Add output samples for stages 009 and 016. Stage 071 has no sample because
      all the tasks in 016's sample are up to date in ES.
    - Update READMEs.
    - Add symbolic link for stage 071's input.
    Evildoor committed Apr 19, 2019
    Configuration menu
    Copy the full SHA
    181fb14 View commit details
    Browse the repository at this point in the history
  3. Update the dataflow README.

    Add information about the new stage and dataflow to the README.
    Evildoor committed Apr 19, 2019
    Configuration menu
    Copy the full SHA
    7117242 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2019

  1. Configuration menu
    Copy the full SHA
    165c5d2 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2019

  1. Change messages formatting.

    Type of _id is unknown - it can be str or int for task, and str for dataset.
    Evildoor committed May 28, 2019
    Configuration menu
    Copy the full SHA
    8f84d22 View commit details
    Browse the repository at this point in the history
  2. Add _parent field handling.

    The field is required to get child documents such as output datasets.
    Evildoor committed May 28, 2019
    Configuration menu
    Copy the full SHA
    d195650 View commit details
    Browse the repository at this point in the history
  3. Remove service fields before checking.

    Service fields are different from data fields and shouldn't be checked.
    Evildoor committed May 28, 2019
    Configuration menu
    Copy the full SHA
    612bf52 View commit details
    Browse the repository at this point in the history
  4. Remove interpreter directives from lib files.

    These are unnecessary because library files are not supposed to be executed.
    Evildoor committed May 28, 2019
    Configuration menu
    Copy the full SHA
    01ae258 View commit details
    Browse the repository at this point in the history
  5. Simplify a field retrieval.

    Evildoor committed May 28, 2019
    Configuration menu
    Copy the full SHA
    8f86ddd View commit details
    Browse the repository at this point in the history