Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[obsolete] DF/095: safe dataset metadata update #246

Open
wants to merge 4 commits into
base: 91-ds-safe
Choose a base branch
from

Commits on Apr 26, 2019

  1. pyDKB/common: rename custom_readline to utils.

    There can possibly be different functions, useful for some specific
    operations. It is possible to keep them all in the individual files, but
    it is not very comfortable to use (one have to type something like
    `from pyDKB.common.my_function_name import my_function_name`, or in
    `__init__` file have a list of imports like this:
    `from my_function_name import my_function_name`).
    
    So I think it will be fine to put them all to a common file ('utils' or
    'misc' or whatever) and use like `from pyDKB.common.utils import
    my_function_name`). Looks a little better (for me, at least).
    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    4dd8a17 View commit details
    Browse the repository at this point in the history
  2. pyDKB/common: move read_es_config() function from stage to library.

    Function like this seem to be needed in multiple places, so why not to
    have it in the common library.
    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    2b4a50e View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2019

  1. DF/095: allow stage to use DKB ES storage as second (backup) source.

    If `--es-config FILE` parameter is specified, use DKB ES storage
    as a backup metadata source in case that in primary source (AMI)
    information was removed.
    
    I am not sure under what curcumstances information can be removed from
    AMI, so for now we just check if there are empty/missed fields in the
    data taken from AMI and then, if such firlds found, check ES for
    (possibly) already known values.
    
    The problem is that there are quite a lot of "missed" values, so almost
    for every record we have to check both AMI and ES. Maybe there is more
    delicate trigger, like "No data at all", or "if dataset property
    'deleted' is set to True", or...
    
    Or maybe AMI just doesn't remove data at all?..
    mgolosova committed Apr 27, 2019
    Configuration menu
    Copy the full SHA
    a9d10ac View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2019

  1. DF/095: explicitly specify document type for ES get() request.

    In theory it should make the request slightly faster.
    mgolosova committed Apr 28, 2019
    Configuration menu
    Copy the full SHA
    7d31da3 View commit details
    Browse the repository at this point in the history