Wildcards broken in write block

[soniamitchell]

When attempting to generate a working config from the following write block:

write:
- data_product: data_product/write/wildcard/b873cb1e1a51f1b8c9306c35b4d34cb08b82526f/*
  description: A csv file
  file_type: csv
  use:
    version: ${{MAJOR}}

A working config is generated without a write block and the following warning is returned:

WARNING:FAIRDataPipeline.ConfigYAML:Missing use:data_product in {'data_product': 'data_product/write/wildcard/b873cb1e1a51f1b8c9306c35b4d34cb08b82526f/1/2', 'description': 'A csv file', 'file_type': 'csv', 'use': {'namespace': 'testing', 'version': '${{MAJOR}}'}, 'public': True}
WARNING:FAIRDataPipeline.ConfigYAML:Missing use:data_product in {'data_product': 'data_product/write/wildcard/b873cb1e1a51f1b8c9306c35b4d34cb08b82526f/1', 'description': 'A csv file', 'file_type': 'csv', 'use': {'namespace': 'testing', 'version': '${{MAJOR}}'}, 'public': True}

[jwscook]

After marking & Christmas @richardreeve to write down logic of this here so that @kzscisoft can integrate into the current architecture.

[jwscook]

@soniamitchell, when you're back after the break would you mind putting in more details in here and #137 about how you generated those warnings please?

[soniamitchell]

Sure! What other information do you need?

[jwscook]

Any and all details needed for @kzscisoft to be able to reproduce those warnings/error messages and how he will know that he's fixed the problem.

[richardreeve]

@soniamitchell - it turns out the the examples I got Kristian to incorporate into the testing when he refactored fair didn't include any wildcard examples, so if there are any that use read and write wildcards (or if it's easy to create them) then that would be perfect. My mistake - sorry!

[soniamitchell]

All information necessary to implement it is in my comment above (minus the run_metadata block of course, which will depend on where you're running it from).

The outputs should look something like the examples given here and here.

The only thing we need to think about is the fact that wildcards can appear anywhere in the string..

I hope that helps!

[kzscisoft]

I am still somewhat confused, the example does not include any source reference, i.e. where data_product/write/wildcard/b873cb1e1a51f1b8c9306c35b4d34cb08b82526f/* obtains its info from. What input/setup is needed such that the expansion of this wildcard retrieves the correct data to then give the listing you intend?

I am guessing I need a job/model run which makes these items?

[soniamitchell]

I don't know how you've set up your unit tests, but I assumed you'd been using the python DP API?

[soniamitchell]

I would just generate some simple examples, first writing data_product/1 and data_product/1/2 to the registry, then requesting data_product/* in the read: block, to return data_product/1 and data_product/1/2 in the working config. Then requesting data_product/* in the write: block to return data_product/1, data_product/1/2, and data_product/*. But also bearing in mind that the wildcard can appear at any point in the string.

[kzscisoft]

I don't know how you've set up your unit tests, but I assumed you'd been using the python DP API?

Not necessarily, the ideal unit test would hone in on just the wildcard expansion without any need for any calling of external codes. If someone could give me the exact scenarios they want to be covered using the registry example data that would be great.

[soniamitchell]

I'm afraid I don't know anything about registry example data. Is it preinstalled? or are you talking about a db.sqlite3 file?

[richardreeve]

Hi @kzscisoft - sorry I've been out of this discussion. I was trying to get my post-Christmas marking done, but it's taking longer than I hoped (as always!). However, I'm afraid I'm with @soniamitchell - I don't know what's in the registry example data... can you point to where that is explained (if anywhere) or instructions on how to install it?

[kzscisoft]

fair registry install
cd $HOME/.fair/registry
venv/bin/python manage.py add_example_data

Note this does not add actual files, so you may encounter issues (I use mocking in tests to overwrite the file retrieval capability)