Verify spec

[FynnBe]

also contains:

• changes to adhere to format version 0.3.0, which is not yet finalized.
• draft of a runner for scikit-learn models (we are not planning to officially support sklearn models at this point)

[oeway]

Nice, we should also add this verification to the compilation script and generate a report for each model (e.g. highlight as an error icon on the model card if error detected).

[oeway]

@FynnBe Can you generate an error report after verification, not only throw error but output details about the error?

[FynnBe]

@FynnBe Can you generate an error report after verification, not only throw error but output details about the error?

example from travis:

$ git clone --depth=50 https://github.com/ilastik/bioimage-io-models.git ilastik/bioimage-io-models
0.01s0.01s$ source ~/virtualenv/python3.7/bin/activate
$ python --version
Python 3.7.1
$ pip --version
pip 20.1.1 from /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/pip (python 3.7)
install
12.57s$ pip install git+https://github.com/bioimage-io/python-bioimage-io.git@verify_spec
2.01s$ python -m pybio.spec verify-bioimageio-manifest manifest.bioimage.io.yaml
invalid model: https://raw.githubusercontent.com/bioimage-io/pytorch-bioimage-io/v0.1.1/specs/models/unet2d/nuclei_broad/UNet2DNucleiBroad.model.yaml
{'inputs': {0: {'description': ['Missing data for required field.']}},
 'optional_kwargs': ['Unknown field.'],
 'outputs': {0: {'description': ['Missing data for required field.']}},
 'prediction': ['Unknown field.'],
 'test_input': ['Unknown field.'],
 'test_output': ['Unknown field.'],
 'training': ['Unknown field.'],
 'weights': ['Missing data for required field.'],
 'weights_format': ['Missing data for required field.']}
invalid model: https://raw.githubusercontent.com/wolny/pytorch-3dunet/37f186c80f4d64b1dab5d165d8c2aae15b5aede1/bioimage-io/UNet3DArabidopsisOvules.model/UNet3DArabidopsisOvules.model.yaml
{'inputs': {0: {'description': ['Missing data for required field.']}},
 'optional_kwargs': ['Unknown field.'],
 'outputs': {0: {'description': ['Missing data for required field.']}},
 'prediction': ['Unknown field.'],
 'test_input': ['Unknown field.'],
 'test_output': ['Unknown field.'],
 'weights': ['Missing data for required field.'],
 'weights_format': ['Missing data for required field.']}
invalid model: https://raw.githubusercontent.com/platybrowser/platybrowser/3711f1c26e5db8c38c3faff4cccb3110560e3c67/segmentation/cells/UNet3DPlatyCellProbs.model/UNet3DPlatyCellProbs.model.yaml
{'inputs': {0: {'description': ['Missing data for required field.']}},
 'optional_kwargs': ['Unknown field.'],
 'outputs': {0: {'description': ['Missing data for required field.']}},
 'prediction': ['Unknown field.'],
 'test_input': ['Unknown field.'],
 'test_output': ['Unknown field.'],
 'weights': ['Missing data for required field.'],
 'weights_format': ['Missing data for required field.']}
invalid model: https://raw.githubusercontent.com/subeeshvasu/hbp-DL-seg-codes/0.1.2/UNetDA.model.yaml
{'inputs': {0: {'description': ['Missing data for required field.']}},
 'optional_kwargs': ['Unknown field.'],
 'outputs': {0: {'description': ['Missing data for required field.']}},
 'prediction': ['Unknown field.'],
 'test_input': ['Unknown field.'],
 'test_output': ['Unknown field.'],
 'weights': ['Missing data for required field.'],
 'weights_format': ['Missing data for required field.']}
invalid model: https://raw.githubusercontent.com/subeeshvasu/hbp-DL-seg-codes/0.1.2/2sUNetDA.model.yaml
{'inputs': {0: {'description': ['Missing data for required field.']}},
 'optional_kwargs': ['Unknown field.'],
 'outputs': {0: {'description': ['Missing data for required field.']}},
 'prediction': ['Unknown field.'],
 'test_input': ['Unknown field.'],
 'test_output': ['Unknown field.'],
 'weights': ['Missing data for required field.'],
 'weights_format': ['Missing data for required field.']}
The command "python -m pybio.spec verify-bioimageio-manifest manifest.bioimage.io.yaml" exited with 1.
Done. Your build exited with 1.

[oeway]

This looks great!

[constantinpape]

The checker works for my most current v3 spec:
https://github.com/hci-unihd/batchlib/blob/master/misc/models/torch/bioimageio/UNetCovidIf.model.yaml

What's left to do before merging?

[FynnBe]

What's left to do before merging?

get the tests to pass...

[m-novikov]

Use Dict instead of OrderedDict

[FynnBe]

I went with MutableMapping instead, but seem to have forgotten it here... I'll fix it

[m-novikov]

I don't know if MutableMapping implies ordering, while python Dict does.

[FynnBe]

I am a bit puzzled why this fails:

self = PosixPath('/home/travis/build/bioimage-io/python-bioimage-io/tests/conftest.py/../../specs/models/sklearn/RandomForestClassifier_v0_1.model.yaml')
name = '/home/travis/build/bioimage-io/python-bioimage-io/tests/conftest.py/../../specs/models/sklearn/RandomForestClassifier_v0_1.model.yaml'
flags = 524288, mode = 438
    def _opener(self, name, flags, mode=0o666):
        # A stub for the opener argument to built-in open()
>       return self._accessor.open(self, flags, mode)
E       NotADirectoryError: [Errno 20] Not a directory: '/home/travis/build/bioimage-io/python-bioimage-io/tests/conftest.py/../../specs/models/sklearn/RandomForestClassifier_v0_1.model.yaml'

[FynnBe]

I am a bit puzzled why this fails:

for PosixPaths this does not seem legal: posix_path/a_file.ext/../elsewhere