Update validation pipeline to use a json schema #420
Conversation
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
…e into 403-json-schema-validate
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
|
@jpmckinney the json schema check is failing due to camel case codelist and fields, but I think that we cannot avoid this, right? |
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
|
@jpmckinney I'm still having some json schema errors |
|
@yolile No worries - I'll relax the validations today. (They are strict because so far we've only used JSON Schema for the standard.) |
kingfisher_scrapy/pipelines.py
Outdated
| for item in ['file', 'file_error', 'file_item']: | ||
| filename = os.path.join(schema_path, f'{item}.json') | ||
| with open(filename) as f: | ||
| schema = jsonref.load(f, base_uri=pathlib.Path(os.path.join(schema_path), 'item_schema').as_uri()) |
There was a problem hiding this comment.
Hmm, 'item_schema' is outside the os.path.join call (which we don't need if it's only one argument), and in any case schema_path already contains item_schema. I'm not sure how this works.
Since we're re-using the base_uri parameter value, we can set schema_path to it above.
There was a problem hiding this comment.
@jpmckinney I changed it but for some reason jsonref fails if I don't put 'item_schema' again in base_uri I'm not sure why
There was a problem hiding this comment.
Aha, I had to add a trailing slash to the URI.
…e into 403-json-schema-validate Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
…e into 403-json-schema-validate Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
Signed-off-by: Yohanna Lisnichuk <yohanitalisnichuk@gmail.com>
… values from arrays to strings.
…moved during dereferencing
closes #403