Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make validator GitHub Action #191

Merged

Conversation

CasperWA
Copy link
Member

@CasperWA CasperWA commented Feb 24, 2020

This PR adds the possibility of using the OPTiMaDe validator optimade_validator developed by @ml-evs as a GitHub Action.

It is a Docker-based action that supports both "local" servers running within the GitHub Actions runner, as well as external, deployed servers.

There are the following available inputs at this point (taken from the updated README.md):


GitHub Action - OPTiMaDe validator

This action runs optimade_validator from this repository on a running server.

Example usage

To run optimade_validator for an index meta-database at http://gh_actions_host:5001/optimade/v0 do the following:
Within the same job, first, start a server, e.g., using the docker-compose.yml setup from this repository, and then add the step

uses: Materials-Consortia/optimade-python-tools@master
with:
  port: 5001
  path: /optimade/v0
  index: yes

To run optimade_validator for a regular OPTiMaDe deployed implementation, testing all possible versioned base URLs:

  • https://example.org:443/optimade/example/v0
  • https://example.org:443/optimade/example/v0.10
  • https://example.org:443/optimade/example/v0.10.1
uses: Materials-Consortia/optimade-python-tools@master
with:
  protocol: https
  domain: example.org
  port: 443
  path: /optimade/example
  all versioned paths: True

Inputs

protocol

Optional Protocol for the OPTiMaDe URL.
Default: http

domain

Optional Domain for the OPTiMaDe URL (defaults to the GitHub Actions runner host).
Default: gh_actions_host

port

Optional Port for the OPTiMaDe URL.
Default: 5000

path

Optional Path for the OPTiMaDe (versioned) base URL - MUST start with /
Note: If all versioned paths is true, this MUST be un-versioned, e.g., /optimade, otherwise it MUST be versioned, e.g., the default value.
Default: /optimade/v0

all versioned paths

Optional Whether to test all possible versioned base URLs:

  • /vMAJOR
  • /vMAJOR.MINOR
  • /vMAJOR.MINOR.PATCH

If this is 'true', the input 'path' MUST exempt the version part (e.g., '/optimade' instead of '/optimade/v0').
If this is 'false', the input 'path' MUST include the version part (e.g., '/optimade/v0' instead of '/optimade').
Default: false

index

Optional Whether or not this is an index meta-database.
Default: false


Furthermore, the PR adds a separate CI file for the action, as well as updated invoke tasks to update the action with the newest supported OPTiMaDe version.

@CasperWA CasperWA added enhancement New feature or request validator Related to the OPTIMADE validator labels Feb 24, 2020
@CasperWA CasperWA requested a review from ml-evs February 24, 2020 19:20
Update README and invoke tasks.

The action runs the `optimade_validator` in a docker, with support for
validating "local" running servers within the GitHub Action runner, as
well as external, deployed servers.
Copy link
Member

@ml-evs ml-evs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really nice, thanks @CasperWA. Previously I had a similar thing set up from in separate repo using GitHub's secrets but I could never get the syntax working... This will encourage me to update the validator and try to verify it against the spec itself, rather than the subset we have implemented. I'll review it properly tomorrow when I'm able to test it.

@CasperWA
Copy link
Member Author

CasperWA commented Feb 25, 2020

(...) This will encourage me to update the validator and try to verify it against the spec itself, rather than the subset we have implemented. I'll review it properly tomorrow when I'm able to test it.

Wonderful!
I indeed also wanted to have the test run against provider.optimade.org as an index meta-database, testing deployed implementations, but the validator is of course too strict to allow that to pass.
I think we need to introduce the concept of optionality properly - without having to change too much in the models (preferably nothing at all).

ml-evs
ml-evs previously approved these changes Mar 2, 2020
Copy link
Member

@ml-evs ml-evs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @CasperWA, not sure I can test it on my server repo until we've merged here, but I can't see any problems :)

@CasperWA
Copy link
Member Author

CasperWA commented Mar 2, 2020

Thanks for this @CasperWA, not sure I can test it on my server repo until we've merged here, but I can't see any problems :)

You can use the validator temporarily for tests via the following:

uses: CasperWA/optimade-python-tools@make_validator_github_action
with:
  # ...

I would encourage you to test it before merging - if you have time :)

@ml-evs ml-evs self-requested a review March 3, 2020 18:14
action.yml Outdated Show resolved Hide resolved
@CasperWA CasperWA merged commit 71348db into Materials-Consortia:master Mar 4, 2020
@CasperWA CasperWA deleted the make_validator_github_action branch March 4, 2020 08:22
@CasperWA CasperWA mentioned this pull request Mar 5, 2020
CasperWA added a commit that referenced this pull request Mar 6, 2020
Up to v0.6.0.

**New features**:
- GitHub Action validator that runs `optimade_validator` for a locally running OPTiMaDe server (#191, @CasperWA, tested by @ml-evs)
- Support filter queries for `HAS ALL`, `HAS ANY` and `HAS ONLY` and value lists on MongoDB backends (#173, @ml-evs)
  Note: `OPERATOR` use in value lists are still _not_ supported.
- Debug mode. Start the server in debug mode to enable `debug` log-level in `uvicorn` and get a full Python traceback in the JSON response (#190, @CasperWA)
- Add testing of mandatory `filter` queries to `optimade_validator` (#205, @ml-evs)

**Updates**:
- Allow Cross-Origin requests from anywhere (`*`), i.e., enable CORS for both servers (#194, @CasperWA)
- Updates to models (correct misspelling, more transparent model class naming, streamline models with respect to python class inheritance, update field descriptions) (#195, @CasperWA)
- API change: Rename `optimade.models.toplevel.py` to `optimade.models.responses.py` (#195, @CasperWA)
- Update dependencies to newest versions (as of 02.03.2020) (#202, #196, @CasperWA)
- Move imports from `starlette` to `fastapi`, where possible (#202, @ml-evs)
- Remove custom middleware to redirect slashed URLs in favor of `starlette` implementation (#202, @ml-evs)
- CI tests are now performed with a real MongoDB in the backend. CI tests are also performed with a `mongomock` backend for the tests in `server/test_middleware.py`, `server/test_server_validation.py`, and `server/test_config.py` (#196, @ml-evs, additional testing by @CasperWA )
- Remove `/optimade` from base URLs. This was especially important for the OpenAPI schema (#201, #216, @CasperWA, @ml-evs)
- Check integrity of query part of the raw URL using a custom middleware (#209, @CasperWA)
- New `optimade/server/exceptions.py` to contain custom `HttpException`s, moved `BadRequest` here (#209, @CasperWA)
- Pattern and regex testing for `data.available_api_versions` parts in `/info` endpoint and fix tests for the same (#211, @CasperWA)
- Restructure import of test data for regular server (#212, @shyamd)

**Bug fixes**:
- New retrieval URL for Materials-Consortia's list of providers (#187, @CasperWA)
- Skip local `HAS ONLY` tests with a `mongomock` backend, since v3.19.0 does not support these (#206, @ml-evs)
- Resource ID's can now contain slashes (`/`) (#183, @ml-evs, @CasperWA)
- Remove _valid_ version part of base URL in `meta.query.representation` (#201, @CasperWA)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request validator Related to the OPTIMADE validator
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants