Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests: Make resources locally accessible #76

Closed
adamjanovsky opened this issue May 13, 2021 · 5 comments
Closed

Tests: Make resources locally accessible #76

adamjanovsky opened this issue May 13, 2021 · 5 comments
Assignees
Labels
CI/CD Stuff related to tests, deployment, docker, ... refactoring Need to cleanup and refactor

Comments

@adamjanovsky
Copy link
Collaborator

For now, many of tests rely on downloads from commoncriteria.org, which itself is quite unreliable webpage. Apart from test that check the correctness of such downloads, we should rely more on local resources, so that the tests don't fail due to inaccessible cc web.

@adamjanovsky adamjanovsky added the CI/CD Stuff related to tests, deployment, docker, ... label May 13, 2021
@adamjanovsky adamjanovsky self-assigned this May 13, 2021
@J08nY
Copy link
Member

J08nY commented May 13, 2021

We could maybe cache the commoncriteria contents somewhere (I could setup a caching proxy at seccerts.org) or just dump it periodically (again at seccerts.org) and point tests there.

@adamjanovsky
Copy link
Collaborator Author

adamjanovsky commented May 13, 2021

No need really, I was just being comfy. Basically we distinguish two situations:

  1. We want to test download from CC site: We should really download from CC site
  2. We want to test something else: Usually one pdf will do, no more than 5 distinct are needed, we can store them in git, they do not change.

The problem is that even in the second case, I'm now fetching the pdfs from commoncriteria.org, which sometimes fails.

@adamjanovsky
Copy link
Collaborator Author

Temporary workaround -- make those test not fail the pipeline.

@adamjanovsky adamjanovsky mentioned this issue Feb 12, 2022
5 tasks
@adamjanovsky
Copy link
Collaborator Author

@mmstanone Can you imagine dividing your tests into two parts: (i) tests of downloads; (ii) test of other functionality? Where:

  • (i) test that downloading works properly from the website, these are allowed to fail
  • (ii) tests anything else and uses local pre-downloaded resources that. Alternatively, these don't have to be hosted in git, but on seccerts.org. Probably, we would need some toy datasets, otherwise we get too large downloads.

@adamjanovsky adamjanovsky added the refactoring Need to cleanup and refactor label Feb 12, 2022
@adamjanovsky
Copy link
Collaborator Author

Tests that require remote and unreliable resources can fail now. This was done as a part of #275

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Stuff related to tests, deployment, docker, ... refactoring Need to cleanup and refactor
Projects
None yet
Development

No branches or pull requests

2 participants