Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor FIPS & unify CLI actions #275

Merged
merged 83 commits into from
Dec 8, 2022
Merged
Show file tree
Hide file tree
Changes from 75 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
a900b90
unify CLI actions
adamjanovsky Oct 25, 2022
0e12ba4
merge dataset constructors, from_web_latest code
adamjanovsky Oct 25, 2022
7aff8bf
unify _get_certs_by_name methods
adamjanovsky Oct 25, 2022
d0d9f91
unify get_keywords_df method
adamjanovsky Oct 25, 2022
1b150aa
unify and generalize dataset method get_keywords_df()
adamjanovsky Oct 27, 2022
5110b52
root_dir setter for FIPSDataset
adamjanovsky Oct 27, 2022
39c89c1
WiP: refactor FIPS get_certs_from_web()
adamjanovsky Oct 27, 2022
9433658
implement artifact download FIPS
adamjanovsky Oct 27, 2022
933b469
refactor tests unittest -> pytest
adamjanovsky Nov 4, 2022
80af3a2
add type hint for json serialization
adamjanovsky Nov 9, 2022
6134479
new object to hold auxillary datasets
adamjanovsky Nov 9, 2022
fad4fbf
use temp folders for cc analysis test data
adamjanovsky Nov 9, 2022
e26fb0c
mark further download tests with xfail
adamjanovsky Nov 9, 2022
14c369b
fix xfail marker on cpe_dset_from_web test
adamjanovsky Nov 9, 2022
8482d82
pandas tests, cve_dset, cpe_dset unify json_path approach
adamjanovsky Nov 10, 2022
c6913dc
merge main
adamjanovsky Nov 10, 2022
24f11fb
test maintenance updates
adamjanovsky Nov 10, 2022
9052138
fix paths handling in CPEDataset, CVEDataset
adamjanovsky Nov 11, 2022
67c6295
cleanup path issues
adamjanovsky Nov 11, 2022
1bc1c7d
auxillary dataset processing CC
adamjanovsky Nov 11, 2022
ca5d4c8
fix mypy error in cli.py
adamjanovsky Nov 11, 2022
b41fa12
fix error in mu dset tests
adamjanovsky Nov 11, 2022
d51c65c
FIPS policy pdf convert refactoring
adamjanovsky Nov 11, 2022
f7c5915
cleanup in fips code structure
adamjanovsky Nov 11, 2022
9fbba9e
common interface for Dataset.analyze_certificates()
adamjanovsky Nov 16, 2022
c0ca076
merge main
adamjanovsky Nov 16, 2022
2a694c4
delete plot_graph() of FIPSDataset
adamjanovsky Nov 16, 2022
9462624
analyce_certificate() interface, delete dead code
adamjanovsky Nov 16, 2022
01e4156
FIPSDataset new parsing of html modules
adamjanovsky Nov 17, 2022
315270a
fix tests
adamjanovsky Nov 23, 2022
2871384
refactor algorithm extraction from policy tables
adamjanovsky Nov 23, 2022
048f3f6
delete InternalState.errors of cert objects
adamjanovsky Nov 23, 2022
4e582e1
deduplicate FIPSAlgorithm data structures
adamjanovsky Nov 23, 2022
0568ca4
remove graphviz requirement
adamjanovsky Nov 23, 2022
d7603e1
move AlgorithmDataset to AuxillaryDatasets class
adamjanovsky Nov 23, 2022
35c5734
Refactor FIPSAlgorithm objects
adamjanovsky Nov 25, 2022
25d42fc
update flake8 CI workflow
adamjanovsky Nov 25, 2022
67fc667
update flake8 config
adamjanovsky Nov 25, 2022
97dce48
cleanup
adamjanovsky Nov 25, 2022
4d0ae40
clean-up, update docs, cli
adamjanovsky Nov 25, 2022
cb879f3
fix json objects for fips test
adamjanovsky Nov 29, 2022
5895c85
rename dependency -> references of transitive vulns
adamjanovsky Nov 29, 2022
c6d826c
fips refactor reference computation
adamjanovsky Nov 30, 2022
fc49b6a
implement transitive vuln. search for FIPS
adamjanovsky Nov 30, 2022
cae2dc2
restrict usage of fresh bool param
adamjanovsky Nov 30, 2022
e062f3e
improve dataset processing logging
adamjanovsky Nov 30, 2022
5b0a7cb
fix table extraction from fips policies
adamjanovsky Dec 2, 2022
f681244
fix reference computation fips
adamjanovsky Dec 2, 2022
08ff031
update readme
adamjanovsky Dec 2, 2022
3fbf5f0
random fixes for cc pipeline
adamjanovsky Dec 2, 2022
6953dfb
fix CC notebooks
adamjanovsky Dec 2, 2022
6d7a907
random fixes in FIPS notebooks
adamjanovsky Dec 2, 2022
2f21854
move label studio interface layout file
adamjanovsky Dec 2, 2022
91b0973
update readme
adamjanovsky Dec 2, 2022
4ddae8a
introduce pyupgrade
adamjanovsky Dec 2, 2022
6d66552
bump scipy, dependabot errors on it
adamjanovsky Dec 2, 2022
40206cd
bump pillow lib
adamjanovsky Dec 2, 2022
25dcec9
bump Github action versions
adamjanovsky Dec 2, 2022
ca0c4e2
convert examples to notebooks
adamjanovsky Dec 2, 2022
86a62cb
fips normalize embodiment string
adamjanovsky Dec 2, 2022
6ce7007
unify from __future__ import annotations
adamjanovsky Dec 5, 2022
4e62ae1
Update sec_certs/dataset/common_criteria.py
adamjanovsky Dec 5, 2022
1a3502a
Update sec_certs/dataset/fips.py
adamjanovsky Dec 5, 2022
8f7a14b
entry guard
adamjanovsky Dec 5, 2022
4085c61
revive tests settings
adamjanovsky Dec 5, 2022
b37eaaf
fix here, fix there
adamjanovsky Dec 5, 2022
6c02383
rename dataset of maintenance updates
adamjanovsky Dec 5, 2022
8ac389a
Update sec_certs/dataset/common_criteria.py
adamjanovsky Dec 5, 2022
bc5a532
Update sec_certs/model/cpe_matching.py
adamjanovsky Dec 5, 2022
4712279
chain.from_iterable() now working with generator expessions
adamjanovsky Dec 5, 2022
f14dfe3
fix getitem on fips dataset
adamjanovsky Dec 5, 2022
a1ec986
test config global fixture
adamjanovsky Dec 6, 2022
577300e
add pyupgrade into linter pipeline
adamjanovsky Dec 6, 2022
ed8813e
reimplement dataset serialization constraints
adamjanovsky Dec 7, 2022
29dd48c
delete pp dataset json
adamjanovsky Dec 7, 2022
52bddce
update docs
adamjanovsky Dec 7, 2022
30ef160
attempt to fix pipelines
adamjanovsky Dec 8, 2022
0bcda6b
don't download spacy model test pipeline
adamjanovsky Dec 8, 2022
7710ef8
test pipeline ubuntu 20.04
adamjanovsky Dec 8, 2022
7af69ca
disable CPE from web test
adamjanovsky Dec 8, 2022
be7f6d7
try ubuntu 22.04 test runner
adamjanovsky Dec 8, 2022
7d59063
cli print -> click.echo()
adamjanovsky Dec 8, 2022
4574a3d
FIPSCertificate no longer hashable
adamjanovsky Dec 8, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions .flake8
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[flake8]
max-line-length = 120
exclude =
exclude =
.git,
__pycache__,
build,
Expand All @@ -13,10 +13,13 @@ max-complexity = 10

# Ignore complexity for CLI
per-file-ignores =
cc_cli.py: C901,
cc_cli.py: C901,
fips_cli.py: C901

ignore =
E501, # line length, should be handleded by black
W503, # line break before binary operator, depracated
E203, # whitespace before :, not PEP8 compliant
ignore =
# line length, should be handleded by black
E501,
# line break before binary operator, depracated
W503,
# whitespace before :, not PEP8 compliant
E203,
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v2
- uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Install external dependencies
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/draft_release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
if: (github.repository == 'crocs-muni/sec-certs') && (github.ref == 'refs/heads/main')
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v3
- name: Release
uses: softprops/action-gh-release@v1
with:
Expand Down
38 changes: 27 additions & 11 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ jobs:
run-mypy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name : Setup python
uses: actions/setup-python@v2
- uses: actions/checkout@v3
- name: Setup python
uses: actions/setup-python@v4
with:
python-version: '3.9'
python-version: "3.9"
- name: Install external dependencies
run: sudo apt-get install build-essential libpoppler-cpp-dev pkg-config python3-dev -y
- name: Install python dependencies
Expand All @@ -22,27 +22,43 @@ jobs:
black:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- uses: psf/black@stable
isort:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: 3.8
- uses: isort/isort-action@master
with:
requirementsFiles: "requirements/requirements.txt requirements/dev_requirements.txt"
requirementsFiles: "requirements/requirements.txt requirements/dev_requirements.txt"
pyupgrade:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup python
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install python dependencies
run: |
pip install -r requirements/requirements.txt
pip install -r requirements/dev_requirements.txt
- name: Run pyupgrade
run: pre-commit run pyupgrade --all-files
flake8-lint:
runs-on: ubuntu-latest
name: Flake8
steps:
- name: Check out source repository
uses: actions/checkout@v2
uses: actions/checkout@v3
- name: Set up Python environment
uses: actions/setup-python@v1
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: flake8 Lint
uses: py-actions/flake8@v2
uses: py-actions/flake8@v2
with:
plugins: "flake8-future-annotations"
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install build dependencies
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ jobs:
sudo add-apt-repository ppa:hlprasu/poppler
sudo apt-get update
sudo apt-get install -y build-essential libpoppler-cpp-dev pkg-config python3-dev
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Setup python
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: "3.8"
- name: Install python dependencies
Expand Down
25 changes: 16 additions & 9 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,24 +1,31 @@
repos:
- repo: https://github.com/asottile/pyupgrade
rev: v3.2.3
hooks:
- id: pyupgrade
args: ["--py38-plus"]
adamjanovsky marked this conversation as resolved.
Show resolved Hide resolved
- repo: https://github.com/psf/black
rev: 22.6.0
hooks:
- id: black
args: ['--check', '--target-version', 'py38']
args: ["--check", "--target-version", "py38"]
- repo: https://github.com/pycqa/isort
rev: 5.10.1
hooks:
- id: isort
args: ['--check-only']
args: ["--check-only"]
- repo: https://github.com/pre-commit/mirrors-mypy
rev: 'v0.982'
rev: "v0.982"
hooks:
- id: mypy
additional_dependencies:
- 'numpy'
- 'types-PyYAML'
- 'types-python-dateutil'
- 'types-requests'
- "numpy"
- "types-PyYAML"
- "types-python-dateutil"
- "types-requests"
- repo: https://github.com/pycqa/flake8
rev: '4.0.1'
rev: "4.0.1"
hooks:
- id: flake8
- id: flake8
additional_dependencies:
- "flake8-future-annotations"
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ All commits shall pass the lint pipeline of the following tools:
- Black (see [pyproject.toml](https://github.com/crocs-muni/sec-certs/blob/main/pyproject.toml) for settings)
- isort (see [pyproject.toml](https://github.com/crocs-muni/sec-certs/blob/main/pyproject.toml) for settings)
- Flake8 (see [.flake8](https://github.com/crocs-muni/sec-certs/blob/main/.flake8) for settings)
- PyUpgrade

These tools can be installed via [dev_requirements.txt](https://github.com/crocs-muni/sec-certs/blob/main/dev_requirements.txt) You can use [pre-commit](https://pre-commit.com/) tool register git hook that will evalute these checks prior to any commit and abort the commit for you. Note that the pre-commit is not meant to automatically fix the issues, just warn you.

Expand All @@ -65,6 +66,7 @@ To ivoke the tools manually, you can, in the repository root, use:
- Black: `black --check .` (without the flag to reformat)
- isort: `isort --check-only .` (without the flag to actually fix the issue)
- Flake8: `flake8 .`
- PyUpgrade: `pyupgrade --py38-plus 'find ./sec_certs/ -name "*.py" -type f'`

## Documentation

Expand Down
4 changes: 0 additions & 4 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ RUN DEBIAN_FRONTEND="noninteractive" apt-get -y install tzdata
RUN apt-get install build-essential libpoppler-cpp-dev pkg-config python3-dev -y
RUN apt-get install libqpdf-dev -y
RUN apt-get install default-jdk -y
RUN apt-get install graphviz -y
J08nY marked this conversation as resolved.
Show resolved Hide resolved
RUN apt-get install tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu tesseract-ocr-fra -y


Expand Down Expand Up @@ -51,9 +50,6 @@ RUN \
pip3 install --no-cache notebook jupyterlab && \
pip3 install -e .

# Download spacy language model
RUN python3 -m spacy download en_core_web_sm
adamjanovsky marked this conversation as resolved.
Show resolved Hide resolved

# #just to be sure that pdftotext is in $PATH
ENV PATH /usr/bin/pdftotext:${PATH}

Expand Down
Loading