Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added asynchronous cvedb module. #758

Merged
merged 54 commits into from Jun 30, 2020
Merged

Added asynchronous cvedb module. #758

merged 54 commits into from Jun 30, 2020

Conversation

Niraj-Kamdar
Copy link
Contributor

@Niraj-Kamdar Niraj-Kamdar commented Jun 24, 2020 •
edited

Changes in this PR:

  • Use of aiohttp instead of multiprocessing to download NVD dataset in cvedb
  • Split cvedb into cvedb and cve_scanner
  • Renamed scanner to version_scanner
  • Renamed cve_id to cve_number since that's what NVD calls it.
  • Fixed eventloop bug.
  • No need to inherit from object now since Python 3 implicitly do that.
  • beautiful progressbar while downloading and updating database
  • It is only taking 1 minutes to update database 🎉
  • Using pytest for every tests in CI since pytest can run unittest and asyncio function can only be (easily) tested on pytest.
  • CI is only taking 20 minutes to run long tests!

SinghHrmn
SinghHrmn reviewed Jun 24, 2020
View reviewed changes
cve_bin_tool/cli.py Outdated Show resolved Hide resolved
@Niraj-Kamdar
Copy link
Contributor Author

This will fix bug of PR #755

@codecov-commenter
Copy link

codecov-commenter commented Jun 24, 2020 •
edited

Codecov Report

Merging #758 into master will increase coverage by 0.71%.
The diff coverage is 87.30%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #758      +/-   ##
==========================================
+ Coverage   87.62%   88.33%   +0.71%     
==========================================
  Files         136      137       +1     
  Lines        2473     2478       +5     
  Branches      304      302       -2     
==========================================
+ Hits         2167     2189      +22     
+ Misses        237      225      -12     
+ Partials       69       64       -5
Flag Coverage Δ
#longtests 88.33% <87.30%> (+0.71%) ⬆️
Impacted Files Coverage Δ
cve_bin_tool/csv2cve.py 63.72% <50.00%> (-2.28%) ⬇️
cve_bin_tool/file.py 95.45% <50.00%> (+23.04%) ⬆️
cve_bin_tool/strings.py 95.65% <50.00%> (+22.31%) ⬆️
cve_bin_tool/async_utils.py 89.65% <66.66%> (-2.02%) ⬇️
cve_bin_tool/version.py 76.47% <75.00%> (-23.53%) ⬇️
cve_bin_tool/cli.py 86.17% <77.77%> (-1.88%) ⬇️
cve_bin_tool/cvedb.py 85.47% <85.98%> (+2.70%) ⬆️
cve_bin_tool/OutputEngine.py 58.08% <100.00%> (-0.31%) ⬇️
cve_bin_tool/cve_scanner.py 100.00% <100.00%> (ø)
cve_bin_tool/extractor.py 61.06% <100.00%> (+0.92%) ⬆️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3632e74...a64ab12. Read the comment docs.

Niraj-Kamdar and others added 10 commits June 25, 2020 16:00
@Niraj-Kamdar
update yml
f07ee92 
don't set exit status breaks CI


black format


update yml
fix bugs
typo!
fix CI
try
yml don't have comment
@Niraj-Kamdar
continue if error
a054bdd
@Niraj-Kamdar
try fix
9bc9c42
@Niraj-Kamdar
fix cvedb
7eafe48
@Niraj-Kamdar
fix test
1afcf70
@Niraj-Kamdar
DRY
8a286d0
@Niraj-Kamdar
Merge pull request #3 from Niraj-Kamdar/paral
a9a5c35 
Fix eventloop bug and create helper function to run coroutine and fix CI
@Niraj-Kamdar
bug fix
e890fa8
@Niraj-Kamdar
cli has to be sync
e737855
@Niraj-Kamdar
Merge branch 'paral' into new
dc9ff4b
@Niraj-Kamdar
well indent!
e6d23be
@Niraj-Kamdar
update requirements
fead34c 
bs4 is dummy package beautifulsoup4 is real
Niraj-Kamdar
Niraj-Kamdar commented Jun 25, 2020
View reviewed changes
cve_bin_tool/OutputEngine.py Show resolved Hide resolved
cve_bin_tool/OutputEngine.py Show resolved Hide resolved
cve_bin_tool/async_utils.py Show resolved Hide resolved
cve_bin_tool/async_utils.py Show resolved Hide resolved
cve_bin_tool/async_utils.py Outdated Show resolved Hide resolved
cve_bin_tool/cli.py Show resolved Hide resolved
cve_bin_tool/cli.py Outdated Show resolved Hide resolved
cve_bin_tool/csv2cve.py Show resolved Hide resolved
cve_bin_tool/version.py Show resolved Hide resolved
pdxjohnny
pdxjohnny reviewed Jun 26, 2020
View reviewed changes
.github/workflows/pythonapp.yml Outdated Show resolved Hide resolved
cve_bin_tool/async_utils.py Outdated Show resolved Hide resolved
cve_bin_tool/cvedb.py Show resolved Hide resolved
cve_bin_tool/cvedb.py Outdated Show resolved Hide resolved
cve_bin_tool/version_scanner.py Outdated Show resolved Hide resolved
test/test_cvedb.py Outdated Show resolved Hide resolved
cve_bin_tool/cve_scanner.py Show resolved Hide resolved
cve_bin_tool/cve_scanner.py Outdated Show resolved Hide resolved
.github/workflows/pythonapp.yml Outdated Show resolved Hide resolved
@Niraj-Kamdar
Copy link
Contributor Author

@pdxjohnny I have done mentioned changes

pdxjohnny
pdxjohnny reviewed Jun 30, 2020
View reviewed changes
test/test_scanner.py Outdated Show resolved Hide resolved
@pdxjohnny pdxjohnny merged commit ca4b2d9 into intel:master Jun 30, 2020
@Niraj-Kamdar Niraj-Kamdar deleted the new branch July 1, 2020 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdxjohnny pdxjohnny pdxjohnny left review comments

@SinghHrmn SinghHrmn SinghHrmn left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@Niraj-Kamdar @codecov-commenter @pdxjohnny @SinghHrmn