integrate cveXplore settings #884

BenediktMKuehne · 2023-11-15T10:36:17Z

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
integrates settings environment variables
What is the current behavior? (You can also link to an open issue here)
cve_search db update is not working since we have settings, that no longer work with the new tool
What is the new behavior (if this is a feature change)? If possible add a screenshot.
working cve_search database with it's search functionality
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
nope

m-1-k-3 · 2023-11-15T18:26:57Z

closes #886

BenediktMKuehne · 2023-11-16T09:19:27Z

Now we are getting a different error:

└─$ MONGODB_HOST=172.36.0.1 ./external/cve-search/sbin/db_updater.py -fv
Warning! Could not write log to /home/cylox/emba/emba/external/cve-search/log/update_populate.log. Disabling temporarily.
2023-11-15 04:22:41,284 - DBUpdater - INFO     - Dropping metadata
2023-11-15 04:22:41,289 - DBUpdater - INFO     - ==========================
2023-11-15 04:22:41,289 - DBUpdater - INFO     - Wed 15 November 2023 14:22
2023-11-15 04:22:41,290 - DBUpdater - INFO     - ==========================
2023-11-15 04:22:41,313 - CveXplore.database.maintenance.Sources_process - INFO     - CPE Database population started
2023-11-15 04:22:41,313 - CveXplore.database.maintenance.Sources_process - INFO     - CPE Database population started
2023-11-15 04:22:41,370 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('id', 1)] on cpe
2023-11-15 04:22:41,370 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('id', 1)] on cpe
2023-11-15 04:22:41,375 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vendor', 1)] on cpe
2023-11-15 04:22:41,375 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vendor', 1)] on cpe
2023-11-15 04:22:41,380 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('product', 1)] on cpe
2023-11-15 04:22:41,380 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('product', 1)] on cpe
2023-11-15 04:22:41,385 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cpeNameId', 1)] on cpe
2023-11-15 04:22:41,385 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cpeNameId', 1)] on cpe
2023-11-15 04:22:41,390 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('deprecated', 1)] on cpe
2023-11-15 04:22:41,390 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('deprecated', 1)] on cpe
2023-11-15 04:22:41,395 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cpeName', 1)] on cpe
2023-11-15 04:22:41,395 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cpeName', 1)] on cpe
2023-11-15 04:22:41,400 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('title', 1)] on cpe
2023-11-15 04:22:41,400 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('title', 1)] on cpe
2023-11-15 04:22:41,401 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-15 04:22:41,401 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-15 04:22:42,382 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1195362 CPE entries
2023-11-15 04:22:42,382 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1195362 CPE entries
Downloading and processing content: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1195362/1195362 [04:30<00:00, 4411.87it/s]
2023-11-15 04:27:13,352 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:04:31.951544                                                                                                                             
2023-11-15 04:27:13,352 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:04:31.951544
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.Sources_process - INFO     - Finished CPE database population
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.Sources_process - INFO     - Finished CPE database population
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.main_updater - INFO     - Sleeping for 30 seconds between CPE and CVE database population..
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.main_updater - INFO     - Sleeping for 30 seconds between CPE and CVE database population..
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO     - CVE database population started
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO     - CVE database population started
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO     - Starting CVE database population starting from year: 2002
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO     - Starting CVE database population starting from year: 2002
2023-11-15 04:27:43,757 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('id', 1)] on cves
2023-11-15 04:27:43,757 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('id', 1)] on cves
2023-11-15 04:27:43,762 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_configuration', 1)] on cves
2023-11-15 04:27:43,762 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_configuration', 1)] on cves
2023-11-15 04:27:43,766 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_product', 1)] on cves
2023-11-15 04:27:43,766 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_product', 1)] on cves
2023-11-15 04:27:43,772 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('modified', 1)] on cves
2023-11-15 04:27:43,772 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('modified', 1)] on cves
2023-11-15 04:27:43,776 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('published', 1)] on cves
2023-11-15 04:27:43,776 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('published', 1)] on cves
2023-11-15 04:27:43,781 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('lastModified', 1)] on cves
2023-11-15 04:27:43,781 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('lastModified', 1)] on cves
2023-11-15 04:27:43,786 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cvss', 1)] on cves
2023-11-15 04:27:43,786 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cvss', 1)] on cves
2023-11-15 04:27:43,791 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cvss3', 1)] on cves
2023-11-15 04:27:43,791 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cvss3', 1)] on cves
2023-11-15 04:27:43,796 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('summary', 'text')] on cves
2023-11-15 04:27:43,796 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('summary', 'text')] on cves
2023-11-15 04:27:43,804 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vendors', 1)] on cves
2023-11-15 04:27:43,804 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vendors', 1)] on cves
2023-11-15 04:27:43,810 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('products', 1)] on cves
2023-11-15 04:27:43,810 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('products', 1)] on cves
2023-11-15 04:27:43,816 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('assigner', 1)] on cves
2023-11-15 04:27:43,816 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('assigner', 1)] on cves
2023-11-15 04:27:43,821 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cwe', 1)] on cves
2023-11-15 04:27:43,821 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('cwe', 1)] on cves
2023-11-15 04:27:43,826 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('status', 1)] on cves
2023-11-15 04:27:43,826 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('status', 1)] on cves
2023-11-15 04:27:43,831 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_product_stems', 1)] on cves
2023-11-15 04:27:43,831 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_product_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO     - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-15 04:27:44,991 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 217132 CVE entries
2023-11-15 04:27:44,991 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 217132 CVE entries
Downloading and processing content: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 217132/217132 [05:41<00:00, 636.35it/s]
2023-11-15 04:33:26,206 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:05:42.370443                                                                                                                             
2023-11-15 04:33:26,206 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:05:42.370443
2023-11-15 04:33:26,555 - CveXplore.database.maintenance.Sources_process - INFO     - Finished CVE database population
2023-11-15 04:33:26,555 - CveXplore.database.maintenance.Sources_process - INFO     - Finished CVE database population
2023-11-15 04:33:26,597 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-11-15 04:33:26,597 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-11-15 04:33:26,598 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-15 04:33:26,598 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
Traceback (most recent call last):
  File "/home/cylox/emba/emba/./external/cve-search/sbin/db_updater.py", line 122, in <module>
    cvex.database.initialize()
  File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 169, in initialize
    self.update()
  File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 67, in update
    up.update()
  File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 218, in update
    self.process_downloads()
  File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 168, in process_downloads
    total_results = self.api_handler.get_count(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 196, in get_count
    raise ApiDataRetrievalFailed
CveXplore.errors.apis.ApiDataRetrievalFailed

BenediktMKuehne added 6 commits November 15, 2023 09:51

update cvexplore version

4c6c730

switch to cvexplore directly for updating

da22e64

update

c18de8f

make mongodb IP global var

3d95ee4

add mongodb_host var to installer

bd4bc76

go back to db_updater

13e1eb3

m-1-k-3 mentioned this pull request Nov 15, 2023

Installation Fails Due to CVE-Search Looking at 127.0.0.1 instead of 172.36.0.1 #886

Closed

BenediktMKuehne marked this pull request as ready for review November 16, 2023 10:41

m-1-k-3 approved these changes Nov 16, 2023

View reviewed changes

m-1-k-3 merged commit 7b5ae5b into e-m-b-a:master Nov 16, 2023

This was referenced Nov 17, 2023

CVE seacher error #888

Closed

The NVD plans to retire the remaining legacy data feeds as well as all 1.0 APIs. #725

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

integrate cveXplore settings #884

integrate cveXplore settings #884

Uh oh!

BenediktMKuehne commented Nov 15, 2023 •

edited

Loading

Uh oh!

m-1-k-3 commented Nov 15, 2023

Uh oh!

BenediktMKuehne commented Nov 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

integrate cveXplore settings #884

integrate cveXplore settings #884

Uh oh!

Conversation

BenediktMKuehne commented Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m-1-k-3 commented Nov 15, 2023

Uh oh!

BenediktMKuehne commented Nov 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BenediktMKuehne commented Nov 15, 2023 •

edited

Loading