Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError: epss #908

Closed
brainstorm opened this issue Nov 23, 2023 · 12 comments · Fixed by #913
Closed

KeyError: epss #908

brainstorm opened this issue Nov 23, 2023 · 12 comments · Fixed by #913
Assignees
Labels
cve-search Some cve-search question/issue EMBA in progress Someone is working on this Installation Installation issues

Comments

@brainstorm
Copy link

Today's clean re-installation of emba (rm -rf ./external) yields the CveExplorer error: KeyError: epss:

emba$ git reflog
95d469d4 (HEAD -> master, origin/master, origin/HEAD) HEAD@{0}: pull: Fast-forward
7b5ae5b5 HEAD@{1}: pull: Fast-forward
bc8f63e9 HEAD@{2}: clone: from https://github.com/e-m-b-a/emba.git

emba$ sudo ./installer.sh -d
(...)
2023-11-23 13:14:21,853 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:06:03.180899
2023-11-23 13:14:21,854 - CveXplore.database.maintenance.Sources_process - INFO     - Finished VIA4 database update
Traceback (most recent call last):
  File "/home/rvalls/dev/personal/emba/external/emba_venv/bin/cvexplore", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 28, in initialize_cmd
    ctx.obj["data_source"].database.initialize()
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 187, in initialize
    self.update()
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 72, in update
    up = source["updater"]()
         ^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 1051, in __init__
    self.feed_url = Configuration.getFeedURL(self.feed_type.lower())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/common/config.py", line 113, in getFeedURL
    return cls.SOURCES[source]
           ~~~~~~~~~~~^^^^^^^^
KeyError: 'epss'
------------------------------------------------------------------------------------------------
Error detected - status code 1
Command:  echo -e "Command:  ${ORANGE}${BASH_COMMAND}${NC}"
Location:  ./installer/IF20_cve_search.sh, line 203
Stack Trace:
    [1] IF20_cve_search(): ./installer/IF20_cve_search.sh, line 203 -> IF20_cve_search
    [2] main(): ./installer.sh, line 358 -> main -d

Important: Consider filling out a bug report at https://github.com/e-m-b-a/emba/issues

------------------------------------------------------------------------------------------------
@m-1-k-3
Copy link
Member

m-1-k-3 commented Nov 23, 2023

Could you tell us which OS you are using and when you updated your local git repo?

@brainstorm
Copy link
Author

Ubuntu Linux 23.10.

BenediktMKuehne added a commit that referenced this issue Nov 23, 2023
@m-1-k-3
Copy link
Member

m-1-k-3 commented Nov 23, 2023

I have also seen multiple instabilities of the NIST API today. Please try to update your cve database manually or use the latest EMBA master for another try. EMBA now tries the update multiple times ....

@brainstorm
Copy link
Author

Dropped the entire MongoDB for good measure and re-tried, exactly the same epss error today.

@brainstorm
Copy link
Author

brainstorm commented Nov 23, 2023

Here is a bit of context for the mongodb.log if that helps?:

{"t":{"$date":"2023-11-24T10:18:01.431+11:00"},"s":"I",  "c":"INDEX",    "id":20440,   "ctx":"conn107","msg":"Index build: waiting for index build to complete","attr":{"buildUUID":{"uuid":{"$uuid":"d1b8bd03-8aaf-4c23-a844-2e90d7ec33eb"}},"deadline":{"$date":{"$numberLong":"9223372036854775807"}}}}
{"t":{"$date":"2023-11-24T10:18:01.432+11:00"},"s":"I",  "c":"INDEX",    "id":20447,   "ctx":"conn107","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"d1b8bd03-8aaf-4c23-a844-2e90d7ec33eb"}}}}
{"t":{"$date":"2023-11-24T10:18:01.432+11:00"},"s":"I",  "c":"INDEX",    "id":20438,   "ctx":"conn107","msg":"Index build: registering","attr":{"buildUUID":{"uuid":{"$uuid":"7411fb82-d670-4e21-bf66-4b8caa24af3f"}},"namespace":"cvedb.cves","collectionUUID":{"uuid":{"$uuid":"e106ee13-22d5-4025-844b-c9353f6328fb"}},"indexes":1,"firstIndex":{"name":"epss"}}}
{"t":{"$date":"2023-11-24T10:18:01.437+11:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"conn107","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"cvedb.cves","index":"epss","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2023-11-24T10:18:01.437+11:00"},"s":"I",  "c":"INDEX",    "id":20440,   "ctx":"conn107","msg":"Index build: waiting for index build to complete","attr":{"buildUUID":{"uuid":{"$uuid":"7411fb82-d670-4e21-bf66-4b8caa24af3f"}},"deadline":{"$date":{"$numberLong":"9223372036854775807"}}}}
{"t":{"$date":"2023-11-24T10:18:01.437+11:00"},"s":"I",  "c":"INDEX",    "id":20447,   "ctx":"conn107","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"7411fb82-d670-4e21-bf66-4b8caa24af3f"}}}}

Welp, later on during an:

emba$ ./external/cve-search/sbin/db_updater.py -f

I was greeted with:

2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=420000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=420000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=430000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=430000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=440000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=440000 failed....

And that's after registering a NIST API key (not anonymous)... I'd guess that they'd be more lenient with registered API key users?

Sigh, anyway, I guess I'll try another day :-S

@brainstorm
Copy link
Author

Small observation... why are those messages repeated twice for every action?:

emba$ ./external/cve-search/sbin/db_updater.py -f
2023-11-24 10:42:56,305 - DBUpdater - INFO     - Dropping metadata
2023-11-24 10:42:56,320 - CveXplore.database.maintenance.main_updater - INFO     - Starting Database initialization....
2023-11-24 10:42:56,320 - CveXplore.database.maintenance.main_updater - INFO     - Starting Database initialization....
2023-11-24 10:42:56,325 - CveXplore.database.maintenance.Sources_process - INFO     - CPE Database population started
2023-11-24 10:42:56,325 - CveXplore.database.maintenance.Sources_process - INFO     - CPE Database population started
2023-11-24 10:42:56,326 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-24 10:42:56,326 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-24 10:43:17,086 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1199455 CPE entries
2023-11-24 10:43:17,086 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1199455 CPE entries
Downloading and processing content:   0%|                                                                                                                                  | 0/1199455 [00:00<?, ?it/s]

@brainstorm
Copy link
Author

Adding a print where it fails to see the contents of cls.SOURCES:

cls.SOURCES is: {'cwe': 'https://cwe.mitre.org/data/xml/cwec_latest.xml.zip', 'capec': 'https://capec.mitre.org/data/xml/capec_latest.xml', 'via4': 'https://www.cve-search.org/feeds/via4.json'}

Context:

emba$ ./external/cve-search/sbin/db_updater.py -f
(...)
2023-11-24 21:28:45,632 - CveXplore.database.maintenance.Sources_process - INFO     - VIA4 database update started
2023-11-24 21:28:45,632 - CveXplore.database.maintenance.Sources_process - INFO     - VIA4 database update started
2023-11-24 21:28:45,633 - CveXplore.database.maintenance.Sources_process - INFO     - Downloading files (max 10 workers)
2023-11-24 21:28:45,633 - CveXplore.database.maintenance.Sources_process - INFO     - Downloading files (max 10 workers)
Downloading files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [03:41<00:00, 221.42s/it]
Processing downloaded files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.41s/it]
Transferring queue to database: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [11:46<00:00, 47.12s/it]
2023-11-24 21:44:20,594 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:15:34.961643
2023-11-24 21:44:20,594 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:15:34.961643
2023-11-24 21:44:20,691 - CveXplore.database.maintenance.Sources_process - INFO     - Finished VIA4 database update
2023-11-24 21:44:20,691 - CveXplore.database.maintenance.Sources_process - INFO     - Finished VIA4 database update
getFeedURL source argument contains: epss
 and cls.SOURCES is: {'cwe': 'https://cwe.mitre.org/data/xml/cwec_latest.xml.zip', 'capec': 'https://capec.mitre.org/data/xml/capec_latest.xml', 'via4': 'https://www.cve-search.org/feeds/via4.json'}

Traceback (most recent call last):
  File "/home/rvalls/dev/personal/emba/./external/cve-search/sbin/db_updater.py", line 122, in <module>
    cvex.database.initialize()
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 187, in initialize
    self.update()
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 72, in update
    up = source["updater"]()
         ^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 1051, in __init__
    self.feed_url = Configuration.getFeedURL(self.feed_type.lower())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/common/config.py", line 114, in getFeedURL
    return cls.SOURCES[source]
           ~~~~~~~~~~~^^^^^^^^
KeyError: 'epss'

@brainstorm
Copy link
Author

@m-1-k-3 Would it make sense to download the EPSS part(s) from FIRST instead of NIST or are there disparities between those two sources of information?:

https://www.first.org/epss/api

@m-1-k-3
Copy link
Member

m-1-k-3 commented Nov 24, 2023

I have reopened the original issue here #725

@m-1-k-3 m-1-k-3 self-assigned this Nov 24, 2023
@m-1-k-3 m-1-k-3 added the in progress Someone is working on this label Nov 24, 2023
@m-1-k-3
Copy link
Member

m-1-k-3 commented Nov 24, 2023

Check on this #913

@brainstorm
Copy link
Author

brainstorm commented Nov 26, 2023

Seems to work @m-1-k-3, analysis finished after 19h (:-!) and 12 cores with this on the MIPS router firmware:

(...)
[+] Identified 1274 CVE entries.](file:///home/rvalls/dev/personal/emba/results/html-report/f20_vul_aggregator.html#anchor_collectcveandexploitdetails)
    Identified 981 High rated CVE entries / Exploits: 114
    Identified 249 Medium rated CVE entries / Exploits: 7
    Identified 44 Low rated CVE entries /Exploits: 41
   [ 162 possible exploits available (63 Metasploit modules).](file:///home/rvalls/dev/personal/emba/results/html-report/f20_vul_aggregator.html#anchor_minimalreportofexploitsandcves)

    Remote exploits: 47 / Local exploits: 41 / DoS exploits: 32 / Github PoCs: 0 / Known exploited vulnerabilities: 0 / Verified Exploits: 0

Intense use of disk space, btw:

results$ du -hs .
6.1G	.

Screenshot from 2023-11-26 20-39-07

@m-1-k-3
Copy link
Member

m-1-k-3 commented Nov 26, 2023

s99_grepit module is very disk intense. If you are runnning the default profile this module should not run. The firmware is also stored and could be removed automatically via the -r option

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cve-search Some cve-search question/issue EMBA in progress Someone is working on this Installation Installation issues
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants