Cve search 399 #508

P-T-I · 2020-10-23T11:58:17Z

@adulau As promised the first version of the API re-write; could you test this in the beta instance?

Work done:

restructure of web gui to facilitate new API setup;
documented API via redoc (flask-restx);
bumped bootstrap, jquery, font-awesome versions;
moved table to datatables.js and incorporated server-side processing of all queries related to tables;
added JWT for token based api calls to admin endpoints;
started first setup for database plugin/abstraction;
moved to gevent webserver instead of tornado;
added websockets to speedup certain long lasting requests.

Install instructions:

I've added some indexes for query performance, so please re-run the indexer;
There's no more minimal.py; this behaviour is controled via the configuration.ini file; setting WebInterface to something other then 'Full' should mimic this behaviour;
Web interface should be started via the index.py (as before)

Love to hear your comments / further thoughts!

fix #399

…od instead of a fixed connection to mongodb

adulau · 2020-11-10T15:56:30Z

Whoaaa, you are fast.

That's great. The vendor browsing works as expected.

https://cvepremium.circl.lu/browse

We are still doing some tests with the API especially the legacy. We will also update the CPE module to use the new API interface.

P-T-I · 2020-11-10T17:37:47Z

Awesome!

P-T-I · 2020-11-11T11:00:00Z

We are still doing some tests with the API especially the legacy. We will also update the CPE module to use the new API interface.

I see the CPE module you mentioned is the one from MISP; great!

Once this change makes it to master the first thing I would like to do is to update the (gh-pages) documentation. I would like to move to sphinx to facilitate automatic documentation building on master commits; any objections?

adulau · 2020-11-12T05:47:25Z

We are still doing some tests with the API especially the legacy. We will also update the CPE module to use the new API interface.

I see the CPE module you mentioned is the one from MISP; great!

Once this change makes it to master the first thing I would like to do is to update the (gh-pages) documentation. I would like to move to sphinx to facilitate automatic documentation building on master commits; any objections?

Good idea!

adulau · 2020-11-12T05:49:58Z

Another small issue, the search on top right seems to be broken.

P-T-I · 2020-11-12T07:56:24Z

Another small issue, the search on top right seems to be broken.

I see what you mean! On it!

P-T-I · 2020-11-12T08:12:36Z

Done!

adulau · 2020-11-12T08:47:21Z

Thanks!

Another question regarding sanity check on the query fields before passing it to MongoDB. Would it be safer to have a set of rules to do some sanity check before passing it to MongoDB.

  File "/usr/local/lib/python3.6/dist-packages/flask_restx/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/adulau/git/cve-search/web/restapi/query.py", line 337, in post
    JSONApiRequest(headers=headers, body=body)
  File "/home/adulau/git/cve-search/lib/DatabaseHandler.py", line 29, in handle_api_json_query
    results = request.process(self.connection)
  File "/home/adulau/git/cve-search/lib/ApiRequests.py", line 80, in process
    results = database_connection.query_docs(**self.body)
  File "/home/adulau/git/cve-search/lib/DatabasePlugins/mongodb.py", line 168, in query_docs
    .limit(limit)
  File "/usr/local/lib/python3.6/dist-packages/pymongo/cursor.py", line 476, in limit
    raise TypeError("limit must be an integer")
TypeError: limit must be an integer

P-T-I · 2020-11-12T14:06:25Z

Yes; definitely; good catch! I'll start working on it!

P-T-I · 2020-11-12T14:15:26Z

Done! Some sanity checks where already in place; added the specific integer checking to the existing checks in JSONApiRequest

adulau · 2020-11-14T12:21:38Z

Cool!

Found something weird in the current branch with the updater which stops at random interval:

Downloading files:   0%|                                                                                                                                                                     | 0/1 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "./sbin/db_updater.py", line 158, in <module>
    up.update()
  File "/home/adulau/git/cve-search/sbin/../lib/Sources_process.py", line 119, in update
    [self.feed_url], collection=self.feed_type.lower(),
  File "/home/adulau/git/cve-search/sbin/../lib/DownloadHandler.py", line 83, in process_downloads
    thread_map(self.download_site, sites, desc="Downloading files")
  File "/usr/local/lib/python3.6/dist-packages/tqdm/contrib/concurrent.py", line 91, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tqdm/contrib/concurrent.py", line 73, in _executor_map
    ex.map(fn, *iterables, **map_args), **kwargs))
  File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1165, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/adulau/git/cve-search/sbin/../lib/DownloadHandler.py", line 265, in download_site
    response.headers["last-modified"], ignoretz=True
  File "/home/adulau/.local/lib/python3.6/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'last-modified'

I find it strange that the HTTP response is missing the last-modified.

P-T-I · 2020-11-14T12:53:27Z

That is indeed weird..... Hard to troubleshoot as well; there’s not much control over the headers being send back to us.... We could add some exception handling and if the header is missing the last-modified field use the current update timestamp instead?

P-T-I · 2020-11-14T12:56:22Z

On second thought; using the current timestamp might conflict at some point with the last-modified header field if it is present on the next update cycle... So using a value to explicitly state it was not received is probably better, don’t you think?

P-T-I · 2020-11-21T19:13:41Z

@adulau Did you encounter any other issues during your tests? If not I might be able to add some exception handling on the updaters' response headers this weekend; once that is done do you consider the PR a beta version? If so we might consider dropping a message on Gitter for the community to take a sneak peak at this version and do some testing/evaluation as well?

adulau · 2020-11-22T13:34:07Z

Thanks a lot. Sure the updater's exception handler is a good idea (also for intermittent issue like SSL/TLS handshake. The only remaining part is the CAPEC entries but the rest is fine and we could consider this as beta.

P-T-I · 2020-11-22T16:08:38Z

Yes the capec issue; did you get your head around that one because I cannot reproduce it on my end....

… of the last_modified header and one for general download failure which will solve issue 513.

adulau · 2020-11-30T14:35:20Z

I think we can merge and fix the minor part later before doing an official release. Ok for a merge?

P-T-I · 2020-11-30T14:42:05Z

Yes; definitely Ok! The minor part you are referring to is that still the CAPEC?

adulau · 2020-11-30T14:47:04Z

Yep but it's not critical.

adulau · 2020-11-30T14:47:51Z

Thanks a lot for PR, it's a great improvement.

P-T-I added 30 commits October 4, 2020 13:35

moved update scripts to separate log file handler

301ac3c

initial setup for a post query endpoint via json body

39ae2c0

rebase

119177b

remodelling web interface to facilitate new api

04ef71e

unignored plugin folder in web

b96a42f

black formatting and swithed to central logging

f6ed2ba

black formatting and swithed to central logging

c06badf

new jquery, bootstrap and font-awesome

2e6ff12

restructure of webgui

8c7c317

added cve logo to all versions of web gui

e949f8c

updated requirements.txt to latest versions and refactor after changes

3178dd2

working on converting api into a flask-restx documented api

caee213

working on API

18a0b3a

Working on restructure API

53fbeeb

moved to _old_

2809319

working on api

b128601

Basic API functionality done

8d43fb5

Black formatting

77ea527

updated requirements.txt

0e14b85

renamed mongo to database to make it more backend agnostic

3b468d2

rewritten POST query endpoint to make use of the database plugin meth…

c7d5ae9

…od instead of a fixed connection to mongodb

version bump

f2dde5a

added formatting and javascript code

e993b7c

Alterations for datatables server side processing

a9dd5a7

refactor

ab2ea0e

updated requirements.txt

53d63c2

restructured logging

b08aefb

restructured logging

77127dd

restructured logging

242216b

setup datatables and filtering

03e00e9

P-T-I added 2 commits November 10, 2020 15:23

removed added additional fields from cve display

e8e589f

added comment

67158c9

chrisr3d referenced this pull request in MISP/misp-modules Nov 10, 2020

chg: [cpe] Support of the new CVE-Search API

b98562a

fix for search field

3984de2

Added limit and skip integer checking and exception handling

5b49755

minor

18fd494

fix cve-search#513; added two exception handlers; one for the absence…

1525fb1

… of the last_modified header and one for general download failure which will solve issue 513.

P-T-I linked an issue Nov 23, 2020 that may be closed by this pull request

Note to myself : updater crash when an exception is at SSL/TLS level #513

Closed

adulau merged commit bc2f206 into cve-search:master Nov 30, 2020

P-T-I deleted the cve-search-399 branch November 30, 2020 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cve search 399 #508

Cve search 399 #508

P-T-I commented Oct 23, 2020 •

edited

adulau commented Nov 10, 2020

P-T-I commented Nov 10, 2020

P-T-I commented Nov 11, 2020

adulau commented Nov 12, 2020

adulau commented Nov 12, 2020

P-T-I commented Nov 12, 2020

P-T-I commented Nov 12, 2020

adulau commented Nov 12, 2020

P-T-I commented Nov 12, 2020

P-T-I commented Nov 12, 2020 •

edited

adulau commented Nov 14, 2020

P-T-I commented Nov 14, 2020

P-T-I commented Nov 14, 2020

P-T-I commented Nov 21, 2020

adulau commented Nov 22, 2020

P-T-I commented Nov 22, 2020

adulau commented Nov 30, 2020

P-T-I commented Nov 30, 2020

adulau commented Nov 30, 2020

adulau commented Nov 30, 2020

Cve search 399 #508

Cve search 399 #508

Conversation

P-T-I commented Oct 23, 2020 • edited

adulau commented Nov 10, 2020

P-T-I commented Nov 10, 2020

P-T-I commented Nov 11, 2020

adulau commented Nov 12, 2020

adulau commented Nov 12, 2020

P-T-I commented Nov 12, 2020

P-T-I commented Nov 12, 2020

adulau commented Nov 12, 2020

P-T-I commented Nov 12, 2020

P-T-I commented Nov 12, 2020 • edited

adulau commented Nov 14, 2020

P-T-I commented Nov 14, 2020

P-T-I commented Nov 14, 2020

P-T-I commented Nov 21, 2020

adulau commented Nov 22, 2020

P-T-I commented Nov 22, 2020

adulau commented Nov 30, 2020

P-T-I commented Nov 30, 2020

adulau commented Nov 30, 2020

adulau commented Nov 30, 2020

P-T-I commented Oct 23, 2020 •

edited

P-T-I commented Nov 12, 2020 •

edited