Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix offensive-security repo links, CSV parsing #112

Merged
merged 1 commit into from
Nov 25, 2022
Merged

Fix offensive-security repo links, CSV parsing #112

merged 1 commit into from
Nov 25, 2022

Conversation

eyalkraft
Copy link
Contributor

@eyalkraft eyalkraft commented Nov 18, 2022

What did you implement:

The offensive-security/exploitdb Github repo moved to Gitlab (see last commit).
This cause some hard coded links to break, which I fixed.
Additionally I set the CSV parser to use the "lazy parser" because of a backslash-escaped quote in the data that resulted with an error:

INFO[11-18|04:42:40] Fetching                                 URL=https://gitlab.com/exploit-database/exploitdb/-/raw/main/files_exploits.csv
Failed to fetch Exploit. err: parse error on line 3138, column 303: extraneous or missing " in quoted-field

I did not implement:

  • Changing structure names referring to Github
  • Changing structure names referring to Offensive Security

Fixes #110

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Manually on my machine.

➜  go-exploitdb git:(fix-links) ✗ go-exploitdb fetch exploitdb --debug
INFO[11-18|05:29:55] Fetching Offensive Security Exploit
INFO[11-18|05:29:55] Fetching                                 URL=https://cve.mitre.org/data/downloads/allitems-cvrf.xml
INFO[11-18|05:42:43] Fetching                                 URL=https://gitlab.com/exploit-database/exploitdb/-/raw/main/files_shellcodes.csv
INFO[11-18|05:42:44] Fetching                                 URL=https://gitlab.com/exploit-database/exploitdb/-/raw/main/files_exploits.csv
INFO[11-18|05:42:45] Fetching                                 URL=https://gitlab.com/exploit-database/exploitdb-papers/-/raw/main/files_papers.csv
INFO[11-18|05:42:46] Offensive Security Exploit               count=49373
INFO[11-18|05:42:46] Insert Exploit into go-exploitdb.        db=sqlite3
INFO[11-18|05:42:46] Inserting 49373 Exploits
INFO[11-18|05:42:46] Inserting new Exploits
49373 / 49373 [---------------------------------------------------------------------] 100.00% 182396 p/s
INFO[11-18|05:42:47] No CveID Exploit Count                   count=36506
INFO[11-18|05:42:47] CveID Exploit Count

Checklist:

  • Write tests
  • Write documentation
  • Check that there aren't other open pull requests for the same issue/feature
  • Format your source code by make fmt
  • Pass the test by make test
  • Provide verification config / commands
  • Enable "Allow edits from maintainers" for this PR
  • Update the messages below

Is this ready for review?: Yes

@arrow003
Copy link

arrow003 commented Nov 25, 2022

issues/110

Confirm that the problem has been solved

Unfortunately, the main branch has not been updated

thank you very much

Copy link
Collaborator

@MaineK00n MaineK00n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@MaineK00n MaineK00n merged commit 4d40a8e into vulsio:master Nov 25, 2022
@Jonher937 Jonher937 mentioned this pull request Feb 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants