Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cve search 399 #508

Merged
merged 105 commits into from Nov 30, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
301ac3c
moved update scripts to separate log file handler
P-T-I Oct 4, 2020
39ae2c0
initial setup for a post query endpoint via json body
P-T-I Oct 4, 2020
119177b
rebase
P-T-I Oct 4, 2020
04ef71e
remodelling web interface to facilitate new api
P-T-I Oct 6, 2020
b96a42f
unignored plugin folder in web
P-T-I Oct 6, 2020
f6ed2ba
black formatting and swithed to central logging
P-T-I Oct 6, 2020
c06badf
black formatting and swithed to central logging
P-T-I Oct 6, 2020
2e6ff12
new jquery, bootstrap and font-awesome
P-T-I Oct 6, 2020
8c7c317
restructure of webgui
P-T-I Oct 6, 2020
e949f8c
added cve logo to all versions of web gui
P-T-I Oct 6, 2020
3178dd2
updated requirements.txt to latest versions and refactor after changes
P-T-I Oct 6, 2020
caee213
working on converting api into a flask-restx documented api
P-T-I Oct 6, 2020
18a0b3a
working on API
P-T-I Oct 7, 2020
53fbeeb
Working on restructure API
P-T-I Oct 8, 2020
2809319
moved to _old_
P-T-I Oct 9, 2020
b128601
working on api
P-T-I Oct 9, 2020
8d43fb5
Basic API functionality done
P-T-I Oct 9, 2020
77ea527
Black formatting
P-T-I Oct 9, 2020
0e14b85
updated requirements.txt
P-T-I Oct 10, 2020
3b468d2
renamed mongo to database to make it more backend agnostic
P-T-I Oct 10, 2020
c7d5ae9
rewritten POST query endpoint to make use of the database plugin meth…
P-T-I Oct 10, 2020
f2dde5a
version bump
P-T-I Oct 10, 2020
e993b7c
added formatting and javascript code
P-T-I Oct 12, 2020
a9dd5a7
Alterations for datatables server side processing
P-T-I Oct 12, 2020
ab2ea0e
refactor
P-T-I Oct 12, 2020
53d63c2
updated requirements.txt
P-T-I Oct 13, 2020
b08aefb
restructured logging
P-T-I Oct 13, 2020
77127dd
restructured logging
P-T-I Oct 13, 2020
242216b
restructured logging
P-T-I Oct 13, 2020
03e00e9
setup datatables and filtering
P-T-I Oct 13, 2020
428d78e
setup datatables and filtering
P-T-I Oct 13, 2020
d89cba9
setup datatables and filtering
P-T-I Oct 13, 2020
04cad86
setup logging
P-T-I Oct 13, 2020
3b5b5d7
formatted admin page and index page
P-T-I Oct 13, 2020
757a052
version bump
P-T-I Oct 13, 2020
c30136b
refactor
P-T-I Oct 13, 2020
d5b3dea
Merge branch 'up_master' into cve-search-399
P-T-I Oct 13, 2020
f18eeab
req update
P-T-I Oct 14, 2020
aecf661
added api docs link to default menubar
P-T-I Oct 14, 2020
3c0807d
typo
P-T-I Oct 14, 2020
621ebc5
working on website restructure
P-T-I Oct 14, 2020
16ca5e2
reformatted to bootstrap 4
P-T-I Oct 15, 2020
1ffa8a8
alterations made to facilitate port to bootstrap 4
P-T-I Oct 15, 2020
4919ecf
added separate breadcrumbs.html subpage to ease breadcrumbs import in…
P-T-I Oct 15, 2020
4cecb67
reformatting
P-T-I Oct 15, 2020
f412789
version bump
P-T-I Oct 15, 2020
a252b0f
added socket.io scripts
P-T-I Oct 16, 2020
ce368d6
removed clipboard
P-T-I Oct 16, 2020
839fec5
added requirements
P-T-I Oct 16, 2020
634ed52
renamed master-page to master-page.html
P-T-I Oct 16, 2020
98c663c
altered hashing mechanism's
P-T-I Oct 16, 2020
c805cc5
working on login
P-T-I Oct 16, 2020
7edbb53
minor
P-T-I Oct 16, 2020
518b67b
version bump
P-T-I Oct 16, 2020
2cdaa3e
login forms formatting
P-T-I Oct 19, 2020
f8ea748
minimal setting ignoring admin blueprint
P-T-I Oct 19, 2020
af4fff6
bump
P-T-I Oct 19, 2020
a6d6ecd
Merge branch 'up_master' into cve-search-399
P-T-I Oct 19, 2020
18ad79c
bump
P-T-I Oct 19, 2020
89e2857
added sweetalert2.min.js
P-T-I Oct 19, 2020
8e71bfe
added sweetalert2.min.js
P-T-I Oct 19, 2020
25034e8
password check fix
P-T-I Oct 19, 2020
02eb6ed
minor
P-T-I Oct 19, 2020
93415bf
reformatted admin, login with local database fixed; working on white …
P-T-I Oct 19, 2020
c6b566f
working on admin api
P-T-I Oct 20, 2020
df8b1ae
status adjustments
P-T-I Oct 20, 2020
b1d7846
version bump
P-T-I Oct 20, 2020
f0ddae2
working on admin part
P-T-I Oct 20, 2020
d339624
working on admin api
P-T-I Oct 20, 2020
c61b016
version bump
P-T-I Oct 20, 2020
98dcbb2
finished porting admin api to new api
P-T-I Oct 22, 2020
3b2c309
finished porting admin api to new api
P-T-I Oct 22, 2020
72b8ce7
moved old files to _old_ folder
P-T-I Oct 22, 2020
69bd761
version bump
P-T-I Oct 22, 2020
b391089
defined limit and skip as integers
P-T-I Oct 22, 2020
549b812
moved to gevent as webserver
P-T-I Oct 22, 2020
32c7a6d
working on websockets
P-T-I Oct 22, 2020
eeeb1ed
version bump
P-T-I Oct 22, 2020
5de310b
cleanup & black formatting
P-T-I Oct 23, 2020
650b0e9
cleanup
P-T-I Oct 23, 2020
aa23126
bump
P-T-I Oct 23, 2020
4654752
merge up_master
P-T-I Oct 23, 2020
366414c
fix for python path
P-T-I Oct 23, 2020
d398139
fix for python path
P-T-I Oct 23, 2020
4dc878e
fix for python path
P-T-I Oct 23, 2020
bfdfd14
disabled loggers
P-T-I Oct 23, 2020
600e2b9
added allow_disk_use to query_docs queries
P-T-I Oct 23, 2020
9fe743c
removed allow_disk_use to query_docs queries
P-T-I Oct 23, 2020
240a09e
version bump
P-T-I Oct 23, 2020
d230cbf
refactor
P-T-I Oct 23, 2020
02ca1ca
moved user functions to mongodb.py
P-T-I Oct 23, 2020
918e839
possible fix for travis failing to build
P-T-I Oct 29, 2020
34c4eb8
working on new indexes and import fields
P-T-I Nov 8, 2020
3eaba79
query optimalization on cvesForCPE query
P-T-I Nov 9, 2020
e76ea1f
minor
P-T-I Nov 10, 2020
dcd4b71
fix for not displaying results on api/cvefor
P-T-I Nov 10, 2020
9b04883
added htmlescape function
P-T-I Nov 10, 2020
04a247e
removed websockets
P-T-I Nov 10, 2020
17fcad4
moved vendor table to DataTable with ajax processing
P-T-I Nov 10, 2020
e8e589f
removed added additional fields from cve display
P-T-I Nov 10, 2020
67158c9
added comment
P-T-I Nov 10, 2020
3984de2
fix for search field
P-T-I Nov 12, 2020
5b49755
Added limit and skip integer checking and exception handling
P-T-I Nov 12, 2020
18fd494
minor
P-T-I Nov 12, 2020
1525fb1
fix #513; added two exception handlers; one for the absence of the la…
P-T-I Nov 23, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Expand Up @@ -19,3 +19,4 @@ etc/plugins.txt
.idea
venv
.coverage
!/web/plugins/
2 changes: 1 addition & 1 deletion .travis.yml
Expand Up @@ -24,7 +24,7 @@ jobs:
- redis-server
- mongodb
install:
- pip install -q -U pip
- pip install -q -U pip setuptools wheel
- pip install -q -r requirements.txt
- pushd sbin
- ./db_mgmt_json.py -p
Expand Down
18 changes: 16 additions & 2 deletions bin/search.py
Expand Up @@ -73,6 +73,14 @@ def replace_special_chars(cpe):
help='With this option, "-p" will only return vulnerabilities directly assigned to the product. I.e. it will not '
'consider "windows_7" if it is only mentioned as affected OS in an adobe:reader vulnerability. ',
)
argParser.add_argument(
"--strict_vendor_product",
dest="strict_vendor_product",
default=False,
action="store_true",
help='With this option, a strict vendor product search is executed. The values in "-p" should be formatted as '
'vendor:product, e.g. microsoft:windows_7',
)
argParser.add_argument(
"--lax",
default=False,
Expand Down Expand Up @@ -116,6 +124,7 @@ def replace_special_chars(cpe):
pyReq = args.q
vSearch = args.p
relaxSearch = args.lax
strict_vendor_product = args.strict_vendor_product
vulnerableProductSearch = args.vulnProdSearch
cveSearch = [x.upper() for x in args.c] if args.c else None
vOutput = args.o
Expand Down Expand Up @@ -149,7 +158,12 @@ def print_job(item):


def search_product(prod):
ret = cvesForCPE(prod, lax=relaxSearch, vulnProdSearch=vulnerableProductSearch)
if strict_vendor_product:
search = prod.split(":")
search = (search[0], search[1])
ret = cvesForCPE(search, lax=relaxSearch, vulnProdSearch=vulnerableProductSearch, strict_vendor_product=True)
else:
ret = cvesForCPE(prod, lax=relaxSearch, vulnProdSearch=vulnerableProductSearch)
for item in ret["results"]:
if not last_ndays:
print_job(item)
Expand Down Expand Up @@ -474,7 +488,7 @@ def search_in_summary(item):

# Search Product (best to use CPE notation, e.g. cisco:ios:12.2
if vSearch:
# Serach multiple products in one query
# Search multiple products in one query
for cpe in vSearch:
search_product(cpe)
if htmlOutput:
Expand Down
10 changes: 4 additions & 6 deletions bin/search_cpe.py
Expand Up @@ -7,18 +7,16 @@
# Copyright (c) 2015-2018 Pieter-Jan Moreels - pieterjan.moreels@gmail.com
# Copyright (c) 2015-2019 Alexandre Dulaunoy - a@foo.be

# Imports
import argparse
import json
import os
import re
import sys
import urllib.parse

runPath = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(runPath, ".."))

import re
import argparse
import json
import urllib.parse

import lib.DatabaseLayer as db

runPath = os.path.dirname(os.path.realpath(__file__))
Expand Down
281 changes: 161 additions & 120 deletions bin/search_irc.py
Expand Up @@ -19,143 +19,184 @@

# Imports
import argparse
import irc.bot
import irc.strings
import json
import os
import signal
import ssl
import sys

import irc.bot
import irc.strings
from bson import json_util

from web.templates._old_.api import API

runPath = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(runPath, ".."))

# BSON MongoDB include ugly stuff that needs to be processed for standard JSON
from bson import json_util

from web.api import API

argParser = argparse.ArgumentParser(description='IRC bot to query cve-search')
argParser.add_argument('-s', type=str, help='server ip', default='localhost')
argParser.add_argument('-p', type=int, help='server port)', default=6667)
argParser.add_argument('-n', type=str, help='nickname', default='cve-search')
argParser.add_argument('-w', type=str, help='password')
argParser.add_argument('-u', type=str, help='username', default='cve-search')
argParser.add_argument('-c', nargs="*", help='channel list', default=['cve-search'])
argParser.add_argument('-t', type=str, help='trigger prefix', default='.')
argParser.add_argument('-v', action='store_true', help='channel list', default=['cve-search'])
argParser.add_argument('-m', type=int, help='maximum query amount', default=20)
argParser.add_argument('--ssl', action='store_true', help='Use SSL')
args = argParser.parse_args()

class IRCBot(irc.bot.SingleServerIRCBot):
def __init__(self, channel, nickname, server, port, password=None, username=None, **kwargs):
if not username:
username=nickname
irc.bot.SingleServerIRCBot.__init__(self, [(server, port)], nickname, username, **kwargs)
self.channel = channel
self.api = API()
argParser = argparse.ArgumentParser(description="IRC bot to query cve-search")
argParser.add_argument("-s", type=str, help="server ip", default="localhost")
argParser.add_argument("-p", type=int, help="server port)", default=6667)
argParser.add_argument("-n", type=str, help="nickname", default="cve-search")
argParser.add_argument("-w", type=str, help="password")
argParser.add_argument("-u", type=str, help="username", default="cve-search")
argParser.add_argument("-c", nargs="*", help="channel list", default=["cve-search"])
argParser.add_argument("-t", type=str, help="trigger prefix", default=".")
argParser.add_argument(
"-v", action="store_true", help="channel list", default=["cve-search"]
)
argParser.add_argument("-m", type=int, help="maximum query amount", default=20)
argParser.add_argument("--ssl", action="store_true", help="Use SSL")
args = argParser.parse_args()

def on_nicknameinuse(self, c, e):
c.nick(c.get_nickname() + "_")

def on_welcome(self, c, e):
if args.v:
print("Server welcomed us")
for chan in self.channel:
if not chan.startswith('#'):chan=("#%s"%chan)
if args.v:
print("joining %s"%chan)
c.join(chan)

def on_privmsg(self, c, e):
self.do_command(e, e.arguments[0])

def on_pubmsg(self, c, e):
line = e.arguments[0]
if line.startswith(args.t):
self.do_command(e, line[len(args.t):])
return

def reply(self, e, reply):
if type(reply) in [dict, list]:
#reply = json.dumps(reply, sort_keys=True, indent=4, default=json_util.default, ensure_ascii=True)
reply = json.dumps(reply, sort_keys=True, ensure_ascii=True, default=json_util.default)
else:
reply = str(reply)
class IRCBot(irc.bot.SingleServerIRCBot):
def __init__(
self, channel, nickname, server, port, password=None, username=None, **kwargs
):
if not username:
username = nickname
irc.bot.SingleServerIRCBot.__init__(
self, [(server, port)], nickname, username, **kwargs
)
self.channel = channel
self.api = API()

def on_nicknameinuse(self, c, e):
c.nick(c.get_nickname() + "_")

def on_welcome(self, c, e):
if args.v:
print("Server welcomed us")
for chan in self.channel:
if not chan.startswith("#"):
chan = "#%s" % chan
if args.v:
print("joining %s" % chan)
c.join(chan)

def on_privmsg(self, c, e):
self.do_command(e, e.arguments[0])

def on_pubmsg(self, c, e):
line = e.arguments[0]
if line.startswith(args.t):
self.do_command(e, line[len(args.t) :])
return

def reply(self, e, reply):
if type(reply) in [dict, list]:
# reply = json.dumps(reply, sort_keys=True, indent=4, default=json_util.default, ensure_ascii=True)
reply = json.dumps(
reply, sort_keys=True, ensure_ascii=True, default=json_util.default
)
else:
reply = str(reply)

if e.target == self.connection.nickname:
target = e.source.nick
else:
target = e.target
_list = reply.split("\n")
chunk_size = (
512 - 12 - len(e.target)
) # 512 - len("PRIVMSG") - len(" :") - CR/LF - target

_list = [
[x[i : i + chunk_size] for i in range(0, len(x), chunk_size)] for x in _list
]
_list = [item for sublist in _list for item in sublist] # flatten list
for r in _list[:4]:
self.connection.privmsg(target, r)

def do_command(self, e, cmd):
def last(option):
limit = int(option) if option else 10
if limit > args.m or limit < 1:
self.reply(e, "Request not in range 0-%d" % args.m)
self.reply(e, self.api.api_last(limit))

def cve(option):
if option is None:
return "A cve-id must be specified"
return self.api.api_cve(option)

if not cmd:
pass
parts = cmd.split(" ", 1)
cmd = parts[0]
option = parts[1] if len(parts) == 2 else None

if cmd == "die":
self.die()
elif cmd in ["last", "recent"]:
self.reply(e, last(option))
elif cmd in ["get", "cve"]:
self.reply(e, cve(option))
elif cmd in ["browse", "vendor"]:
self.reply(e, self.api.api_browse(option))
elif cmd in ["search", "product"]:
parts = option.split()
if len(parts) < 2:
return self.reply(e, "Usage: search <vendor> <product>")
return self.reply(e, self.api.api_search(parts[0], parts[1]))
elif cmd in ["cvetweet", "tweet"]:
text = ""
cves = []
if option.lower().startswith("cve-"):
cves.append(cve(option))
else:
cves = last(option)
for t in cves:
text += (
str(t["id"])
+ " , "
+ str(t["summary"])
+ " "
+ " , ".join(t["references"])
+ "\n"
)
return self.reply(e, text)
else:
self.reply(e, "Not understood: " + cmd)

if e.target == self.connection.nickname:
target=e.source.nick
else:
target=e.target
_list = reply.split('\n')
chunk_size = 512 - 12 - len(e.target) # 512 - len("PRIVMSG") - len(" :") - CR/LF - target

_list = [[x[i:i+chunk_size] for i in range(0, len(x), chunk_size)] for x in _list]
_list = [item for sublist in _list for item in sublist] # flatten list
for r in _list[:4]:
self.connection.privmsg(target, r)

def do_command(self, e, cmd):
def last(option):
limit = int(option) if option else 10
if limit > args.m or limit < 1:
self.reply(e, "Request not in range 0-%d" % args.m)
self.reply(e, self.api.api_last(limit))
def cve(option):
if option is None:
return "A cve-id must be specified"
return self.api.api_cve(option)

if not cmd: pass
parts = cmd.split(' ', 1)
cmd = parts[0]
option = parts[1] if len(parts) == 2 else None

if cmd == "die": self.die()
elif cmd in ["last", "recent"]: self.reply(e, last(option))
elif cmd in ["get", "cve"]: self.reply(e, cve(option))
elif cmd in ["browse", "vendor"]: self.reply(e, self.api.api_browse(option))
elif cmd in ["search", "product"]:
parts = option.split()
if len(parts) < 2:
return self.reply(e, "Usage: search <vendor> <product>")
return self.reply(e, self.api.api_search(parts[0], parts[1]))
elif cmd in ["cvetweet", "tweet"]:
text = ""
cves = []
if option.lower().startswith("cve-"): cves.append(cve(option))
else: cves = last(option)
for t in cves:
text += str(t['id']) + " , " + str(t['summary']) + " " + " , ".join(t['references']) + "\n"
return self.reply(e, text)
else:
self.reply(e, "Not understood: " + cmd)

# signal handlers
def sig_handler(sig, frame):
print('Caught signal: %s\nShutting down' % sig)
print("Caught signal: %s\nShutting down" % sig)
bot.die()


def main():
server = args.s
port = args.p
nick = args.n
password = args.w
user = args.u
chans = args.c
global bot

if args.ssl:
print("using ssl")
ssl_factory = irc.connection.Factory(wrapper=ssl.wrap_socket)
bot=IRCBot(chans, nick, server, port, password=password,username=user, connect_factory=ssl_factory)
else:
bot=IRCBot(chans, nick, server, port, password=password,username=user)
signal.signal(signal.SIGTERM, sig_handler)
signal.signal(signal.SIGINT, sig_handler)
if args.v:
print("Connecting to server")
bot.start()
server = args.s
port = args.p
nick = args.n
password = args.w
user = args.u
chans = args.c
global bot

if args.ssl:
print("using ssl")
ssl_factory = irc.connection.Factory(wrapper=ssl.wrap_socket)
bot = IRCBot(
chans,
nick,
server,
port,
password=password,
username=user,
connect_factory=ssl_factory,
)
else:
bot = IRCBot(chans, nick, server, port, password=password, username=user)
signal.signal(signal.SIGTERM, sig_handler)
signal.signal(signal.SIGINT, sig_handler)
if args.v:
print("Connecting to server")
bot.start()


if __name__ == "__main__":
main()
main()