[WIP] 441/bugfix/slow report generation with plenty of machines #447
Conversation
ShayNehmad
added
Bug
Lowers amount of deps
…n every run to check if something is a node or a monkey.
| @@ -16,7 +16,8 @@ def parse_creds(attempt): | |||
| if attempt[key]: | |||
| return '%s ; %s : %s' % (username, | |||
| cred['type'], | |||
| cred['output']) | |||
| # TODO Figure out why this is causing an exception with Vakaris | |||
| to_id = NodeService.get_monkey_by_id(edge["to"]) | ||
| if to_id is None: | ||
| to_label = NodeService.get_node_label(NodeService.get_node_by_id(edge["to"])) | ||
| if Monkey.is_monkey(to_id): |
There was a problem hiding this comment.
Not sure if this needs to be changed now, but this is hacky. We should have the Monkey check inside the NodeService
There was a problem hiding this comment.
If we create a Node model and think about the data structure - this should be changed (I think Monkey should be an extension of Node, instead of 2 different models). Until then this patch can remain IMO
| for node in mongo.db.node.find({'exploited': True}, {'_id': 1})] | ||
| mongo.db.monkey.find({}, {'_id': 1}) if | ||
| not NodeService.get_monkey_manual_run(NodeService.get_monkey_by_id(monkey['_id']))] | ||
|
|
There was a problem hiding this comment.
This entire section should probably be rewritten (not now?)
it's basically iterating twice over mongo.db.monkey.find({}, {'_id': 1} and then asking different questions.
I'd rather we pull it once (meaning)
nodes_with_monkeys = [NodeService.get_displayed_node_by_id(monkey['_id'], True) for monkey in mongo.db.monkey.find({}, {'_id': 1})]
And then filter.
This requires an ugly rewrite to use an inner function or have the report not be static :/
There was a problem hiding this comment.
Or a different "exploited_nodes_fetcher" which can be stateful. Again - I agree, but this is not for now. Added to long-term planning on the board.
| nodes_with_monkeys = [NodeService.get_displayed_node_by_id(monkey['_id'], True) for monkey in | ||
| mongo.db.monkey.find({}, {'_id': 1})] | ||
| nodes = nodes_without_monkeys + nodes_with_monkeys | ||
| return nodes | ||
|
|
||
| @staticmethod | ||
| def get_exploited(): | ||
| exploited = \ | ||
| exploited_with_monkeys = \ | ||
| [NodeService.get_displayed_node_by_id(monkey['_id'], True) for monkey in | ||
| mongo.db.monkey.find({}, {'_id': 1}) | ||
| if not NodeService.get_monkey_manual_run(NodeService.get_monkey_by_id(monkey['_id']))] \ | ||
| + [NodeService.get_displayed_node_by_id(node['_id'], True) | ||
| for node in mongo.db.node.find({'exploited': True}, {'_id': 1})] | ||
| mongo.db.monkey.find({}, {'_id': 1}) if | ||
| not NodeService.get_monkey_manual_run(NodeService.get_monkey_by_id(monkey['_id']))] | ||
|
|
||
| exploited_without_monkeys = [NodeService.get_displayed_node_by_id(node['_id'], True) for node in | ||
| mongo.db.node.find({'exploited': True}, {'_id': 1})] |
There was a problem hiding this comment.
Same comment just on node find.
| @@ -699,6 +706,8 @@ def generate_report(): | |||
| cross_segment_issues = ReportService.get_cross_segment_issues() | |||
| monkey_latest_modify_time = Monkey.get_latest_modifytime() | |||
|
|
|||
| scanned_nodes = ReportService.get_scanned() | |||
There was a problem hiding this comment.
Why moved outside? Readability?
There was a problem hiding this comment.
Yes. Not critical and can be undone
ShayNehmad
deleted the
441/bugfix/slow-report-generation-with-plenty-of-machines
branch
Feature / Fixes
Fixes #441.
Screenshots:
From:
To (Middle of progress):
To (final):
Changes
get_displayed_node_by_idfunction via cachingget_scannedfunction via algorithmic improvments