SpiderFootCorrelator: Add error handling, annotations, docstrings #1654
Conversation
| # Strip any extra newlines that may have creeped into meta | ||
| for rule in self.rules: | ||
| for k in rule['meta'].keys(): | ||
| if isinstance(rule['meta'][k], str): | ||
| rule['meta'][k] = rule['meta'][k].strip() | ||
| else: | ||
| rule['meta'][k] = rule[k] | ||
| continue |
There was a problem hiding this comment.
Moved this out of the loop as there's no need to check the entirety of self.rules upon every iteration - it can wait until after the loop completes. Also the continue on the last line of the loop was redundant.
| raise SyntaxError("Sanity check of correlation rules failed, aborting.") | ||
| raise SyntaxError("Sanity check of correlation rules failed.") |
There was a problem hiding this comment.
Removing "aborting" as this isn't the right place to report state. This initialisation function doesn't know where it is being called from and hasn't "started" anything which could be "aborted" (yet). Also, for comparison, the raise above doesn't say "aborting" yet would have the same effect by raising during init.
| @@ -332,6 +418,7 @@ def event_keep(self, event: dict, field: str, patterns: str, patterntype: str) - | |||
| if value == pattern: | |||
| return False | |||
| else: | |||
| ret = False | |||
There was a problem hiding this comment.
I'm pretty sure the absence of this line was a bug. Looking at the conditional above fir comparison, the ret needs to be initialised. Otherwise, in theory, ret could be set to True during one iteration of the loop, then fall through here on the next iteration without being reset.
| @@ -735,71 +894,97 @@ def create_correlation(self, rule: dict, data: list, readonly=False) -> bool: | |||
|
|
|||
| return True | |||
|
|
|||
| # Syntax-check rules | |||
| def check_ruleset_validity(self, rules: list) -> bool: | |||
There was a problem hiding this comment.
The changes from here to end of the file are simply splitting the check_ruleset_validity function into two functions:
- check_ruleset_validity
- check_rule_validity
The former loops through rules and validates them with the latter.
This remove some of the levels of nesting and offers a function for validating a single rule, rather than all rules, if desired.
|
Failed test is related to DNS as usual. Other tests all passed. |
bcoles
force-pushed
the
spiderfoot-correlation
branch
from
a664ab9
to
87e1752
Compare
| self.__setStatus("FINISHED", None, time.time() * 1000) | ||
| self.runCorrelations() | ||
| self.__sf.status(f"Scan [{self.__scanId}] completed.") | ||
| self.__setStatus("FINISHED", None, time.time() * 1000) |
There was a problem hiding this comment.
Scans and correlations are two different things. Once a scan is complete we can run correlations as many times as we want. Updating the scan status to FINISHED before running correlations makes sense, and lets the Correlator validate that the scan has been completed to prevent attempting to run correlations on running scans.
Codecov Report
@@ Coverage Diff @@
## master #1654 +/- ##
==========================================
- Coverage 52.92% 52.91% -0.01%
==========================================
Files 501 501
Lines 42896 42923 +27
==========================================
+ Hits 22702 22714 +12
- Misses 20194 20209 +15
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
bcoles
changed the title
No description provided.