Automatic analysis tasks #663
Conversation
…etch into 662-auto-similarity-scorer
timesketch/api/v1/resources.py
Outdated
| task_directory = {u'plaso': run_plaso, | ||
| u'csv': run_csv_jsonl, | ||
| u'jsonl': run_csv_jsonl} | ||
| from timesketch.lib import tasks |
There was a problem hiding this comment.
Why not just import this at the top?
There was a problem hiding this comment.
This is the way Celery works together with Flask. In order to avoid circular imports this is needed.
timesketch/api/v1/resources.py
Outdated
| # If enabled, run sketch analyzers when timeline is added. | ||
| try: | ||
| if current_app.config[u'ENABLE_SKETCH_ANALYZERS']: | ||
| from timesketch.lib import tasks |
| sketch_id: The Sketch ID. | ||
| """ | ||
| self.id = sketch_id | ||
| self.sql_sketch = SQLSketch.query.get(sketch_id) |
There was a problem hiding this comment.
How safe is this? Should this get wrapped in a try/except block? or will an appropriate exception be raised already?
There was a problem hiding this comment.
SQLalchemy returns None if the sketch doesn't exist. This will raise later in the code path but I have added an early raise here as well.
There was a problem hiding this comment.
Cool. Maybe add a Raises: section in the docstring too then? :)
timesketch/lib/tasks.py
Outdated
| analyzer_class = manager.AnalysisManager.get_analyzer(analyzer_name) | ||
| analyzer = analyzer_class(index_name=index_name, **kwargs) | ||
| result = analyzer.run_wrapper() | ||
| logging.info('[%s] result: %s' % (analyzer_name, result)) |
There was a problem hiding this comment.
Any reason you're not using {0:s} and .format() here? I thought it might be because it's a logging line, but there are instances of using format() in logging lines in this PR.
| # Run psort.py | ||
| try: | ||
| cmd_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) | ||
| subprocess.check_output(cmd, stderr=subprocess.STDOUT) |
There was a problem hiding this comment.
Don't you need to add shell=True here in order for psort_path to work when it is set to just psort.py?
There was a problem hiding this comment.
Good point, but no, it seems to work without shell=True
|
|
||
| return cmd_output | ||
| return index_name |
There was a problem hiding this comment.
I guess the return type should be changed in the docstrings then?
|
|
||
| return {u'Events processed': total_events} | ||
| return index_name |
There was a problem hiding this comment.
Change the return type in docstrings?
| db_session.commit() | ||
|
|
||
| # If enabled, run sketch analyzers when timeline is added. | ||
| from timesketch.lib import tasks |
There was a problem hiding this comment.
Why not import at the top of the file? Maybe add a comment about that?
There was a problem hiding this comment.
Same comment as above. Added comment to explain as well.
|
@aarontp PTAL |
|
Re-approved, :) |
This PR implements automatic analysis of timelines added to Timesketch. It supports analysis plugins to run after indexing, or when a new timeline is added to a sketch.
It implements a simple abstraction for creating new analysis workers taking care of all background tasks scheduling etc.