New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hadoop (Yarn tasks) compromise analysis #243

Merged
merged 26 commits into from Sep 19, 2018

Conversation

Projects
None yet
2 participants
@rgayon
Collaborator

rgayon commented Aug 29, 2018

This analysis task looks into the Hadoop AppRoot directory, where Yarn tasks are saved. In case of a compromise, those usually also save the malicious tasks that were created.
Currently, we naively run strings on each of these files and search for post-compromission trivial artifacts (curl or wget, to pull malware).

This depends on the extract_artifacts() methods from turbinia/lib/utils.py which is going to be added by #226
I'm not using FileArtifactExtractionTask as I want to extract all the AppRoot files as one evidence to generate only one report (it is expected that a large amount of Yarn tasks are going to be analyzed while only a handful of them will contain malicious commands)

rgayon added some commits Aug 29, 2018

@rgayon rgayon changed the title from WIP : Hadoop compromise analysis to Hadoop (Yarn tasks) compromise analysis Sep 7, 2018

rgayon added some commits Sep 7, 2018

@aarontp

This comment has been minimized.

Show comment
Hide comment
@aarontp

aarontp Sep 7, 2018

Member

Cool, looking good. I see it's no longer tagged with WIP, but I don't see reviewers assigned yet, so ping me (or assign reviewers) when you're ready. Thanks!

Member

aarontp commented Sep 7, 2018

Cool, looking good. I see it's no longer tagged with WIP, but I don't see reviewers assigned yet, so ping me (or assign reviewers) when you're ready. Thanks!

@rgayon

This comment has been minimized.

Show comment
Hide comment
@rgayon

rgayon Sep 10, 2018

Collaborator

I wanted to let the tests fail miserably (and make sure they do because #226 is not merged yet)

Collaborator

rgayon commented Sep 10, 2018

I wanted to let the tests fail miserably (and make sure they do because #226 is not merged yet)

@rgayon

This comment has been minimized.

Show comment
Hide comment
@rgayon

rgayon Sep 10, 2018

Collaborator

Seems like I actually can't assign a reviewer.

Collaborator

rgayon commented Sep 10, 2018

Seems like I actually can't assign a reviewer.

@aarontp aarontp self-requested a review Sep 18, 2018

@aarontp

For some reason I don't see this failing on lint errors, but that might be because they are hidden due to it failing on other hard errors like util not being there yet (because #226 isn't submitted yet). I'll see if we can get that merged so that we can clear up the errors on this one. I'm adding some other review comments for now, but most of it is just small nits. Thanks!

Show resolved Hide resolved turbinia/jobs/__init__.py Outdated
Show resolved Hide resolved turbinia/jobs/hadoop.py Outdated
Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
Show outdated Hide outdated turbinia/workers/hadoop.py Outdated

rgayon added some commits Sep 18, 2018

@aarontp

This comment has been minimized.

Show comment
Hide comment
@aarontp

aarontp Sep 18, 2018

Member

FYI, #226 is now submitted. It looks like tests are failing on a separate issue more specific to your PR now. LMK when you want me to want me to PTAL, :).

Member

aarontp commented Sep 18, 2018

FYI, #226 is now submitted. It looks like tests are failing on a separate issue more specific to your PR now. LMK when you want me to want me to PTAL, :).

rgayon added some commits Sep 18, 2018

@rgayon

This comment has been minimized.

Show comment
Hide comment
@rgayon

rgayon Sep 18, 2018

Collaborator

Please PTAL a look

Collaborator

rgayon commented Sep 18, 2018

Please PTAL a look

@aarontp

LG, just a couple small things.

Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
Show outdated Hide outdated turbinia/workers/hadoop.py Outdated

rgayon added some commits Sep 19, 2018

Show resolved Hide resolved turbinia/workers/hadoop.py Outdated
@aarontp

LGTM Thanks!

@aarontp aarontp merged commit 6f031ea into google:master Sep 19, 2018

3 checks passed

CodeFactor 6 issues fixed. 2 issues found.
Details
cla/google All necessary CLAs are signed
continuous-integration/travis-ci/pr The Travis CI build passed
Details

Onager added a commit to Onager/turbinia that referenced this pull request Sep 21, 2018

Hadoop (Yarn tasks) compromise analysis (google#243)
* empty shell

* more scaffholding

* make all of this more simple

* Things appear to be working

* typo

* undo some filepath manipulations

* register the new hadoop Job

* cleanup

* renamed

* fix output evidence

* add tests

* typo

* styleguide

* full path to strings

* comments

* sync

* fix tests

* fix py3 tests

* fix py3 harder

* I call this 'bruteforce programming'

* Make _AnalyzeHadoopAppRoot return a list of string, to use the first one as status on the result Evidence

* remove extra tab

* also check at the beggining of the line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment