Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Evidence local_path when it's saved #319

Merged
merged 1 commit into from
Dec 10, 2018
Merged

Conversation

aarontp
Copy link
Member

@aarontp aarontp commented Dec 5, 2018

See #304 for details, but after #282, Tasks that held evidence in the tmp_dir saved the temp path instead of the locally saved path (thereby breaking when different Tasks processed the newly created Evidence). This fixes that.

FYI @ericzinnikas , LMK what you think.

@ericzinnikas
Copy link
Contributor

LGTM -- will try a local run later today.

@aarontp aarontp merged commit 1bb55f8 into master Dec 10, 2018
@aarontp aarontp deleted the evidence-local-path branch December 10, 2018 18:20
ericzinnikas pushed a commit to ericzinnikas/turbiniafb that referenced this pull request Jan 10, 2019
* Pin specific Celery/Kombu/Redis versions (google#305)

* in TurbiniaTaskResult, input_evidence doesn't have to be a list (google#309)

* input_evidence doesn't have to be a list

* Fix exception

* current pypi version of google-cloud-storage (1.13.0) requires google-cloud-core 0.28.1 (before the rename of google.cloud.iam (core) to google.api_core.iam (google#315)

* Use a link to a "parent Evidence" instead of subclassing (google#296)

* parent evidence

* undo some simplifications for the sake of a simpler CL

* Add some serialization

* update docstring

* Initialize attribute

* set parent evidence if Evidence type is context dependant

* don't pass parent_evidence at instantiation

* undo linter stuff

* comments

* fix aim lib breaking tests

* typo

* Print version on start 3 (google#320)

* Add files via upload

* Delete turbiniactl.py

* Delete turbiniactl.py

* Add files via upload

* Delete turbiniactl.py

* Add files via upload

* Update turbiniactl.py

* Caps

* Quick update to evidence docstrings (google#317)

... to disambiguate between _preprocess() and preprocess().

* Add Job filters (google#247)

* Add job filters

* fix docstrings.

* update docstring

* Get jobs filters working with new job manager

* Refactor out FilterJobObjects into new method

* Update YAPF

* remove missed confict markers

* Docstrings and renaming

* Migrate job graph generator to use new manager (google#321)

* Update Evidence local_path when it's saved (google#319)

* Pin google-cloud-storage to 1.13.0 (google#326)

Fixes google#325

Looks like google-cloud-storage was updated in:
googleapis/google-cloud-python#6741

Which just got released as 1.13.1:
https://pypi.org/project/google-cloud-storage/#history

* Set image export to process all partitions (google#324)

* Add --partitions all to image_export invocations

* Fix typo

* Explicitly set saved_paths to list (google#323)

* Move version print after log level setup (google#322)

* Move version print after log level setup

* Remove extra whitespace

* update the pystyle link (google#333)

* Undefined name: Define 'unicode' in Python 3 (google#337)

* Undefined name: Define 'unicode' in Python 3

__unicode()__ was removed in Python 3 because all __str__ are Unicode.

[flake8](http://flake8.pycqa.org) testing of https://github.com/google/turbinia on Python 3.7.1

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./tools/turbinia_job_graph.py:47:40: F821 undefined name 'unicode'
  parser.add_argument('filename', type=unicode, help='where to save the file')
                                       ^
1     F821 undefined name 'unicode'
1
```

* Placate PyLint

* Added PSQ timeout to 1 week (google#336)

* Error when worker version doesn't match server google#307 (google#327)

* Added turbina_version to TurbinaTask

* First approach

* Changed to no rise error and return instead

* Restored the run from run_wrapper

* Changed format of strings

* Changed words fixed line too long

* bump version
@ericzinnikas
Copy link
Contributor

I've been caught up with a few things, but just gave this another test and it is not working for me. I'm using a fresh master clone.

@aarontp I see in #304 you mentioned:

FWIW I verified it by making sure that the Plaso run was writing to /tmp and the Psort run was operating on the evidence in the permanent output directory.

But I am still seeing the old behavior (reading plaso file from tmp dir):

[INFO] Running psort as [psort.py --status_view none --logfile /data/users/ericwz/turbinia-output/1547302957-fb3cfba43eb04d69a931c5ec0c353b35-PsortTask/fb3cfba43eb04d69a931c5ec0c353b35.log -w /data/users/ericwz/turbinia-output/1547302957-fb3cfba43eb04d69a931c5ec0c353b35-PsortTask/fb3cfba43eb04d69a931c5ec0c353b35.csv /tmp/1547302757-8a95cced682a45bbbec04f7491b70c11-PlasoTask/8a95cced682a45bbbec04f7491b70c11.plaso]

Any thoughts?

@ericzinnikas
Copy link
Contributor

Seems this is no longer occurring (working as expected).

@aarontp
Copy link
Member Author

aarontp commented Feb 7, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants