Uploading 1GB plus plaso file to timesketch fails

[shannaniggans]

I am running timesketch in docker with docker-compose. When i log onto the UI and upload plaso files, files over 1GB failed with the following messages. The standard psort messages, but the last 2 lines are only present on files that are larger than 1GB and the file upload times out.

Ubuntu server 18.04, latest version of docker, ran update/upgrade on everything before running up docker and docker-compose. Have 8GB assigned to the VM, updated vm.max_map_count=262144 else it doesn't run anyway.

timesketch_1 | [2019-12-17 05:22:54,924: INFO/MainProcess] Received task: timesketch.lib.tasks.run_plaso[1fde0509-4b67-4c16-b490-7d7f3ee09b6a] timesketch_1 | [2019-12-17 05:22:54,929: INFO/ForkPoolWorker-1] Index timeline [PCIC_191127_PSG_004] to index [66e678b734cc4e3d8f9cf50ad48c72be] (source: plaso) timesketch_1 | [2019-12-17 05:23:00,096: INFO/ForkPoolWorker-1] Task timesketch.lib.tasks.run_plaso[1fde0509-4b67-4c16-b490-7d7f3ee09b6a] succeeded in 5.169042253000043s: '2019-12-17 05:22:59,007 [INFO] (MainProcess) PID:49 <data_location> Determined data location: /usr/share/plaso timesketch_1 | 2019-12-17 05:22:59,127 [INFO] (MainProcess) PID:49 <timesketch_out> Timeline name: <redacted> timesketch_1 | 2019-12-17 05:22:59,128 [INFO] (MainProcess) PID:49 <timesketch_out> Owner of the timeline: None timesketch_1 | Traceback (most recent call last): timesketch_1 | File "/usr/bin/psort.py", line 85, in <module> timesketch_1 | if not Main(): timesketch_1 | File "/usr/bin/psort.py", line 67, in Main timesketch_1 | tool.ProcessStorage() timesketch_1 | File "/usr/lib/python3/dist-packages/plaso/cli/psort_tool.py", line 501, in ProcessStorage timesketch_1 | self._CheckStorageFile(self._storage_file_path) timesketch_1 | File "/usr/lib/python3/dist-packages/plaso/cli/psort_tool.py", line 115, in _CheckStorageFile timesketch_1 | logger.warning('Appending to an already existing storage file.') timesketch_1 | File "/usr/lib/python3.6/logging/__init__.py", line 1320, in warning timesketch_1 | self._log(WARNING, msg, args, **kwargs) timesketch_1 | File "/usr/lib/python3.6/logging/__init__.py", line 1444, in _log timesketch_1 | self.handle(record) timesketch_1 | File...' timesketch_1 | [2019-12-17 05:45:41 +0000] [29] [CRITICAL] WORKER TIMEOUT (pid:35) timesketch_1 | [2019-12-17 05:45:49 +0000] [57] [INFO] Booting worker with pid: 57

[shannaniggans]

It may be that nothing is working, i have "indexing in progress" for a timeline but when i run curl on the elastic instance there are no indexes.

[kiddinn]

have you tried using the file import? scp the file to an import location?

what version of timesketch are you using?

[gaelmuller]

There seems to be an issue with the docker installation. Launching celery as root by removing --uid nobody from the docker-entrypoint does the trick for me. I am still not sure how to explain this.

[shannaniggans]

have you tried using the file import? scp the file to an import location?

No, not sure how to do that?

what version of timesketch are you using?

latest version from git as of yesterday (docker-compose)

There seems to be an issue with the docker installation. Launching celery as root by removing --uid nobody from the docker-entrypoint does the trick for me. I am still not sure how to explain this.

I am going to try and install timesketch again with docker and see how i go starting again, if i get the same issue ill look at trying this.

[shannaniggans]

When i install, the only errors ive noticed (running docker-compose up for the first time) are:

Step 13/23 : RUN cd /tmp/timesketch && yarn install && yarn run build
---> Running in d52857376018
yarn install v1.21.1
[1/4] Resolving packages...
[2/4] Fetching packages...
warning sha.js@2.4.8: Invalid bin entry for "sha.js" (in "sha.js").
info fsevents@1.1.2: The platform "linux" is incompatible with this module.
info "fsevents@1.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning "@angular/compiler-cli > tsickle@0.23.6" has incorrect peer dependency "typescript@2.3.4".
warning " > codelyzer@3.2.0" has incorrect peer dependency "@angular/compiler@^2.3.1 || >=4.0.0-beta <5.0.0".
warning " > codelyzer@3.2.0" has incorrect peer dependency "@angular/core@^2.3.1 || >=4.0.0-beta <5.0.0".
warning " > karma-jasmine@1.1.0" has unmet peer dependency "jasmine-core@*".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 76.82s.
yarn run v1.21.1
$ webpack

[gaelmuller]

Actually, this is a duplicate of #870

The issue is that when executed as nobody (which is the case in the Docker installation), psort fails trying to create its log file in /.