Uploading 1GB plus plaso file to timesketch fails #1060
Comments
|
It may be that nothing is working, i have "indexing in progress" for a timeline but when i run curl on the elastic instance there are no indexes. |
|
have you tried using the file import? scp the file to an import location? what version of timesketch are you using? |
|
There seems to be an issue with the docker installation. Launching celery as root by removing |
No, not sure how to do that?
latest version from git as of yesterday (docker-compose)
I am going to try and install timesketch again with docker and see how i go starting again, if i get the same issue ill look at trying this. |
|
When i install, the only errors ive noticed (running docker-compose up for the first time) are: Step 13/23 : RUN cd /tmp/timesketch && yarn install && yarn run build |
|
Actually, this is a duplicate of #870 The issue is that when executed as nobody (which is the case in the Docker installation), |
I am running timesketch in docker with docker-compose. When i log onto the UI and upload plaso files, files over 1GB failed with the following messages. The standard psort messages, but the last 2 lines are only present on files that are larger than 1GB and the file upload times out.
Ubuntu server 18.04, latest version of docker, ran update/upgrade on everything before running up docker and docker-compose. Have 8GB assigned to the VM, updated vm.max_map_count=262144 else it doesn't run anyway.
timesketch_1 | [2019-12-17 05:22:54,924: INFO/MainProcess] Received task: timesketch.lib.tasks.run_plaso[1fde0509-4b67-4c16-b490-7d7f3ee09b6a] timesketch_1 | [2019-12-17 05:22:54,929: INFO/ForkPoolWorker-1] Index timeline [PCIC_191127_PSG_004] to index [66e678b734cc4e3d8f9cf50ad48c72be] (source: plaso) timesketch_1 | [2019-12-17 05:23:00,096: INFO/ForkPoolWorker-1] Task timesketch.lib.tasks.run_plaso[1fde0509-4b67-4c16-b490-7d7f3ee09b6a] succeeded in 5.169042253000043s: '2019-12-17 05:22:59,007 [INFO] (MainProcess) PID:49 <data_location> Determined data location: /usr/share/plaso timesketch_1 | 2019-12-17 05:22:59,127 [INFO] (MainProcess) PID:49 <timesketch_out> Timeline name: <redacted> timesketch_1 | 2019-12-17 05:22:59,128 [INFO] (MainProcess) PID:49 <timesketch_out> Owner of the timeline: None timesketch_1 | Traceback (most recent call last): timesketch_1 | File "/usr/bin/psort.py", line 85, in <module> timesketch_1 | if not Main(): timesketch_1 | File "/usr/bin/psort.py", line 67, in Main timesketch_1 | tool.ProcessStorage() timesketch_1 | File "/usr/lib/python3/dist-packages/plaso/cli/psort_tool.py", line 501, in ProcessStorage timesketch_1 | self._CheckStorageFile(self._storage_file_path) timesketch_1 | File "/usr/lib/python3/dist-packages/plaso/cli/psort_tool.py", line 115, in _CheckStorageFile timesketch_1 | logger.warning('Appending to an already existing storage file.') timesketch_1 | File "/usr/lib/python3.6/logging/__init__.py", line 1320, in warning timesketch_1 | self._log(WARNING, msg, args, **kwargs) timesketch_1 | File "/usr/lib/python3.6/logging/__init__.py", line 1444, in _log timesketch_1 | self.handle(record) timesketch_1 | File...' timesketch_1 | [2019-12-17 05:45:41 +0000] [29] [CRITICAL] WORKER TIMEOUT (pid:35) timesketch_1 | [2019-12-17 05:45:49 +0000] [57] [INFO] Booting worker with pid: 57The text was updated successfully, but these errors were encountered: