timesketch_importer - OSError: Format version: 20230327 is too new and not yet supported

[Bloggzy]

Describe the bug
It appears that timesketch_importer doesn't like the plaso files generated by the 20230717 version of log2timeline/plaso, but is will process plaso files generated by older versions of log2timeline/plaso e.g. 20230311.

Actions: Having generated a .plaso (20230717) file using, and attempting to import it into Timesketch, I get the following error:

"OSError: Format version: 20230327 is too new and not yet supported, minimum supported version: 20230107"

To Reproduce
Steps to reproduce the behavior:

1. Using the current Docker version of log2timeline/plaso log2timeline I process some Windows 10 artefacts e.g. sudo docker run --rm -v /home/user/data:/data log2timeline/plaso log2timeline --timezone 'UTC' --parsers 'win7_slow,!filestat' --storage_file /data/host01.plaso /data/host01
2. I then attempt to import that plaso file into Timesketch e.g. timesketch_importer -u testuser -p password --host http://127.0.0.1 --timeline_name timeline-test-01 --sketch_name sketch-test-01 /home/user/data/host01.plaso
3. The importer will upload the file, and then display the following message: Checking file upload status: .[FAIL]
4. Checking the Timesketch GUI there is the following error message:
   Original filename: host01
File on disk: /usr/share/timesketch/upload/70da71c984844d499f0c57ffc8d3bd45
File size: 1.15GB
Uploaded by: testuser
Provider: CLI importer tool
Context: /home/user/.local/bin/timesketch_importer -u testuser -p password --host http://127.0.0.1 --timeline_name timeline-test-01 --sketch_name sketch-test-01 /home/user/data/host01.plaso
Data label: plaso
Status: fail
Total File Events: 0
Error message: Traceback (most recent call last): File "/usr/bin/pinfo.py", line 85, in <module> if not Main(): File "/usr/bin/pinfo.py", line 62, in Main tool.PrintStorageInformation() File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 1673, in PrintStorageInformation storage_reader = self._GetStorageReader(self._storage_file_path) File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 587, in _GetStorageReader storage_factory.StorageFactory.CreateStorageReaderForFile(path)) File "/usr/lib/python3/dist-packages/plaso/storage/factory.py", line 49, in CreateStorageReaderForFile return sqlite_reader.SQLiteStorageFileReader(path) File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/reader.py", line 20, in __init__ self._store.Open(path=path) File "/usr/lib/python3/dist-packages/acstore/sqlite_store.py", line 999, in Open self._ReadAndCheckStorageMetadata(check_readable_only=True) File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 231, in _ReadAndCheckStorageMetadata self._CheckStorageMetadata( File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 53, in _CheckStorageMetadata super(SQLiteStorageFile, self)._CheckStorageMetadata( File "/usr/lib/python3/dist-packages/acstore/sqlite_store.py", line 199, in _CheckStorageMetadata raise IOError(( OSError: Format version: 20230327 is too new and not yet supported, minimum supported version: 20230107.

Expected behavior
Would be to import the events, this can still be achieved, but you have to use an older version of log2timeline/plaso e.g. log2timeline/plaso:20230311

Desktop (please complete the following information):

• OS: Ubuntu 20.04 & 22.04
• Browser: Firefox
• Version: 116

[jaegeral]

This is not a bug, Timesketch can only import Plaso files up to the version installed on Timesketch. If Plaso files are generated with. newer version, there can be dozen of things in the Plaso files that Timesketch does not know how to handle, which is why you see the error message.

So the solution is to update Plaso on the Timesketch, e.g. by updating Timesketch.

Hope that helps

[Bloggzy]

Hey @jaegeral, thanks for the response.

As far as I knew, I was using the most up to date version of Timesketch i.e. "TIMESKETCH_VERSION=latest".

Is there something obvious that I'm missing?

My docker-compose.yml file says:
version: "3.7"
services:
timesketch-web:
container_name: timesketch-web
image: us-docker.pkg.dev/osdfir-registry/timesketch/timesketch:${TIMESKETCH_VERSION}

And the config.env file says:
# Timesketch version to run. Latest is build from the master branch and a release
# number is build from a release tag. Using latest means that you are running
# the bleeding edge version and we cannot guarantee that it will not be broken.
TIMESKETCH_VERSION=latest

[jaegeral]

Yeah this is something that comes up from time to time, see:
#2589
#2859

I added a few infos / data points in here: #2866 on how to get the versions.

[Bloggzy]

When I execute the following on the timesketch-web container:
sudo docker exec timesketch-web log2timeline.py --troubles
I get returned:
plaso - log2timeline version 20230311

And when I query, the version from the log2timeline/plaso container:
sudo docker run --rm -v /home/user/data:/data log2timeline/plaso psort --version
I get this returned:
plaso - psort version 20230717

[Bloggzy]

@jaegeral, please forgive my ignorance, but is there a way to call log2timeline/plaso from inside the Timesketch Docker container?

So that you know you’re using the same version as will be ingesting the plaso file later on?

[jaegeral]

You mean docker in docker? I think that is not possible. But yeah your output in #2865 (comment) confirms, the file you uploaded was created with a to new Plaso version.

So one thing you can do is check which version you have installed in your Timesketch system and have Plaso only create the plaso file you will upload later with that very Plaso version. So you could process your evidence with:
docker pull log2timeline/plaso:20230311 (see: https://hub.docker.com/r/log2timeline/plaso/tags)

[Bloggzy]

I saw the docker exec call to log2timeline.py and wondered if it extended further.
sudo docker exec timesketch-web log2timeline.py --troubles
But not to worry, it was just wishful thinking.

I’ve been using the older docker tag to work around it, I’ll add it to our documentation in case anyone else runs into the same issue.

Thanks for your efforts, much appreciated.

[jaegeral]

And to be clear, I understand that it is a problem and that it needs some better integration / documentation to not run into those situations, but at the moment we do not have a short result for this :-/

[Bloggzy]

It’s all good, I’ll sure you’ll work out a better long term solution, it’s an awesome project, a few minor things like this will always crop up.