The results of uploading CSV files between timesketch_importer CLI and GUI are different

[Camel0101]

Describe the bug
I have a CSV file. When I import it via the GUI, only the non-empty events are displayed:

However, when I import the same CSV file using the timesketch_importer CLI with this command:

sudo timesketch_importer sample_test.csv --timeline_name "TIMESKETCH_IMPORTER_CSV" --sketch_name "TEST" 

all events, including the empty ones, are shown:

Do you know why there is a difference between the two, and is it possible to achieve the same output as the GUI (excluding empty keys) with the timesketch_importer CLI ?

Desktop:

• OS: Ubuntu 22.04
• Browser: Firefox

[cichyx96]

Looks like when uploading via GUI it goes through Celery task which uses pandas to read CSV and it drops all NAN values.

timesketch/timesketch/lib/utils.py

Lines 339 to 351 in d17f984

	for _, row in chunk.iterrows():
	_scrub_special_tags(row)
	# Remove all NAN values from the pandas.Series.
	row.dropna(inplace=True)
	# Make sure we always have a timestamp
	if not "timestamp" in row:
	row["timestamp"] = int(
	pandas.Timestamp(row["datetime"]).value / 1000
	)
	yield row.to_dict()

CLI importer only reads flie and uploads it as it is.

timesketch/cli_client/python/timesketch_cli_client/commands/importer.py

Lines 42 to 56 in d17f984

	with import_client.ImportStreamer() as streamer:
	click.echo("Uploading to server .. ", nl=False)
	streamer.set_sketch(sketch)
	streamer.set_timeline_name(name)
	streamer.set_provider("Timesketch CLI client")
	# TODO: Consider using the whole command as upload context instead
	# of the file path.
	streamer.set_upload_context(file_path)
	streamer.add_file(file_path)
	timeline = streamer.timeline
	if not timeline:
	click.echo("Error creating timeline, please try again.")
	sys.exit(1)
	click.echo("Done")