Post upload file in the new frontend

[marcobrotto]

Post file upload in the new frontend

Improvements

• Non fully indexed files are constantly monitored to give the user an idea of the "indexing-progress". This feature allows the user to upload a large file (e.g., a large Plaso file) and having an idea about the status of the file on the server (e.g., 40% indexed).
• Add a status dialog that summarize the information of the file (how many events, the status, i.e., ready, processing, or fail).
• The user can directly add a new timeline from the Explore tab. The user can only enable the timelines that have been successfully uploaded.

Implementation and Design Idea

When a timeline is uploaded on the server, it may have 3 different status: ready, processing or fail. We always show to the user the status of his timelines.

1. Timeline is ready: the user can explore it.

2. Timeline is failed: the user can not explore it. A failed timeline can be spotted because it has a red background color and it cannot be opened.

3. Timeline is processing: the user can check how many events have already been indexed and the remaining time for the timeline to be ready. A "processing" timeline can be spotted because is not fully colored and it cannot be opened.

Every 5 seconds, we do an API request for the non ready/failed timelines. The server sends us the status of the timeline plus the 2 extra parameters: the total events of that timeline and the current number of indexed events.

We will call the first variable total_events and the second one indexed_events. The first one can be computed only once, for example when the file is uploaded (see uplaod.py, get_total_events). The second one needs to be computed every time we ask the server if the timeline is ready or not.

total_events

We modified the schema of the SQL table Timeline. In particular, we added the attribute total_events that represents how many events the timeline has.

How total_events is computed

For efficiency, we run subprocess Python command to extract this information.

1. Plaso file: we run the command pinfo.py and we extract the total events. This command gives us more useful information such as which parser has been used.
2. CSV/JSONL file: we run the command wc -l file_name.

Why do we need another attribute in the Timeline table?

Adding a new attribute to the schema of a table was not an easy choice because it implies modifying the schema of Timesketch infrastructure. We considered another option that is passing the total_events value in the UPLOAD API response, and, on the client, store this value in the VUE store. However this solution has the main disadvantage that the UPLOAD API is called only once, i.e., after the file is uploaded on the server. If the user refresh the page, then we lose this value on the server. On the contrary, having total_events stored in the schema of the Timeline, allows us to retrieve it every time we perform an API request api/v1/sketches/ID_scketch/timelines/ID_timeline.

The third solution is to compute this value every time we perform the above API request. However, this idea is not efficient since total_events will always be the same value.

indexed_events

This information is retrieved by querying the Opensearch database when we send the API api/v1/sketches/ID_scketch/timelines/ID_timeline.

Progress percentage and remaining time

The progress percentage P_perc is equal to:

P_perc = indexed_events/total_events

The remaining time R_t is equal to:

t0 := {timestamp when the user uploaded the file}
t_now := {current timestamp}
Delta_now = t_now - t_0
Delta_tot = (Delta_now * 100) / P_perc
Delta_tot =: t_end - t0 => t_end = Delta_tot + t0
=> R_t = t_end - t_now

Major file updates

Frontend

2. TimelinePicker.vue: This component is responsible for showing ALL the timelines of the sketch (also those ones that are not ready).
3. TimelineChip.vue: This component is responsible for showing the "status-color" of a timeline. For each timeline, depending on its status, this component allows the user to perform a certain set of actions such as observing the status, explore the timeline...
4. [NEW!] TimelineStatus.vue: this component receives from TimelineChip.vue some props such as timeline, indexed events, and status. Although some parameters are "redundant", we decided to pass them anyway because they are both used from TimelineStatus and TimelineChip. This components is responsible to show the user the status information of a timeline.

Backend

1. upload.py: we compute the total_events for a single timeline. This value is added to the table timeline
2. timeline.py: we compute the number of the indexed events when the timeline is ready or processing. We add this value in the meta field of the response.

[jaegeral]

Please check the merge conflict

[marcobrotto]

New PR #2326