Get processing timeline events #3241

jbaptperez · 2024-12-05T15:57:41Z

IMPORTANT: All Pull Requests should be connected to an issue, if you don't
have an issue, please start by creating an issue and link it to the PR.

Please provide enough information so that others can review your pull request:

What existing problem does this PR solve?
- Impossible to get timeline events when the one is being updated (new Plaso ingestion for the timeline).
What new feature is being introduced with this PR?
- Allows to get timeline events when the timeline is being updated (new Plaso ingestion) i.e. allows to get partial events, already indexed,
- Adds a setting to enable or disable this feature,
- The UI clearly communicates to the user that search results against processing timelines may be incomplete and subject to change.
Overview of changes to existing functions if required.
- Backend: In the Sketch model, the active_timelines property includes timelines with a status set to processing,
- Backend: In the utils.py, the get_validated_indices function includes timelines with a status set to processing,
- Backend: The POST /sketches/{{sketch_id}}/aggregation/explore/ endpoint includes indices which timeline has a status set to processing.

Checks

All tests succeed.
Unit tests added.
e2e tests added.
Documentation updated.

Closing issues

Closes #3219.

jkppr · 2024-12-12T09:42:24Z

Thanks for adding a draft PR for early feedback. If I see this correctly, you are also having the changes submitted in your PR #3226 in this PR. Can you please separate this out and only commit the changes relevant to the issue #3219 in this PR?

This would make reviewing the code much easier, thanks.

jbaptperez · 2024-12-14T15:44:12Z

@jkppr, I removed the work of the other PR.
I also rebased the branch onto master.

The only change I made for now is at the backend side: It unlocks the desired behaviour: Getting timeline events while in the processing status.
Note for now I did not added any frontend or backend setting to properly toggle this feature, I'll work on it at the end.

At the frontend side, for now, all timeline events appear even if some of them are is in the processing state.
It is easy to reproduce such a situation by forcing a timeline state in the database.
This results in an infinite spinning wheel in the timeline chip.

However, I face difficulties when attempting to change the timeline chip so that a processing one behaves like a ready/failed one, with an additional spinning wheel as the only visual difference.

The complementarity between TimelineChip.vue and TimelineComponent.vue is not trivial.

Do you think someone can help me to accomplish this precise part?

Annoraaq · 2024-12-16T15:53:36Z

TimelineComponent.vue contains a slot that is shown if the timeline status is processing. My understanding is that you want to change that and instead want to render the other slot (called "processed" in the code).

Now one thing to be careful about is that TimelineComponent.vue is used by Explore/TimelineChip.vue and Analyzer/TimelineChip.vue and whatever you change in TimelineComponen.vue will impact both types of timeline chips.

jbaptperez · 2024-12-23T17:38:30Z

Hi @jkppr, I have made progress, but still WIP.

For now:

I added a user setting (false by default): showProcessingTimelineEvents,
I adapted the frontend-ng so that it properly reads the setting,
The frontend can display processing timeline events with a consistent TimelineComponent/TimelineChip set,
I have manually checked that there is no issue when displaying the timeline chip in the_Analyser results_ and the Visualizations pages.

I am currently developing the following:

Updating the backend so that the current hard-coded changes in my branch become variable, depending on a default value as a method parameter or coming from the related endpoint forms (at least GET /sketches/{sketch_id}/ and POST /sketches/{sketch_id}/explore/),
Updating the frontend (store.js, RestApiClient.js) so that the setting is properly dispatched over the views.

Feel free to give me advice or instructions to fine tune my change, if necessary.

jbaptperez · 2025-01-08T15:42:31Z

Hi @jkppr, I just pushed a first version of the feature.

Changes are:

POST /sketch/{sketchId}/explore: Add an optional includeProcessingTimelines parameter,
sketch model: Update the active_timelines property to include timelines with status processing,
POST /sketches/{sketch_id}/aggregation/explore/ : Include indices of timelines with status processing,
Add a frontend user setting: showProcessingTimelineEvents,
Update the frontend to take into account the showProcessingTimelineEvents,
Add a frontend banner when displaying processing timeline events (at least one processing timeline must be selected).

I will try to deploy those changes in my company in order to let the PO check the feature.
However, I would like your opinion about the code I produced, in the meantime.

I'll update the current code in the case we find a bug or the PO wants to change something.
I'll pull out the draft status of the PR when we'll consider it as final.

jkppr

This feature turned out great already during testing, thanks for all the effort!

Here is the first part of a partial review. It covers all files except some files in components/Explore/.

Some general topics I noticed:

With the current implementation the decision if the feature will be used is solely on the user. What I'm missing is an option for a server admin to control the functionality. So adding a config option to data/timesketch.conf that allows admins to disable/enable the feature for their instance. Check server side functionality against that setting and handle accordingly. This does not only give the admin of an instance the decision what features they want to support on the deployment but also allows us later to easily add default settings for all users.
- E.g. SEARCH_PROCESSING_TIMELINES = True
There seems to be no processing progress shown anymore on the Data Source dialog, when the feature is turned on. Is this on purpose?

Otherwise I just noticed some errors and added suggestions or questions below.

timesketch/models/sketch.py

timesketch/lib/utils.py

timesketch/frontend-ng/src/components/Explore/AggregateDialog.vue

timesketch/frontend-ng/src/components/Explore/EventActionMenu.vue

jkppr · 2025-02-21T22:37:01Z

timesketch/frontend-ng/src/components/Visualization/VisualizationEditor.vue

@@ -438,7 +441,8 @@ export default {
            xTitle: this.selectedXTitle,
            yTitle: this.selectedYTitle,
          },
-        }
+        },
+        include_processing_timelines: !!this.settings.showProcessingTimelineEvents,


What is this change supposed to do? I did not notice any effect on the VisualizationEditor.

It adds the "processing yes/no" boolean for a further backend API call (to propagate the user setting up to the API call).

timesketch/frontend-ng/src/components/Visualization/SavedVisualization.vue

timesketch/frontend-ng/src/components/SettingsDialog.vue

timesketch/frontend-ng/src/components/Visualization/SavedVisualization.vue

timesketch/frontend-ng/src/components/LeftPanel/TimelinesTable.vue

timesketch/frontend-ng/src/components/Explore/AggregateDialog.vue

timesketch/frontend-ng/src/store.js

Annoraaq · 2025-02-27T12:08:40Z

timesketch/frontend-ng/src/components/Explore/EventList.vue

+      // least one enabled timeline is the "processing" state.
+      this.showBanner =
+        !!this.settings.showProcessingTimelineEvents &&
+        this.$store.state.sketch.active_timelines


optional: you can simplify this by using this.sketch() instead of this.$store.state.sketch.

Done (rebased).

timesketch/frontend-ng/src/store.js

jbaptperez · 2025-03-18T14:47:36Z

@jkppr, @Annoraaq, we (a colleague and I) did the requested changes.
We rebased/squashed the result to minimize the amount of useless commits.
We only put the addition of the new system (backend) setting into a new commit.

The PR is now ready for a final review.

The POST /sketches/{sketchId}/explore/ endpoint can now be parameterized to include processing timeline indices.

Default value is "false" (legacy behaviour).

User settings must be loaded before the sketch to transmit the new parameter to POST /sketches/{id}/aggregation/explore/.

The setting name is SEARCH_PROCESSING_TIMELINES. The related user setting (showProcessingTimelineEvents) only appears when this system setting is enabled.

jbaptperez force-pushed the get-processing-timeline-events branch 2 times, most recently from 42403d9 to de22d67 Compare December 5, 2024 17:09

jkppr assigned jbaptperez and jkppr Dec 12, 2024

jkppr added the Backend label Dec 12, 2024

jbaptperez force-pushed the get-processing-timeline-events branch from de22d67 to 281bb34 Compare December 14, 2024 15:09

jbaptperez force-pushed the get-processing-timeline-events branch 3 times, most recently from 7aad31c to 0177d57 Compare December 23, 2024 17:25

jbaptperez force-pushed the get-processing-timeline-events branch 2 times, most recently from e5e1caf to 901a0b6 Compare January 8, 2025 14:27

jbaptperez force-pushed the get-processing-timeline-events branch from 901a0b6 to a3d287e Compare January 9, 2025 09:30

jbaptperez force-pushed the get-processing-timeline-events branch 2 times, most recently from bd02f79 to a308002 Compare January 21, 2025 11:43

jbaptperez force-pushed the get-processing-timeline-events branch from a308002 to 91be24c Compare February 5, 2025 08:42

jbaptperez closed this Feb 5, 2025

jbaptperez deleted the get-processing-timeline-events branch February 5, 2025 08:54

jbaptperez restored the get-processing-timeline-events branch February 5, 2025 08:59

jbaptperez reopened this Feb 5, 2025

jbaptperez force-pushed the get-processing-timeline-events branch 2 times, most recently from b0b1266 to 2f41b26 Compare February 21, 2025 17:27

jkppr self-requested a review February 21, 2025 19:01

jkppr requested changes Feb 21, 2025

View reviewed changes

Annoraaq reviewed Feb 27, 2025

View reviewed changes

jbaptperez force-pushed the get-processing-timeline-events branch from 2f41b26 to 38a69c9 Compare March 18, 2025 14:05

jbaptperez force-pushed the get-processing-timeline-events branch from 38a69c9 to b8b2507 Compare March 18, 2025 14:40

jbaptperez marked this pull request as ready for review March 18, 2025 14:48

jbaptperez force-pushed the get-processing-timeline-events branch from b8b2507 to e749a4f Compare March 19, 2025 15:26

jaegeral requested a review from jkppr March 19, 2025 15:31

jbaptperez and others added 6 commits March 20, 2025 09:12

Allow showing processing timeline events (backend)

c7fb08a

The POST /sketches/{sketchId}/explore/ endpoint can now be parameterized to include processing timeline indices.

Add a user setting to enable processing timeline events

ac6cc63

Default value is "false" (legacy behaviour).

Allow showing processing timeline events (frontend)

b0f626d

Add a banner when displaying processing timeline events

dd44c5e

Updates aggregation/explore to fit user settings

d01da68

User settings must be loaded before the sketch to transmit the new parameter to POST /sketches/{id}/aggregation/explore/.

Add a system setting to enable processing timeline events

Loading
Loading status checks…

f2c14b6

The setting name is SEARCH_PROCESSING_TIMELINES. The related user setting (showProcessingTimelineEvents) only appears when this system setting is enabled.

jbaptperez force-pushed the get-processing-timeline-events branch from e749a4f to f2c14b6 Compare March 20, 2025 08:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get processing timeline events #3241

Get processing timeline events #3241

jbaptperez commented Dec 5, 2024 •

edited

Loading

jkppr commented Dec 12, 2024

jbaptperez commented Dec 14, 2024

Annoraaq commented Dec 16, 2024

jbaptperez commented Dec 23, 2024 •

edited

Loading

jbaptperez commented Jan 8, 2025

jkppr left a comment

jkppr Feb 21, 2025

jbaptperez Mar 18, 2025

Annoraaq Feb 27, 2025

jbaptperez Mar 18, 2025

jbaptperez commented Mar 18, 2025

Get processing timeline events #3241

Are you sure you want to change the base?

Get processing timeline events #3241

Conversation

jbaptperez commented Dec 5, 2024 • edited Loading

jkppr commented Dec 12, 2024

jbaptperez commented Dec 14, 2024

Annoraaq commented Dec 16, 2024

jbaptperez commented Dec 23, 2024 • edited Loading

jbaptperez commented Jan 8, 2025

jkppr left a comment

Choose a reason for hiding this comment

jkppr Feb 21, 2025

Choose a reason for hiding this comment

jbaptperez Mar 18, 2025

Choose a reason for hiding this comment

Annoraaq Feb 27, 2025

Choose a reason for hiding this comment

jbaptperez Mar 18, 2025

Choose a reason for hiding this comment

jbaptperez commented Mar 18, 2025

jbaptperez commented Dec 5, 2024 •

edited

Loading

jbaptperez commented Dec 23, 2024 •

edited

Loading