[DOCS] Add Terminal output view information to Session view doc #2566
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,35 +1,42 @@ | ||
| [[session-view]] | ||
| == Session View | ||
|
|
||
| Session View is an investigation tool that allows you to examine Linux process data organized | ||
| in a tree-like structure according to the Linux logical event model, with processes organized by parentage and time of execution. | ||
| It displays events in a highly readable format that is inspired by the terminal. This makes it a powerful tool for monitoring | ||
| and investigating session activity on your Linux infrastructure and understanding user and service behavior. | ||
|
|
||
| NOTE: Session View requires an https://www.elastic.co/pricing[Enterprise subscription]. | ||
|
|
||
| [float] | ||
| [[session-view-data]] | ||
| ====== Session View displays: | ||
| * *Interactive and non-interactive processes:* Processes and services with or without a controlling terminal. | ||
| * *User information:* The Linux user that executed each session or process, and any exec user changes. | ||
| * *Process and event telemetry:* Process information included in the Linux logical event model. | ||
| * *Nested sessions:* Sessions started by processes descended from the entry session. | ||
| * *Alerts:* Alerts in the context of the processes which caused them. | ||
| * *Terminal output:* Terminal output associated with each process in the session. | ||
|
|
||
| NOTE: To view Linux session data from your Kubernetes infrastructure, you'll need to set up the <<kubernetes-dashboard,Kubernetes dashboard>>. | ||
|
|
||
| [float] | ||
| [[enable-session-view]] | ||
| === Enable Session View data | ||
| Session View uses process data collected by the {elastic-defend} integration, | ||
| but this data is not always collected by default. To confirm that Session View data is enabled: | ||
|
|
||
| . Go to *Manage* -> *Policies*, and edit one or more of your {elastic-defend} integration policies. | ||
| . Select the *Policy settings* tab, then | ||
| scroll down to the Linux event collection section near the bottom. | ||
| Turn on the *Include session data* toggle if it isn't on already. | ||
| . To enable terminal output capture, turn on the *Capture terminal output* toggle. | ||
|
|
||
| Session View can only display data that was collected by {elastic-defend} when *Include session data* was enabled. For more information about the additional | ||
| fields collected by {elastic-defend} when this setting is enabled, refer to the https://github.com/elastic/ecs/blob/main/rfcs/text/0030-linux-event-model.md[Linux event model RFC]. | ||
|
|
||
|
|
||
|
|
||
| [float] | ||
| [[open-session-view]] | ||
| === Open Session View | ||
|
|
@@ -40,13 +47,10 @@ Events and sessions that you can investigate in Session View have a rectangular | |
| * On the Alerts page, scroll down to view the Alerts table. | ||
| Look for alerts that have the **Open Session View** button in the **Actions** column: | ||
| [role="screenshot"] | ||
| image::images/session-view-action-icon-detail.png[Detail of the Open Session View button,width=75%] | ||
|
|
||
| * On the Hosts page (*Explore* -> *Hosts*), select the *Sessions* or the *Events* tab. | ||
| From either of these tabs, click the *Open Session View* button for an event or session. | ||
|
|
||
| [discrete] | ||
| [[session-view-ui]] | ||
|
|
@@ -58,7 +62,7 @@ image::images/session-view-terminal-labeled.png[Detail of Session view with labe | |
|
|
||
| 1. The *Close Session* and *Full screen* buttons. | ||
| 2. The search bar. Use it to find and highlight search terms within the current session. | ||
| The left and right arrows allow you to navigate through search results. | ||
| 3. The *display settings* button. Click to toggle Timestamps and Verbose mode. | ||
| With Verbose mode enabled, Session View shows all processes created in a session, including shell startup, | ||
| shell completion, and forks caused by built-in commands. | ||
|
|
@@ -73,8 +77,10 @@ You can also expand collapsed alerts and scripts where they appear. | |
| Collapsed processes will automatically expand when their contents match a search. | ||
| 7. The *Alerts* button. Click to show alerts caused by the parent process. Note the red line to the left | ||
| of the process that caused the alert. | ||
| 8. The *Terminal output* button. Hover to see how much output data has been captured from the session. Click to open the terminal output view, which is described in detail below. | ||
| 9. The *Refresh session* button. Click to check for any new data from the current session. | ||
|
|
||
| Session View includes additional badges not pictured above: | ||
| // | ||
| //* The *Script* button allows you to expand or collapse executed scripts: | ||
| // | ||
|
|
@@ -84,4 +90,53 @@ Session View includes an additional badge not pictured above: | |
| * The *Exec user change* badge highlights exec user changes, such as when a user escalates to root: | ||
| + | ||
| [role="screenshot"] | ||
| image::images/session-view-exec-user-change-badge.png[The Exec user change badge,width=80%,height=80%] | ||
|
|
||
| * The *Output* badge appears next to commands that generated terminal output. Click it to view that command's output in terminal output view. | ||
| + | ||
| [role="screenshot"] | ||
| image::images/session-view-output-badge.png[The Output badge,width=80%,height=80%] | ||
|
|
||
| [[session-view-output]] | ||
| [discrete] | ||
| === Terminal output view UI | ||
|
|
||
| beta::[] | ||
|
|
||
| In general, terminal output is the text that appears in interactive Linux shell sessions. This generally includes user-entered text (terminal input), which appears as output to facilitate editing commands, as well as the text output of executed programs. In certain cases such as password entry, terminal input is not captured as output. | ||
|
|
||
| From a security perspective, terminal output is important because it offers a means of exfiltrating data. For example, a command like `cat tls-private-key.pem` could output a web server's private key. Thus, terminal output view can improve your understanding of commands executed by users or adversaries, and assist with auditing and compliance. | ||
|
|
||
| To enable terminal output data capture: | ||
|
|
||
| . Go to *Manage* -> *Policies*, then select one or more of your {elastic-defend} integration policies to edit. | ||
| . On the *Policy settings* tab, scroll down to the Linux event collection section near the bottom of the page | ||
| and select the *Include session data* and *Capture terminal output* options. | ||
|
|
||
| You can configure several additional settings by clicking *Advanced settings* at the bottom of the page: | ||
|
|
||
| * `linux.advanced.tty_io.max_kilobytes_per_process`: The maximum number of kilobytes of output to record from a single process. Default: 512 KB. Process output exceeding this value will not be recorded. | ||
| * `linux.advanced.tty_io.max_kilobytes_per_event`: The maximum number of kilobytes of output to send to {es} as a single event. Default: 512 KB. Additional data is captured as a new event. | ||
| * `linux.advanced.tty_io.max_event_interval_seconds`: The maximum interval (in seconds) during which output is batched. Default: 30 seconds. Output will be sent to {es} at this interval (unless it first exceeds the `max_kilobytes_per_event` value, in which case it might be sent sooner). | ||
|
|
||
| [role="screenshot"] | ||
|
Comment on lines
+121
to
+122
There was a problem hiding this comment. Do we have any docs on writing alerting rules based on matching on terminal output? It is important to note that when writing Alerting Rules querying Elasticsearch for matching terminal output, that multi-line matching will not behave consistently because the output text may be stored in more than one output event. For example, Foo\nBar - that is,
Contributor
Author
There was a problem hiding this comment. We do not have any docs specifically about alerting rules based on terminal output, yet. But we could mention something here, or in our query rules docs, about the issue you outlined. Any thoughts on how to work around that limitation? I guess one thing would be to increase the maximum size of output events to reduce the number of unexpected newlines, but that would still be inconsistent. There was a problem hiding this comment. Yes, increasing the max event buffering size and 30 second capture to be larger (60s?) create bigger events that are more likely to work for multi-line matches but there are still chances for inconsistency. Also, IIRC we flush the output immediately if the program writing to the terminal changes - e.g. I ctrl-Z a program that is producing output and the shell then echos a prompt (the shell is another program writing to the terminal, causing a flush), then put the original program back into the foreground with If this important for some customers, the best way is to post process these output events to group all the output events of a given process, and while they're at it ideally scrub out terminal control codes for color, etc. There was a problem hiding this comment. As for rules that match against process.io.text, it is advised that all rules match with leading/trailing wildcards (*). e.g EQL equivalent: This will ensure a less strict rule which will have a much better chance of firing. Without the * wildcards, it would try to match an event with exactly the string "hello world" which in most cases will not match on anything, since an output event can have multiple lines in it. This also opens up another potential bit of confusion for the end user. E.g if their event size is quite large. e.g 512kb (due to some long cat'd file), there will be a single alert for this document no matter how little the text we are matching against. This could make it difficult to figure out which part of the proces.io.text value actually triggered the alert. We may add some UX down the road to help with this. e.g showing alerts where event.action: 'text_output' inside the tty player itself (potentially highlighting the bit of text responsible for the alert). Reducing the timeout and batch size configs could help limit the amount of text per event, thus alleviating this issue. |
||
| image::images/session-view-output-viewer.png[Terminal output view] | ||
|
|
||
benironside marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| 1. Search bar. Use to find and highlight search terms within the current session. | ||
| The left and right arrows allow you to navigate through search results. | ||
|
Comment on lines
+125
to
+126
There was a problem hiding this comment. Detail on why search may behave unexpectedly would help as well as emphasis that this is only search over terminal output held in the browser (ie it is not an Elasticsearch query - @mitodrummer pls correct me if I'm wrong). E.g.: "Use to find and highlight search terms within the current session's terminal output held by the browser. Note that because terminal output may include terminal codes for colorizing the text and moving the cursor, some searches may not find a match. For example, if one is searching output for a command like "sudo" (echoed command input) and the user mis-entered "sudo" as "sdo", and moved the cursor back to insert the missing "u", the terminal output search for "sudo" will not match. This challenge can also be present in visual editors like vi and emacs where cursor movement for editing is commonplace" There was a problem hiding this comment. You are correct @m-sample, though, for the browser side "plain text" search, I strip out all ansi chars on a a line before matching the user's search against it. This is required so that I get an accurate string index position for each match, so that when I seek to this "line" in the tty player, it's able to highlight the correct match (if there are multiple on a single line). That being said, you are correct in that if a user mistypes a command and then hits backspace to correct it, the resulting output text will not be the "corrected" version. e.g The \b is an escape code for backspace, so you can see why searching for the corrected command will not match. That being said, most of the formating ansi codes for color and text weight would be stripped out and should not interfere with most plain text searches. This is in contrast to using KQL or EQL to match on values in process.io.text as it will be the un altered version of the value that will potentially contain many escape sequences between words in the output text. Here is a link to a resource which describes these codes. https://gist.github.com/fnky/458719343aabd01cfb17a3a4f7296797 |
||
| 2. Right-side scroll bar. Use along with the bottom scroll bar to navigate output data that doesn't fit on a single screen. | ||
| 3. Playback controls and progress bar. Use to advance or rewind the session's commands and output. Click anywhere on the progress bar to jump to that part of the session. The marks on the bar represent processes that generated output. Click them or the *Prev* and *Next* buttons to skip between processes. | ||
| 4. *Fit screen*, *Zoom in*, and *Zoom out* buttons. Use to adjust the text size. | ||
|
|
||
| TIP: Use Session view's *Fullscreen* button (located next to the *Close session viewer* button) to better fit output with long lines, such as for graphical programs like `vim`. | ||
|
|
||
| [discrete] | ||
| [[terminal-output-limitations]] | ||
| ==== Terminal output limitations for search and alerting | ||
| You should understand several current limitations before building rules based on terminal output data: | ||
|
|
||
| * Terminal output that appears in the `process.io.text` field includes https://gist.github.com/fnky/458719343aabd01cfb17a3a4f7296797[ANSI codes] that represent, among other things, text color, text weight, and escape sequences. This can prevent EKS queries from matching as expected. Queries of this data will have more success matching single words than more complex strings. | ||
| * Queries of this data should include leading and trailing wildcards (for example `process where process.io.text : "*sudo*"`), since output events typically include multiple lines of output. | ||
| * The search functionality built into terminal output view is subject to similar limitations. For example, if a user accidentally entered `sdo` instead of `sudo`, then pressed backspace twice to fix the typo, the recorded output would be `sdo\b\budo`. This would appear in the terminal output view as `sudo`, but searching terminal output view for `sudo` would not result in a match. | ||
| * Output that seems like it should be continuous may be split into multiple events due to the advanced settings described above, which may prevent a query or search from matching as expected. | ||
| * Rules based on output data will identify which output event's `process.io.text` value matched the alert query, without identifying which specific part of that value matched. For example, the rule query `process.io.text: "*test*"` could match a large, multi-line log file due to a single instance of `test`, without identifying where in the file the instance occurred. | ||
Uh oh!
There was an error while loading. Please reload this page.