Merge remote-tracking branch 'upstream/master'

_static/versions.json

@@ -1,10 +1,18 @@
 [
   {
-    "name": "v5.4.4 (latest)",
-    "version": "v5.4.4",
+    "name": "v5.4.6 (latest)",
+    "version": "v5.4.6",
     "url": "https://docs.gravwell.io/",
     "preferred": true
   },
+  {
+    "version": "v5.4.5",
+    "url": "https://docs.gravwell.io/v5.4.5/"
+  },
+  {
+    "version": "v5.4.4",
+    "url": "https://docs.gravwell.io/v5.4.4/"
+  },
   {
     "version": "v5.4.3",
     "url": "https://docs.gravwell.io/v5.4.3/"

alerts/alerts.md

@@ -33,6 +33,10 @@ Every event generated by your dispatchers will be ingested into the Target Tag i
 
 In this example, we have chosen the tag `_alerts_admin_logins`.
 
+```{note}
+If the owner of the *alert* does not have permission to ingest (either via the `Webserver-Ingest-Groups` configuration in `gravwell.conf` or through [CBAC](/cbac/cbac)), no events will be ingested.
+```
+
 ### Max Events
 
 The "Max Events" configuration option is an important safeguard against accidentally sending yourself thousands of emails. Basically, when a dispatcher fires, Gravwell will only process *up to* Max Events results from the search. Suppose you have a scheduled search dispatcher which normally generates one or two results, which are emailed out via a flow consumer. If a new data source is added and the scheduled search suddenly returns thousands of results each time, you could be getting thousands of emails -- unless you've been cautious and set Max Events to a low value!
@@ -81,6 +85,10 @@ Having verified that events are being detected and logged, we can now move on to
 Dispatcher searches should use either `table`, `text`, or `raw` as their renderer. If table is used, the alert will use the columns of the table as the "fields" of each event. If text or raw are used, the alert will use the enumerated values attached to each entry as fields.
 ```
 
+```{warning}
+If you have been granted access to a scheduled search that you don't own, you will be able to select it as an alert dispatcher, but be aware that if the owner of the search decides to change it, your alert may stop working!
+```
+
 ## Setting a Validation Schema
 
 If you refer back to the queries used in our dispatchers, you'll note that we were careful to emit the same three field names in both: Hostname, remotehost, and service.

api/api.md

@@ -1,6 +1,9 @@
 # REST API
 
-This section documents part of the REST API provided by the Gravwell webserver. For complete, up-to-date information about the REST API, please refer to the Gravwell Go Client <https://pkg.go.dev/github.com/gravwell/gravwell/v3/client>
+This section contains links to documentation about direct query REST APIs, token authentication, and some scripting API interfaces.  More complete API documentation is available on the interactive [Swagger OpenAPI documentation instance](https://api.docs.gravwell.io).
+
+
+Complete open source clients are available for [Golang](https://github.com/gravwell/gravwell) with [hosted documentation](https://pkg.go.dev/github.com/gravwell/gravwell/v3/client).
 
 
 The test API located at `/api/test` can be used to verify that the webserver is up and functioning. The test API is unauthenticated and always responds with a StatusOK 200 and an empty body if the webserver is available.

cbac/cbac.md

@@ -78,7 +78,7 @@ In practice, it is less common to grant capabilities to individual users; instea
 | ScheduleRead | View a flow, script, or scheduled search and its results. |
 | ScheduleWrite | Create and edit a flow, script, or scheduled search. |
 | SOARLibs | Import an external library into a script. |
-| SOAREmail | Send an email in a script. |
+| SOAREmail | Send an email in a script or a flow. |
 | PlaybookRead | View a playbook. |
 | PlaybookWrite | Create and edit a playbook. |
 | LicenseRead | View the license. |

changelog/5.4.5.md

@@ -0,0 +1,32 @@
+# Changelog for version 5.4.5
+
+## Released 09 February 2024
+
+## Gravwell
+
+### Additions
+
+* Added the ability to share write access with a group for Query Library items.
+
+### Bug Fixes
+
+* Fixed an issue with permissions checking when de-referencing a Query Library item in a Scheduled Search.
+* Fixed an issue with dereferencing a Query Library item in a Scheduled Search when there are multiple versions installed via kits (e.g. one kit installed globally by an admin and the same kit installed/edited by a user).
+* Fixed an issue with Scheduled Searches prompting for unsaved changes when no change was made.
+* Fixed an issue with emitting entries outside of the end of a time window when using the `dump` module . 
+* Fixed an issue where installing with an expired license would not properly display the validation error.
+* Fixed an issue where the ingesters list would re-sort. 
+* Fixed an issue where the explore entry in the Query Studio details pane would sometimes not show. 
+* Fixed an issue where the backend would respond twice to explore requests on the websocket in Query Studio. 
+* Fixed an issue where the browser would hang while navigating away from Query Library. 
+
+## Ingesters
+
+### Additions
+
+* Added a new `Trim` flag to Windows File Follower.
+* Added more logging to the S3 ingester.
+
+### Bug Fixes
+
+* Fixed an issue where Windows File Follower would not respect `Regex-Delimiter` configuration.  

changelog/5.4.6.md

@@ -0,0 +1,48 @@
+# Changelog for version 5.4.6
+
+## Released 15 March 2024
+
+## Gravwell
+
+### Additions
+
+* Added a button in Query Studio to Apply a timeframe without launching a search. 
+* Added the ability for Actionable readers to access the form page with disabled inputs.
+* Added the ability to enter multi-line Secrets.
+* Added a retry for a failed attempt to pull results for an Alert.
+* Added the ability to share write access with a group for Scheduled Searches, Flows, and Alerts.
+* Any Scheduled Search (dispatcher) or Flow (consumer) that you have access to can be added to an Alert - even if you do not own the Scheduled Search or Flow. 
+
+### Bug Fixes
+
+* Fixed an issue where ingest would fail and retry with overly dramatic logs when attempting to write to a block that was actively aging out.
+* Fixed an issue where a search far into the future would consume significant CPU on the webserver in a cluster environment. 
+* Fixed an issue where a "beginning of line" regex delimiter could cause a dropped buffer while waiting for next delimiter and potentially cause data loss in File Follower.
+* Fixed an issue where a user could see cached webpages using an expired license and the browser Back button. 
+* Fixed an issue where uploading a kit could show a duplicate in a different state. 
+* Fixed an issue where a Gravwell API tokens were not respected when hitting an Alerts endpoint.
+* Fixed an issue with writing back to files when performing searches that caused stress on COW file systems. 
+* Fixed an issue with failover well feeder locking when aborting queries.
+* Fixed an incorrect type assertion that could cause a crash in the `slice` module.
+* Fixed an issue with indexer shutdown related to timeouts in network connectivity.
+* Fixed an issue with detecting and handling oversized blocks in the ingest server.
+* Fixed an issue with bounds checking in the ipfix packet parser.
+* Fixed an issue with tile metadata in Dashboards. 
+* Fixed an issue with creating a Scheduled Search from Query Studio when using a custom duration timeframe. 
+* Fixed an issue with performance on the Persistent Searches page when there are a large number of searches. 
+* Improved problems with extremely long launch delays when replication was backed by very low IOP storage.
+* Improved the way the webserver shuts down.
+* Made IP-based filters with no CIDR notation imply a /32 or /128.
+
+
+## Ingesters
+* Updated HTTP ingester to use AWS Firehose naming schemes.
+
+### Additions
+
+* For a timestamp that is zero or some very low value, the HEC ingester will now use the ingest time instead.
+
+### Bug Fixes
+
+* Fixed an issue with the HTTP ingester running out of memory upon mass reconnect or failure to ingest.
+* Improved Federator throughput when lots of indexers are present.

changelog/list.md

@@ -7,7 +7,7 @@
 maxdepth: 1
 caption: Current Release
 ---
-5.4.4 <5.4.4>
+5.4.6 <5.4.6>
 ```
 
 ## Previous Versions
@@ -18,6 +18,8 @@ maxdepth: 1
 caption: Previous Releases
 ---
 
+5.4.5 <5.4.5>
+5.4.4 <5.4.4>
 5.4.3 <5.4.3>
 5.4.2 <5.4.2>
 5.4.1 <5.4.1>

conf.py

@@ -21,7 +21,7 @@
 project = "Gravwell"
 copyright = f"Gravwell, Inc. {date.today().year}"
 author = "Gravwell, Inc."
-release = "v5.4.4"
+release = "v5.4.6"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

configuration/ageout.md

@@ -133,6 +133,14 @@ Storage-based constraints are not an instant hard limit.  Be sure to leave a lit
 
 ## Storage Available-Based Ageout Rules
 
+```{attention}
+Be careful when configuring storage available-based ageout rules. These rules apply to the available storage of the disk partition the well is configured on. If disk space is used by another data source, or even a process outside of Gravwell, the rules will still be enforced. This means that external data can cause data deletion when using this option.
+```
+
+```{attention}
+Be careful when configuring storage available-based ageout rules with multiple wells. If multiple wells are configured on the same disk partition, it's possible to have well configurations "compete" for available disk space. This is especially true if two wells have different storage available-based ageout rules. This means that wells on the same disk that use this option can cause each other to delete data.
+```
+
 Well storage constraints can also be applied based on availability of storage.  Some wells may be low priority, consuming storage only when it is available.  Using the storage reserve directives allows a well to consume as much space as it wants, so long as some percentage of available storage is maintained on the volume.  A storage availability rule essentially allows the well to act as a "second class" well, consuming disk space only when no one else is.  Specifying `Hot-Storage-Reserve=5` ensures that should the hot storage volume drop below 5% free space the well will begin migrating or deleting its oldest shards.  The reserve directives apply to the underlying volume hosting the storage location, meaning that if the volume is also hosting other wells or other arbitrary file storage, the well can pull back its storage usage as needed.
 
 An example well configuration which will use the hot location as long as there is 30% free space on the volume, and will use the cold volume as long as there is 10% free space:

configuration/environment-variables.md

@@ -6,7 +6,7 @@ The indexer, webserver, and ingester components support configuring some paramet
 GRAVWELL_INGEST_SECRET=MyIngestSecret /opt/gravwell/bin/gravwell_federator
 ```
 
-### Loading Values From Files
+## Loading Values From Files
 
 If "_FILE" is added to the end of the environment variable name, Gravwell assumes the variable contains the path to a file which in turn contains the desired data. This is particularly useful in combination with [Docker's "secrets" feature](https://docs.docker.com/engine/swarm/secrets/).
 

packages.nix → default.nix

@@ -62,7 +62,19 @@ let
       sphinx-notfound-page
     ]);
 
-in {
-  inherit pkgs;
-  chosenPackages = [ pythonBundle pkgs.gnumake pkgs.git custom-aspell ];
+in pkgs.stdenv.mkDerivation {
+  name = "gravwell-wiki";
+  src = ./.;
+
+  buildInputs = [ pythonBundle pkgs.gnumake pkgs.git custom-aspell ];
+  buildPhase = ''
+    make clean html
+  '';
+
+  installPhase = ''
+    mkdir -p $out
+    cp -r _build/html $out
+  '';
+
+  LOCALE_ARCHIVE = "${pkgs.glibcLocales}/lib/locale/locale-archive";
 }

ingesters/collectd.md

@@ -20,6 +20,23 @@ The collectd ingester uses the unified global configuration block described in t
 
 The configuration file is at `/opt/gravwell/etc/collectd.conf`. The ingester will also read configuration snippets from its [configuration overlay directory](configuration_overlays) (`/opt/gravwell/etc/collectd.conf.d`).
 
+The Gravwell collectd ingester is designed to accept the native binary collectd data formats as exported by the `network` plugin which uses the UDP transport.  A basic `network` plugin definition which ships data to a Gravwell ingester might look like so:
+
+```
+<Plugin network>
+	<Server "10.0.0.70" "25826">
+		SecurityLevel Encrypt
+		Username "user"
+		Password "secret"
+		ResolveInterval 14400
+	</Server>
+	CacheFlush 1800
+	MaxPacketSize 1452
+	ReportStats false
+	TimeToLive 128
+</Plugin>
+```
+
 ## Collector Examples
 
 ```
@@ -88,6 +105,20 @@ By default the collectd ingester reads a configuration file located at _/opt/gra
 
 Each Collector block must contain a unique name and non-overlapping Bind-Strings.  You cannot have multiple Collectors that are bound to the same interface on the same port.
 
+| Parameter            | Type         | Required | Default Value | Description  |
+|----------------------|--------------|----------|---------------|--------------|
+| Bind-String          | string       | YES      |               | An IP:Port combination specifying an address to listen on for the interface.  `0.0.0.0:25826` is a good default. |
+| Tag-Name             | string       | NO       |               | Tag to be assigned to data ingested on this listener. |
+| Source-Override      | string       | NO       |               | Override the source IP assigned to entries ingested on this listener. |
+| Security-Level       | string       | YES      |               | Collectd data transport security encoding, must match the value in the network Plugin. |
+| User                 | string       | YES      |               | Collectd data transport username, must match the value in the network Plugin. |
+| Password             | string       | YES      |               | Collectd data transport password, must match the value in the network Plugin. |
+| Encoder              | string       | NO       | json          | Output data format, default is JSON and published Gravwell kits expect JSON. |
+| Tag-Plugin-Override  | string array | NO       |               | Optional set of plugin-to-tag mappings. |
+| Preprocessor         | string array | NO       |               | Set of preprocessors to apply to entries. |
+
+
+
 #### Bind-String
 
 Bind-String controls the address and port which the Collector uses to listen for incoming collectd samples.  A valid Bind-String must contain either an IPv4 or IPv6 address and a port.  To listen on all interfaces use the "0.0.0.0" wildcard address.
@@ -162,3 +193,99 @@ Tag-Plugin-Override=memory:memstats # Map the memory plugin data to the "memstat
 Tag-Plugin-Override= df : diskdata  # Map the df plugin data to the "diskdata" tag.
 Tag-Plugin-Override = disk : diskdata  # Map the disk plugin data to the "diskdata" tag.
 ```
+
+
+## Example Collect Configuration
+
+The Collectd system is a plugin-based system instrumentation and testing framework. There is no standard Collectd deployment and every plugin can send a unique set of fields and structures.   The only hard requirement for configuring the Collectd system with Gravwell is a proper `network` Plugin definition with matching username, password, and Security-Level.  Here are basic configurations for both the Gravwell Collectd ingester and the Collectd service that will collect some reasonable metrics and send them to Gravwell:
+
+
+### /etc/collectd/collectd.conf
+
+```
+Hostname "server.example.com"
+FQDNLookup false
+
+AutoLoadPlugin true
+
+CollectInternalStats false
+
+Interval 10
+
+<Plugin network>
+	<Server "10.0.0.70" "25826">
+		SecurityLevel Encrypt
+		Username "user"
+		Password "secret"
+		ResolveInterval 14400
+	</Server>
+	CacheFlush 1800
+	MaxPacketSize 1452
+	ReportStats false
+	TimeToLive 128
+</Plugin>
+
+<Plugin cpu>
+	ReportByCpu true
+	ReportByState true
+	ValuesPercentage false
+	ReportNumCpu false
+	ReportGuestState false
+	SubtractGuestState true
+</Plugin>
+
+<Plugin df>
+	# ignore rootfs; else, the root file-system would appear twice, causing
+	# one of the updates to fail and spam the log
+	FSType rootfs
+	# ignore the usual virtual / temporary file-systems
+	FSType sysfs
+	FSType proc
+	FSType devtmpfs
+	FSType devpts
+	FSType tmpfs
+	FSType fusectl
+	FSType cgroup
+	IgnoreSelected true
+
+	ValuesPercentage true
+</Plugin>
+
+<Plugin ethstat>
+	Interface "eno1"
+	Map "rx_csum_offload_errors" "if_rx_errors" "checksum_offload"
+	Map "multicast" "if_multicast"
+	MappedOnly false
+</Plugin>
+
+<Plugin load>
+	ReportRelative true
+</Plugin>
+
+<Plugin memory>
+	ValuesAbsolute false
+	ValuesPercentage true
+</Plugin>
+```
+
+### /opt/gravwell/etc/collectd.conf
+
+```
+[Global]
+Ingester-UUID="5cb70d8d-3800-4044-bae1-308f00b6f7b5"
+Ingest-Secret = "SuperHardSecrets"
+Connection-Timeout = 0
+Insecure-Skip-TLS-Verify=false
+Cleartext-Backend-Target=10.0.0.42 #example of adding a cleartext connection
+Ingest-Cache-Path=/opt/gravwell/cache/collectd.cache
+Max-Ingest-Cache=1024 #Number of MB to store, localcache will only store 1GB before stopping.  This is a safety net
+Log-Level=INFO
+Log-File=/opt/gravwell/log/collectd.log
+
+[Collector "default"]
+	Bind-String=0.0.0.0:25826
+	Tag-Name=collectd
+	Security-Level=encrypt
+	User=user
+	Password=secret
+```