diff --git a/.github/workflows/docs-publish.yml b/.github/workflows/docs-publish.yml deleted file mode 100644 index 8e07d61f61..0000000000 --- a/.github/workflows/docs-publish.yml +++ /dev/null @@ -1,21 +0,0 @@ -name: Publish documentation to GitHub Wiki - -on: - push: - branches: - - main - paths: - - 'docs/**' - -jobs: - build: - runs-on: ubuntu-20.04 - - steps: - - uses: actions/checkout@v3 - - name: Upload Documentation to Wiki - uses: brimdata/github-wiki-publish-action@v1 - with: - path: "docs" - env: - GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} diff --git a/.github/workflows/notify-docs-update.yaml b/.github/workflows/notify-docs-update.yaml new file mode 100644 index 0000000000..4d4b797c13 --- /dev/null +++ b/.github/workflows/notify-docs-update.yaml @@ -0,0 +1,16 @@ +name: Notify docs update + +on: + push: + branches: + - main + paths: + - 'docs/**' + +jobs: + build: + runs-on: ubuntu-20.04 + steps: + - name: Send dispatch event + run: | + curl -XPOST -u "${{ secrets.PAT_USERNAME }}:${{ secrets.PAT_TOKEN }}" -H "Accept: application/vnd.github.v3+json" -H "Content-Type: application/json" https://api.github.com/repos/brimdata/zui-docs-site/dispatches --data '{"event_type":"zui-docs-update"}' diff --git a/docs/Home.md b/docs/Home.md deleted file mode 100644 index 2ea92be889..0000000000 --- a/docs/Home.md +++ /dev/null @@ -1,32 +0,0 @@ -# Welcome - -Welcome to the Brim wiki. These pages provide additional information for your -effective use of the Brim desktop application and related tools. - -> **Note:** In Brim release `v0.25.0` and newer, functionality related to -> pcap processing (including the generation of Zeek and Suricata summary logs -> from pcaps and pcap import/extraction workflows) is now handled by -> [Brimcap](https://github.com/brimdata/brimcap), which is bundled with Brim. -> A separate [Brimcap wiki](https://github.com/brimdata/brimcap/wiki) -> contains current docs related to pcap processing. - -## Support Resources - -- [[Installation]] -- [[Filesystem Paths]] -- [[Supported Platforms]] -- [[Troubleshooting]] - -## User Documentation - -- [[Migration for Version 0.29]] -- [[Importing Parquet and VNG]] - -## Cookbooks - -- [[Remote Lakes]] -- [[Joining Data]] - -## Developer Resources - -- [[Adding Migrations]] diff --git a/docs/Importing-Parquet-and-VNG.md b/docs/Importing-Parquet-and-VNG.md deleted file mode 100644 index 02aa807e28..0000000000 --- a/docs/Importing-Parquet-and-VNG.md +++ /dev/null @@ -1,61 +0,0 @@ -# Importing Parquet and VNG - -- [Summary](#summary) -- [Example](#example) -- [Contact us!](#contact-us) - -# Summary - -Brim's ability to import data of diverse file formats via simple drag & drop -depends on the backend [Zed](https://github.com/brimdata/zed) platform's -ability to "auto-detect" these formats. While several formats are covered by -the current auto-detection implementation, -[Parquet](https://parquet.apache.org/) and -[VNG](https://zed.brimdata.io/docs/next/formats/vng) are not -yet (see [zed/2517](https://github.com/brimdata/zed/issues/2517)). - -This article shows how the Zed CLI tools can be used to preprocess data of -these formats into [ZNG](https://zed.brimdata.io/docs/formats/zng/) -that can be easily imported into Brim. - -# Example - -Preprocessing can be performed with the [`zq`](https://zed.brimdata.io/docs/commands/zq/) -CLI tool. These Zed CLI tools are included with Brim and can be found in the -`zdeps` directory under the Brim [application binaries](https://github.com/brimdata/brim/wiki/Filesystem-Paths#application-binaries) -path. - -Input file format can be specified via the `-i` option. - -``` -$ zq -h -NAME - zq - process data with Zed queries - -USAGE - zq [ options ] [ zed-query ] file [ file ... ] - -OPTIONS -... - -i format of input data [auto,zng,vng,json,zeek,zjson,csv,parquet,line] (default "auto") -... -``` - -Here's an example `zq` command line on macOS to preprocess the test data -from [userdata1.parquet](https://github.com/Teradata/kylo/raw/master/samples/sample-data/parquet/userdata1.parquet) -into ZNG. - -``` -$ /Applications/Brim.app/Contents/Resources/app.asar.unpacked/zdeps/zq -i parquet userdata1.parquet > userdata1.zng -``` - -Now our `userdata1.zng` can be imported into Brim. - -![Imported Parquet](media/Imported-Parquet.png) - -# Contact us! - -If you're struggling with importing alternate file formats or just have -questions, we'd like to hear from you! Please join our -[public Slack](https://www.brimdata.io/join-slack/) -and speak up, or [open an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue). Thanks! diff --git a/docs/Installation.md b/docs/Installation.md index d83fe159b8..f97ab2e8c9 100644 --- a/docs/Installation.md +++ b/docs/Installation.md @@ -1,9 +1,14 @@ +--- +sidebar_position: 2 +sidebar_label: Installation +--- + # Installation -Brim is an interactive desktop application for Windows, macOS, and Linux. +Zui is an interactive desktop application for Windows, macOS, and Linux. To install, click the link at the -[Brim download](https://www.brimdata.io/download/) page for your platform -and double-click to launch the installer. +[Zui download](https://www.brimdata.io/download/) page for your platform +and double-click the downloaded package to launch the installer. The notes and short videos below show how to successfully install on each platform. @@ -13,69 +18,68 @@ platform. * [Linux Installation](#linux-installation) Once installed, no additional configuration is necessary. For your reference, -a separate article describes the [[filesystem paths]] where unpacked Brim +a separate article describes the [filesystem paths](./support/Filesystem-Paths.md) where unpacked Zui binaries and saved user data are stored on each platform. -Details are provided in each section below about when Brim may "auto-update" to +Details are provided in each section below about when Zui may "auto-update" to newer releases and when manual upgrades are necessary. In all cases, updating to a newer version of the app will preserve your existing -[user data](https://github.com/brimdata/brim/wiki/Filesystem-Paths#user-data). As the way user data +[user data](./support/Filesystem-Paths.md#user-data). Because the way user data is stored sometimes changes in newer releases, downgrades to older releases are _not_ currently supported. -For a walk through some typical operations in Brim to get started, watch the -[Brim Demo video](https://www.youtube.com/watch?v=InT-7WZ5Y2Y). +For a walk through some typical operations in Zui to get started, watch the +[Zui demo video](https://www.youtube.com/watch?v=InT-7WZ5Y2Y). -If you run into any problems, you may want to browse the -[wiki articles](https://github.com/brimdata/brim/wiki), perhaps starting with -[[Troubleshooting]]. If you get stuck, [join our public Slack](https://www.brimdata.io/join-slack/) +If you run into any problems, you may want to browse the +[troubleshooting docs](./support/Troubleshooting.md). If you get stuck, [join our public Slack](https://www.brimdata.io/join-slack/) and we'll be happy to help. ## Windows Installation -* Download the Brim installer via the Windows link at the [Brim download](https://www.brimdata.io/download/) page +* Download the Zui installer via the Windows link at the [Zui download](https://www.brimdata.io/download/) page * Launch the downloaded `.exe` file to begin installation * When prompted, click "I Agree" to accept the terms of the license agreement -* Brim will start automatically when install completes -* Click the Brim icon on the Desktop or Start menu to relaunch in the future +* Zui will start automatically when install completes +* Click the Zui icon on the Desktop or Start menu to relaunch in the future ![Windows Installation](media/Windows-installation.gif) -As newer Windows releases are published, Brim will automatically download them +As newer Windows releases are published, Zui will automatically download them and pop up a notification offering to restart the app to apply the update. ## macOS Installation -* Download the Brim installer via the macOS link at the [Brim download](https://www.brimdata.io/download/) page +* Download the Zui installer via the macOS link at the [Zui download](https://www.brimdata.io/download/) page * Launch the downloaded `.dmg` file to begin installation -* Drag the Brim icon into the Applications folder -* Click the Brim icon in the Applications folder to start Brim now and in the future -* You may need to click through an "Are you sure?" prompt the first time Brim is launched +* Drag the Zui icon into the Applications folder +* Click the Zui icon in the Applications folder to start Zui now and in the future +* You may need to click through an "Are you sure?" prompt the first time Zui is launched ![macOS Installation](media/macOS-installation.gif) -As newer macOS releases are published, Brim will automatically download them +As newer macOS releases are published, Zui will automatically download them and pop up a notification offering to restart the app to apply the update. ## Linux Installation -* Uninstall any currently-installed Brim release, using a tool like `yum`, +* Uninstall any currently-installed Zui release, using a tool like `yum`, `dnf`, or `apt` (saved data and settings will not be disturbed by uninstall, as these are stored under a - [user data](https://github.com/brimdata/brim/wiki/Filesystem-Paths#user-data) + [user data](./support/Filesystem-Paths.md#user-data) path that's separate from the - [application binaries](https://github.com/brimdata/brim/wiki/Filesystem-Paths#application-binaries)) -* Download either the `.deb` or `.rpm` Brim installer at the - [Brim download](https://www.brimdata.io/download/) page, as appropriate + [application binaries](support/./Filesystem-Paths.md#application-binaries)) +* Download either the `.deb` or `.rpm` Zui installer at the + [Zui download](https://www.brimdata.io/download/) page, as appropriate for your Linux distribution * Open the downloaded `.deb` or `.rpm` file in the Software Install utility and click "Install" * Enter administration credentials, if prompted -* Click the Brim icon in the applications menu to start the app now and in the future +* Click the Zui icon in the applications menu to start the app now and in the future ![Linux Installation](media/Linux-installation.gif) -As newer Linux releases are published, Brim will pop up a notification of their +As newer Linux releases are published, Zui will pop up a notification of their availability, but the Linux releases do _not_ auto-update. Follow the link in -the pop-up to the [Download](https://www.brimdata.io/download/) page to +the pop-up to the [Zui download](https://www.brimdata.io/download/) page to get the latest release and update it using the appropriate package manager for your distribution. diff --git a/docs/Joining-Data.md b/docs/Joining-Data.md deleted file mode 100644 index d7d3b1b8ed..0000000000 --- a/docs/Joining-Data.md +++ /dev/null @@ -1,65 +0,0 @@ -# Joining Data - -- [Summary](#summary) -- [About Cookbooks](#about-cookbooks) -- [Example Usage](#example-usage) -- [Contact us!](#contact-us) - -# Summary - -Similar to the well-known [SQL construct](), -the Zed language provides a [`join`](https://zed.brimdata.io/docs/language/operators/join/) operator that can be used to combine data -sources. This cookbook describes how it can be used with the Brim application. - -# About Cookbooks - -Brim cookbooks provide an opportunity to "test drive" new/experimental -features in the Brim application and related [Zed](https://github.com/brimdata/zed) -tools. They also walk through details of how Brim and Zed tools function and -therefore may inspire other creative configurations. - -All efforts are made to disclose known caveats and limitations that are -relevant to the configurations shown. However, due to the potential to -encounter bugs in evolving functionality, it is recommended that you initially -follow cookbooks in a non-production, lab-style setting. As such features -become more complete and stable, cookbooks may be retired and replaced with -regular [User Documentation](https://github.com/brimdata/brim/wiki#user-documentation). - -Please report any bugs or usability issues you find when working with cookbooks -by [opening an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue) -or reaching out on the [Brim public Slack](https://www.brimdata.io/join-slack/). -We'd also love to hear your success stories and variations, so please don't be -shy! - -# Example Usage - -The same operations shown in the Zed language [`join` tutorial](https://zed.brimdata.io/docs/tutorials/join/) -can be performed in the Brim app. However, the more sophisticated logic of a -`join` operation is often best expressed in a multi-line Zed script A multi-line -editor is planned for a future Brim release -(see [brim/2264](https://github.com/brimdata/brim/issues/2264) for details). -In the meantime, multi-line Zed scripts can be pasted and run in the current -single-line search bar. You may find it helpful to edit your multi-line Zed -scripts in an external editor and paste them into Brim to run them. - -By default, the source data used for Zed a query in Brim will be whichever pool -you've currently selected in the **Pools** list in the left panel. Therefore, -the example shown in the [Self Joins](https://zed.brimdata.io/docs/tutorials/join/#self-joins) -section of the tutorial can be run as shown if both the example data sources -have been imported into the same pool (e.g., `cat fruit.ndjson people.ndjson > all.ndjson` -at the shell before dragging `all.ndjson` into Brim). - -![Self Join Example](media/Self-Join.png) - -If the data sources are in separate pools (such as if we'd dragged -`fruit.ndjson` and `people.ndjson` into Brim separately), specify each `pool` -source in `from`. Below is the [Inner Join](https://zed.brimdata.io/docs/tutorials/join/#inner-join) -example from the tutorial run in Brim this way. - -![Inner Join Example](media/Inner-Join.png) - -# Contact us! - -If you have questions or feedback about this cookbook, we'd like to hear from -you! Please join our [public Slack](https://www.brimdata.io/join-slack/) or -[open an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue). Thanks! diff --git a/docs/Migration-for-Version-0.29.md b/docs/Migration-for-Version-0.29.md deleted file mode 100644 index 5ae32331b9..0000000000 --- a/docs/Migration-for-Version-0.29.md +++ /dev/null @@ -1,41 +0,0 @@ -# Pool Migration for Version 0.29 - -1. Install the latest version of Brim. -2. Download the [migration script](https://raw.githubusercontent.com/brimdata/brim/main/scripts/util/migration-v0.29.sh). -3. Run it. `sh migration-v0.29` -4. Open the app and see your pools. - -The migration script can be executed while Brim v0.29 is already open. Click **View > Reload** from the Brim pull-down menu to see your migrated pools after the script completes. -> **Windows Users**: See the section below titled "[How do I run this on Windows?](/#how-do-i-run-this-on-windows)". - -## Will it delete anything? - -No. - -## Do I absolutely need to migrate? - -No. Only run the steps above if you care to keep your old pool data. If you prefer a fresh start, you can delete your old pools by following the steps [below](#how-do-i-delete-the-old-pools). - -## What does the script do? - -The Zed lake service that powers Brim made breaking changes to its data storage format in this release. Your pools are currently stored in a `data/lake/` subdirectory under the Brim [user data](https://github.com/brimdata/brim/wiki/Filesystem-Paths#user-data) path. The new version expects the pools to be in a `lake/` subdirectory (i.e., one level higher) under the user data path. The script will read in the old data and write it to the new location. The script does not attempt to delete anything on your system. If you want to delete the old pools after this script finishes, see [below](#how-do-i-delete-the-old-pools). - -## How do I delete the old pools? - -You can remove the old pools by deleting the directory shown in the table below for your OS. - -|**OS Platform**|**Location** | -|---------------|------------------------------------------------------| -| **Windows** | `%APPDATA%\Brim\data\lake` | -| **macOS** | `$HOME/Library/Application Support/Brim/data/lake` | -| **Linux** | `$HOME/.config/Brim/data/lake` | - -## How do I run this on Windows? - -The script must be run in a Windows `sh` variant like `BusyBox`, `Cygwin`, or `MSYS2`. If you do not have any of these already setup, we recommend downloading [busybox.exe](https://frippery.org/files/busybox/busybox.exe) as it seems to be the easiest. Once downloaded, start the shell. See the [Busy Box documentation](https://frippery.org/busybox/) for more detail. - -``` -C:\path\to\busybox.exe sh -l -``` - -This will drop you into a `sh` environment where you can execute [step 3](#pool-migration-for-version-029). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..a0eb9604a8 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,66 @@ +--- +sidebar_position: 1 +sidebar_label: Introduction +--- + +# The Zui Desktop Application + +Zui (formerly called Brim) is a graphical user interface for exploring data in +[Zed lakes](https://zed.brimdata.io/docs/commands/zed). While these docs are +specific to the Zui app experience, you may want to browse the separate +[docs for the Zed project](https://zed.brimdata.io/docs) on topics such as: + +* the [Zed language](https://zed.brimdata.io/docs/language) that's used in Zui for queries and analytics, +* the [command-line tools](https://zed.brimdata.io/docs/commands) that work alongside Zui, and +* the [Zed data formats](https://zed.brimdata.io/docs/formats). + +## Getting Started + +Trying out Zui is easy: just [install](./Installation.md) the package for +your operating system. Once you launch the app, click **Import Data** +on the welcome screen and drag & drop a file of one of the formats +shown. Simple operations such as filtering and "count by" are available in a +menu that appears when you right-click on values in your data. When you're ready +to perform more sophisticated queries and aggregations, consult the +[Zed language](https://zed.brimdata.io/docs/language) docs. + +## Behind the Name Change + +When Brim began as a company, we intended to build a security-focused desktop +application powered by our backend called [Zed](https://zed.brimdata.io/). The +company's main product was the app, so we called it the Brim app. + +As we developed Zed, we started to realize we had something big on our hands. +Zed's data model, language, query engine, and storage formats provided +revolutionary new ways to work with all kinds of data. In April 2021, we +decided to pivot, making Zed the company's flagship technology. Brim Security +became [Brim Data](https://www.brimdata.io/). + +This change made us question the purpose of the app. We were now leading with +the [Zed lake](https://zed.brimdata.io/docs/commands/zed), so we decided +the app would support the lake. Users would be able to explore a lake's data +and manage its configuration with the app. It would be a GUI for Zed. + +At this point, having the app named after the company was confusing. We needed +a name that clearly tied the app to Zed. After many fun naming sessions, we +played off the word "GUI" and landed on "Zui": Zed user interface. + +:::tip pcap processing +Functionality related to pcap processing (including the generation of Zeek and +Suricata summary logs from pcaps and pcap import/extraction workflows) is +handled by [Brimcap](https://github.com/brimdata/brimcap), which is bundled +with Zui. A separate [Brimcap wiki](https://github.com/brimdata/brimcap/wiki) +contains current docs related to pcap processing. +::: + +## Zui Insiders + +While GA versions of Zui will be released periodically, it's evolving fast. +If you're interested in keeping up with the very latest enhancements and bug +fixes, you can download the +[Zui - Insiders](https://github.com/brimdata/zui-insiders) app. It can run +side-by-side with regular Zui, stores data in a different location, and runs +the Zed lake service on a different port. It receives automatic updates every +weekday built from the tip of our `main` Zui branch. This is our take on a beta +program. If you find bugs or think of improvements, please chime in on the +[#zui-insiders](https://brimdata.slack.com/archives/C03MW6XT7HC) Slack channel. diff --git a/docs/_Footer.md b/docs/_Footer.md deleted file mode 100644 index 30e61da842..0000000000 --- a/docs/_Footer.md +++ /dev/null @@ -1 +0,0 @@ -Join our [public Slack](https://www.brimdata.io/join-slack/) workspace for announcements, Q&A, and to trade tips! diff --git a/docs/_Sidebar.md b/docs/_Sidebar.md deleted file mode 100644 index fc7e13e156..0000000000 --- a/docs/_Sidebar.md +++ /dev/null @@ -1,20 +0,0 @@ -**Support Resources** - -- [[Installation]] -- [[Filesystem Paths]] -- [[Supported Platforms]] -- [[Troubleshooting]] - -**User Documentation** - -- [[Migration for Version 0.29]] -- [[Importing Parquet and VNG]] - -**Cookbooks** - -- [[Remote Lakes]] -- [[Joining Data]] - -**Developer Resources** - -- [[Adding Migrations]] diff --git a/docs/advanced/README.md b/docs/advanced/README.md new file mode 100644 index 0000000000..065ca41a40 --- /dev/null +++ b/docs/advanced/README.md @@ -0,0 +1,11 @@ +--- +sidebar_position: 4 +--- + +# Advanced Guides + +While Zui enables many powerful workflows by simply clicking in the app, some +advanced functionality may require performing operations at the shell. Topics +described here include: + +* Working with [remote Zed lakes](Remote-Zed-Lakes.md) diff --git a/docs/Remote-Lakes.md b/docs/advanced/Remote-Zed-Lakes.md similarity index 51% rename from docs/Remote-Lakes.md rename to docs/advanced/Remote-Zed-Lakes.md index 7bbdd61159..b0ca59940a 100644 --- a/docs/Remote-Lakes.md +++ b/docs/advanced/Remote-Zed-Lakes.md @@ -1,41 +1,15 @@ -# Remote Lakes +--- +sidebar_position: 1 +--- -- [Summary](#summary) -- [About Cookbooks](#about-cookbooks) -- [Limitations](#limitations) -- [Background: Brim & Zed Lakes](#background-brim--zed-lakes) -- [Starting a Remote Zed Lake](#starting-a-remote-zed-lake) -- [Accessing Our Remote Lake](#accessing-our-remote-lake) -- [Contact us!](#contact-us) +# Remote Zed Lakes -# Summary - -By default, the Brim application connects to a [Zed lake](https://zed.brimdata.io/docs/commands/zed/#1-the-lake-model) -on the system on which it is launched. However, Brim is capable of accessing -data stored in a Zed lake on a remote system. This cookbook describes the +By default, the Zui application connects to a [Zed lake](https://zed.brimdata.io/docs/commands/zed/#1-the-lake-model) +on the system on which it is launched. However, Zui is capable of accessing +data stored in a Zed lake on a remote system. This article describes the available options and current limitations. -# About Cookbooks - -Brim cookbooks provide an opportunity to "test drive" new/experimental -features in the Brim application and related Zed tools. They also walk through -details of how Brim and Zed tools function and therefore may inspire other -creative configurations. - -All efforts are made to disclose known caveats and limitations that are -relevant to the configurations shown. However, due to the potential to -encounter bugs in evolving functionality, it is recommended that you initially -follow cookbooks in a non-production, lab-style setting. As such features -become more complete and stable, cookbooks may be retired and replaced with -regular [User Documentation](https://github.com/brimdata/brim/wiki#user-documentation). - -Please report any bugs or usability issues you find when working with cookbooks -by [opening an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue) -or reaching out on the [Brim public Slack](https://www.brimdata.io/join-slack/). -We'd also love to hear your success stories and variations, so please don't be -shy! - -# Limitations +## Limitations Before diving into the specifics of what's possible, here's an overview of some rough edges you may encounter as you work through the configurations @@ -50,10 +24,10 @@ described in this article. authentication, individual logins, or roles/permissions. Care should be taken to avoid the accidental exposure or loss of centrally-stored data. -# Background: Brim & Zed Lakes +## Background: Zui & Zed Lakes -Since it's presented as an icon that can be double-clicked to launch it on -your desktop, it's easy to think of Brim as a simple standalone application. +Since it's presented as an icon that can be double-clicked to launch on +your desktop, it's easy to think of Zui as a simple standalone application. However, the overall app experience is powered by a distributed "backend" architecture that includes multiple components. @@ -61,25 +35,24 @@ One essential component is the Zed lake which is accessed via a [`zed serve`](https://zed.brimdata.io/docs/commands/zed/#213-serve) process that manages the storage and querying of imported data. Operations on the Zed lake are invoked via a [REST API](https://zed.brimdata.io/docs/lake/api/) -that's utilized by a "client", such as the Brim app. The +that's utilized by a "client", such as the Zui app. The [`zed`](https://zed.brimdata.io/docs/commands/zed/) command is also available as a command line client that can perform many of the same operations as the -Brim app, and therefore may be useful in scripting and automation. +Zui app, and therefore may be useful in scripting and automation. -![Brim zed and the Zed lake](media/Brim-zed-cli-zed-serve.png) +![Zui zed and the Zed lake](media/Zui-zed-cli-zed-serve.png) -The default location where Zed stores imported data is a `lake` subdirectory -under the Brim [user data](https://github.com/brimdata/brim/wiki/Filesystem-Paths#user-data) +By default, the Zed backend that runs behind Zui stores imported data in a +`lake` subdirectory under the [user data](../support//Filesystem-Paths.md#user-data) path. -If you examine the process table while Brim is running, you can observe the +If you examine the process table while Zui is running, you can observe the command line that was used to start the backend Zed process. For example, here is the process on a Mac laptop being operated by username "phil". ``` -macOS# ps auxww | grep zed -phil 5289 0.0 0.0 34122868 868 s002 S+ 4:37PM 0:00.00 grep zed -phil 5254 0.0 0.1 34867788 15340 ?? S 4:37PM 0:00.03 /Applications/Brim.app/Contents/Resources/app.asar.unpacked/zdeps/zed serve -l localhost:9867 -lake /Users/phil/Library/Application Support/Brim/lake -log.level=info -log.filemode=rotate -log.path /Users/phil/Library/Application Support/Brim/logs/zlake.log -brimfd=3 +macOS# ps auxww | grep zed | grep -v grep +phil 2409 0.0 0.1 34880932 14784 ?? S 6:38PM 0:00.02 /Applications/Zui.app/Contents/Resources/app.asar.unpacked/zdeps/zed serve -l localhost:9867 -lake /Users/phil/Library/Application Support/Zui/lake -log.level=info -log.filemode=rotate -log.path /Users/phil/Library/Application Support/Zui/logs/zlake.log -brimfd=3 ``` Some useful information revealed in this command line: @@ -91,79 +64,81 @@ Some useful information revealed in this command line: 1. The `-lake` option points to the `lake` user data subdirectory for macOS. 1. The `-brimfd=3` is an option unique to when `zed serve` is launched by - Brim. This helps ensure that if Brim is killed abruptly, the `zed` process will + Zui. This helps ensure that if Zui is killed abruptly, the `zed` process will also be terminated (see [zed/1184](https://github.com/brimdata/zed/pull/1184) for details). -1. We can see the full path to the `zed` binary that's packaged with Brim. This - binary and other dependencies that are typically launched by Brim can be found - in the `zdeps` directory under Brim's [application binaries](https://github.com/brimdata/brim/wiki/Filesystem-Paths#application-binaries) +1. We can see the full path to the `zed` binary that's packaged with Zui. This + binary and other dependencies that are typically launched by Zui can be found + in the `zdeps` directory under Zui's [application binaries](../support/Filesystem-Paths.md#application-binaries) path. -Now that we know Brim is simply connecting to Zed locally, next we'll vary +Now that we know Zui is simply connecting to Zed locally, next we'll vary this approach to instead start a remote `zed serve` and connect to it to access the data stored there. -# Starting a Remote Zed Lake +## Starting a Remote Zed Lake -For our example remote host, we'll use a Linux Ubuntu 18.04 VM running in -Amazon AWS. Because Brim interacts with `zed serve` over a REST API that -is still evolving, care should be taken to ensure the Brim version being +For our example remote host, we'll use a Linux Ubuntu 22.04 VM running in +Amazon AWS. Because Zui interacts with `zed serve` over a REST API that +is still evolving, care should be taken to ensure the Zui version being installed on the remote side matches the version being run locally. In this -cookbook we'll use Brim v0.29.0, which includes Zed v1.0.0. +article we'll use Zui v1.0.0, which includes Zed v1.6.0. Even though our VM on AWS has no graphical interface, we'll install the full -Brim package because it includes the compatible Zed binaries as well as a +Zui package because it includes the compatible Zed binaries as well as a bundled [Brimcap](https://github.com/brimdata/brimcap) that will prove useful if we want to import packet capture data. ``` -ubuntu# wget --quiet https://github.com/brimdata/brim/releases/download/v0.29.0/Brim-0.29.0.deb +ubuntu# wget --quiet https://github.com/brimdata/zui/releases/download/v1.0.0/Zui-1.0.0.deb ubuntu# sudo apt update -ubuntu# sudo apt install -y ./Brim-0.29.0.deb +ubuntu# sudo apt install -y ./Zui-1.0.0.deb ``` The following additional steps are also currently necessary to work around -issue [brim/1701](https://github.com/brimdata/brim/issues/1701). +issue [zui/1701](https://github.com/brimdata/brim/issues/1701). ``` -ubuntu# sudo find /opt/Brim/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \; -ubuntu# /opt/Brim/resources/app.asar.unpacked/zdeps/suricata/suricataupdater +ubuntu# sudo find /opt/Zui/resources/app.asar.unpacked/zdeps/suricata -exec chmod go+w {} \; +ubuntu# /opt/Zui/resources/app.asar.unpacked/zdeps/suricata/suricataupdater ``` -> **Variation:** Rather than the full Brim package, we could instead -> [download a Zed package](https://www.brimdata.io/download/). The Zed -> package includes the binaries that could be used to construct command lines -> similar to those shown below. However, as the Zed tools are part of a general -> data system, they do not include Brimcap. This means such a configuration -> would either lack the ability to import packet data or would need to be -> augmented with a separate Brimcap install. The [Custom Brimcap Config](https://github.com/brimdata/brimcap/wiki/Custom-Brimcap-Config) -> article in the Brimcap wiki provides relevant guidance for this. +:::tip Variation +Rather than the full Zui package, we could instead +[download a Zed package](https://www.brimdata.io/download/). The Zed +package includes the binaries that could be used to construct command lines +similar to those shown below. However, as the Zed tools are part of a general +data system, they do not include Brimcap. This means such a configuration +would either lack the ability to import packet data or would need to be +augmented with a separate Brimcap install. The [custom Brimcap config](https://github.com/brimdata/brimcap/wiki/Custom-Brimcap-Config) +article in the Brimcap wiki provides relevant guidance for this. +::: Since there's no desktop environment on this VM, there's no "app" interface to see. Therefore we'll start `zed serve` manually from the -[application binaries](https://github.com/brimdata/brim/wiki/Filesystem-Paths#application-binaries-v0250) path for the Linux +[application binaries](../support/Filesystem-Paths.md#application-binaries) path for the Linux platform as follows: ``` -ubuntu# mkdir -p ~/.config/Brim/lake ~/.config/Brim/data/brimcap-root ~/.config/Brim/logs -ubuntu# /opt/Brim/resources/app.asar.unpacked/zdeps/zed serve \ +ubuntu# mkdir -p ~/.config/Zui/lake ~/.config/Zui/data/brimcap-root ~/.config/Zui/logs +ubuntu# /opt/Zui/resources/app.asar.unpacked/zdeps/zed serve \ -l :9867 \ - -lake $HOME/.config/Brim/lake \ + -lake $HOME/.config/Zui/lake \ -log.level=info \ -log.filemode=rotate \ - -log.path $HOME/.config/Brim/logs/zlake.log + -log.path $HOME/.config/Zui/logs/zlake.log ``` Building on what we learned earlier, we've made two adjustments here compared -to the command line Brim would have invoked: +to the command line that Zui would have invoked: 1. `localhost` was dropped from the `-l` option. By providing only the port `:9867` specification, `zed serve` is now prepared to accept remote connections as well. 2. The `-brimfd=3` was dropped, since we're controlling the start/stop of Zed - rather than the Brim app. + rather than the Zui app. At this point `zed serve` is ready to accept remote connections. However, the network between clients and our remote Zed lake needs to permit this @@ -175,20 +150,18 @@ IP address. ![Security Group](media/Security-Group.png) -# Accessing Our Remote Lake +## Accessing Our Remote Lake -Now that our remote Zed lake is running, we'll access it from the Brim app +Now that our remote Zed lake is running, we'll access it from the Zui app that's running on our Mac laptop. The option to initiate a remote connection is available by clicking the -pull-down above the pools list in the left panel and selecting **Add -Lake**. +"**+**" near the top of the left sidebar and selecting **Add Lake**. ![Add Lake](media/Add-Lake.png) A window will pop up into which we can specify a name for our remote connection -along with its hostname or IP address. An optional port specification may also -be included if it's listening on a port other than the default `9867`. +along an HTTP URL including its hostname or IP address and listening port. ![Add Lake window](media/Add-Lake-window.png) @@ -196,7 +169,7 @@ Existing pools in our remote Zed lake will appear in the left panel just as we'r accustomed to seeing when working with local data. We can now enter Zed queries and perform normal workflows. -You can import data (such as the logs shown below) directly from your Brim app to +You can import data (such as the logs shown below) directly from your Zui app to the remote Zed lake in the same manner as you've been doing locally. ![Importing logs to remote Zed lake](media/Remote-Zed-Lake-Import.gif) @@ -218,19 +191,19 @@ shown above. See [brimcap/105](https://github.com/brimdata/brimcap/issues/105) f ``` Another_User_macOS# wget --quiet https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz Another_User_macOS# gunzip wrccdc.2018-03-23.010014000000000.pcap.gz -Another_User_macOS# export PATH="/Applications/Brim.app/Contents/Resources/app.asar.unpacked/zdeps:$PATH" -Another_User_macOS# brimcap index -root "$HOME/Library/Application Support/Brim/data/brimcap-root" -r wrccdc.2018-03-23.010014000000000.pcap +Another_User_macOS# export PATH="/Applications/Zui.app/Contents/Resources/app.asar.unpacked/zdeps:$PATH" +Another_User_macOS# brimcap index -root "$HOME/Library/Application Support/Zui/data/brimcap-root" -r wrccdc.2018-03-23.010014000000000.pcap ``` A connection to a remote lake can be removed by selecting the **Get Info** option in the pull-down and clicking **Logout**. This only removes the config -in your Brim app that references the remote lake. It does not shutdown the +in your Zui app that references the remote lake. It does not shutdown the remote `zed serve` nor does it delete any data stored there. ![Get Lake info](media/Lake-Get-Info.png) -# Contact us! +## Contact us! -If you have questions or feedback about this cookbook, we'd like to hear from +If you have questions or feedback about this article, we'd like to hear from you! Please join our [public Slack](https://www.brimdata.io/join-slack/) or -[open an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue). Thanks! +[open an issue](../support/Troubleshooting.md#opening-an-issue). Thanks! diff --git a/docs/media/Add-Lake-window.png b/docs/advanced/media/Add-Lake-window.png similarity index 100% rename from docs/media/Add-Lake-window.png rename to docs/advanced/media/Add-Lake-window.png diff --git a/docs/advanced/media/Add-Lake.png b/docs/advanced/media/Add-Lake.png new file mode 100644 index 0000000000..c0b3b04d82 Binary files /dev/null and b/docs/advanced/media/Add-Lake.png differ diff --git a/docs/advanced/media/Imported-Parquet.png b/docs/advanced/media/Imported-Parquet.png new file mode 100644 index 0000000000..ca36ed8100 Binary files /dev/null and b/docs/advanced/media/Imported-Parquet.png differ diff --git a/docs/advanced/media/Lake-Get-Info.png b/docs/advanced/media/Lake-Get-Info.png new file mode 100644 index 0000000000..bd493833d2 Binary files /dev/null and b/docs/advanced/media/Lake-Get-Info.png differ diff --git a/docs/advanced/media/Remote-Flow-Wireshark.gif b/docs/advanced/media/Remote-Flow-Wireshark.gif new file mode 100644 index 0000000000..4f8bd4ca17 Binary files /dev/null and b/docs/advanced/media/Remote-Flow-Wireshark.gif differ diff --git a/docs/advanced/media/Remote-Zed-Lake-Import.gif b/docs/advanced/media/Remote-Zed-Lake-Import.gif new file mode 100644 index 0000000000..2e9c15320e Binary files /dev/null and b/docs/advanced/media/Remote-Zed-Lake-Import.gif differ diff --git a/docs/media/Security-Group.png b/docs/advanced/media/Security-Group.png similarity index 100% rename from docs/media/Security-Group.png rename to docs/advanced/media/Security-Group.png diff --git a/docs/advanced/media/Zui-zed-cli-zed-serve.png b/docs/advanced/media/Zui-zed-cli-zed-serve.png new file mode 100644 index 0000000000..870626799c Binary files /dev/null and b/docs/advanced/media/Zui-zed-cli-zed-serve.png differ diff --git a/docs/Adding-Migrations.md b/docs/developer/Adding-Migrations.md similarity index 89% rename from docs/Adding-Migrations.md rename to docs/developer/Adding-Migrations.md index c8f399f6b5..31ec947e05 100644 --- a/docs/Adding-Migrations.md +++ b/docs/developer/Adding-Migrations.md @@ -1,6 +1,12 @@ +--- +sidebar_position: 1 +--- + +# Adding Migrations + Because we persist state on a user's computer, we must migrate that state each time we release a version that changes the schema expected by the app. If any of the reducers in `src/js/state` are changed, we need to write a migration. When the app starts, it will automatically run all the pending migrations. -### Creating a Migration +## Creating a Migration Let's say we've added a property called `nickname` to each lake connection in the app. We'll use the migration generator tool to create the necessary files. From the root directory, run `bin/gen migration `. For example: @@ -28,7 +34,7 @@ export default function addNicknameToConnection(state) { As of this writing, the schema of the persisted state is called `SessionState` and is defined in: `src/js/electron/tron/formatSessionState.js` -### Testing Your Migration +## Testing Your Migration The migrations only ever run when we release a new version. To test your migration, you'll need a sample of the session state from the most recent release tag. Let's say the last release was `v0.13.1`. We can easily get a snapshot of the state at that version following these steps. @@ -45,13 +51,13 @@ Now get the app into a state that you want to test against. In this example, we When ready, go to the App Menu and click `Developer => Save Session for Testing Migrations`. -Save Session Menu Item +![Save Session Menu Item](media/save-session-menu-item.png) This will save the session state and inform you where the file is. -Session Saved Popup +![Session Saved Popup](media/session-saved-popup.png) -The name of the json file is just today's date, so it might be helpful to rename it to the version that you're targeting like `v0.13.1.json` +The name of the JSON file is just today's date, so it might be helpful to rename it to the version that you're targeting like `v0.13.1.json` ```bash mv src/js/test/states/202007091803.json src/js/test/states/v0.13.1.json @@ -83,4 +89,4 @@ test("202007101024_addNicknameToConnection", () => { }) ``` -Once you've got that test passing, you're home free. The app will see the new migration and run it automatically next time someone upgrades. +Once you've got that test passing, you're home free. The app will see the new migration and run it automatically the next time someone upgrades. diff --git a/docs/developer/README.md b/docs/developer/README.md new file mode 100644 index 0000000000..ec52550b5b --- /dev/null +++ b/docs/developer/README.md @@ -0,0 +1,10 @@ +--- +sidebar_position: 5 +--- + +# Developer Resources + +Developers that wish to extend or customize the Zui application code should +read the [contributing](https://github.com/brimdata/brim/blob/main/CONTRIBUTING.md) +guide and also understand how to [add state migrations](Adding-Migrations.md) +between app versions. diff --git a/docs/media/save-session-menu-item.png b/docs/developer/media/save-session-menu-item.png similarity index 100% rename from docs/media/save-session-menu-item.png rename to docs/developer/media/save-session-menu-item.png diff --git a/docs/media/session-saved-popup.png b/docs/developer/media/session-saved-popup.png similarity index 100% rename from docs/media/session-saved-popup.png rename to docs/developer/media/session-saved-popup.png diff --git a/docs/media/Add-Lake.png b/docs/media/Add-Lake.png deleted file mode 100644 index 87ced069c9..0000000000 Binary files a/docs/media/Add-Lake.png and /dev/null differ diff --git a/docs/media/Brim-Task-Manager.png b/docs/media/Brim-Task-Manager.png deleted file mode 100644 index 913aa7f7fe..0000000000 Binary files a/docs/media/Brim-Task-Manager.png and /dev/null differ diff --git a/docs/media/Brim-zed-cli-zed-serve.png b/docs/media/Brim-zed-cli-zed-serve.png deleted file mode 100644 index 1bc2f5d7f4..0000000000 Binary files a/docs/media/Brim-zed-cli-zed-serve.png and /dev/null differ diff --git a/docs/media/Brim-zed-serve-9867.png b/docs/media/Brim-zed-serve-9867.png deleted file mode 100644 index b127a98593..0000000000 Binary files a/docs/media/Brim-zed-serve-9867.png and /dev/null differ diff --git a/docs/media/Conn-Err-could-not-be-reached.png b/docs/media/Conn-Err-could-not-be-reached.png deleted file mode 100644 index a08cdeeaa6..0000000000 Binary files a/docs/media/Conn-Err-could-not-be-reached.png and /dev/null differ diff --git a/docs/media/Imported-Parquet.png b/docs/media/Imported-Parquet.png deleted file mode 100644 index 45c4a2fb35..0000000000 Binary files a/docs/media/Imported-Parquet.png and /dev/null differ diff --git a/docs/media/Inner-Join.png b/docs/media/Inner-Join.png deleted file mode 100644 index b4fb1f24c1..0000000000 Binary files a/docs/media/Inner-Join.png and /dev/null differ diff --git a/docs/media/Lake-Get-Info.png b/docs/media/Lake-Get-Info.png deleted file mode 100644 index 0ee81afd4b..0000000000 Binary files a/docs/media/Lake-Get-Info.png and /dev/null differ diff --git a/docs/media/Open_Dev_Tools.png b/docs/media/Open_Dev_Tools.png deleted file mode 100644 index 2e908252c6..0000000000 Binary files a/docs/media/Open_Dev_Tools.png and /dev/null differ diff --git a/docs/media/Prep_Dev_Tools.png b/docs/media/Prep_Dev_Tools.png deleted file mode 100644 index 98fff3d160..0000000000 Binary files a/docs/media/Prep_Dev_Tools.png and /dev/null differ diff --git a/docs/media/Remote-Flow-Wireshark.gif b/docs/media/Remote-Flow-Wireshark.gif deleted file mode 100644 index 090f188f4d..0000000000 Binary files a/docs/media/Remote-Flow-Wireshark.gif and /dev/null differ diff --git a/docs/media/Remote-Zed-Lake-Import.gif b/docs/media/Remote-Zed-Lake-Import.gif deleted file mode 100644 index d5296024fb..0000000000 Binary files a/docs/media/Remote-Zed-Lake-Import.gif and /dev/null differ diff --git a/docs/media/Self-Join.png b/docs/media/Self-Join.png deleted file mode 100644 index 7e65934303..0000000000 Binary files a/docs/media/Self-Join.png and /dev/null differ diff --git a/docs/media/zed-listening.png b/docs/media/zed-listening.png deleted file mode 100644 index 9aecf993f7..0000000000 Binary files a/docs/media/zed-listening.png and /dev/null differ diff --git a/docs/support/Brim-Zui-Transition.md b/docs/support/Brim-Zui-Transition.md new file mode 100644 index 0000000000..3542c97a79 --- /dev/null +++ b/docs/support/Brim-Zui-Transition.md @@ -0,0 +1,69 @@ +--- +sidebar_position: 4 +--- + +# Brim/Zui Transition + +Due to the large number of changes, the move from Brim v0.31.0 to Zui v1.0.0 +requires some manual steps to fully upgrade. + +Normally when the app is updated, the older version of the app is replaced +with the new version. But for this specific transition both the +[application binaries](https://zui.brimdata.io/docs/support/Filesystem-Paths#application-binaries) +and [user data](https://zui.brimdata.io/docs/support/Filesystem-Paths#user-data) +for the older Brim are left in place separate from the new Zui. With this in +mind, the sections below provide guidance on how to proceed from this state to +complete the transition. + +## Zed Lake Migration + +When Zui v1.0.0 first launches, the **Pools** list will be empty, but the data +in your pools from when you were running Brim v0.31.0 are still present in +the Brim [user data](https://zui.brimdata.io/docs/support/Filesystem-Paths#user-data) +directory. If you don't need your old pools anymore, you can leave the old +Brim user data where it is or backup/delete it as you choose. Zui will +not access it. However, if you want to migrate your pools to Zui, tools are +available in the [Zed Lake Migration](https://github.com/brimdata/zed-lake-migration) +repository. See its [README](https://github.com/brimdata/zed-lake-migration/blob/main/README.md) +for details. + +## Platform-Specific Guidance + +When Zui v1.0.0 is released, you will receive a notification of its +availability in Brim v0.31.0 as usual, but you'll need to complete some +additional manual steps at this point. See the section below for details +regarding the transition on your operating system. + +Those interested in the technical details of the testing that led to this +guidance can read more +[here](https://github.com/brimdata/brim/issues/2459#issuecomment-1442316859). + +### Windows + +When presented with the update notice and **Restart** is clicked, Brim will +close and Zui installation will proceed. Once Zui is installed, _both_ the Brim +and Zui apps will be present in the installed **Programs** list. It is now +recommended to complete the [Zed lake migration](#zed-lake-migration) if you wish +and then manually uninstall Brim. From this point forward each new release +of Zui can be installed via auto-update and will replace any prior Zui entry +in the **Programs** list. + +### macOS + +When presented with the update notice and **Restart** is clicked, the app will +not actually restart. Instead, manually quit the app and +download/[install](https://zui.brimdata.io/docs/Installation#macos-installation) +the Zui v1.0.0 package for macOS. Once Zui is installed, _both_ the Brim and +Zui apps will be present in the installed **Applications** list. It is now +recommended to complete the [Zed lake migration](#zed-lake-migration) if you wish +and then manually uninstall Brim. From this point forward each new release +of Zui can be installed via auto-update and will replace any prior Zui entry +in the **Applications** list. + +### Linux + +Zui is similar to Brim in that only manual upgrades are possible on Linux. After +you manually install Zui v1.0.0, _both_ the Brim and Zui apps will be present +in the list of installed applications. It is now recommended to complete the +[Zed lake migration](#zed-lake-migration) if you wish and then manually +uninstall Brim. diff --git a/docs/Filesystem-Paths.md b/docs/support/Filesystem-Paths.md similarity index 74% rename from docs/Filesystem-Paths.md rename to docs/support/Filesystem-Paths.md index 14b50456a0..099376c338 100644 --- a/docs/Filesystem-Paths.md +++ b/docs/support/Filesystem-Paths.md @@ -1,28 +1,28 @@ -# Filesystem Paths +--- +sidebar_position: 2 +--- -- [Application Binaries](#application-binaries) -- [User Data](#user-data) -- [Temporary Storage](#temporary-storage) +# Filesystem Paths -Occasionally when using Brim or troubleshooting problems with it, you may need +Occasionally when using Zui or troubleshooting problems with it, you may need to know the location on the filesystem where the application binaries and/or user data is stored. Below you'll find these details for each supported platform. -# Application Binaries +## Application Binaries -When installed, the binaries and other supporting files that make up the Brim +When installed, the binaries and other supporting files that make up the Zui application are unpacked to the following path on each platform: |**OS Platform**|**Location** | |---------------|---------------------------------------------| -| **Windows** | `%USERPROFILE%\AppData\Local\Programs\Brim` | -| **macOS** | `/Applications/Brim.app` | -| **Linux** | `/opt/Brim` | +| **Windows** | `%USERPROFILE%\AppData\Local\Programs\Zui` | +| **macOS** | `/Applications/Zui.app` | +| **Linux** | `/opt/Zui` | This filesystem location should be treated as read-only and is typically only modified when a newer version of the app is being installed. -Since Brim is most often launched directly from the GUI environment in your OS, +Since Zui is most often launched directly from the GUI environment in your OS, you'll typically not need to access the application binaries directly. One exception is if you wish to make use of the command line binaries that are bundled with the app, such as [`zq`](https://zed.brimdata.io/docs/commands/zq/), @@ -32,38 +32,38 @@ found in the `zdeps` subdirectory as follows: |**OS Platform**|**Location**| |---------------|------------| -| **Windows** | `%USERPROFILE%\AppData\Local\Programs\Brim\resources\app.asar.unpacked\zdeps` | -| **macOS** | `/Applications/Brim.app/Contents/Resources/app.asar.unpacked/zdeps` | -| **Linux** | `/opt/Brim/resources/app.asar.unpacked/zdeps` | +| **Windows** | `%USERPROFILE%\AppData\Local\Programs\Zui\resources\app.asar.unpacked\zdeps` | +| **macOS** | `/Applications/Zui.app/Contents/Resources/app.asar.unpacked/zdeps` | +| **Linux** | `/opt/Zui/resources/app.asar.unpacked/zdeps` | While these binaries are also available for download as standalone release packages for [Zed](https://github.com/brimdata/zed/releases) or [Brimcap](https://github.com/brimdata/brimcap/releases), if you're using CLI commands to interact with a pool in the Zed lake that runs behind -Brim, you should typically use these bundled `zdeps` binaries since they've -been tested with the same version of Brim and can be considered API-compatible +Zui, you should typically use these bundled `zdeps` binaries since they've +been tested with the same version of Zui and can be considered API-compatible with that version of the app. -# User Data +## User Data -Saved data from your work in Brim is stored in a read/write filesystem path +Saved data from your work in Zui is stored in a read/write filesystem path that's separate from the application binaries. The top-level user data path on each supported platform is: |**OS Platform**|**Location** | |---------------|------------------------------------------------------| -| **Windows** | `%APPDATA%\Brim` | -| **macOS** | `$HOME/Library/Application Support/Brim` | -| **Linux** | `$HOME/.config/Brim` | +| **Windows** | `%APPDATA%\Zui` | +| **macOS** | `$HOME/Library/Application Support/Zui` | +| **Linux** | `$HOME/.config/Zui` | Particular categories of saved data are held under this path. Specific categories of interest include: - * `data` (subdirectory) - Storage used by tools bundled with Brim, such as + * `data` (subdirectory) - Storage used by tools bundled with Zui, such as [Brimcap](https://github.com/brimdata/brimcap). - * `lake` (subdirectory) - Storage for the local Zed lake that holds data imported into Brim. + * `lake` (subdirectory) - Storage for the local Zed lake that holds data imported into Zui. - * `logs` (subdirectory) - Logs for the running Brim app and the Zed backend + * `logs` (subdirectory) - Logs for the running Zui app and the Zed backend that it launches for data storage and query. * `appState.json` (file) - Persistent app settings such as changes you've @@ -71,15 +71,15 @@ categories of interest include: in the **History** panel, and so forth. Generally you should not need to directly access the saved user data, though -it may prove necessary during [[Troubleshooting]] or if you use +it may prove necessary during [troubleshooting](Troubleshooting.md) or if you use [`zed`](https://zed.brimdata.io/docs/commands/zed/) or [`brimcap`](https://github.com/brimdata/brimcap) CLI tools to perform scripted operations from outside the app against data that's stored -behind Brim. +behind Zui. -# Temporary Storage +## Temporary Storage -Brim and the bundled Zed/Brimcap tools often make use of temporary storage. +Zui and the bundled Zed/Brimcap tools often make use of temporary storage. Some examples: * The **Zed** backend may use temporary storage to "spill to disk" when @@ -96,11 +96,11 @@ Zeek and Suricata) before they're loaded into a pool in a Zed lake. For this, Brimcap uses a directory with a name that starts with `brimcap-`. The directory is automatically deleted when `brimcap analyze` finishes. -* In **Brim** when the **Packets** button is clicked to extract a flow from +* In **Zui** when the **Packets** button is clicked to extract a flow from a larger pcap, the extracted packets are placed in a temporary file that's -opened in Wireshark. For this, Brim uses a file with a name of the format +opened in Wireshark. For this, Zui uses a file with a name of the format `packets-[date+timestamp].pcap` (e.g., `packets-2020-03-09T17_27_53.399273Z.pcap`). -This file is _not_ deleted automatically, as Brim does not make any assumptions +This file is _not_ deleted automatically, as Zui does not make any assumptions about what you may continue to do with the pcap file outside the app. Standard conventions are used for creating these directories/files in the @@ -132,11 +132,13 @@ and is typically of a format like `/var/folders/yn/jbkxxkpd4vg142pc3_bd_krc0000g used, if it were set. However, in default test environments we've found this variable is typically not set, and therefore the `/tmp` path is used. -> **Note:** Detailed explanations for the logic used for locating the +:::tip +Detailed explanations for the logic used for locating the temporary storage location can be found in the -[Electron docs](https://www.electronjs.org/docs/latest/api/app#appgetpathname) (for Brim) +[Electron docs](https://www.electronjs.org/docs/latest/api/app#appgetpathname) (for Zui) and the [Go docs](https://pkg.go.dev/os#TempDir) (for Zed/Brimcap). +::: -If you'd like to change the base temporary storage path that's used by Brim -and the Zed/Brimcap tools, just ensure that Brim is launched after having set +If you'd like to change the base temporary storage path that's used by Zui +and the Zed/Brimcap tools, just ensure that Zui is launched after having set the environment variable mentioned above that's appropriate for your platform. diff --git a/docs/support/README.md b/docs/support/README.md new file mode 100644 index 0000000000..6d833c05b8 --- /dev/null +++ b/docs/support/README.md @@ -0,0 +1,11 @@ +--- +sidebar_position: 3 +--- + +# Support Resources + +The Zui support resources explain: + +* the [platforms](Supported-Platforms.md) on which the app is supported, +* the [filesystem paths](Filesystem-Paths.md) where application binaries and user data are stored on different platforms, and +* [troubleshooting](Troubleshooting.md) tips in case you experience problems. diff --git a/docs/Supported-Platforms.md b/docs/support/Supported-Platforms.md similarity index 57% rename from docs/Supported-Platforms.md rename to docs/support/Supported-Platforms.md index 40f9842bbe..a5c31bdba6 100644 --- a/docs/Supported-Platforms.md +++ b/docs/support/Supported-Platforms.md @@ -1,23 +1,12 @@ +--- +sidebar_position: 1 +--- + # Supported Platforms -- [Summary](#summary) -- [Per-Platform Details](#per-platform-details) - - [Windows](#windows) - - [macOS](#macos) - - [Software](#software) - - [Hardware](#hardware) - - [Linux](#linux) -- [Establishing Platform Supportability](#establishing-platform-supportability) - - [Development Tools](#development-tools) - - [Automated Testing](#automated-testing) - - [Smoke Testing](#smoke-testing) - - [Non-Recommended Platforms](#non-recommended-platforms) - -# Summary - -[Downloadable packages](https://www.brimdata.io/download/) for Brim are +[Downloadable packages](https://www.brimdata.io/download/) for Zui are available that run on Windows, macOS, and Linux. Our current platform -recommendations on which to run Brim: +recommendations on which to run Zui: - Windows - Windows 10 or newer @@ -34,46 +23,46 @@ recommendations on which to run Brim: The sections below provide details regarding these guidelines and how they are established. -# Per-Platform Details +## Per-Platform Details -## Windows +### Windows -Brim's [test automation](#automated-testing) runs on Windows Server +Zui's [test automation](#automated-testing) runs on Windows Server 2019 and therefore this is the platform on which we are best able to ensure quality and prevent regressions. [Microsoft support statements](https://docs.microsoft.com/en-us/windows/release-information/status-windows-10-20h2) simultaneously target both Windows 10 and Windows Server, so our quality expectations between Windows 10 and Windows Server 2019 are equivalent. Several -Brim developers also run Windows 10 desktops and regularly perform ad hoc +Zui developers also run Windows 10 desktops and regularly perform ad hoc testing with it to reproduce reported issues. -Basic [smoke testing](#smoke-testing) has also validated that Brim appears to -work on Windows 8.1 as well. Similar testing has also confirmed that Brim does -_not_ work on Windows 7. Therefore we do _not_ recommend attempting to run Brim +Basic [smoke testing](#smoke-testing) has also validated that Zui appears to +work on Windows 8.1 as well. Similar testing has also confirmed that Zui does +_not_ work on Windows 7. Therefore we do _not_ recommend attempting to run Zui on releases older than Windows 8.1. -## macOS +### macOS -### Software +#### Software -Brim's [test automation](#automated-testing) runs on Monterey 12 and +Zui's [test automation](#automated-testing) runs on Monterey 12 and therefore this is the macOS version on which we are best able to ensure quality -and prevent regressions. Several Brim developers also run macOS Ventura 13.1 +and prevent regressions. Several Zui developers also run macOS Ventura 13.1 and regularly perform ad hoc testing with it to reproduce reported issues. -Basic [smoke testing](#smoke-testing) has also validated that Brim appears to +Basic [smoke testing](#smoke-testing) has also validated that Zui appears to work on macOS Big Sur 11.7.2 as well. We do _not_ recommend attempting to run Brim on macOS releases older than macOS Big Sur 11.7.2. -### Hardware +#### Hardware -The build procedure for Brim's macOS releases creates binaries intended to +The build procedure for Zui's macOS releases creates binaries intended to run on the Intel-based Mac hardware that make up many of the Macs in -use today. Brim releases are not yet available that are built specifically for +use today. Zui releases are not yet available that are built specifically for the recently-released [M1-based hardware](https://en.wikipedia.org/wiki/Apple_M1). However, Apple's [Rosetta 2](https://support.apple.com/en-us/HT211861) makes it possible to run Intel-targeted binaries on M1-based Macs, and -[smoke testing](#smoke-testing) has indicated that current Brim releases can +[smoke testing](#smoke-testing) has indicated that current Zui releases can leverage this to run on M1-based Macs. Due to issues of @@ -82,27 +71,27 @@ our automated tests for macOS run today on Apple hardware that is Intel-based but not M1-based. Therefore if you have your choice of Mac hardware platform, Intel-based is more strongly recommended. However, as we know the M1-based Macs will become more popular in the future, please -[open issues](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue) -for problems you experience with Brim on M1-based Macs as you would any other. +[open issues](./Troubleshooting.md#opening-an-issue) +for problems you experience with Zui on M1-based Macs as you would any other. If we should begin to accumulate bugs that are specific to M1-based hardware, this will help guide the prioritization of our goal to deliver M1-specific -builds ([brim/1266](https://github.com/brimdata/brim/issues/1266)). +builds ([zui/1266](https://github.com/brimdata/brim/issues/1266)). -## Linux +### Linux -Brim's [test automation](#automated-testing) runs on Ubuntu 20.04 (`.deb` +Zui's [test automation](#automated-testing) runs on Ubuntu 20.04 (`.deb` package) and therefore this is the Linux distribution on which we are best able to ensure quality and prevent regressions. -An `.rpm` package is also provided that is periodically [smoke tested](https://github.com/brimdata/brim/wiki/Supported-Platforms#smoke-testing) +An `.rpm` package is also provided that is periodically [smoke tested](#smoke-testing) on Rocky 8.3. The [Rocky Linux site](https://rockylinux.org/about) explains that Rocky seeks to maintain the [RHEL](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux)-compatibility mission formerly championed by [CentOS](https://www.centos.org/). Therefore the Rocky versions -listed in this article provide the basis of the Brim supportability expectation for the +listed in this article provide the basis of the Zui supportability expectation for the equivalent RHEL version. Basic [smoke testing](#smoke-testing) has also validated the _oldest_ -release on which Brim appeared to work for each common distribution, as +release on which Zui appeared to work for each common distribution, as follows: - Ubuntu 20.04 @@ -111,10 +100,10 @@ follows: - Rocky 8.3 - CentOS 8 1905 (deprecated) -Therefore we do _not_ recommend attempting to run Brim on distributions older +Therefore we do _not_ recommend attempting to run Zui on distributions older than those listed above. -# Establishing Platform Supportability +## Establishing Platform Supportability The determination of the specific versions of platforms for which we can set expectations of quality are based on multiple factors. These include: @@ -124,27 +113,26 @@ expectations of quality are based on multiple factors. These include: and/or [smoke testing](#smoke-testing) The following sections provide more detail, along with guidance if you feel -strongly about trying to get Brim running on a [non-recommended platform](#non-recommended-platforms). +strongly about trying to make Zui run on a [non-recommended platform](#non-recommended-platforms). -## Development Tools +### Development Tools -There are two primary development tools on which Brim depends: +There are two primary development tools on which Zui depends: [Electron](https://www.electronjs.org/docs/latest/development/README) and [Go](https://github.com/golang/go/wiki/MinimumRequirements). Their support -statements cite older platform releases than the Brim-specific ones cited above. -Therefore the recommendations in the [Summary](#summary) section above should -be followed. +statements cite older platform releases than the Zui-specific ones cited above. +Therefore the recommendations at the top of this page should be followed. -## Automated Testing +### Automated Testing -The most extensive testing of Brim is provided via automation that is run on +The most extensive testing of Zui is provided via automation that is run on [GitHub Actions](https://github.com/features/actions). Specific platform versions of -hosted [runners](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) are referenced in the automation for Brim's -[Continuous Integration tests](https://github.com/brimdata/brim/blob/main/.github/workflows/ci.yml) +hosted [runners](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) are referenced in the automation for Zui's +[continuous integration tests](https://github.com/brimdata/brim/blob/main/.github/workflows/ci.yml) and [release workflow](https://github.com/brimdata/brim/blob/main/.github/workflows/release.yml). -## Smoke Testing +### Smoke Testing Due to the large number of permutations of platforms (especially as relates to Linux distributions), it is currently infeasible to provide the same exhaustive @@ -153,30 +141,30 @@ manual "smoke testing" has been performed on a wider number of platforms to confirm basic functionality. Such a smoke test consists of the following: - Install the base platform while accepting the defaults on any offered install-time config options -- Install the Brim app using the standard package install procedure for the platform -- Import a test pcap into Brim and confirm the bundled Zeek and Suricata both produce records from it +- Install the Zui app using the standard package install procedure for the platform +- Import a test pcap into Zui and confirm the bundled Zeek and Suricata both produce records from it - Click the **Packets** button to extract a slice from the pcap into Wireshark This exercise was most recently performed in August, 2022 in preparation for -the transition from Brim to GA Zui release v1.0.0. For more details on -previously-performed smoke testing exercises, review [brim/1263](https://github.com/brimdata/brim/issues/1263), -[brim/2481](https://github.com/brimdata/brim/pull/2481), and [brim/2482](https://github.com/brimdata/brim/issues/2482). +Zui release v1.0.0. For more details on +previously-performed smoke testing exercises, review [zui/1263](https://github.com/brimdata/brim/issues/1263), +[zui/2481](https://github.com/brimdata/brim/pull/2481), and [zui/2482](https://github.com/brimdata/brim/issues/2482). -## Non-Recommended Platforms +### Non-Recommended Platforms While we welcome you to -[open an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue) -about any problem you experience with Brim regardless of platform version, -the priority of the core Brim development team is to maintain stability and +[open an issue](./Troubleshooting.md#opening-an-issue) +about any problem you experience with Zui regardless of platform version, +the priority of the core Zui development team is to maintain stability and introduce new features on the modern platforms that are most widely used. Therefore we may be limited in our ability to provide timely fixes (or any fix at all) for platform versions older/different from those recommended above. -We also understand that certain users may have a strong motivation to make Brim -work on other platforms. As Brim is open source, community members are welcomed -to perform their own research regarding such platforms and submit -[cookbooks](https://github.com/brimdata/brim/wiki#cookbooks) that may be of +We also understand that certain users may have a strong motivation to make Zui +work on other platforms. As Zui is open source, community members are welcomed +to perform their own research regarding such platforms and contribute +[advanced guides](../advanced/README.md) that may be of use to other users seeking to run on the same platform. Before embarking on such an effort, we recommend -[opening an issue](https://github.com/brimdata/brim/wiki/Troubleshooting#opening-an-issue) +[opening an issue](./Troubleshooting.md#opening-an-issue) to check if we're aware of any existing efforts regarding that platform. diff --git a/docs/Troubleshooting.md b/docs/support/Troubleshooting.md similarity index 57% rename from docs/Troubleshooting.md rename to docs/support/Troubleshooting.md index 002f4af2d4..0cce9f192a 100644 --- a/docs/Troubleshooting.md +++ b/docs/support/Troubleshooting.md @@ -1,46 +1,44 @@ +--- +sidebar_position: 2 +--- + # Troubleshooting -If you're having a problem with Brim, please browse the following sections +If you're having a problem with Zui, please browse the following sections before you [open an issue](#opening-an-issue). -- [Common Problems](#common-problems) - * [I've clicked to open a packet capture in Brim, but it failed to open](#ive-clicked-to-open-a-packet-capture-in-brim-but-it-failed-to-open) - * [I've clicked in Brim to extract a flow from my pcap into Wireshark, but the flow looks different than when I isolate it in the original pcap file in Wireshark](#ive-clicked-in-brim-to-extract-a-flow-from-my-pcap-into-wireshark-but-the-flow-looks-different-than-when-i-isolate-it-in-the-original-pcap-file-in-wireshark) - * [Brim seems unable to restart normally, such as after a bad crash](#brim-seems-unable-to-restart-normally-such-as-after-a-bad-crash) - * [Brim shows "Connection Error: The service at localhost:9867 could not be reached"](#brim-shows-connection-error-the-service-at-localhost9867-could-not-be-reached) - * [My antivirus software has flagged Brim as potentially malicious](#my-antivirus-software-has-flagged-brim-as-potentially-malicious) -- [Gathering Info](#gathering-info) - * [Sensitive Information (important!)](#sensitive-information-important) - * [Screenshots/Videos](#screenshotsvideos) - * [Developer Tools](#developer-tools) - * [Logs](#logs) - * [Sample Packet Captures](#sample-packet-captures) - * [Large Files](#large-files) -- [Opening an Issue](#opening-an-issue) - -# Common Problems - -## I've clicked to open a packet capture in Brim, but it failed to open - -Functionality related to pcaps in Brim releases `v0.25.0` and newer is +## Common Problems + +The following are some of the more commonly-reported Zui problems. You may +also want to review the [current open issues](https://github.com/brimdata/brim/issues). + +* [I've clicked to open a packet capture in Zui, but it failed to open](#ive-clicked-to-open-a-packet-capture-in-zui-but-it-failed-to-open) +* [I've clicked in Zui to extract a flow from my pcap into Wireshark, but the flow looks different than when I isolate it in the original pcap file in Wireshark](#ive-clicked-in-zui-to-extract-a-flow-from-my-pcap-into-wireshark-but-the-flow-looks-different-than-when-i-isolate-it-in-the-original-pcap-file-in-wireshark) +* [Zui seems unable to restart normally, such as after a bad crash](#zui-seems-unable-to-restart-normally-such-as-after-a-bad-crash) +* [Zui shows an error: "The service could not be reached"](#zui-shows-an-error-the-service-could-not-be-reached) +* [My antivirus software has flagged Zui as potentially malicious](#my-antivirus-software-has-flagged-zui-as-potentially-malicious) + +#### I've clicked to open a packet capture in Zui, but it failed to open + +Functionality related to pcaps is provided via [Brimcap](https://github.com/brimdata/brimcap#usage-with-brim-desktop-app). This question is covered [here](https://github.com/brimdata/brimcap/wiki/Troubleshooting#ive-clicked-to-open-a-packet-capture-in-brim-but-it-failed-to-open) in the Brimcap wiki. -## I've clicked in Brim to extract a flow from my pcap into Wireshark, but the flow looks different than when I isolate it in the original pcap file in Wireshark +#### I've clicked in Zui to extract a flow from my pcap into Wireshark, but the flow looks different than when I isolate it in the original pcap file in Wireshark -Functionality related to pcaps in Brim releases `v0.25.0` and newer is +Functionality related to pcaps is provided via [Brimcap](https://github.com/brimdata/brimcap#usage-with-brim-desktop-app). This question is covered [here](https://github.com/brimdata/brimcap/wiki/Troubleshooting#ive-clicked-in-brim-to-extract-a-flow-from-my-pcap-into-wireshark-but-the-flow-looks-different-than-when-i-isolate-it-in-the-original-pcap-file-in-wireshark) in the Brimcap wiki. -## Brim seems unable to restart normally, such as after a bad crash +#### Zui seems unable to restart normally, such as after a bad crash -Though we attempt to fix bad bugs in Brim soon after they're identified, +Though we attempt to fix bad bugs in Zui soon after they're identified, occasionally you may encounter a new bug that crashes the app in a way that -leaves it in a bad state. In these situations Brim will seem "stuck" each time -it starts, either at at a blank white screen (such as in previously-fixed issue -[brim/1099](https://github.com/brimdata/brim/issues/1099)) or showing an error +leaves it in a bad state. In these situations Zui will seem "stuck" each time +it starts, either at a blank white screen (such as in previously-fixed issue +[zui/1099](https://github.com/brimdata/brim/issues/1099)) or showing an error dump similar to the one below: ![Example crash from issue #652](media/Crash-652.png) @@ -49,24 +47,24 @@ If you experience such a crash, please gather any error dump text and [open an issue](#opening-an-issue) with as much detail as possible regarding the steps you followed that led up to the crash. -Then to clear the condition, click **Window > Reset State** from the Brim -pull-down menu. This will clear some cached data from your previous use of Brim +Then to clear the condition, click **Window > Reset State** from the Zui +pull-down menu. This will clear some cached data from your previous use of Zui (e.g., the contents of the **History** panel), but the data in your pools will remain intact. -Before resuming normal work in Brim, this would be a good opportunity to +Before resuming normal work in Zui, this would be a good opportunity to retrace your steps and confirm that you've captured the reproduction steps accurately in your [issue](#opening-an-issue), since you won't lose any history -if you repro/crash Brim and clear it one more time. Your assistance in helping +if you repro/crash Zui and clear it one more time. Your assistance in helping us squash these types of bad bugs is much appreciated. Thank you! -## Brim shows "Connection Error: The service at localhost:9867 could not be reached" +#### Zui shows an error: "The service could not be reached" -The following screenshot shows how this error is presented in Brim. +The following screenshot shows how this error is presented in Zui. ![Service could not be reached](media/Conn-Err-could-not-be-reached.png) -These failures are quite rare. Brim is regularly tested for successful +These failures are quite rare. Zui is regularly tested for successful installation in "out-of-the-box" configurations of all supported platforms, so it's likely that this type of failure may be due to some unique configuration on your system that's preventing normal operation. Therefore, even if the tips @@ -75,116 +73,117 @@ share the details with us so we can provide guidance to ensure other users avoid the problem in the future. There have been a couple reports from Windows users who experienced these -symptoms after an upgrade to a newer Brim release. In each case the users +symptoms after an upgrade to a newer Zui release. In each case the users ultimately resolved the problem via system reboot. We've unfortunately been unable to reproduce this problem or study a system "live" that's in this state to determine the root cause or why the reboot fixed it. An issue -[brim/1490](https://github.com/brimdata/brim/issues/1490) is being kept open to +[zui/1490](https://github.com/brimdata/brim/issues/1490) is being kept open to track this specific case. Even though a reboot might be an effective fix for you also, your assistance would be much appreciated in still running through the troubleshooting steps below and gathering the debug information. If you ultimately find a reboot fixes the symptom for you, please add your logs -and details to [brim/1490](https://github.com/brimdata/brim/issues/1490). +and details to [zui/1490](https://github.com/brimdata/brim/issues/1490). In all other cases, please [open a new issue](#opening-an-issue). -To begin troubleshooting this, it helps to understand the "backend" of Brim. +To begin troubleshooting this, it helps to understand the "backend" of Zui. One essential component is a [Zed lake](https://zed.brimdata.io/docs/commands/zed/#1-the-lake-model), a server-style process that manages the storage and querying of imported data. Operations in the pools of a Zed lake are invoked via a [REST API](https://zed.brimdata.io/docs/lake/api/) that's -utilized by a "client", such as the Brim app. When run on your desktop, -`zed serve` is launched at startup by Brim and then listens for such API +utilized by a "client", such as the Zui app. When run on your desktop, +`zed serve` is launched at startup by Zui and then listens for such API connections on local TCP port `9867`. -![Brim connecting to zed serve on 9867](media/Brim-zed-serve-9867.png) +![Zui connecting to zed serve on 9867](media/Zui-zed-serve-9867.png) -Therefore, this particular error message indicates that for some reason Brim +Therefore, this particular error message indicates that for some reason Zui was not able to successfully communicate with a `zed serve` process via `localhost:9867`. In brief, the root cause is likely one of the following: 1. [A `zed serve` crash](#a-zed-serve-crash) 2. [A failure to launch `zed serve`](#a-failure-to-launch-zed-serve) -3. [No local communications between Brim and `zed serve`](#no-local-communications-between-brim-and-zed-serve) +3. [No local communications between Zui and `zed serve`](#no-local-communications-between-zui-and-zed-serve) In all cases, you should plan to [open an issue](#opening-an-issue) with -Brim/Zed logs attached, as these may include crash/failure details. However, +Zui/Zed logs attached, as these may include crash/failure details. However, for situations other than simple crashes, the problem may be related to antivirus tools that could block the install/upgrade of software and/or firewalls that block communications between unfamiliar processes. You may therefore need to work with your IT department to examine logs from such -utilities and determine if settings can be adjusted to allow Brim to operate. +utilities and determine if settings can be adjusted to allow Zui to operate. The sections below describe additional debug you can perform to narrow down among the above root causes. Please include your findings from these steps in -any issue you open, along with Brim/Zed logs and details on how you +any issue you open, along with Zui/Zed logs and details on how you reproduced the error. -### A `zed serve` crash +##### A `zed serve` crash -If you experience the error during normal operations in Brim (e.g., after you've +If you experience the error during normal operations in Zui (e.g., after you've attempted to execute a query in the middle of an otherwise healthy session), it's possible that the `zed serve` process suffered a fatal crash. If you can reliably reproduce the crash by executing a particular query and/or with a particular data source, please include this detail in your issue along with attached logs. -### A failure to launch `zed serve` +##### A failure to launch `zed serve` -If you experience the error when launching Brim (perhaps after an auto-update +If you experience the error when launching Zui (perhaps after an auto-update to a new release), it's possible that system utilities prevented the `zed serve` process from fully installing or starting. To narrow down if this is occurring, it may help to compare the current running state to that of a healthy installation. -Once you've launched Brim and have observed the error message, check the +Once you've launched Zui and have observed the error message, check the running processes on your system and compare to the outputs shown below as appropriate for your platform. -**Windows (Task Manager)** +###### Windows (Task Manager) -![Healthy Brim in Task Manager](media/Brim-Task-Manager.png) +![Healthy Zui in Task Manager](media/Zui-Task-Manager.png) -**Linux [`pstree`](https://man7.org/linux/man-pages/man1/pstree.1.html)** +###### Linux [`pstree`](https://man7.org/linux/man-pages/man1/pstree.1.html) ``` $ pstree ... - | | | |-gnome-session-b-+-gnome-shell-+-brim-+-brim---brim-+-brim - | | | | | | | `-4*[{brim}] - | | | | | | |-brim---brim - | | | | | | |-brim---5*[{brim}] - | | | | | | |-2*[brim---12*[{brim}]] - | | | | | | |-zed---9*[{zed}] - | | | | | | `-31*[{brim}] + | | |-zui-+-zed---10*[{zed}] + | | | |-zui---zui-+-zui + | | | | `-4*[{zui}] + | | | |-zui---zui + | | | |-zui---5*[{zui}] + | | | |-zui---11*[{zui}] + | | | |-zui---10*[{zui}] + | | | `-31*[{zui}] ``` -**macOS [`pstree`](https://man7.org/linux/man-pages/man1/pstree.1.html)** +###### macOS [`pstree`](https://man7.org/linux/man-pages/man1/pstree.1.html) ``` $ pstree ... - |-+= 67726 phil /Applications/Brim.app/Contents/MacOS/Brim - | |--- 67727 phil /Applications/Brim.app/Contents/Frameworks/Brim Helper (GPU).app/Contents/MacOS/Brim Helper (GPU) --type=gpu-process --field-trial-handle=1718379636,158416190604165762 - | |--- 67729 phil /Applications/Brim.app/Contents/Resources/app.asar.unpacked/zdeps/zed serve -l localhost:9867 -R /Users/phil/Library/Application Support/Brim/data/lake -log.level - | |--- 67730 phil /Applications/Brim.app/Contents/Frameworks/Brim Helper.app/Contents/MacOS/Brim Helper --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-hand - | |--- 67731 phil /Applications/Brim.app/Contents/Frameworks/Brim Helper (Renderer).app/Contents/MacOS/Brim Helper (Renderer) --type=renderer --field-trial-handle=1718379636,15841619060 - | \--- 67732 phil /Applications/Brim.app/Contents/Frameworks/Brim Helper (Renderer).app/Contents/MacOS/Brim Helper (Renderer) --type=renderer --field-trial-handle=1718379636,15841619060 + |-+= 00899 phil /Applications/Zui.app/Contents/MacOS/Zui + | |--- 00903 phil /Applications/Zui.app/Contents/Frameworks/Zui Helper (GPU).app/Contents/MacOS/Zui Helper (GPU) --type=gpu-process --field-trial-handle=1718379636,3073461579650436005,9081109346180051893,131072 --disable-features=SpareRendererForSitePerProcess --user-data-dir=/Users/phil/Library/Application Support/Zui --gpu-preferences=UAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAAAABgAAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJgEAAAAAAAAmAQAAAAAAACIAQAAMAAAAIABAAAAAAAAiAEAAAAAAACQAQAAAAAAAJgBAAAAAAAAoAEAAAAAAACoAQAAAAAAALABAAAAAAAAuAEAAAAAAADAAQAAAAAAAMgBAAAAAAAA0AEAAAAAAADYAQAAAAAAAOABAAAAAAAA6AEAAAAAAADwAQAAAAAAAPgBAAAAAAAAAAIAAAAAAAAIAgAAAAAAABACAAAAAAAAGAIAAAAAAAAgAgAAAAAAACgCAAAAAAAAMAIAAAAAAAA4AgAAAAAAAEACAAAAAAAASAIAAAAAAABQAgAAAAAAAFgCAAAAAAAAYAIAAAAAAABoAgAAAAAAAHACAAAAAAAAeAIAAAAAAACAAgAAAAAAAIgCAAAAAAAAkAIAAAAAAACYAgAAAAAAAKACAAAAAAAAqAIAAAAAAACwAgAAAAAAALgCAAAAAAAAwAIAAAAAAADIAgAAAAAAANACAAAAAAAA2AIAAAAAAADgAgAAAAAAAOgCAAAAAAAA8AIAAAAAAAD4AgAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAGAAAAEAAAAAAAAAAAAAAABwAAABAAAAAAAAAAAAAAAAgAAAAQAAAAAAAAAAAAAAAKAAAAEAAAAAAAAAAAAAAACwAAABAAAAAAAAAAAAAAAA0AAAAQAAAAAAAAAAAAAAAOAAAAEAAAAAAAAAABAAAAAAAAABAAAAAAAAAAAQAAAAYAAAAQAAAAAAAAAAEAAAAHAAAAEAAAAAAAAAABAAAACAAAABAAAAAAAAAAAQAAAAoAAAAQAAAAAAAAAAEAAAALAAAAEAAAAAAAAAABAAAADQAAABAAAAAAAAAAAQAAAA4AAAAQAAAAAAAAAAQAAAAAAAAAEAAAAAAAAAAEAAAABgAAABAAAAAAAAAABAAAAAcAAAAQAAAAAAAAAAQAAAAIAAAAEAAAAAAAAAAEAAAACgAAABAAAAAAAAAABAAAAAsAAAAQAAAAAAAAAAQAAAANAAAAEAAAAAAAAAAEAAAADgAAABAAAAAAAAAABwAAAAAAAAAQAAAAAAAAAAcAAAAGAAAAEAAAAAAAAAAHAAAABwAAABAAAAAAAAAABwAAAAgAAAAQAAAAAAAAAAcAAAAKAAAAEAAAAAAAAAAHAAAACwAAABAAAAAAAAAABwAAAA0AAAAQAAAAAAAAAAcAAAAOAAAAEAAAAAAAAAAIAAAAAAAAABAAAAAAAAAACAAAAAYAAAAQAAAAAAAAAAgAAAAHAAAAEAAAAAAAAAAIAAAACAAAABAAAAAAAAAACAAAAAoAAAAQAAAAAAAAAAgAAAALAAAAEAAAAAAAAAAIAAAADQAAABAAAAAAAAAACAAAAA4AAAAQAAAAAAAAAAoAAAAAAAAAEAAAAAAAAAAKAAAABgAAABAAAAAAAAAACgAAAAcAAAAQAAAAAAAAAAoAAAAIAAAAEAAAAAAAAAAKAAAACgAAABAAAAAAAAAACgAAAAsAAAAQAAAAAAAAAAoAAAANAAAAEAAAAAAAAAAKAAAADgAAAAgAAAAAAAAACAAAAAAAAAA= --use-gl=swiftshader-webgl --shared-files --seatbelt-client=41 + | |--- 00905 phil /Applications/Zui.app/Contents/Resources/app.asar.unpacked/zdeps/zed serve -l localhost:9867 -lake /Users/phil/Library/Application Support/Zui/lake -log.level=info -log.filemode=rotate -log.path /Users/phil/Library/Application Support/Zui/logs/zlake.log -brimfd=3 + | |--- 00907 phil /Applications/Zui.app/Contents/Frameworks/Zui Helper.app/Contents/MacOS/Zui Helper --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=1718379636,3073461579650436005,9081109346180051893,131072 --disable-features=SpareRendererForSitePerProcess --lang=en-US --service-sandbox-type=network --user-data-dir=/Users/phil/Library/Application Support/Zui --shared-files --seatbelt-client=47 + | |--- 00908 phil /Applications/Zui.app/Contents/Frameworks/Zui Helper (Renderer).app/Contents/MacOS/Zui Helper (Renderer) --type=renderer --user-data-dir=/Users/phil/Library/Application Support/Zui --app-path=/Applications/Zui.app/Contents/Resources/app.asar --enable-experimental-web-platform-features --no-sandbox --no-zygote --field-trial-handle=1718379636,3073461579650436005,9081109346180051893,131072 --disable-features=SpareRendererForSitePerProcess --disable-gpu-compositing --lang=en-US --num-raster-threads=4 --enable-zero-copy --enable-gpu-memory-buffer-compositor-resources --enable-main-frame-before-activation --renderer-client-id=4 --no-v8-untrusted-code-mitigations --shared-files + | \--- 00928 phil /Applications/Zui.app/Contents/Frameworks/Zui Helper (Renderer).app/Contents/MacOS/Zui Helper (Renderer) --type=renderer --user-data-dir=/Users/phil/Library/Application Support/Zui --app-path=/Applications/Zui.app/Contents/Resources/app.asar --no-sandbox --no-zygote --field-trial-handle=1718379636,3073461579650436005,9081109346180051893,131072 --disable-features=SpareRendererForSitePerProcess --disable-gpu-compositing --lang=en-US --num-raster-threads=4 --enable-zero-copy --enable-gpu-memory-buffer-compositor-resources --enable-main-frame-before-activation --renderer-client-id=5 --no-v8-untrusted-code-mitigations --shared-files ``` If `zed` is absent from these outputs, it's possible that another process is already running that's listening on TCP port 9867. To confirm this, completely -exit Brim and run `netstat -an` and look for instances of `:9867`. +exit Zui and run `netstat -an` and look for instances of `:9867`. If no such process is found, examine the logs of system utilities (perhaps with assistance from an IT department) to look for indications of the `zed` binary not having been fully installed (such as during auto-update to a new release) or prevented from being launched. -### No local communications between Brim and `zed serve` +##### No local communications between Zui and `zed serve` If the process tables indicated `zed` is running, the next thing to consider is the local network connectivity. In an "out-of-the-box" desktop environment, -not only would Brim have been able to connect, but a direct connection from a +not only would Zui have been able to connect, but a direct connection from a browser to `http://localhost:9867` would cause the following informational message to be returned by `zed serve`. @@ -198,12 +197,12 @@ communications. It may be necessary to examine firewall logs (perhaps with assistance from an IT department) to look for evidence of the connection having been blocked. -## My antivirus software has flagged Brim as potentially malicious +#### My antivirus software has flagged Zui as potentially malicious We've been made aware that the Windows binary `suricata-update.exe` has been flagged by several antivirus engines, as summarized in [this VirusTotal entry](https://www.virustotal.com/gui/file/b184511d689d547fa5fd524c7ca120586fcffc60f338f932d41800c63961d723/detection). -If your antivirus software should flag a different Brim-related binary as +If your antivirus software should flag a different Zui-related binary as potentially malicious, please [open an issue](#opening-an-issue), as we'd want to investigate that as well. @@ -213,26 +212,26 @@ still have concerns. Thankfully the code and tools involved in building the binary are all open source, so we encourage users to perform their own assessment as to whether they feel safe installing and using the software. As we don't claim deep expertise in antivirus matters, we'd also welcome input -from experienced members of the Brim community on ways we can best address +from experienced members of the user community on ways we can best address the topic. Please come talk to us on [public Slack](https://www.brimdata.io/join-slack/) if you feel you could offer further assistance. It may be helpful to first understand the role of this binary. As the name suggests, the binary is part of the [Suricata threat detection engine](https://suricata.io/) -that's bundled with Brim as part of the [Brimcap](https://github.com/brimdata/brimcap) +that's bundled with Zui as part of the [Brimcap](https://github.com/brimdata/brimcap) tool that generates summary logs from packet captures. When used in a default -configuration, Brim invokes Suricata's update utility each time the app +configuration, Zui invokes Suricata's update utility each time the app launches to ensure the most recent threat rules are locally cached such that -accurate alerts can be generated from pcaps you import into Brim. +accurate alerts can be generated from pcaps you import into Zui. -Considering how the tool functions and how Brim uses it, it becomes plausible +Considering how the tool functions and how Zui uses it, it becomes plausible that a conservative antivirus engine may perceive the observed network behavior as consistent with that of ["backdoor"](https://en.wikipedia.org/wiki/Backdoor_(computing)) malware that may reach out to arbitrary network destinations in the name of transmitting sensitive information or providing network channels for nefarious remote logins. Of course, the [functionality of Suricata-Update](https://suricata.readthedocs.io/en/suricata-6.0.0/rule-management/suricata-update.html) as described by the OISF is quite benign: In its default -configuration as it's used by Brimcap and Brim, the network destinations it +configuration as it's used by Brimcap and Zui, the network destinations it reaches out to are servers associated with providing the [Emerging Threats Open](https://rules.emergingthreats.net/open/) ruleset. @@ -242,8 +241,8 @@ Indeed, the Suricata-Update tool in its original form is written in Python and in a normal Suricata install, users are responsible for having a compatible Python installation that's capable of running the tool. This seems a reasonable expectation for the types of servers on which Suricata is often installed. -However, since it is our intent for Brim to be desktop software that's easy -to install, we felt that bundling Python with Brim would bring excessive +However, since it is our intent for Zui to be desktop software that's easy +to install, we felt that bundling Python with Zui would bring excessive bloat, and we felt requiring users to maintain a working Python installation as a prerequisite for having valid alerts would be a heavy burden. It's for this reason that we made the decision to use the popular [PyInstaller](https://www.pyinstaller.org/) @@ -254,20 +253,19 @@ If you wish to perform your own audit to confirm that the executable has not been tampered with in the build process, below are some pointers to trace its origin. -* A Brim-managed [fork of suricata-update](https://github.com/brimdata/suricata-update) holds a +* A Brim-maintained [fork of suricata-update](https://github.com/brimdata/suricata-update) holds a copy of the [original OISF repo](https://github.com/OISF/suricata-update). - As of Brim - release `v0.25.0`, the fork is holding a copy of the OISF repo as of tag + As of Zui release `v1.0.0`, the fork is holding a copy of the OISF repo as of tag `1.2.0`. * A branch [`fix-windows`](https://github.com/brimdata/suricata-update/tree/fix-windows) - exists in the Brim-managed repo that includes minimal changes that we found + exists in the Brim-maintained repo that includes minimal changes that we found necessary for the executable to build and run on Windows. The total set of changes can be reviewed [here](https://github.com/brimdata/suricata-update/compare/1.2.0...0b10a83). * A [build-suricata repo](https://github.com/brimdata/build-suricata) contains automation that's used to create the Suricata packages that are bundled with Brimcap. This - includes [script code](https://github.com/brimdata/build-suricata/blob/9847f9ba72b24a132aa7068a15b9d3ff9e63af1b/.github/workflows/windows-build.yaml#L103-L109) + includes [script code](https://github.com/brimdata/build-suricata/blob/406a18362e55efb9b2948d22ba02db2e2a5e3453/.github/workflows/windows-build.yaml#L103-L109) that unpacks the enhanced Suricata-Update and runs PyInstaller to generate the `suricata-update.exe` executable. The same automation also creates the Suricata artifacts on the repo's @@ -275,11 +273,11 @@ its origin. * The [Brimcap repo](https://github.com/brimdata/brimcap) then has its own [release automation](https://github.com/brimdata/brimcap/blob/4fc6d62e8e12f0949074225274c3f9ea3074344c/.github/workflows/ci.yaml#L43) - which [downloads the Suricata artifact](https://github.com/brimdata/brimcap/blob/4fc6d62e8e12f0949074225274c3f9ea3074344c/Makefile#L41) which is bundled + which [downloads the Suricata artifact](https://github.com/brimdata/brimcap/blob/03b90badb7ea1f3a97c793b21f97d4f525cdd65f/Makefile#L39) which is bundled into [Brimcap releases](https://github.com/brimdata/brimcap/releases). -* Each Brim release points at a [Brimcap release](https://github.com/brimdata/brim/blob/b288fae1654bd6b848a0714583f6570d22e91d97/package.json#L68) - that's bundled by Brim's own [release automation](https://github.com/brimdata/brim/blob/main/.github/workflows/release.yml). +* Each Zui release points at a [Brimcap release](https://github.com/brimdata/brim/blob/4c74bdcdb38b84ff15c20926a7a05bfb7fe61d92/package.json#L69) + that's bundled by Zui's own [release automation](https://github.com/brimdata/brim/blob/main/.github/workflows/release.yml). To summarize, the executable consists of a minimally-enhanced Suricata-Update that's been turned into an executable by other open source tools. If your conclusion @@ -287,13 +285,13 @@ matches our own that this is a "false positive", you can help others by [flagging it as harmless in VirusTotal](https://support.virustotal.com/hc/en-us/articles/115002146769-Comments), as multiple community members have already done. -# Gathering Info +## Gathering Info Before [opening an issue](#opening-an-issue), or while debugging a previously-opened issue, there's detail you can gather that may help us more quickly provide you with a resolution. -## Sensitive Information (important!) +### Sensitive Information (important!) For all information described in this section, we understand that some of it may be of a sensitive nature such that you don't feel you can attach it @@ -302,28 +300,28 @@ the information before attaching, please do. If you feel you could share such information only privately, please notify us on [public Slack](https://www.brimdata.io/join-slack/) and we can arrange to receive it from you in a private 1-on-1 chat. -## Screenshots/Videos +### Screenshots/Videos If you think you've encountered a bug, a screenshot or recorded desktop video -of your Brim app that includes the error message or unexpected behavior may +of your Zui app that includes the error message or unexpected behavior may help us a lot. If you can annotate your media with arrows, text, or other detail, even better. The same is true if you're submitting a detailed feature request. -## Developer Tools +### Developer Tools -Most of Brim's error messages should be surfaced within the app and include +Most of Zui's error messages should be surfaced within the app and include enough detail for us to figure out the problem. However, if you've uncovered a challenging bug, we may need your help to gather some additional detail. For instance, this may come up with bugs where you're clicking in the app and you expect a certain outcome but nothing seems to be happening. A more detailed level of debug info may be hiding in a _Developer_ _Tools_ area -that's available in Brim, and it may help us a lot if you could check there to +that's available in Zui, and it may help us a lot if you could check there to see if more error info is being generated when you reproduce a problem. This will work best with bugs that you can reliably reproduce. -To prepare, get to the point in the Brim app where you're _about_ _to_ trigger +To prepare, get to the point in the Zui app where you're _about_ _to_ trigger the problem, but have not triggered it yet. Then from the menu bar, select **View > Toggle Developer Tools**. This will reveal a new panel on the right side of the app window, as outlined in red below. @@ -340,30 +338,30 @@ Now perform the operation in the app that triggers the problem. Hopefully this will reveal some error messages in the **Console** area of the panel. If this is the case, please take a screenshot and include it with your issue. -## Logs +### Logs -Logs are generated by the Brim app and the `zed serve` process it +Logs are generated by the Zui app and the `zed serve` process it launches (the engine for querying your data). These are located in the -`logs` subdirectory under the Brim [user data](https://github.com/brimdata/brim/wiki/Filesystem-Paths#user-data) +`logs` subdirectory under the Zui [user data](./Filesystem-Paths.md#user-data) path. Please include a ZIP of the contents of this directory along with any issue you open to report a bug. -## Sample Packet Captures +### Sample Packet Captures If your problem involves a pcap file, we'll likely be able to debug much quicker if you can attach a sample pcap file to your issue. If you are able to reproduce the issue using a small pcap that you're certain does not contain sensitive information, that would be ideal. -## Large Files +### Large Files If you have a video or sample pcap that is too large to attach to a GitHub Issue, please upload it to the file-sharing service of your choosing and paste a link to it in your issue. -# Opening an Issue +## Opening an Issue Before/when [opening a new issue](https://github.com/brimdata/brim/issues/new/choose), you can help us out by doing the following: @@ -374,4 +372,4 @@ you can help us out by doing the following: * Feel free to chat with the team on the [public Slack](https://www.brimdata.io/join-slack/) before/after opening an issue. * When you open a new issue, its initial text will include a template with standard info that will almost always be needed. Please include detail for all areas of the template. -Thanks for helping us support the Brim community! +Thanks for helping us support the user community! diff --git a/docs/support/media/Conn-Err-could-not-be-reached.png b/docs/support/media/Conn-Err-could-not-be-reached.png new file mode 100644 index 0000000000..f5e9e1fca2 Binary files /dev/null and b/docs/support/media/Conn-Err-could-not-be-reached.png differ diff --git a/docs/media/Crash-652.png b/docs/support/media/Crash-652.png similarity index 100% rename from docs/media/Crash-652.png rename to docs/support/media/Crash-652.png diff --git a/docs/support/media/Open_Dev_Tools.png b/docs/support/media/Open_Dev_Tools.png new file mode 100644 index 0000000000..65068f44fa Binary files /dev/null and b/docs/support/media/Open_Dev_Tools.png differ diff --git a/docs/support/media/Prep_Dev_Tools.png b/docs/support/media/Prep_Dev_Tools.png new file mode 100644 index 0000000000..c9b4374d63 Binary files /dev/null and b/docs/support/media/Prep_Dev_Tools.png differ diff --git a/docs/support/media/Zui-Task-Manager.png b/docs/support/media/Zui-Task-Manager.png new file mode 100644 index 0000000000..8a37c0f5c5 Binary files /dev/null and b/docs/support/media/Zui-Task-Manager.png differ diff --git a/docs/support/media/Zui-zed-serve-9867.png b/docs/support/media/Zui-zed-serve-9867.png new file mode 100644 index 0000000000..5f8b1709ec Binary files /dev/null and b/docs/support/media/Zui-zed-serve-9867.png differ diff --git a/docs/support/media/zed-listening.png b/docs/support/media/zed-listening.png new file mode 100644 index 0000000000..92098e8e90 Binary files /dev/null and b/docs/support/media/zed-listening.png differ