Understanding SEQ data storage and indexing and practical limits

[mattchristenson]

Hello!

We're considering using SEQ at our organization. Initial tests show some great performance when searching and sorting and getting to the logs that are important to us, but I'm a little concerned about how the scalability of this works? When we get to having many GB , 100s of GB, or 1TB+ of logs, is there indexing that allows searches to continue to work well? How does archiving older records - say anything older than 90 days, but with some ability to bring that back in for analysis later work?

Can you share your high volume experiences with me?

Thanks for your time!

-Matt Christenson

[nblumhardt]

Hi Matt! Thanks for checking out Seq.

RE archiving and retrieving old data, the usual tactic with Seq is to use Seq.App.JsonArchive to stream incoming logs to a set of newline-delimited JSON files on a volume or file share accessible to the Seq server.

These can be read back into the same Seq instance, but it's usually better to grab files of interest and analyze them on a local/individual instance of Seq, rather than to import them back into the original server. The seqcli ingest --json command can work with these (and seqcli print actually supports a few tricks available for querying them directly, without importing them into any Seq instance at all.)

Many-GB with Seq is typical; 100s of GB is also comfortable with a good configuration on suitable hardware.

1TB+ stores aren't super-unusual, though the comfort level of doing this depends heavily on the hardware, and the appropriate hardware will depend heavily:

• on what your retrieval requirements are (i.e. will you need fast access to the most recent 100 GB, or to the whole thing?),
• on the shape and quality of event data (structured events are easier to construct fast queries for than plain text; average event size needs to be around the 1KB mark),
• on the quality of signal indexes being used (indexes partition the event stream into overlapping regions that are searched in linear time; the more specific the index, the faster it can be queried),
• on the kinds of searches being performed, and
• on the number of concurrent users.

Doing this is usually a matter of tuning over time - as you'd expect, it's not as straightforward as installing the software and dumping in a terabyte of data 😅. There unfortunately isn't a great alternative to testing with the expected load, but you might look at something like an AWS r5.4xlarge or r5.8xlarge as a reasonable starting point.

Seq is also heavily geared towards avoiding the need for massive online data volumes; if you haven't spotted it already, you might find some interesting background in our recent blog post on storage management:

https://blog.datalust.co/storage-view/

Quite often when teams end up with massive log volumes, it's due to a disproportionate amount of noise mixed in with the useful log data. This isn't always avoidable, but over time we've worked hard on fine-grained ingestion and retention controls to help battle this and keep costs under control.

I'm nblumhardt@datalust.co, in case you would like to dig into the specifics of your use case a bit further.

Hope this helps!
Nick

[MattMofDoom]

@mattchristenson In my workplace we're using a decent size production Seq instance ... currently ~782GB indexed. We've implemented retention policies for some very noisy non-prod data (7 days), very noisy prod data (30 days) and all other data (90 days). We apply a 'retention value' consideration against our logs; that is, logs with a lot of 'noise' that isn't useful are in the 30 day basket, while some high volume logging has no 'noise' but high value to hold online for the 90 days.

We have Json archives set which export data based on signal, which allows us to have complete archives for each application that is logging to Seq, and gives us long term storage). One of my periodic efforts is to circle back on any additions to our Seq logging and ensure that a Json archive has been allocated. The Json archive app is excellent for purpose, and if we use an app like Agent Ransack to search the resulting Json files, we can generally even do without importing them back into a Seq instance.

As touched on for the Json archives, we use signals to index our logs. Obviously this powers our ability to query logs - queries against signals always perform better - but also powers our monitoring and alerting, so that using signals, we can build 'views' that dashboards and apps can watch and alert on.

I have a strong focus on trying to foster good structured logging in our environment, which has spilled over into the apps I've been creating and contributing to. This helps greatly with monitoring and alerting in particular, but as noted by @nblumhardt, structured events are much easier to query.

This really is a matter of tuning over time ... for example while writing this, I found one of our devs had changed a signal into something that was less than ideal, so after correcting that, I've protected the signal to avoid future pain! I've also moved over time to a separate system drive, data drive (for Seq logs), archive drive, and page file to minimise IO impact on the data drive.

We have an enterprise license, so my goal is to shift to a HA pair from 2021.3 onward, a separate non-prod instance, and probably a DR instance running on the free license for the sake of allowing apps in DR to log to Seq.

Performance is dependent upon, in order - CPU, memory, and disk speed. @nblumhardt might well say that memory goes first - he'd know - but I've certainly found more CPU is of great value. In our environment, feeding Seq enough RAM to hold everything in memory isn't really viable, and once we resort to disk, queries will slow. A few tips;

1. More precise queries will give you good performance, even from disk. For example, narrow a time range down to the same day and a matter of hours. Multi-day queries on disk can be slow, particularly if the index is quite broad.
2. You can have a broad index (ie. AppName = 'Blah') for the purpose of (for example) archiving and retention, and specific indexes (AppName = 'Blah', Property1 IN ['Value1','Value2','Value3'] to narrow your focus and improve speed. This is especially true in monitoring and alerting, where our intere
   st may be AppName = 'Blah', @Level = 'Error' to be able to send an alert to OpsGenie.
3. The storage view is your friend - you can get to know what your typical storage characteristics look like at each retention milestone. The aforementioned change to a signal explains something I'd puzzled over somewhat, which is an increase in the per-day storage past 30 days. I hadn't been bothered enough to dig deep, but I expect to see this return to closer to what the previous levels were.
4. Json Archive is your friend. It's the whole Json event in a Json file, readily searchable with an Agent Ransack or similar to pinpoint the events you want, and then optionally load it into a Seq instance.
5. Always remember you can have a free Seq license on your local machine for the purpose of importing Json logs or, in my case, testing apps

I feel it's obvious that I love Seq, and I'm always finding opportunities to add more and more apps into our Seq logging, along with contributing back to the community by creating to new apps and collaborating on enhancements to other apps. I can't recommend it highly enough - cost effective, efficient, powerful, plenty of open source apps to leverage ... the list goes on. I used it as a single-user POC instance for some years before my workplace finally agreed to make the purchase, but that was actually to our advantage - I had the background knowledge on Seq, had done some development on POC implementations of Seq logging in our applications, and had a clear picture of what was needed (including mandating logging to a centralised application logging server, Seq). This is an ongoing effort - I've created common C# and Java log libraries, Linux scripts, Seq apps to suit needs and niches, extended other apps to improve the monitoring and alerting story, and dug down into how to make Seq apps interoperable. It needs a mindset focused towards logging that ultimately pays off in spades.

Hope that's of some help.

Cheers,

Matt

[nblumhardt]

Wow, thanks for the detailed write-up, Matt!