Welcome to Substation! #1
Replies: 2 comments 2 replies
-
|
Very interesting architecture! I'm working on a similar project that uses Rust on Lambda, Apache Arrow and CDK under the hood to let users deploy a serverless security data lake to AWS. https://github.com/matanolabs/matano Great to see other projects taking a similar approach! |
Beta Was this translation helpful? Give feedback.
-
Hey @shaeqahmed 👋 Thanks for sharing, our team saw Matano when it was released and it certainly is interesting! Based on the design and documentation I think the projects have different use cases, but it's always exciting to see more effort put into the security data engineering space. https://github.com/benthosdev/benthos is another cool project in this space that others may be interested in (and also has some overlap, but otherwise addresses different use cases). |
Beta Was this translation helpful? Give feedback.
-
|
Hey @jshlbrd I checked out substation blog recently and came here. I come from a Security Engg background and I have been a user of Streamalert for sometime and have been recently trying my hands on matano which @shaeqahmed had mentioned above. At an overview level it seems like substation has been written in a generic fashion which can accomodate any serverless pipeline based use cases (not just security use cases) unlike above mentioned tools. I really liked the idea as it allows you to play with it the way you want. But I am really curious and wanted to understand what drove you and your team to write such a generic framework instead of using framework like benthos (another generic data pipeline framework supporting serverless use cases - mentioned above by you ) , AWS step functions (with supported serverless orchestrator) , streamalert , matano etc.. that existed way before ? Was there any challenge or any special cases that you encountered which led to this ? It would be nice to know as it would give another perspective on use case of serverless pipelines in detection engineering / general. Also great work on the tool, I think I would be hooked on experimenting with it in coming days. Thanks for making this open source !! 👍👍🙂🙂 |
Beta Was this translation helpful? Give feedback.
-
|
👋 @rams3sh
Yep, in my opinion it solves an entirely different use case compared to the other tools you mentioned.
To be honest, when we started this project we didn't know that Benthos existed. (We ran an internal, pre-release version of Substation for more than a year before discovering Benthos). If we had known about Benthos, then Substation might not exist at all. With that mentioned, the two projects are different when you look at low-level details (for example, Substation is serverless by default with infrastructure as code deployment support and does not have a "domain language").
This never came up during initial design, but I suspect that it could work as a replacement. We'd still have to invest a lot of engineering effort into data transformation, which is Substation's main feature (in my opinion). The design we have today was also selected to keep cloud costs as low as possible (for example, that's why we enforce data aggregation in AWS Kinesis), I'm not sure if that would be true if we used Step Functions at scale.
I mentioned this earlier but I think that StreamAlert solves an entirely different use case (alerting). The closest comparison in StreamAlert is their unannounced data normalization feature -- Substation is that, except much larger in scope and capability. Substation + StreamAlert would be great together, and easy to integrate (put Substation in front of StreamAlert as a data source).
Same as StreamAlert, Matano solves a different use case -- in fact, architecturally they look similar, except that Matano is more explicit in some areas (e.g., use of Apache Iceberg). It's an all-in-one solution compared to what Substation does.
We've been developing and running Substation internally at Brex for almost 2 years -- it's been around a long time ago, we just never told anyone about it. We actually quietly open sourced it in April 2022, but very few people discovered it on their own. 😆
Nothing in particular, but I've worked on very large systems like this at other companies and had a good idea of what the project needed to be -- affordable at scale, low maintenance, and able to support use cases that even we can't think of today. For references on problems that the project solves, I'd refer folks to these sources:
If you use it, then let us know how it goes! |
Beta Was this translation helpful? Give feedback.
-
👋 Welcome!
The @brexhq/substation team is using Discussions as a place to connect with members of the community. We hope that you use this space to:
If you're interested, then comment below with an introduction of yourself and share why you're interested in the project!
Beta Was this translation helpful? Give feedback.
All reactions