Demo components #7
Conversation
Temporarly pinn to a branch in Benthos while we incubate these changes.
Introduce top level filter component with choices. Move resource filter to a choice of the top level filter. Add constructors for the components.
This will be used to issue events instead of relying on the caller to forward them to the destination.
Handler now sends events using a producer instead of returning them.
The handler setup is now a component that aggregates all possible plugins for the app. Producer and HTTP client plugins are introduced. Add an env var switch for activating Lambda mode.
Codecov Report
@@ Coverage Diff @@
## master #7 +/- ##
========================================
- Coverage 100% 92.2% -7.8%
========================================
Files 3 5 +2
Lines 46 77 +31
========================================
+ Hits 46 71 +25
- Misses 0 5 +5
- Partials 0 1 +1
Continue to review full report at Codecov.
|
| panic(err.Error()) | ||
| } | ||
| ctx := context.Background() | ||
| runner := new(func(context.Context, settings.Source) error) |
There was a problem hiding this comment.
Question: How does this work/what is it doing? It looks like you're creating an empty function signature here.
There was a problem hiding this comment.
The new built-in creates a pointer to an new memory allocation that contains the zero value of a type. The function signature is an instance of a type. For example, we can do things like this:
func takesAFunc(fn func() error) (func() error, error) {
return fn
}
type AFunction func() error
var fn func() errorWhen you omit the body of the function is becomes a type like any other in the language. In this case, we're creating a new pointer to a function type with a specific signature. Since the component we're loading returns a function with this signature then we can pass the new pointer into the loader. At the end of loading, the pointer now points to an actual instance of the function signature rather than the zero value.
There was a problem hiding this comment.
This makes sense to me. It might be helpful to add a comment here, as the magic that makes this work is inside the settings library.
| # (bool) Use the Lambda SDK to start the system. | ||
| AWSCONFIGFILTERD_LAMBDAMODE="false" | ||
| # ([]string) List of enabled filters. | ||
| AWSCONFIGFILTERD_FILTER_ENABLED="resourcetype" |
There was a problem hiding this comment.
Question: How would you go about configuring an AnyMatch filter which takes a ResourceType filter in this context?
There was a problem hiding this comment.
Currently, you don't. This is set up such that additional filters are all combined using the AnyMatch underneath. The way I understood how this system was working is that it discards all events by default. A filter in this system is something that allows a message to pass through to the next stage. Because of this, I defaulted it to AnyMatch for whichever filters are enabled.
If there's a case for complex structures like AnyMatch and AllMatch being chained to form boolean expressions then we'd need to re-evaluate the loading strategy for settings. I don't believe the high-level component form can handle this today and we'd likely need to drop down in the the lower level APIs to make that happen.
There was a problem hiding this comment.
This makes perfect sense. I missed that the AnyMatch filter was being used in FilterComponent.New, and thus mistakenly believed it was unused.
|
|
||
| // New generates a filter. | ||
| func (c *FilterComponent) New(ctx context.Context, conf *FilterConfig) (domain.ConfigFilterer, error) { | ||
| f := make(AnyMatch, 0, len(conf.Enabled)) |
There was a problem hiding this comment.
nit: Because I missed this on my first reading of this, might be worth tossing a comment here to draw extra attention to the use of AnyMatch instead of the vanilla []domain.ConfigFilterer.
Following from the example of asecurityteam/awsconfig-filterd#7, this pulls in the individual settings components libraries to manage the configuration and changes the behavior of the system to produce events itself rather than relying on the caller.
Following from the example of asecurityteam/awsconfig-filterd#7, this pulls in the individual settings components libraries to manage the configuration and changes the behavior of the system to produce events itself rather than relying on the caller.
* Update to use component libraries Following from the example of asecurityteam/awsconfig-filterd#7, this pulls in the individual settings components libraries to manage the configuration and changes the behavior of the system to produce events itself rather than relying on the caller. * Fix typo in README * Update comment for AttributeHandler Handle func to match new usage
I broke things down by commit for easier review. This change set increases the use of settings components to manage the configuration and changes the behavior of the system to produce events itself rather than relying on the caller.
The event production is moved into the function because this pattern works best for Lambda. The event source, whether SQS or Kinesis, can be attached to the Lambda such that AWS handles consume the events for us and calls the lambda for each event. I don't believe there's a corresponding mechanism to take the Lambda return value and automatically place it on another queue/stream. As a result, we now produce the events directly.
Events are produced using an interface that has two options at runtime. One will take a Benthos configuration and and leverage that library to send events to some destination. The other is an HTTP client that will POST the payload to some endpoint. We can continue to use the HTTP client wherever we run Benthos as a side-car. Additionally, the HTTP client may be either the default HTTP client or one that accepts a transportd configuration when we need smart features.