Make service.RunWithOpts public again

[joao3101]

On V3, we could start the service with RunWithOpts. On V4, we can only start it passing options if we use the StreamBuilder. Although StreamBuilder is great, RunWithOpts is more suitable for most of our use cases, so it would be nice to get this public again :)

P.S: This was already discussed on the Discord channel a few time ago.

[Jeffail]

Hey @joao3101, which specific options were you using with v3? There are some that I'm happy to add but others that I'd rather avoid if possible.

[joao3101]

Hi @Jeffail. That's the function that we actually have to start Benthos:

func (be *environment) run(configLocation string) {
	c := config.Type{}
	if lints, _ := config.Read(configLocation, true, &c); len(lints) > 0 {
		log.Error().Msg("Shutting down due to linter errors, to prevent shutdown run Benthos with --chilled")
		return
	}
	service.RunWithOpts(
		service.OptOverrideConfigDefaults(func(c *config.Type) {
			_, err := config.Read(configLocation, true, c)
			if err != nil {
				log.Error().Msg(fmt.Sprintf("error loading conf file: %v", err))
			}
		}))
}

[joao3101]

Hi @Jeffail ,

Any news on this?

We run our services normally, but as the RunWithOpts is not public anymore, we're running our integration tests as streams, as suggested. But we have some services that receives http_servers as inputs, and on those integration tests, when we send the POST request, we're having the following problem:
| Actual: 'Post "http://localhost:33065/ingest": dial tcp 127.0.0.1:33065: connect: connection refused'

When I run it normally on my PC (without streams), the response is ok:

If you need more information, tell me and I'll send you a snippet of a simplified version of the pipeline we have.

Thank you in advance!

[Jeffail]

Hey @joao3101 if it's for testing purposes I'd recommend setting https://pkg.go.dev/github.com/benthosdev/benthos/v4@v4.9.0/public/service#StreamBuilder.SetHTTPMux and then calling that multiplexer directly as it saves you needing to spin up and call real HTTP servers.

Alternatively you can still set the config field http.enabled to true and it'll behave the same as a regular stream, it's false by default when running a stream builder as it avoids your process accidentally exposing endpoints and unintended port bind collisions.

[joao3101]

Hi @Jeffail,

I've tried using http.enabled = true, but it didn't work also, still on the same connection refused error. Do you have any ideia on what else could work?

Bellow is a sample (I've removed what is not important for this specific part):

http:
  address: ${HTTP_ADDRESS:0.0.0.0:4195}
  enabled: true

input:
  broker:
    inputs:
    - label: input_httpserver
      http_server:
        path: /ingest
        sync_response:
          status: ${!meta("status")}
      processors:
      - bloblang: |
          root = {"keep":"alive"}

    batching:
      count: ${KAFKA_BROKER_BATCH_COUNT:0}
      period: ${KAFKA_BROKER_BATCH_PERIOD:""}

buffer:
  type: memory

[Jeffail]

Which version of Benthos are you pulling? The was a bug fix in 4.6.0 you might be missing: https://github.com/benthosdev/benthos/blob/main/CHANGELOG.md#fixed-3

[joao3101]

This solved a problem, but came up with another (don't know if they're related):
kafka: invalid configuration (Consumer.Group.Rebalance.GroupStrategies and Consumer.Group.Rebalance.Strategy cannot be set at the same time. I'll look into it, but I think this is not related to streams anymore, as we already have some kafka outputs running on v4. Thank you!

[mihaitodor]

@joao3101 You'll want to pull the latest version. There were some issues with the underlying Sarama library which had to be reverted to an older version for now...