Add profile setting to enable serverless in compose

[jsoriano]

Supersedes #1231.

[ruflin]

I think there are also some flags that have to be set for Kibana: https://github.com/elastic/kibana/blob/main/config/serverless.oblt.yml#L17

[ruflin]

@jsoriano It would be great to get this moving forward to make testing on serverless much easier.

[jsoriano]

@jsoriano It would be great to get this moving forward to make testing on serverless much easier.

Testing on actual serverless is already fully supported by using elastic-package stack up --provider serverless (given that the user has access to serverless).

I could not find any different behaviour in ES by setting es.serverless. How could we check that this is working as expected?

[ruflin]

Testing on actual serverless is already fully supported by using elastic-package stack up --provider serverless (given that the user has access to serverless).

But we are talking here about local testing / during development / latest snapshots.

I could not find any different behaviour in ES by setting es.serverless. How could we check that this is working as expected?

@leehinman Maybe you can help with this one?

[jsoriano]

I think there are also some flags that have to be set for Kibana: https://github.com/elastic/kibana/blob/main/config/serverless.oblt.yml#L17

Added, but it looks like some flags are not available in the builds we use. Maybe we need to use different docker images.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: aa0dc68

History

• 💚 Build #2793 succeeded 50eff68

cc @jsoriano

[ruflin]

@flash1293 Do you know by chance more about what configs must be used?

[flash1293]

Unfortunately I don't know much about running serverless kibana/elasticsearch. The dev environment is using the docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified image

[cmacknz]

When we were adding support to serverless for Beats we found that there was no local development environment that could easily be setup with Docker. That elasticsearch-serverless container has service dependencies expects things like a blob storage service to exist, it isn't (or wasn't) meant to run standalone.

All of the Beats serverless tests run against real Serverless projects provisioned by the agent test framework.

[flash1293]

there was no local development environment that could easily be setup with Docker

It seems like the kibana dev setup is super similar - @afharo where can we learn more about how it works there?

[afharo]

HI! Thanks for the ping. I think the best folks to ask about how our yarn es serverless works are @elastic/kibana-operations, prob @delanni.

[afharo]

AFAIK, just setting serverless: oblt should be enough to load all other settings (if you use the Kibana Serverless docker image).

[Ikuni17]

Most of how we run ES serverless for Kibana development is in this file: https://github.com/elastic/kibana/blob/main/packages/kbn-es/src/utils/docker.ts

The first few logs lines should be

Starting Serverless Elasticsearch...
Serverless default for [stateless.enabled] is overridden to [true]

We pass --env stateless.enabled=true (instead of serverless) so this may be difference, though I think the image starts ES in serverless mode anyways.