Add progress flag if docker compose is used #1628

mrodm · 2024-01-18T14:41:12Z

Latest Buildkite builds that are executing the docker compose for all the commands under elastic-package looks like are not honoring the parameter --ansi never

This --ansi parameter is a valid parameter as it is mentioned in the docs: https://docs.docker.com/engine/reference/commandline/compose/#options

These are two BK buildsexamples where it can be seen these changes in the output of docker compose vs docker-compose commands:

Using docker-compose: https://buildkite.com/elastic/elastic-package/builds/2030#018c5a4b-7d43-4804-bdbe-0adc2ca4fa7b/67-337
Using docker compose: https://buildkite.com/elastic/elastic-package/builds/2134#018d1ca8-6837-4bfb-87fb-151b5da99ec1/63-336

In this PR, it is added a new parameter --progress available just for docker compose command. If used plain as value, results are similar to what --ansi never does. If used quiet as value, there is no output at all.

Using --progress plain: https://buildkite.com/elastic/elastic-package/builds/2138
Using --progress quiet: https://buildkite.com/elastic/elastic-package/builds/2139

A new section is added to the README , to include all the environment variables available that could be used when using elastic-package CLI.

Testing this parameter locally

 $ docker compose --progress=plain down ; echo "-----" ; docker rmi ubuntu:latest ; echo "--------" ; docker compose  --progress=plain up -d --quiet-pull
 Container dockertest-ubuntu-1  Stopping
 Container dockertest-ubuntu-1  Stopped
 Container dockertest-ubuntu-1  Removing
 Container dockertest-ubuntu-1  Removed
 Network dockertest_default  Removing
 Network dockertest_default  Removed
-----
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:e6173d4dc55e76b87c4af8db8821b1feae4146dd47341e4d431118c7dd060a74
Deleted: sha256:e34e831650c1bb0be9b6f61c6755749cb8ea2053ba91c6cda27fded9e089811f
Deleted: sha256:8e87ff28f1b5ff2d5131999ccfa1e674cb252631c50683f5ee43fad59cbea8e1
--------
 ubuntu Pulling 
 ubuntu Pulled 
 Network dockertest_default  Creating
 Network dockertest_default  Created
 Container dockertest-ubuntu-1  Creating
 Container dockertest-ubuntu-1  Created
 Container dockertest-ubuntu-1  Starting
 Container dockertest-ubuntu-1  Started


 $ docker compose --progress=quiet down ; echo "-----" ; docker rmi ubuntu:latest ; echo "--------" ; docker compose  --progress=quiet up -d --quiet-pull
-----
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:e6173d4dc55e76b87c4af8db8821b1feae4146dd47341e4d431118c7dd060a74
Deleted: sha256:e34e831650c1bb0be9b6f61c6755749cb8ea2053ba91c6cda27fded9e089811f
Deleted: sha256:8e87ff28f1b5ff2d5131999ccfa1e674cb252631c50683f5ee43fad59cbea8e1
--------

Docker-compose used for the example:

version: '2.3'
services:
  ubuntu:
    image: ubuntu
    command: tail -F anything
    volumes:
      - type: bind
        source: ./test
        target: /test

mrodm · 2024-01-18T15:19:29Z

internal/compose/compose.go

 	}

+	v, ok = os.LookupEnv(DisableProgressOutputComposeEnv)
+	if !c.dockerComposeV1 && !c.dockerComposeStandalone && ok && strings.ToLower(v) != "false" {


This option is just available using docker compose

Why? This behaves the same in docker-compose (trying with standalone docker-compose v2.23.3).

I was trying with docker-compose with version 2.17.2 (same version that was set previously in CI for it). In that version there is no --progress flag.

I see that in the latest version of docker-compose standalone (v2.24.1), that version is already in place.

mrodm · 2024-01-18T17:30:05Z

.buildkite/pipeline.yml

+  # ELASTIC_PACKAGE_COMPOSE_DISABLE_PROGRESS_OUTPUT: "true"
+  ELASTIC_PACKAGE_COMPOSE_PROGRESS_OUTPUT: "quiet"


@jsoriano any preference about which environment would be best to set ?

ELASTIC_PACKAGE_COMPOSE_DISABLE_PROGRESS_OUTPUT would be just to set true or false, and we would need to decide if we want plain or quiet for the --progress flag.

In ELASTIC_PACKAGE_COMPOSE_PROGRESS_OUTPUT var, a few values would be allowed (auto, plain or quiet) and the user could choose which one.

For CI quite looks good 👍

Or plain, with --quiet-pull also looks good for CI.

mrodm · 2024-01-18T17:31:04Z

README.md

 * `stack.serverless.region` can be used to select the region to use when starting
  serverless projects.

+## Useful environment variables


AFAIK there is no docs about all the environment variables that are defined in elastic-package. Just added this section to include all these references

WDYT ?

mrodm · 2024-01-18T17:32:13Z

internal/compose/compose.go

+	DisableProgressOutputComposeEnv   = environment.WithElasticPackagePrefix("COMPOSE_DISABLE_PROGRESS_OUTPUT")
+	ProgressOutputComposeEnv          = environment.WithElasticPackagePrefix("COMPOSE_PROGRESS_OUTPUT")


We should decide which one to use here.

COMPOSE_DISABLE_PROGRESS_OUTPUT sounds good to me, we don't need to expose all the options.

jsoriano

Latest Buildkite builds that are executing the docker compose for all the commands under elastic-package looks like are not honoring the parameter --ansi never.

I think it is not a matter of docker-compose vs docker compose but about a regression(?) on recent versions. Trying locally with standalone docker-compose v2.23.3 it neither honors --ansi never.

Given the differences between the different versions I wonder if instead of exposing the different flags, we should have a single flag to enable or disable this verbose output, that under the hood uses any flag known to work for each version.

jsoriano · 2024-01-18T22:53:02Z

.buildkite/pipeline.yml

+  # ELASTIC_PACKAGE_COMPOSE_DISABLE_PROGRESS_OUTPUT: "true"
+  ELASTIC_PACKAGE_COMPOSE_PROGRESS_OUTPUT: "quiet"


For CI quite looks good 👍

jsoriano · 2024-01-18T22:53:27Z

README.md

 * `stack.serverless.region` can be used to select the region to use when starting
  serverless projects.

+## Useful environment variables


README.md

jsoriano · 2024-01-18T22:56:14Z

.buildkite/pipeline.yml

+  # ELASTIC_PACKAGE_COMPOSE_DISABLE_PROGRESS_OUTPUT: "true"
+  ELASTIC_PACKAGE_COMPOSE_PROGRESS_OUTPUT: "quiet"


Or plain, with --quiet-pull also looks good for CI.

jsoriano · 2024-01-18T22:59:04Z

internal/compose/compose.go

+	DisableProgressOutputComposeEnv   = environment.WithElasticPackagePrefix("COMPOSE_DISABLE_PROGRESS_OUTPUT")
+	ProgressOutputComposeEnv          = environment.WithElasticPackagePrefix("COMPOSE_PROGRESS_OUTPUT")


COMPOSE_DISABLE_PROGRESS_OUTPUT sounds good to me, we don't need to expose all the options.

jsoriano · 2024-01-18T23:00:39Z

internal/compose/compose.go

 	}

+	v, ok = os.LookupEnv(DisableProgressOutputComposeEnv)
+	if !c.dockerComposeV1 && !c.dockerComposeStandalone && ok && strings.ToLower(v) != "false" {


Why? This behaves the same in docker-compose (trying with standalone docker-compose v2.23.3).

mrodm · 2024-01-22T10:03:14Z

I think it is not a matter of docker-compose vs docker compose but about a regression(?) on recent versions. Trying locally with standalone docker-compose v2.23.3 it neither honors --ansi never.

Just tried with the latest version (v2.24.1) and it happens the same, --ansi never flag is not taken into account.

Given the differences between the different versions I wonder if instead of exposing the different flags, we should have a single flag to enable or disable this verbose output, that under the hood uses any flag known to work for each version.

Should we use a new flag, for instance just ELASTIC_PACKAGE_COMPOSE_VERBOSE_OUTPUT and remove the flags ELASTIC_PACKAGE_COMPOSE_DISABLE_ANSI and ELASTIC_PACKAGE_COMPOSE_PROGRESS_OUTPUT in this PR ? Should it be removed ELASTIC_PACKAGE_COMPOSE_DISABLE_PULL_PROGRESS_INFORMATION?
That new flag in case of docker-compose v1 would just add --ansi never in docker-compose v2 depending on the version --progress flag exists or not.

Not sure if adding just one variable would just force to update docker-compose versions to users. If it is kept different environment variables, probably it is easier to set up by users depending on their versions. Or maybe having all these environment variables make more difficult to know which one to use... WDYT ? @jsoriano

And checking with the latest-version, it looks also that --quiet-pull is ignored in docker-copmpose version v2.24.1 in my tests. Example:

# docker-compose v2.24.1
 $ docker-compose --ansi=never down ; echo "-----" ; docker rmi ubuntu:latest ; echo "--------" ; docker-compose  --ansi=never up -d --quiet-pull
[+] Running 2/2
 ✔ Container dockertest-ubuntu-1  Removed                                                         10.3s 
 ✔ Network dockertest_default     Removed                                                          0.4s 
-----
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:e6173d4dc55e76b87c4af8db8821b1feae4146dd47341e4d431118c7dd060a74
Deleted: sha256:e34e831650c1bb0be9b6f61c6755749cb8ea2053ba91c6cda27fded9e089811f
Deleted: sha256:8e87ff28f1b5ff2d5131999ccfa1e674cb252631c50683f5ee43fad59cbea8e1
--------
[+] Running 1/1
 ✔ ubuntu Pulled                                                                                   5.8s 
[+] Running 1/2
 ⠴ Network dockertest_default     Created                                                          0.4s 
 ✔ Container dockertest-ubuntu-1  Started

mrodm · 2024-01-22T12:28:56Z

buildkite test this

jsoriano

👍

elasticmachine · 2024-01-23T16:34:28Z

💚 Build Succeeded

Buildkite Build
Commit: 13ea63f

History

💛 Build #2162 was flaky 2ba4762
💔 Build #2161 failed 2ba4762
💔 Build #2158 failed aa1c365
💚 Build #2147 succeeded f8f7578
💔 Build #2139 failed 0042aa6

cc @mrodm

mrodm added 2 commits January 18, 2024 15:40

Add progress flag if docker compose is used

198f36c

Add new environment variable

0484777

mrodm commented Jan 18, 2024

View reviewed changes

mrodm added 4 commits January 18, 2024 16:39

Update comment

b32ba68

Set quiet as value of progress flag

0042aa6

Use a different environment variable

f2d48f4

Add section in README for environment variables

93a334d

mrodm force-pushed the update_params_disable_ansi branch from f76c295 to 4662354 Compare January 18, 2024 17:25

mrodm requested a review from a team January 18, 2024 17:26

mrodm self-assigned this Jan 18, 2024

mrodm commented Jan 18, 2024

View reviewed changes

mrodm force-pushed the update_params_disable_ansi branch from 4662354 to 00c34ad Compare January 18, 2024 17:31

mrodm commented Jan 18, 2024

View reviewed changes

mrodm marked this pull request as ready for review January 18, 2024 17:32

Update README with environment variables available

f8f7578

mrodm force-pushed the update_params_disable_ansi branch from 00c34ad to f8f7578 Compare January 18, 2024 17:41

jsoriano reviewed Jan 18, 2024

View reviewed changes

Just use one env variable to disable progress output

aa1c365

mrodm requested a review from jsoriano January 22, 2024 11:56

mrodm added 2 commits January 22, 2024 13:01

Update env. vars in BK pipeline

80141c1

Moved comment

2ba4762

jsoriano approved these changes Jan 23, 2024

View reviewed changes

Merge remote upstream/main into update_params_disable_ansi

13ea63f

jsoriano approved these changes Jan 23, 2024

View reviewed changes

mrodm merged commit 2ec9be9 into elastic:main Jan 23, 2024

mrodm deleted the update_params_disable_ansi branch January 23, 2024 16:41

mrodm mentioned this pull request Jan 23, 2024

[Buildkite] Update elastic-package settings elastic/integrations#8952

Merged

		# ELASTIC_PACKAGE_COMPOSE_DISABLE_PROGRESS_OUTPUT: "true"
		ELASTIC_PACKAGE_COMPOSE_PROGRESS_OUTPUT: "quiet"

		DisableProgressOutputComposeEnv = environment.WithElasticPackagePrefix("COMPOSE_DISABLE_PROGRESS_OUTPUT")
		ProgressOutputComposeEnv = environment.WithElasticPackagePrefix("COMPOSE_PROGRESS_OUTPUT")

Add progress flag if docker compose is used #1628

Add progress flag if docker compose is used #1628

Uh oh!

Conversation

mrodm commented Jan 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing this parameter locally

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm commented Jan 22, 2024

Uh oh!

mrodm commented Jan 22, 2024

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 23, 2024

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented Jan 18, 2024 •

edited

Loading