HttpStress execution failure on Windows

[antonfirsov]

HttpStress tests are failing on Windows since 02.17:

The current Compose file version is not compatible with your engine version. Please upgrade your Compose file to a more recent version, or set a COMPOSE_API_VERSION in your environment.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=180524&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

Weirdly, this happens after the actual run is complete.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

HttpStress tests are failing on Windows since 02.17:

The current Compose file version is not compatible with your engine version. Please upgrade your Compose file to a more recent version, or set a COMPOSE_API_VERSION in your environment.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=180524&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

Weirdly, this happens after the actual run is complete.

Author:	antonfirsov
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	8.0.0

[antonfirsov]

@dotnet/dnceng was there any change around the 17th that might have led to this?

[ilyas1974]

Windows images were updated on the 24th (dotnet/arcade#12486) but nothing around the 17th.

[wfurt]

Do you know what Docker version runs in the Nano server @ilyas1974 ?

[ilyas1974]

Do you know VMs the docker nano server container is running on? Otherwise, you can check dotnet-buildtools-prereqs-docker to see what version of docker is installed on the docker containers hosting nano.

/cc: @mthalman

[mthalman]

From what I can tell, it's not running this in a container from dotnet-buildtools-prereqs-docker. It's running compose directly from the build agent which is from the 1es-windows-2022-open pool.

[michellemcdaniel]

I don't know if we can make a helix repro vm directly from the 1es-windows-2022-open pool, but if this job is running docker compose, I wonder if you could just do a docker version to figure out this question. I don't know what is baked into those images, as they come to us from 1es

[wfurt]

maybe draft PR @antonfirsov ? I think we can then adjust the version in docker-compose.yml. It seems like we now ask for version '3'.

It also seems like the recommendation is to use both major and minor version:
https://docs.docker.com/compose/compose-file/compose-versioning/

[MattGal]

Some quick points:

• For any 1es-* image, the version can "float" even when we're not doing rollouts. Thus, when folks notice differences it's best to check out the image version found inside the "Initialize Job" phase. Relevant snippets:

From your failed run:

Image: 1es-windows-2022-open
Image Version: 51.0.0
Pool Name: NetCore-Public
Current image version: '20230214.1'

From the pipeline's last successful run:

Image: 1es-windows-2022-open
Image Version: 50.0.0
Pool Name: NetCore-Public
Current image version: '20230129.1'

... this shows us that indeed the image version being updated is a prime suspect for what broke you.

• For these 1es-windows-2022 images, the "starting point" is https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md . This is the best place to go when something updated and broke you with these guys.
• Reviewing the PR that changed this doc on the image version noted, indeed Docker Compose gets updated from 2.15.1 -> 2.16.0
• Finally, for pedantry's sake, 1es-windows-2022-open is an image name, not a pool (the pool is NetCore-Public)

[antonfirsov]

I think we are hitting actions/runner-images#7080 which is still happening, despite the issue being closed. We can try the workarounds recommended in the discussion, or wait for an upstream fix.

[MattGal]

I think we are hitting actions/runner-images#7080 which is still happening, despite the issue being closed. We can try the workarounds recommended in the discussion, or wait for an upstream fix.

If transitioning to windows.vs2022.amd64 / windows.vs2022.amd64.open works, that's a fine workaround for as long as you like.