Task, that was working yesterday, started to fail today in 20210131.1, the same code

[nanvel]

Description

This workflow: https://github.com/nanvel/tb-manager/actions/runs/536216335

Yesterday it was passing.
Today I restarted it, and it fails.
I did checkout from the revision, created another branch, pushed, and - the same issue.

Errors in pytest while connectiong to postgres service (using aiopg):

self = <selectors.EpollSelector object at 0x7f701a5f81d0>, fileobj = 16
events = 2
data = (None, <Handle Connection._ready(<weakref at 0...x7f701a594e90>) at /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/aiopg/connection.py:97>)

    def modify(self, fileobj, events, data=None):
        try:
            key = self._fd_to_key[self._fileobj_lookup(fileobj)]
        except KeyError:
            raise KeyError(f"{fileobj!r} is not registered") from None
    
        changed = False
        if events != key.events:
            selector_events = 0
            if events & EVENT_READ:
                selector_events |= self._EVENT_READ
            if events & EVENT_WRITE:
                selector_events |= self._EVENT_WRITE
            try:
>               self._selector.modify(key.fd, selector_events)
E               FileNotFoundError: [Errno 2] No such file or directory

/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/selectors.py:389: FileNotFoundError

The only difference I see is environment version: 20210131.1
The tests were passing on 20210123.1.

Postgres service seems to works ok, as I see that my migrations are executed by alembic.

Area for Triage:

Python

Question, Bug, or Feature?:

Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

Image version

20210131.1

Expected behavior

My pytests passing.

Actual behavior

My pytest are not passing since today.

Repro steps

An aiohttp server that uses aiopg to connect to postgresql database.
I can create a small example if it is necessary.

[Darleev]

Hello @nanvel,
We don't have access to the workflow provided. It would be great and speed up the investigation if you provide a small example or exact steps on how to reproduce the issue.
We are looking forward to your reply

[nanvel]

@Darleev reproducing the same error: https://github.com/nanvel/virtual_environments_2630/runs/1838322866

[jhheider]

I am having a similar problem with a Swift 5/Vapor 4 application (it's inexplicably having postgres connection failures when testing). Passing tests under 20210123.1 are now failing under 20210131.1 (and still passing locally on macOS using Xcode).

I can try and build a minimal test case if I get time.

[janpio]

Are you connecting to your databases with the hostnames localhost or ::1 (IPv6)?

We at Prisma suddenly could not connect to local databases running via Docker on our Github Actions workflows, and think it was caused by Docker not binding IPv6 by default any more since Docker Engine 20.10.2 which is part of the 20210131.1 image.

⚠️ Using 127.0.0.1 instead of localhost let us work around the problem temporarily.

Here is our description of the problem on our side: prisma/prisma#5499 Turned out Prisma was not doing the right thing when trying to resolve localhost and got both 127.0.0.1 and ::1 back, and stopped after a failing IPv6 instead of also trying the IPv4. After we fixed that, our tests are again passing with the current image.

Buuuut this is of course still a bug on the Docker side, or at least a unexpected breaking change. This seems to be one of the reports of this problem on the Docker side: moby/moby#41858 There are multiple issues and PRs from January that sounds related.

[jhheider]

Confirming that replacing localhost with 127.0.0.1 also fixes the issue for me.

[janpio]

And I think I have a rather minimal reproduction:
https://github.com/janpio/github-actions-docker-ipv6-repro

This repo starts MSSQL via Docker Compose, shows the problem with localhost not working, downgrades Docker, and then localhost does work with Prisma Client. As control it both time also connects to 127.0.0.1 which works with the old and new Docker version.

[nanvel]

Indeed, 127.0.0.1 solved my problem.

[janpio]

Ugh, that is just a workaround. Could you leave this issue open so someone from GH possible sees and can fix the issue to work properly again? Would appreciate it.

[kundkingan]

We are also experiencing issues with tests that worked last week and now they failing. Also rerun a workflow from 2 weeks ago that passed and they also failed.

• Ubuntu 18-04
• Docker with node:12-alpine
• Mysql 5.6
• Jest
• Localstack 11.4

[janpio]

Now that we upgrade our own software a bit, we can also explicitly show that with the updated Docker from that image, ::1 can not be used as a host to connect to our SQL Server database. After I downgrade Docker to an older version, it works again as it should: https://github.com/janpio/github-actions-docker-ipv6-repro/pull/1/checks?check_run_id=1922720249

[Darleev]

Hello,
As it was confirmed here, the issue is related to one of the latest docker updates. So, we need to wait until the issue is fixed on the docker side.
I'm going to close the issue.
If additional information or assistance is required, do not hesitate to contact us.