Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 on GitHub-hosted runners #668

Closed
BrightRan opened this issue Apr 3, 2020 · 30 comments
Closed

IPv6 on GitHub-hosted runners #668

BrightRan opened this issue Apr 3, 2020 · 30 comments
Labels
Area: Image administration investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu

Comments

@BrightRan
Copy link

BrightRan commented Apr 3, 2020

This issue is transferred from here.

In this ticket, the user is trying to reach the IPv6 address of Google by running the command "curl -6 https://www.google.com" on the GitHub-hosted Ubuntu runner, but it always failed to connect and returned the message "Couldn't connect to server".
The command "curl https://www.google.com" or "curl -4 https://www.google.com" can work fine to reach the IPv4 address.

I also tested directly ping the IPv6 address of Google via the command "ping6 ipv6.google.com", it also failed to connect and returned the message "connect: Network is unreachable".

To check whether the current Linux kernel supports IPv6 via the command,

[ -f /proc/net/if_inet6 ] && echo 'IPv6 ready system!' || echo 'No IPv6 support found! Compile the kernel!!'

It returns "IPv6 ready system!", looks like IPv6 is supported.
Then to check whether the IPv6 module has loaded,

lsmod | grep -qw ipv6 && echo "IPv6 kernel driver loaded and configured." || echo "IPv6 not configured and/or driver loaded on the system."

It returns "IPv6 not configured and/or driver loaded on the system.", looks like the IPv6 module is not loaded.
Load the IPv6 module and check again.

modprobe ipv6
lsmod | grep -qw ipv6 && echo "IPv6 kernel driver loaded and configured." || echo "IPv6 not configured and/or driver loaded on the system."

The returned result still shows that the IPv6 module is not loaded. Looks like this should be the reason for failed to reach the IPv6 address.

Why it can't load the IPv6 module on the GitHub-hosted runner? Whether this is an expected behavior for some limitations set on the hosted runners?

@andy-mishechkin andy-mishechkin added investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu and removed needs triage labels Apr 3, 2020
@miketimofeev
Copy link
Contributor

@BrightRan looks like the images itself are ok.
@alepauly do we have ipv6 configured on the environment side?

@airtower-luna
Copy link

According to the Docker documentation the images would still need to enable IPv6 for Docker containers, even if the VMs start to support it.

@alepauly
Copy link
Contributor

alepauly commented May 5, 2020

Unfortunately we can't support this at this time due to infrastructure constraints.

@lzhecheng
Copy link

Unfortunately we can't support this at this time due to infrastructure constraints.

Is it in the roadmap? I found no opening issues on IPv6 support. Thanks.

@miyurusankalpa
Copy link

GitHub IP address list now has IPv6 address for GitHub Actions.
https://api.github.com/meta

@oscartbeaumont
Copy link

As of a week ago, I could not get IPv6 working in GitHub Actions but let's hope them listing those address's means they are about to release support for it.

@pavelkim
Copy link

pavelkim commented Jan 9, 2021

Tested today.
Under both ubuntu-latest and ubuntu-20.04:

curl -vv --ipv6 "[2607:f8b0:4002:c03::93]:80"

* Rebuilt URL to: [2607:f8b0:4002:c03::93]:80/
*   Trying 2607:f8b0:4002:c03::93...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4002:c03::93: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server

ip route sh

default via 10.1.0.1 dev eth0 proto dhcp src 10.1.0.4 metric 100 
10.1.0.0/16 dev eth0 proto kernel scope link src 10.1.0.4 
168.63.129.16 via 10.1.0.1 dev eth0 proto dhcp src 10.1.0.4 metric 100 
169.254.169.254 via 10.1.0.1 dev eth0 proto dhcp src 10.1.0.4 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

ip addr sh

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:3a:c6:7b:12 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.4/16 brd 10.1.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:3aff:fec6:7b12/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:79:06:ee:ce brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

No IPv6 routes so far.

@jcamiel
Copy link

jcamiel commented Oct 23, 2023

Hi guys, I was trying to add integration test on Hurl a CLI HTTP client based on libcurl.

What does surprise me is that I've been able to make a GET request using curl and ipv6 from a GitHub-hosted macOS runner:

curl on macOS is available under:

  • /usr/bin/curl 8.1.2
  • /usr/local/opt/curl/bin/curl 8.3.0

Both binary have IPv6 feature enabled but only curl 8.3.0 is able to GET https://google.com with --ipv6 on GitHub macOS runners.

$ /usr/local/opt/curl/bin/curl --version
curl 8.3.0 (x86_64-apple-darwin21.6.0) libcurl/8.3.0 (SecureTransport) OpenSSL/3.1.2 zlib/1.2.11 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libssh2/1.11.0 nghttp2/1.56.0 librtmp/2.3 OpenLDAP/2.6.6
Release-Date: 2023-09-13
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
$ /usr/bin/curl --version
curl 8.1.2 (x86_64-apple-darwin21.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

And test with --ipv6 against google.com:

$/usr/local/opt/curl/bin/curl --ipv6 --verbose https://google.com/
...
Connected to google.com (::ffff:142.250.115.101) port 443
...
$/usr/bin/curl --ipv6 --verbose https://google.com/
curl: (7) Couldn't connect to server

So, it seems there is kind of IPv6 support inside GitHub infra. I'm surprised to see that between curl 8.1.2 and curl 8.3.0 there is a difference of usage with --ipv6 option but I can't explain it.

@cross
Copy link

cross commented Oct 25, 2023

You'll note however, that's not really IPv6. It's an IPv4-mapped address. If it were really IPv6, it would look like this, from my desktop Macbook:

$ curl -V
curl 8.1.2 (x86_64-apple-darwin23.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1
[...]
$ curl --ipv6 --verbose https://google.com/
*   Trying [2607:f8b0:4002:c2c::65]:443...
* Connected to google.com (2607:f8b0:4002:c2c::65) port 443 (#0)
[...]

@i18n-now
Copy link

image use this
  jobs:
    ipv6:
      runs-on: ubuntu-latest
      steps:
        - name: Setup WARP
          uses: fscarmen/warp-on-actions@v2
        - name: ipv6
          run: |
            curl -m 9 --ipv6 --verbose https://google.com
            curl -m 9 -6 https://ifconfig.co

titanism added a commit to titanism/upptime that referenced this issue Jan 7, 2024
Recently we submitted a PR for adding `ipv6: true` to the yaml config to check IPv6.  However GitHub actions does not yet support IPv6 unfortunately.  This PR is necessary as it uses Cloudflare WARP network configuration on the runner to add IPv6 support.  See <actions/runner-images#668 (comment)> for more insight.
titanism added a commit to forwardemail/status.forwardemail.net that referenced this issue Jan 7, 2024
Mischback added a commit to Mischback/mischback.de that referenced this issue Mar 19, 2024
Still not working out of the box.

Research actions/runner-images#668 (comment)
alpeb added a commit to linkerd/linkerd2-proxy-init that referenced this issue Mar 28, 2024
## Flags Changes

This adds the proxy-init flag `--iptables-mode` (with possible values `legacy` and `nft`), which supersedes `--firewal-bin-path` and `firewall-save-bin-path` (which still remain supported).
Also the `--ipv6` flag has been added (default `true`).

After the set of rules run via iptables are processed, if `--ipv6` is true (which is the default), the same set of rules will be run via ip6tables.

Analog changes were applied to linkerd-cni as well.

## Backwards-Compatibility

This is backwards-compatible with older control planes and upcoming control planes.
If `--ipv6` is not passed (and thus defaults to true), this doesn't impact operation even if the cluster doesn't support IPv6; the ip6tables rules are applied but they're innocuous.
OTOH if there's no kernel support for IPv6 (which is the case for github runners*) then the ip6tables command will fail but we'll just log the failure and not fail the linkerd-init container (nor the `add` command for linkerd-cni). This avoids having to explicitly set `--ipv6=false`, but it can be set if the user is aware of such limitations and wants to get rid of the errors.

## Testing Improvements

The cni-plugin-integration workflow has been simplified by using a matrix strategy, and enhanced by parameterizing the iptables-mode config.

## Linkerd IPv6 Support

This allows routing IPv6 traffic to the proxy, but is just the first step towards IPv6/dual-stack support. Control plane and proxy changes will come up next.

## (*) Github Runners IPv6 Support

Even though `modinfo` signals support for IPv6, `ip6tables` commands throw modprobe errors. Indeed, according to actions/runner-images#668 support is not there yet.
Also, according to actions/runner#3138 there are issues with hosted runners as well, but that might not affect us if we still expose an IPv4 interface to interact with github. Something to take into account when we get to IPv6 integration testing.
undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 29, 2024
This commit introduces the `force-ipv4` GitHub action to address
connectivity issues caused by the lack of IPv6 support in GitHub
runners. Details:
- actions/runner#3138
- actions/runner-images#668

This change solves connection problems when Node's `fetch` API fails due
to `UND_ERR_CONNECT_TIMEOUT` errors. Details:
- actions/runner-images#9540
- actions/runner#3213

This action disables IPv6 at the system level, ensuring all outging
requests use IPv4. Resolving connectivity issues when running external
URL checks and Docker build checks.

This solution is a temporary workaround until GitHub runners support
IPv6 or Node `fetch` API has a working solution such as Happy Eyeball.
Detais:
- nodejs/node#41625
- nodejs/undici#1531
@undergroundwires
Copy link

Here's my workaround (open-source and documented) that I hope that can help you too:

After days of research and trial/error, this is how I got this working:

  1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
  2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
  3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd

undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 30, 2024
This commit upgrades Node.js version to v20.x in CI/CD environment.

Previously used Node 18.x is moving towards end-of-life, with a planned
date of 2025-04-30. In contrast, Node 20.x has been offering long-term
support (LTS) since 2023-10-24. This makes Node 20.x a stable and
recommended version for production environments.

This commit also configures `actions/setup-node` with the
`check-latest` flag to always use the latest Node 20.x version, keeping
CI/CD setup up-to-date with minimal maintenance.
Details:
- actions/setup-node#165
- actions/setup-node#160

Using Node 20.x in CI/CD environments provides better compatibility with
Electron v29.0 which moves to Node 20.x.
Details:
- electron/electron#40343

This upgrade improves network connection handling in CI/CD pipelines
(where issues occur due to GitHub runners not supporting IPv6).
Details:
- actions/runner#3138
- actions/runner-images#668
- actions/runner#3213
- actions/runner-images#9540

Node 20.x adopts the Happy Eyeballs algorithm for improved IPv6
connectivity.
- nodejs/node#40702
- nodejs/node#41625
- nodejs/node#44731

This mitigates issues like `UND_ERR_CONNECT_TIMEOUT` and localhost DNS
resolution in CI/CD environments:
Details:
- nodejs/node#40537
- actions/runner#3213
- actions/runner-images#9540

Node 20 introduces `setDefaultAutoSelectFamily`, a global function from
Node 19.4.0, enabling better IPv4 support, especially in environments
with limited or problematic IPv6 support.
Details:
- nodejs/node#45777

Node 20.x defaults to the new `autoSelectFamily`, improving network
connection reliability in GitHub runners lacking full IPv6 support.
Details:
- nodejs/node#46790
undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 30, 2024
This commit upgrades Node.js version to v20.x in CI/CD environment.

Previously used Node 18.x is moving towards end-of-life, with a planned
date of 2025-04-30. In contrast, Node 20.x has been offering long-term
support (LTS) since 2023-10-24. This makes Node 20.x a stable and
recommended version for production environments.

This commit also configures `actions/setup-node` with the
`check-latest` flag to always use the latest Node 20.x version, keeping
CI/CD setup up-to-date with minimal maintenance.
Details:
- actions/setup-node#165
- actions/setup-node#160

Using Node 20.x in CI/CD environments provides better compatibility with
Electron v29.0 which moves to Node 20.x.
Details:
- electron/electron#40343

This upgrade improves network connection handling in CI/CD pipelines
(where issues occur due to GitHub runners not supporting IPv6).
Details:
- actions/runner#3138
- actions/runner-images#668
- actions/runner#3213
- actions/runner-images#9540

Node 20.x adopts the Happy Eyeballs algorithm for improved IPv6
connectivity.
- nodejs/node#40702
- nodejs/node#41625
- nodejs/node#44731

This mitigates issues like `UND_ERR_CONNECT_TIMEOUT` and localhost DNS
resolution in CI/CD environments:
Details:
- nodejs/node#40537
- actions/runner#3213
- actions/runner-images#9540

Node 20 introduces `setDefaultAutoSelectFamily`, a global function from
Node 19.4.0, enabling better IPv4 support, especially in environments
with limited or problematic IPv6 support.
Details:
- nodejs/node#45777

Node 20.x defaults to the new `autoSelectFamily`, improving network
connection reliability in GitHub runners lacking full IPv6 support.
Details:
- nodejs/node#46790
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Image administration investigate Collect additional information, like space on disk, other tool incompatibilities etc. OS: Ubuntu
Projects
None yet
Development

No branches or pull requests