set NO_PROXY env var for containerd #2071

alexandre-allard · 2019-11-25T08:11:58Z

Component: salt

Context: Containerd is failing to pull images when a http(s) proxy is set system wide through environment variables in /etc/environment

Summary: Set NO_PROXY environment variable with control, workload plane and K8s internal
networks in containerd systemd unit file, to avoid using system wide defined HTTP(S)
proxy, if any, when trying to pull resources from metalk8s registry.

Acceptance criteria: Be able to install metalk8s with a proxy configured following the doc https://metal-k8s.readthedocs.io/en/latest/quickstart/setup.html#proxies

Closes: #2052

bert-e · 2019-11-25T08:12:00Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2019-11-25T08:12:08Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/2052-set-no-proxy-for-internal-resources into
development/2.4
w/2.5/improvement/2052-set-no-proxy-for-internal-resources will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2019-11-25T08:12:10Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

salt/metalk8s/container-engine/containerd/installed.sls

bert-e · 2019-11-25T10:24:00Z

History mismatch

Merge commit #bb82b0b81f942754794039ac08639127053abae8 on the integration branch
w/2.5/improvement/2052-set-no-proxy-for-internal-resources is merging a branch which is neither the current
branch improvement/2052-set-no-proxy-for-internal-resources nor the development branch
development/2.5.

It is likely due to a rebase of the branch improvement/2052-set-no-proxy-for-internal-resources and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

gdemonet

Some questions...

salt/metalk8s/container-engine/containerd/files/50-metalk8s.conf.j2

gdemonet · 2019-11-25T16:55:27Z

salt/metalk8s/container-engine/containerd/installed.sls

    - user: root
    - group: root
    - mode: 0644
    - makedirs: true
    - dir_mode: 0755
+    - context:
+      environment: >-
+        NO_PROXY=127.0.0.1,localhost,.svc,.default,.local,{{


I don't know the allowed syntax of NO_PROXY, but this will probably won't be enough.

First question: what's the exact symptom? When we have a HTTP_PROXY set up, is containerd failing on retrieval of new container images? Are containers also failing to reach other services/cluster IPs?

Second question: if containerd uses the proxy for container networking as well, then we'll need a lot more than this (I don't even think we could be exhaustive) - all Service DNS names can use .svc.<namespace>, or .svc.<namespace>.cluster, or .svc.<namespace>.cluster.local, or even remove this whole suffix and only use the Service actual name. We naturally cannot specify all of them here, so maybe we could instead try to resolve names with CoreDNS before trying to go through the proxy?

During my tests, only containerd was failing to retrieve container images.
I don't think that these environment variables are used inside running containers.

OK, then I guess we don't need all those rules, just the registry address, no? Something like metalk8s-registry-from-config.invalid (or its actual IP:port, not sure)?

I seem to remember from the Slack chats with TS, the processes inside containers were also using the proxy variables, then failing to connect to e.g. 10.96.0.1 or whatnot. Please validate this more in-depth...

In any case:

Where does .default come from?

Where does .local come from?

Indeed, what about the registry 'DNS name'?

Ebaneck

LGTM, just a few questions that need clarifications

docs/quickstart/bootstrap.rst

salt/metalk8s/container-engine/containerd/installed.sls

alexandre-allard · 2019-12-10T16:44:31Z

/reset

bert-e · 2019-12-10T16:44:39Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2019-12-10T16:44:47Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/2052-set-no-proxy-for-internal-resources into
development/2.4
w/2.5/improvement/2052-set-no-proxy-for-internal-resources will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2019-12-10T16:44:49Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

gdemonet

Small question, looks good to me otherwise, can we open a debt ticket for adding some test behind a proxy?

salt/metalk8s/repo/redhat.sls

alexandre-allard · 2019-12-10T19:20:20Z

Small question, looks good to me otherwise, can we open a debt ticket for adding some test behind a proxy?

@gdemonet Yup, easily doable.
Since we don't need any image from the internet, we can set a fake proxy pointing to nothing (e.g. http://10.10.10.10:1234).
I'll open an issue to track this.

alexandre-allard · 2019-12-11T07:54:01Z

/approve

salt/metalk8s/container-engine/containerd/installed.sls

docs/quickstart/bootstrap.rst

salt/metalk8s/container-engine/containerd/installed.sls

We need to disable proxy usage on metalk8s internal repositories, otherwise we can't reach them if any http(s)_proxy variable is set. Refs: #2052

Refs: #2052

Set HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables in containerd systemd unit file, Refs: #2052

Move the documentation for proxies to the bootstrap configuration section and update it with the new way to setup proxies. Refs: #2052

Set the containerd configuration generation as a requisite for containerd package installation, to avoid having to restart the service just after its installation.

bert-e · 2020-01-03T14:49:37Z

History mismatch

Merge commit #229e41a419210b597e85957825b66dc7fd18a5ef on the integration branch
w/2.5/improvement/2052-set-no-proxy-for-internal-resources is merging a branch which is neither the current
branch improvement/2052-set-no-proxy-for-internal-resources nor the development branch
development/2.5.

It is likely due to a rebase of the branch improvement/2052-set-no-proxy-for-internal-resources and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: approve

bert-e · 2020-01-03T14:51:28Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: approve

bert-e · 2020-01-03T14:51:35Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/2052-set-no-proxy-for-internal-resources into
development/2.4
w/2.5/improvement/2052-set-no-proxy-for-internal-resources will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e · 2020-01-03T14:51:37Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

alexandre-allard · 2020-01-06T09:06:37Z

/approve

Ebaneck

Looks good to go!

bert-e · 2020-01-06T09:09:03Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.4
✔️ development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-01-06T09:09:10Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.4
✔️ development/2.5

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

alexandre-allard requested a review from a team as a code owner November 25, 2019 08:11

alexandre-allard changed the title ~~salt: set NO_PROXY env var for containerd~~ set NO_PROXY env var for containerd Nov 25, 2019

sayf-eddine-scality reviewed Nov 25, 2019

View reviewed changes

salt/metalk8s/container-engine/containerd/installed.sls Outdated Show resolved Hide resolved

alexandre-allard force-pushed the improvement/2052-set-no-proxy-for-internal-resources branch from bb82b0b to 7081d24 Compare November 25, 2019 10:23

gdemonet reviewed Nov 25, 2019

View reviewed changes

alexandre-allard force-pushed the improvement/2052-set-no-proxy-for-internal-resources branch 5 times, most recently from 298effa to 659c742 Compare December 10, 2019 14:40

Ebaneck reviewed Dec 10, 2019

View reviewed changes

docs/quickstart/bootstrap.rst Outdated Show resolved Hide resolved

salt/metalk8s/container-engine/containerd/installed.sls Outdated Show resolved Hide resolved

salt/metalk8s/container-engine/containerd/installed.sls Outdated Show resolved Hide resolved

alexandre-allard force-pushed the improvement/2052-set-no-proxy-for-internal-resources branch from 659c742 to 8c7461e Compare December 10, 2019 16:44

gdemonet reviewed Dec 10, 2019

View reviewed changes

salt/metalk8s/repo/redhat.sls Show resolved Hide resolved

alexandre-allard force-pushed the improvement/2052-set-no-proxy-for-internal-resources branch from 8c7461e to fd9e83b Compare December 10, 2019 23:06

Ebaneck previously approved these changes Dec 11, 2019

View reviewed changes

gdemonet previously approved these changes Dec 11, 2019

View reviewed changes

TeddyAndrieux reviewed Dec 11, 2019

View reviewed changes

salt/metalk8s/container-engine/containerd/installed.sls Show resolved Hide resolved

NicolasT reviewed Dec 13, 2019

View reviewed changes

docs/quickstart/bootstrap.rst Outdated Show resolved Hide resolved

NicolasT reviewed Dec 13, 2019

View reviewed changes

docs/quickstart/bootstrap.rst Outdated Show resolved Hide resolved

NicolasT reviewed Dec 13, 2019

View reviewed changes

salt/metalk8s/container-engine/containerd/installed.sls Outdated Show resolved Hide resolved

scality deleted a comment from bert-e Dec 18, 2019

alexandre-allard added 5 commits January 3, 2020 15:49

salt: add proxy none to metalk8s repositories

a0ebdfa

We need to disable proxy usage on metalk8s internal repositories, otherwise we can't reach them if any http(s)_proxy variable is set. Refs: #2052

salt: add proxies configuration in pillar

cad45e8

Refs: #2052

salt: use proxies from pillar in containerd unit

e4be30d

Set HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables in containerd systemd unit file, Refs: #2052

docs: update documentation regarding proxies

47baa8d

Move the documentation for proxies to the bootstrap configuration section and update it with the new way to setup proxies. Refs: #2052

salt: generate containerd configuration before install

a737ace

Set the containerd configuration generation as a requisite for containerd package installation, to avoid having to restart the service just after its installation.

alexandre-allard force-pushed the improvement/2052-set-no-proxy-for-internal-resources branch from b7cbbc3 to a737ace Compare January 3, 2020 14:49

scality deleted a comment from bert-e Jan 3, 2020

Ebaneck approved these changes Jan 6, 2020

View reviewed changes

bert-e merged commit a737ace into development/2.4 Jan 6, 2020

bert-e deleted the improvement/2052-set-no-proxy-for-internal-resources branch January 6, 2020 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set NO_PROXY env var for containerd #2071

set NO_PROXY env var for containerd #2071

alexandre-allard commented Nov 25, 2019 •

edited

Loading

bert-e commented Nov 25, 2019

bert-e commented Nov 25, 2019

bert-e commented Nov 25, 2019

bert-e commented Nov 25, 2019

gdemonet left a comment

gdemonet Nov 25, 2019

alexandre-allard Nov 26, 2019 •

edited

Loading

gdemonet Nov 26, 2019

NicolasT Nov 27, 2019

Ebaneck left a comment

alexandre-allard commented Dec 10, 2019

bert-e commented Dec 10, 2019

bert-e commented Dec 10, 2019

bert-e commented Dec 10, 2019

gdemonet left a comment

alexandre-allard commented Dec 10, 2019 •

edited

Loading

alexandre-allard commented Dec 11, 2019

bert-e commented Jan 3, 2020

bert-e commented Jan 3, 2020

bert-e commented Jan 3, 2020

bert-e commented Jan 3, 2020

alexandre-allard commented Jan 6, 2020

Ebaneck left a comment

bert-e commented Jan 6, 2020

bert-e commented Jan 6, 2020

set NO_PROXY env var for containerd #2071

set NO_PROXY env var for containerd #2071

Conversation

alexandre-allard commented Nov 25, 2019 • edited Loading

bert-e commented Nov 25, 2019

Hello alexandre-allard-scality,

bert-e commented Nov 25, 2019

Integration data created

bert-e commented Nov 25, 2019

Waiting for approval

bert-e commented Nov 25, 2019

History mismatch

gdemonet left a comment

Choose a reason for hiding this comment

gdemonet Nov 25, 2019

Choose a reason for hiding this comment

alexandre-allard Nov 26, 2019 • edited Loading

Choose a reason for hiding this comment

gdemonet Nov 26, 2019

Choose a reason for hiding this comment

NicolasT Nov 27, 2019

Choose a reason for hiding this comment

Ebaneck left a comment

Choose a reason for hiding this comment

alexandre-allard commented Dec 10, 2019

bert-e commented Dec 10, 2019

Reset complete

bert-e commented Dec 10, 2019

Integration data created

bert-e commented Dec 10, 2019

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

alexandre-allard commented Dec 10, 2019 • edited Loading

alexandre-allard commented Dec 11, 2019

bert-e commented Jan 3, 2020

History mismatch

bert-e commented Jan 3, 2020

Reset complete

bert-e commented Jan 3, 2020

Integration data created

bert-e commented Jan 3, 2020

Waiting for approval

alexandre-allard commented Jan 6, 2020

Ebaneck left a comment

Choose a reason for hiding this comment

bert-e commented Jan 6, 2020

In the queue

bert-e commented Jan 6, 2020

alexandre-allard commented Nov 25, 2019 •

edited

Loading

alexandre-allard Nov 26, 2019 •

edited

Loading

alexandre-allard commented Dec 10, 2019 •

edited

Loading