[SURE-7845] HTTP_PROXY does not work in gitjobs anymore

[Flou21]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When the Gitjob pod creates new Kubernetes jobs to download content from the Git repos, the jobs are missing the HTTP_PROXY environment variables.

Therefore, fleet does not work in situations where this is necessary.

Expected Behavior

This was not a problem in old versions of Fleet, but since the update to 0.9.0, the problem exists.

Expected behavior is that the jobs also inherit the HTTP_PROXY environment variables from the gitjob pod

Steps To Reproduce

No response

Environment

Fleet Version: 0.9.0
Rancher: 2.8.0
Kubernetes Distribution: RKE2
Version: v1.26.10+rke2r2

Logs

No response

Anything else?

I have found a workaround, although this workaround still contains a problem, but this should also be feasible.

The solution is based on the open github issue 348 in rancher/gitjob. (rancher/gitjob#348)

It is possible to revert the gitjob deployment in the cattle-fleet-system namespace to an older version. It is important that it is before version v0.1.85.
In this release the inheritance of the HTTP_PROXY env vars has been removed.
https://github.com/rancher/gitjob/pull/302/files#diff-d8502e4f5b09fbcbcb9b11e66e4e2d72c33d0b70b2def399245866b80fc9d975L272

I have used version v0.1.76, as this is the version in my Rancher v2.7.9 deployment.

In addition to the image tag, the command line arguments must also be modified. All previous arguments gitjob --gitjob-image rancher:VERSION must be completely removed and replaced with --tekton-image rancher/tekton-utils:v0.1.33.

All jobs based on this Gitjob deployment run without problems.

I still have a problem with the solution, which is that when a Rancher pod is restarted, Fleet is deployed, when this happens the manual change resets and Fleet stops working again.

So the bug is not really related to the Fleet repo itself, but to Gitjob or the update of the Gitjob dependency in the Fleet repository.

I hope this can help others who run into a similar problem, and draw some attention to issue 348 in the Gitjob repo.
With the Rancher 2.8.0 release some more people will run into this problem.

[kkaempf]

/cc @raulcabello

[raulcabello]

/backport v2.9.0

[aruiz14]

/forwardport v2.9.0

[raulcabello]

QA Template

Solution

Add HTTP_PROXY, HTTPS_PROXY, NO_PROXY env vars to all containers and initContainer of the Job created by gitjob

Testing

Verify that the HTTP_PROXY, HTTPS_PROXY, NO_PROXY env vars are set in all containers and initContainers of the Job created by gitjob if they are present in the values.yaml used when installing fleet.
This Job is created each time we create/modify a GitRepo or fleet detects changes in the git repository. This Job will then run fleet apply to create Bundles.

Create a proxy and verify that it works as expected.

Additional info

Proxy was tested as part of this. However, we missed this bug because we didn't check the Jobs generated by gitjob. Please make sure the Jobs also reach the proxy.

[sbulage]

---

System Information

• Rancher version: 2.8.3-rc2
• Fleet Version: 0.9.1-rc.5

One upstream cluster, one downstream cluster and one proxy server(used tiny-proxy)

---

Installation and Testing

Rancher helm install command

helm upgrade --install rancher rancher-latest/rancher  \
  --namespace cattle-system \
  --set hostname=<IP> \
  --set replicas=1 \
  --set bootstrapPassword=<PASSWD> \
  --set global.cattle.psp.enabled=false  \
  --version=v2.8.3-rc2 \
  --set proxy="<PROXY_IP>:<PORT_NO>/" \
  --devel

Testing

• Deployed GitRepo
• Checked that application installed correctly by checking it's deployment and pods etc.
• Verified that fleet-controller, fleet-pod, gitjob and job (generated by gitjob) has HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables available.
• Verified that the connection is established through the proxy server by checking tiny-proxy logs.

---

Screenshots

Fleet-Controller

Fleet-agent

Jobs

GitJob

[vincebrannon]

SURE-7845

[thehejik]

I verified also the upgrade scenario from Rancher 2.7.11 -> 2.8.3-rc2 (fleet:v0.8.2-> fleet:v0.9.1-rc.6) on squid with ENV variables: HTTP_PROXY, HTTPS_PROXY and NO_PROXY set in /etc/default/k3s. Rancher was installed/upgraded with appended --set proxy=myproxy:3128.

The env variables are still set after performing the upgrade on Pods fleet-controller and gitjob and also on fleet Job.

Note: For some reason the fleet-agent pod doesn't have the env vars set but it was like that even on 2.7.11 so I don't consider this as a problem.