Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running this via Github Action runner (Docker-In-Docker and host k3s cluster) results in wierd behavior #21

Open
samip5 opened this issue Nov 3, 2022 · 4 comments
Open

Running this via Github Action runner (Docker-In-Docker and host k3s cluster) results in wierd behavior #21

samip5 opened this issue Nov 3, 2022 · 4 comments

Comments

@samip5
Copy link

samip5 commented Nov 3, 2022

Environment:

Running Github Actions runners inside a existing k3s Kubernetes cluster inside CI namespace.

Somehow the readiness things keeps on yelling about no CI namespace? Any ideas, or should I make an issue upstream?

Preparing to install k3d into /usr/local/bin
k3d installed into /usr/local/bin/k3d
Run 'k3d --help' to see what you can do with it.
k3d version v5.4.6
k3s version v1.24.4-k3s1 (default)
Detected k3d-version::v5.4.6
Detected k3s-version::v1.22.12+k3s1
Detected k8s-version::v1.22.12
INFO[0000] Prep: Network                                
INFO[0001] Created network 'k3d-k3s-default'            
INFO[0001] Created image volume k3d-k3s-default-images  
INFO[0001] Starting new tools node...                   
INFO[0002] Creating node 'k3d-k3s-default-server-0'     
INFO[0003] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.4.6' 
INFO[0006] Pulling image 'rancher/k3s:v1.22.12-k3s1'    
INFO[0011] Starting Node 'k3d-k3s-default-tools'        
INFO[0018] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0020] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.4.6' 
INFO[0036] Using the k3d-tools node to gather environment information 
INFO[0038] HostIP: using network gateway 172.18.0.1 address 
INFO[0038] Starting cluster 'k3s-default'               
INFO[0038] Starting servers...                          
INFO[0038] Starting Node 'k3d-k3s-default-server-0'     
INFO[0048] All agents already running.                  
INFO[0048] Starting helpers...                          
INFO[0048] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0057] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap... 
INFO[0060] Cluster 'k3s-default' created successfully!  
INFO[0060] You can now use it like this:                
kubectl cluster-info
Waiting for cluster readiness
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
  Error from server (NotFound): namespaces "ci" not found
@samip5 samip5 changed the title Running this via Github Actio nrunner (Docker-In-Docker) seems wierd behavior Running this via Github Action runner (Docker-In-Docker) results in wierd behavior Nov 3, 2022
@samip5 samip5 changed the title Running this via Github Action runner (Docker-In-Docker) results in wierd behavior Running this via Github Action runner (Docker-In-Docker and host k3s cluster) results in wierd behavior Nov 3, 2022
@nolar
Copy link
Owner

nolar commented Nov 4, 2022

Hi. Thanks for reporting the issue. Can you please provide a repro with the setup that leads to this error?

@samip5
Copy link
Author

samip5 commented Nov 4, 2022 •
edited
Loading

Hi. Thanks for reporting the issue. Can you please provide a repro with the setup that leads to this error?

Hi. Repo as in what exactly? Workflow that failed for example: https://github.com/samipsolutions/helm-charts/actions/runs/3389687583/jobs/5633058490

I'm using: https://github.com/samipsolutions/helm-charts/blob/master/.github/workflows/charts-test.yaml and Github Action runner inside a k3s cluster using my own image ghcr.io/samipsolutions/containers/action-runner.

Related cluster config can be seen at: https://github.com/samip5/k8s-cluster/tree/da4826be25c71bc41ced365fc42a19f46cef8779/cluster/apps/ci

@nolar
Copy link
Owner

nolar commented Nov 4, 2022

Thanks. (I meant repro, not repo — something to reproduce the issue. Sorry for this.)

As for the issue: it is difficult to say without a deep dive into the code. It does not happen in a clean setup for me. So, I might assume it comes from some steps & commands before setup-k3d-k3s — i.e. it is a leftover from some previous config changes.

You can execute a command like kubectl config view or kubectl config view --minify immediately before the k3d setup step — to see the current config. Most likely, the namespace is set by the previous steps, but the cluster is changed by k3d cluster creation, and they are somehow mixed (my blind guesses).

The step has the skip-readiness option to disable this cycle (but the config will be corrupted anyway).

Or you can find a CLI option to disable this behaviour of K3d itself — and pass it via the k3d-args option.

Or you can execute kubectl config set-context --current --namespace=default to change the namespace.

I hope this helps. If not, please provide the output of kubectl config view (with all the secrets redacted/removed).

@samip5
Copy link
Author

samip5 commented Nov 13, 2022

Thanks. (I meant repro, not repo — something to reproduce the issue. Sorry for this.)

As for the issue: it is difficult to say without a deep dive into the code. It does not happen in a clean setup for me. So, I might assume it comes from some steps & commands before setup-k3d-k3s — i.e. it is a leftover from some previous config changes.

You can execute a command like kubectl config view or kubectl config view --minify immediately before the k3d setup step — to see the current config. Most likely, the namespace is set by the previous steps, but the cluster is changed by k3d cluster creation, and they are somehow mixed (my blind guesses).

The step has the skip-readiness option to disable this cycle (but the config will be corrupted anyway).

Or you can find a CLI option to disable this behaviour of K3d itself — and pass it via the k3d-args option.

Or you can execute kubectl config set-context --current --namespace=default to change the namespace.

I hope this helps. If not, please provide the output of kubectl config view (with all the secrets redacted/removed).

Before the setup k3d step, the env indeed has the things one would expect as it's already running inside a cluster.

$ kubectl config view && env
apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
KUBERNETES_SERVICE_PORT_HTTPS=443
DOCKER_TLS_VERIFY=1
GITHUB_WORKSPACE=/runner/_work/helm-charts/helm-charts
KUBERNETES_SERVICE_PORT=443
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_PORT_443_TCP_ADDR=10.96.141.216
PKG_CONFIG_PATH=/opt/hostedtoolcache/Python/3.10.8/x64/lib/pkgconfig
HOSTNAME=samipsolutions-runners-5nvxs-9xxwd
RUNNER_WORKDIR=/runner/_work
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_SERVICE_PORT_HTTPS=443
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_PORT_80_TCP=tcp://10.96.68.73:80
GITHUB_PATH=/runner/_work/_temp/_runner_file_commands/add_path_5937455f-5c9b-4396-97f0-b931f595b11c
GITHUB_ACTION=__run
DOCKERD_IN_RUNNER=false
GITHUB_RUN_NUMBER=62
RUNNER_NAME=samipsolutions-runners-5nvxs-9xxwd
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP_ADDR=10.96.86.157
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_PORT_80_TCP_PORT=80
GITHUB_URL=https://github.com/
RUNNER_ORG=samipsolutions
Python_ROOT_DIR=/opt/hostedtoolcache/Python/3.10.8/x64
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_PORT_443_TCP=tcp://10.96.141.216:443
ACTIONS_RUNNER_HOOK_JOB_COMPLETED=/etc/arc/hooks/job-completed.sh
GITHUB_TRIGGERING_ACTOR=samip5
pythonLocation=/opt/hostedtoolcache/Python/3.10.8/x64
GITHUB_REF_TYPE=branch
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_PORT=tcp://10.96.86.157:8443
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_SERVICE_PORT_METRICS_PORT=8443
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_SERVICE_PORT=80
***
DOCKER_ENABLED=true
GITHUB_ACTIONS=true
CT_CONFIG_DIR=/opt/hostedtoolcache/ct/v3.7.1/x86_64/etc
GITHUB_SHA=585a9c3158fd166d44f5f73862d44e75f2548a75
TZ=Europe/Helsinki
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_SERVICE_PORT=8443
GITHUB_REF=refs/heads/master
RUNNER_OS=Linux
GITHUB_REF_PROTECTED=true
RUNNER_STATUS_UPDATE_HOOK=false
HOME=/home/runner
GITHUB_API_URL=https://api.github.com/
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_PORT_443_TCP_PROTO=tcp
RUNNER_TRACKING_ID=github_75ba4243-4f59-4572-a1cd-d6c99dfb9d75
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_PORT=tcp://10.96.141.216:443
RUNNER_ARCH=X64
VIRTUAL_ENV=/opt/hostedtoolcache/ct/v3.7.1/x86_64/venv
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_SERVICE_HOST=10.96.86.157
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_PORT_80_TCP_ADDR=10.96.68.73
RUNNER_TEMP=/runner/_work/_temp
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_SERVICE_HOST=10.96.68.73
GITHUB_STATE=/runner/_work/_temp/_runner_file_commands/save_state_5937455f-5c9b-4396-97f0-b931f595b11c
GITHUB_ENV=/runner/_work/_temp/_runner_file_commands/set_env_5937455f-5c9b-4396-97f0-b931f595b11c
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_SERVICE_PORT=443
GITHUB_EVENT_PATH=/runner/_work/_temp/_github_workflow/event.json
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP=tcp://10.96.86.157:8443
GITHUB_EVENT_NAME=workflow_dispatch
RUNNER_ASSETS_DIR=/runnertmp
GITHUB_RUN_ID=3453262643
HELM_VERSION=3.9.2
GITHUB_STEP_SUMMARY=/runner/_work/_temp/_runner_file_commands/step_summary_5937455f-5c9b-4396-97f0-b931f595b11c
GITHUB_ACTOR=samip5
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_PORT=tcp://10.96.68.73:80
RUNNER_EPHEMERAL=true
GITHUB_RUN_ATTEMPT=1
GITHUB_GRAPHQL_URL=https://api.github.com/graphql
RUNNER_GROUP=
GITHUB_SERVER_URL=https://github.com/
DOCKER_CERT_PATH=/certs/client
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_SERVICE_PORT_HTTP=80
SHLVL=4
KUBERNETES_PORT_443_TCP_PROTO=tcp
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_PORT_443_TCP_PORT=443
RUNNER_TOOL_CACHE=/opt/hostedtoolcache
Python3_ROOT_DIR=/opt/hostedtoolcache/Python/3.10.8/x64
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
GITHUB_REF_NAME=master
GITHUB_JOB=install-chart
LD_LIBRARY_PATH=/opt/hostedtoolcache/Python/3.10.8/x64/lib
GITHUB_ACTIONS_RUNNER_CONTROLLER_WEBHOOK_SERVICE_HOST=10.96.141.216
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP_PORT=8443
GITHUB_ACTIONS_RUNNER_CONTROLLER_GITHUB_WEBHOOK_SERVER_PORT_80_TCP_PROTO=tcp
ACTIONS_RUNNER_HOOK_JOB_STARTED=/etc/arc/hooks/job-started.sh
DOCKER_HOST=tcp://localhost:2376
GITHUB_REPOSITORY=samipsolutions/helm-charts
Python2_ROOT_DIR=/opt/hostedtoolcache/Python/3.10.8/x64
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
GITHUB_RETENTION_DAYS=90
RUNNER_WORKSPACE=/runner/_work/helm-charts
GITHUB_ACTIONS_RUNNER_CONTROLLER_METRICS_SERVICE_PORT_8443_TCP_PROTO=tcp
GITHUB_ACTION_REPOSITORY=
PATH=/opt/hostedtoolcache/ct/v3.7.1/x86_64/venv/bin:/opt/hostedtoolcache/ct/v3.7.1/x86_64:/opt/hostedtoolcache/Python/3.10.8/x64/bin:/opt/hostedtoolcache/Python/3.10.8/x64:/opt/hostedtoolcache/kubectl/1.22.12/x64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/runner/.local/bin
GITHUB_BASE_REF=
CI=true
ImageOS=ubuntu22
GITHUB_REPOSITORY_OWNER=samipsolutions
GITHUB_HEAD_REF=
GITHUB_ACTION_REF=
RUNNER_LABELS=self-hosted,X64
RUNNER_ENTERPRISE=
GITHUB_WORKFLOW=Pull Request: Validate
DEBIAN_FRONTEND=noninteractive
GITHUB_OUTPUT=/runner/_work/_temp/_runner_file_commands/set_output_5937455f-5c9b-4396-97f0-b931f595b11c
OLDPWD=/
_=/usr/bin/env

@t1m0thyj t1m0thyj mentioned this issue May 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nolar @samip5