Local DNS issue #763
Comments
|
Can you share a way to reproduce this? Also please share the output of the following commands when it's in a broken state: kubectl describe -n kube-system svc/kube-dns
kubectl logs -n kube-system svc/kube-dns
# run this and find the name of the coredns pod, e.g. "coredns-687f7d69ff-qvwph"
kubectl get pod -A
# substitute here
kubectl describe -n kube-system pod/coredns-XYZ
kubectl logs -n kube-system pod/coredns-XYZ |
You can use fluxcd to reproduce the error. Here are the command output ❯ kubectl describe -n kube-system svc/kube-dns
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=CoreDNS
objectset.rio.cattle.io/hash=90462e4597bdcacaa7038840f786450ac707bfa2
Annotations: objectset.rio.cattle.io/applied:
H4sIAAAAAAAA/4yRQY/TMBCF/wqasxOSJm1SSxzQ7gUhoZUWuCAOE2eWmiS25ZkWoSr/HbnNikJV2JutefPpvTdHwGA/U2TrHWg4lKBgsK4HDY8UD9YQKJhIsEdB0EdA57ygWO84fX...
objectset.rio.cattle.io/id:
objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
objectset.rio.cattle.io/owner-name: orb-coredns
objectset.rio.cattle.io/owner-namespace: kube-system
prometheus.io/port: 9153
prometheus.io/scrape: true
Selector: k8s-app=kube-dns
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 192.168.194.138
IPs: 192.168.194.138
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 192.168.194.5:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 192.168.194.5:53
Port: metrics 9153/TCP
TargetPort: 9153/TCP
Endpoints: 192.168.194.5:9153
Session Affinity: None
Events: <none>❯ kubectl describe -n kube-system pod/coredns-687f7d69ff-tmscz
Name: coredns-687f7d69ff-tmscz
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: coredns
Node: orbstack/198.19.249.2
Start Time: Mon, 13 Nov 2023 15:46:47 +1100
Labels: k8s-app=kube-dns
pod-template-hash=687f7d69ff
Annotations: <none>
Status: Running
IP: 192.168.194.5
IPs:
IP: 192.168.194.5
IP: fd07:b51a:cc66:a::5
Controlled By: ReplicaSet/coredns-687f7d69ff
Containers:
coredns:
Container ID: docker://9ad490cd7e154ab0f1129b2b55177750bca5a7464409367eeeace2be881f613c
Image: rancher/mirrored-coredns-coredns:1.10.1
Image ID: docker-pullable://rancher/mirrored-coredns-coredns@sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Running
Started: Mon, 13 Nov 2023 15:46:48 +1100
Ready: True
Restart Count: 0
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/etc/coredns/custom from custom-config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w4qtf (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
custom-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns-custom
Optional: true
kube-api-access-w4qtf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=kube-dns
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m11s default-scheduler Successfully assigned kube-system/coredns-687f7d69ff-tmscz to orbstack
Normal Pulled 2m11s kubelet Container image "rancher/mirrored-coredns-coredns:1.10.1" already present on machine
Normal Created 2m11s kubelet Created container coredns
Normal Started 2m11s kubelet Started container coredns❯ kubectl logs -n kube-system pod/coredns-687f7d69ff-tmscz
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
.:53
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] plugin/reload: Running configuration SHA512 = 2bf9b838333e20143c631516d1a474474b5ddb089d94e66f7831747dc364e3b7a92a481402ecd77264fd9b27c6abeea0e080f04d6bf43e10148b366e3bdd0e19
CoreDNS-1.10.1
linux/arm64, go1.20, 055b2c3
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server |
|
@ylbeethoven Are these logs from when the issue is actively occurring, i.e. while you're getting Also, please provide exact steps to reproduce the issue with fluxcd. |
Yes
The error shows on GitRepository object. The steps are pretty simple if you have the repo set up... Bascially fluxCD I am sorry I can't put a deployment key on this public space but I am happy to show you a demo. I will DM you on mastadon if you are keen. |
|
Can you try restarting OrbStack, immediately running apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
k8s-app: kube-dns
k8slens-edit-resource-version: v1
kubernetes.io/name: CoreDNS
name: coredns
namespace: kube-system
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
k8s-app: kube-dns
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
k8s-app: kube-dns
spec:
containers:
- args:
- -conf
- /etc/coredns/Corefile
image: rancher/mirrored-coredns-coredns:1.10.1
imagePullPolicy: IfNotPresent
name: coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
resources:
limits:
memory: 340Mi
requests:
cpu: 100m
memory: 70Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/coredns
name: config-volume
readOnly: true
- mountPath: /etc/coredns/custom
name: custom-config-volume
readOnly: true
dnsPolicy: Default
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: coredns
serviceAccountName: coredns
terminationGracePeriodSeconds: 30
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
topologySpreadConstraints:
- labelSelector:
matchLabels:
k8s-app: kube-dns
maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
volumes:
- configMap:
defaultMode: 420
items:
- key: Corefile
path: Corefile
- key: NodeHosts
path: NodeHosts
name: coredns
name: config-volume
- configMap:
defaultMode: 420
name: coredns-custom
optional: true
name: custom-config-volume
|
|
emm... this is weird. This fix works only once. (first try) After deleting and starting k8s, it did not work again. |
|
ok, I think I understand how to reproduce it. The fix seems to ONLY work the first time orbstack starts. In order to make it work, I need to
The fix does not work when I use |
|
Thanks for testing! Fixed for the next version. |
|
Released in v1.2.0 Canary 1. |
|
Released in v1.2.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Several pods/services are failing when running locally. I receive the following error:
failed to checkout and determine revision: unable to clone 'https://repo1.dso.mil/big-bang/bigbang.git': Get "https://repo1.dso.mil/big-bang/bigbang.git/info/refs?service=git-upload-pack": dial tcp: lookup repo1.dso.mil on 192.168.194.138:53: read udp 192.168.194.4:55533->192.168.194.138:53: read: connection refusedThe git repo is available, but the local UDP traffic is being refused.
To Reproduce
No response
Expected behavior
No response
Diagnostic report (required)
OrbStack info:
Version: 1.0.1
Commit: 0d4cdcf185489f992cf8e3187884f11f4feab1aa (v1.0.1)
System info:
macOS: 14.0 (23A344)
CPU: arm64, 10 cores
CPU model: Apple M1 Pro
Model: MacBookPro18,1
Memory: 32 GiB
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2023-10-31T13-23-46.810846Z.zip
Screenshots and additional context (optional)
No response
The text was updated successfully, but these errors were encountered: