Fail to deploy KEDA - read udp x.x.x.x:46084->y.y.y.y:53: i/o timeout on MacOS (Apple Silicon) #3377

mhcastro · 2022-10-10T16:19:54Z

General information

OS: macOS (Apple Silicon)
Did you run crc setup before starting it? Yes.
Running CRC on: Laptop

CRC version

CRC version: 2.9.0+589ab2cd
OpenShift version: 4.11.3
Podman version: 4.2.0

CRC status

DEBU CRC version: 2.9.0+589ab2cd
DEBU OpenShift version: 4.11.3
DEBU Podman version: 4.2.0
DEBU Running 'crc status'
DEBU Checking file: /Users/myuser/.crc/machines/crc/.crc-exist
DEBU Checking file: /Users/myuser/.crc/machines/crc/.crc-exist
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1
DEBU Using ssh private keys: [/Users/myuser/.crc/machines/crc/id_ecdsa /Users/myuser/.crc/cache/crc_vfkit_4.11.3_arm64/id_ecdsa_crc]
DEBU SSH command results: err: <nil>, output: 32737570816 13706608640 /sysroot
CRC VM:          Running
OpenShift:       Running (v4.11.3)
Podman:
Disk Usage:      13.71GB of 32.74GB (Inside the CRC VM)
Cache Usage:     36.85GB
Cache Directory: /Users/myuser/.crc/cache

CRC config

- consent-telemetry                     : no

Host Operating System

ProductName:	macOS
ProductVersion:	12.6
BuildVersion:	21G115

Steps to reproduce

Deployed KEDA using helm, per documendation: helm install keda kedacore/keda --namespace keda
Containers going into ErrImagePull.

keda-operator-metrics-apiserver-75d6895d6c-75j5r   0/1     ErrImagePull        0          33s
keda-operator-698598c4d-74k9j                      0/1     ErrImagePull        0          33s

Getting the following error:

Events:
Type     Reason          Age   From               Message
----     ------          ----  ----               -------
Normal   Scheduled       44s   default-scheduler  Successfully assigned keda/keda-operator-metrics-apiserver-75d6895d6c-75j5r to crc-j2d48-master-0 by crc-j2d48-master-0
Normal   AddedInterface  41s   multus             Add eth0 [z.z.z.z/23] from openshift-sdn
Normal   Pulling         41s   kubelet            Pulling image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.1"
Warning  Failed          11s   kubelet            Failed to pull image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.1": rpc error: code = Unknown desc = pinging container registry ghcr.io: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io on x.x.x.x:53: read udp y.y.y.y:46084->x.x.x.x:53: i/o timeout
Warning  Failed          11s   kubelet            Error: ErrImagePull
Normal   BackOff         11s   kubelet            Back-off pulling image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.1"
Warning  Failed          11s   kubelet            Error: ImagePullBackOff

Direct image pull using Podman works fine: podman pull ghcr.io/kedacore/keda-metrics-apiserver:2.8.1

There is an issue in Linux (Issue #253) that sounds similar, but it is in Federa and the resolution may be different.

The text was updated successfully, but these errors were encountered:

praveenkumar · 2022-10-11T02:51:56Z

Direct image pull using Podman works fine: podman pull ghcr.io/kedacore/keda-metrics-apiserver:2.8.1

You mean it works inside the VM? You can ssh to the VM using https://github.com/code-ready/crc/wiki/Debugging-guide document.

mhcastro · 2022-10-11T07:44:40Z

@praveenkumar I mean that I can successfully pull the image directly through Podman, outside CRC. But in CRC, it is failing to pull the image with that dial tcp: look up .... i/o timeout issue.

mhcastro · 2022-10-14T14:28:23Z

This issue is not only for Keda. It is the same even for the simple NGINX deployment.

oc apply -f https://k8s.io/examples/application/deployment.yaml

Same error to pull the image on MacOS (Apple M1 Max chip).

It must be something in OpenShift CRC as it works in minikube.

praveenkumar · 2022-10-14T15:34:56Z

@mhcastro I tried same but didn't have issue pulling the image, have you tried following?

$ crc delete
$ crc cleanup
$ ps aux | grep crc <= make sure there is no crc daemon process
$ crc setup
$ crc start

mhcastro · 2022-10-14T15:35:49Z

Yes, I have tried that. It works in minikube, but not in CRC.

praveenkumar · 2022-10-17T04:56:18Z

@mhcastro Are you trying to run CRC and minikube in parallel?

mhcastro · 2022-10-17T08:34:27Z

@praveenkumar - no, I am not.

mhcastro · 2022-10-17T14:13:03Z

Hi - I noticed that the error message is issued by ../crc/vendor/github.com/containers/image/v5/docker/docker_client.go, line 759, but I am using podman, not docker. Docker is not even installed. Is there a configuration issue where I need to explicitly configure CRC to use podman instead of docker?

praveenkumar · 2022-10-17T14:21:12Z

@mhcastro no, this is just the file name which have logic around pulling the image, I am still not able to reproduce it :( , check #2597 one, looks similar.

mhcastro · 2022-10-17T15:05:45Z

@praveenkumar This seems to be a known, legitimate and recurrent issue, which is also referred by #3373. The trick seems to be in the ability to reproduce.

mhcastro · 2022-10-17T15:53:10Z

This may be relevant info ....

{
  "ARP": {
    "DisabledPacketsReceived": 0,
    "MalformedPacketsReceived": 0,
    "OutgoingRepliesDropped": 0,
    "OutgoingRepliesSent": 0,
    "OutgoingRequestBadLocalAddressErrors": 0,
    "OutgoingRequestInterfaceHasNoLocalAddressErrors": 0,
    "OutgoingRequestsDropped": 0,
    "OutgoingRequestsSent": 7,
    "PacketsReceived": 1,
    "RepliesReceived": 1,
    "RequestsReceived": 0,
    "RequestsReceivedUnknownTargetAddress": 0
  },
  "BytesReceived": 78582484,
  "BytesSent": 7419755,
  "DroppedPackets": 0,
  "ICMP": {
    "V4": {
      "PacketsReceived": {
        "ICMPv4PacketStats": {
          "DstUnreachable": 569,
          "EchoReply": 0,
          "EchoRequest": 0,
          "InfoReply": 0,
          "InfoRequest": 0,
          "ParamProblem": 0,
          "Redirect": 0,
          "SrcQuench": 0,
          "TimeExceeded": 0,
          "Timestamp": 0,
          "TimestampReply": 0
        },
        "Invalid": 0
      },
      "PacketsSent": {
        "Dropped": 0,
        "ICMPv4PacketStats": {
          "DstUnreachable": 0,
          "EchoReply": 0,
          "EchoRequest": 0,
          "InfoReply": 0,
          "InfoRequest": 0,
          "ParamProblem": 0,
          "Redirect": 0,
          "SrcQuench": 0,
          "TimeExceeded": 0,
          "Timestamp": 0,
          "TimestampReply": 0
        },
        "RateLimited": 0
      }
    },
    "V6": {
      "PacketsReceived": {
        "ICMPv6PacketStats": {
          "DstUnreachable": 0,
          "EchoReply": 0,
          "EchoRequest": 0,
          "MulticastListenerDone": 0,
          "MulticastListenerQuery": 0,
          "MulticastListenerReport": 0,
          "NeighborAdvert": 0,
          "NeighborSolicit": 0,
          "PacketTooBig": 0,
          "ParamProblem": 0,
          "RedirectMsg": 0,
          "RouterAdvert": 0,
          "RouterSolicit": 0,
          "TimeExceeded": 0
        },
        "Invalid": 0,
        "RouterOnlyPacketsDroppedByHost": 0,
        "Unrecognized": 0
      },
      "PacketsSent": {
        "Dropped": 0,
        "ICMPv6PacketStats": {
          "DstUnreachable": 0,
          "EchoReply": 0,
          "EchoRequest": 0,
          "MulticastListenerDone": 0,
          "MulticastListenerQuery": 0,
          "MulticastListenerReport": 0,
          "NeighborAdvert": 0,
          "NeighborSolicit": 0,
          "PacketTooBig": 0,
          "ParamProblem": 0,
          "RedirectMsg": 0,
          "RouterAdvert": 0,
          "RouterSolicit": 0,
          "TimeExceeded": 0
        },
        "RateLimited": 0
      }
    }
  },
  "IGMP": {
    "PacketsReceived": {
      "ChecksumErrors": 0,
      "IGMPPacketStats": {
        "LeaveGroup": 0,
        "MembershipQuery": 0,
        "V1MembershipReport": 0,
        "V2MembershipReport": 0
      },
      "Invalid": 0,
      "Unrecognized": 0
    },
    "PacketsSent": {
      "Dropped": 0,
      "IGMPPacketStats": {
        "LeaveGroup": 0,
        "MembershipQuery": 0,
        "V1MembershipReport": 0,
        "V2MembershipReport": 0
      }
    }
  },
  "IP": {
    "DisabledPacketsReceived": 0,
    "Forwarding": {
      "Errors": 0,
      "ExhaustedTTL": 0,
      "ExtensionHeaderProblem": 0,
      "HostUnreachable": 0,
      "LinkLocalDestination": 0,
      "LinkLocalSource": 0,
      "NoMulticastPendingQueueBufferSpace": 0,
      "PacketTooBig": 0,
      "UnexpectedMulticastInputInterface": 0,
      "UnknownOutputEndpoint": 0,
      "Unrouteable": 0
    },
    "IPTablesForwardDropped": 0,
    "IPTablesInputDropped": 0,
    "IPTablesOutputDropped": 0,
    "IPTablesPostroutingDropped": 0,
    "IPTablesPreroutingDropped": 0,
    "InvalidDestinationAddressesReceived": 0,
    "InvalidSourceAddressesReceived": 0,
    "MalformedFragmentsReceived": 0,
    "MalformedPacketsReceived": 0,
    "OptionRecordRouteReceived": 0,
    "OptionRouterAlertReceived": 0,
    "OptionTimestampReceived": 0,
    "OptionUnknownReceived": 0,
    "OutgoingPacketErrors": 9,
    "PacketsDelivered": 52301,
    "PacketsReceived": 52301,
    "PacketsSent": 38271,
    "ValidPacketsReceived": 52301
  },
  "NICs": {
    "DisabledRx": {
      "Bytes": 0,
      "Packets": 0
    },
    "MalformedL4RcvdPackets": 0,
    "Neighbor": {
      "UnreachableEntryLookups": 2
    },
    "Rx": {
      "Bytes": 77849358,
      "Packets": 52300
    },
    "Tx": {
      "Bytes": 7420007,
      "Packets": 38269
    },
    "TxPacketsDroppedNoBufferSpace": 0
  },
  "TCP": {
    "ActiveConnectionOpenings": 1069,
    "ChecksumErrors": 0,
    "CurrentConnected": 47,
    "CurrentEstablished": 46,
    "EstablishedClosed": 302,
    "EstablishedResets": 18,
    "EstablishedTimedout": 0,
    "FailedConnectionAttempts": 4,
    "FailedPortReservations": 0,
    "FastRecovery": 0,
    "FastRetransmit": 0,
    "InvalidSegmentsReceived": 0,
    "ListenOverflowAckDrop": 0,
    "ListenOverflowInvalidSynCookieRcvd": 0,
    "ListenOverflowSynCookieRcvd": 0,
    "ListenOverflowSynCookieSent": 0,
    "ListenOverflowSynDrop": 0,
    "PassiveConnectionOpenings": 1,
    "ResetsReceived": 13,
    "ResetsSent": 2752,
    "Retransmits": 0,
    "SACKRecovery": 0,
    "SegmentSendErrors": 0,
    "SegmentsAckedWithDSACK": 0,
    "SegmentsSent": 34569,
    "SlowStartRetransmits": 0,
    "SpuriousRTORecovery": 0,
    "SpuriousRecovery": 0,
    "TLPRecovery": 0,
    "Timeouts": 0,
    "ValidSegmentsReceived": 47197
  },
  "UDP": {
    "ChecksumErrors": 0,
    "MalformedPacketsReceived": 0,
    "PacketSendErrors": 0,
    "PacketsReceived": 3711,
    "PacketsSent": 3702,
    "ReceiveBufferErrors": 0,
    "UnknownPortErrors": 118
  }
}

mhcastro · 2022-10-17T16:17:56Z

@praveenkumar perhaps one additional piece of information. I didn't have issues with CRC in MacOS using Intel-chip. I started having the problems only in the Apple M1 Max silicon.

mhcastro · 2023-04-17T14:04:39Z

This can be closed. It is working now after upgrading to Mac OS Ventura 13.3 on M1 chip, without any change to DNS.

oc get pods -n openshift-marketplace
NAME                                    READY   STATUS    RESTARTS   AGE
certified-operators-zj64b               1/1     Running   0          194d
community-operators-tr5xs               1/1     Running   0          194d
marketplace-operator-8485c7444b-95prb   1/1     Running   0          193d
redhat-marketplace-wqdpg                1/1     Running   0          194d
redhat-operators-r7zk5                  1/1     Running   0          194d

and

oc get pods -n keda
NAME                                               READY   STATUS    RESTARTS   AGE
keda-operator-698598c4d-nc82g                      1/1     Running   0          44s
keda-operator-metrics-apiserver-75d6895d6c-hv9xp   1/1     Running   0          44s

mhcastro added kind/bug Something isn't working status/need triage labels Oct 10, 2022

mhcastro mentioned this issue Oct 17, 2022

[BUG] All pods failing to pull images #3373

Closed

mhcastro closed this as completed Apr 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fail to deploy KEDA - read udp x.x.x.x:46084->y.y.y.y:53: i/o timeout on MacOS (Apple Silicon) #3377

Fail to deploy KEDA - read udp x.x.x.x:46084->y.y.y.y:53: i/o timeout on MacOS (Apple Silicon) #3377

mhcastro commented Oct 10, 2022

praveenkumar commented Oct 11, 2022

mhcastro commented Oct 11, 2022 •

edited

Loading

mhcastro commented Oct 14, 2022 •

edited

Loading

praveenkumar commented Oct 14, 2022

mhcastro commented Oct 14, 2022

praveenkumar commented Oct 17, 2022

mhcastro commented Oct 17, 2022

mhcastro commented Oct 17, 2022

praveenkumar commented Oct 17, 2022

mhcastro commented Oct 17, 2022 •

edited

Loading

mhcastro commented Oct 17, 2022

mhcastro commented Oct 17, 2022 •

edited

Loading

mhcastro commented Apr 17, 2023 •

edited

Loading

Fail to deploy KEDA - read udp x.x.x.x:46084->y.y.y.y:53: i/o timeout on MacOS (Apple Silicon) #3377

Fail to deploy KEDA - read udp x.x.x.x:46084->y.y.y.y:53: i/o timeout on MacOS (Apple Silicon) #3377

Comments

mhcastro commented Oct 10, 2022

General information

CRC version

CRC status

CRC config

Host Operating System

Steps to reproduce

praveenkumar commented Oct 11, 2022

mhcastro commented Oct 11, 2022 • edited Loading

mhcastro commented Oct 14, 2022 • edited Loading

praveenkumar commented Oct 14, 2022

mhcastro commented Oct 14, 2022

praveenkumar commented Oct 17, 2022

mhcastro commented Oct 17, 2022

mhcastro commented Oct 17, 2022

praveenkumar commented Oct 17, 2022

mhcastro commented Oct 17, 2022 • edited Loading

mhcastro commented Oct 17, 2022

mhcastro commented Oct 17, 2022 • edited Loading

mhcastro commented Apr 17, 2023 • edited Loading

mhcastro commented Oct 11, 2022 •

edited

Loading

mhcastro commented Oct 14, 2022 •

edited

Loading

mhcastro commented Oct 17, 2022 •

edited

Loading

mhcastro commented Oct 17, 2022 •

edited

Loading

mhcastro commented Apr 17, 2023 •

edited

Loading