Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] - DNS Address fails while IP address is working #2752

Closed
toky03 opened this issue May 27, 2021 · 17 comments
Closed

[BUG] - DNS Address fails while IP address is working #2752

toky03 opened this issue May 27, 2021 · 17 comments
Labels
bug Something isn't working

Comments

@toky03
Copy link

toky03 commented May 27, 2021

External

Area
Other Cardano Node Relay Connection

Summary
A clear and specific description of what the bug is.

Steps to reproduce
Start a cardano node inside of a docker container e.g:

cardano-node run \
   --topology path/to/mainnet-topology.json \
   --database-path path/to/db \
   --socket-path path/to/db/node.socket \
   --host-addr 0.0.0.0 \
   --port 3001 \
   --config path/to/mainnet-config.json

while the topology.json files has an entry with a dns name (not an ip address)

{
  "Producers": [
    {
      "addr": "relays-new.cardano-testnet.iohkdev.io",
      "port": 3001,
      "valency": 2
    }
  ]
}

With this configuration it fails:

[seeland-:cardano.node.dns-producers:Notice:5] [2021-05-26 19:08:49.68 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-testnet.iohkdev.io", dstPort = 3001, dstValency = 2}]
[seeland-:cardano.node.ip-producers:Notice:5] [2021-05-26 19:08:49.68 UTC] IPSubscriptionTarget {ispIps = [], ispValency = 0}
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened imm db with immutable tip at genesis (origin) and chunk 0
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened vol db
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Replaying ledger from genesis
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened lgr db
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened db with immutable tip at genesis (origin) and tip genesis (origin)
[seeland-:cardano.node.ChainDB:Notice:34] [2021-05-26 19:08:49.73 UTC] Chain extended, new tip: 1e64e74bd7ac76d6806480a28017deb0aedd356fb61844ec95c429ae2f30c7c3 at slot 0
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:08:49.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:08:59.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:09.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.ChainDB:Notice:76] [2021-05-26 19:09:18.86 UTC] before next, messages elided = 2259510500755060
[seeland-:cardano.node.ChainDB:Notice:76] [2021-05-26 19:09:18.86 UTC] Chain extended, new tip: 1e64e74bd7ac76d6806480a28017deb0aedd356fb61844ec95c429ae2f30c7c3 at slot 0
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:19.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:29.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions

If you use an ip address instead it works.

I checked if there is an issue with Kubernetes and DNS but i am able to resolve the dns from within the pod:

root@seeland-stakepool-relay1-77587998b4-zj6k5:/config# nslookup relays-new.cardano-testnet.iohkdev.io
Server:		10.96.0.10
Address:	10.96.0.10#53

Non-authoritative answer:
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 54.250.11.33
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.129.133.68
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.9.80.183
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.124.178.139
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 175.41.160.108
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 18.159.64.253
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 54.241.77.32
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.128.220.55

Expected behavior
DNSSubscription should work like it works for IP addresses.

System info:

  • OS Name: Ubuntu
  • OS Version 20.04
  • Kubernetes Cluster: v1.21.0
  • Node version cardano-node 1.27.0 - linux-x86_64 - ghc-8.10
  • CLI version cardano-cli 1.27.0 - linux-x86_64 - ghc-8.10

Additional context
The Problem has been disscussed in the Cardano Forum DNS Address fails while IP Address works

@toky03 toky03 added the bug Something isn't working label May 27, 2021
@DiazRock
Copy link

Got the same here. Solved using ip. This is my system info:

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

docker version

Docker version 20.10.2, build 20.10.2-0ubuntu1~18.04.2

I'm using microk8s

@ghost
Copy link

ghost commented May 29, 2021

Same problem here. The node syncs on the mainnet for some time but then fails. I have to delete the state folder /db for the node to restart synchronization from the start.

@weebl2000
Copy link

I can try to have a look if I can reproduce & try to fix.

@weebl2000
Copy link

Tried to reproduce but couldn't. More specific reproduction steps?

@toky03
Copy link
Author

toky03 commented Jun 7, 2021

Hi @weebl2000
I built the Docker Cardano node with the following Dockerfile:

FROM haskell:latest as cardano

RUN cabal update
RUN apt-get update -y
RUN apt-get install -y git libtool build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 -y

RUN git clone https://github.com/input-output-hk/libsodium


WORKDIR libsodium
RUN git checkout 66f017f1 && ./autogen.sh && ./configure &&  make
RUN make install

ENV LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
ENV PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
ENV VERSION_TAG=1.27.0

RUN mkdir /cardano
WORKDIR /cardano
RUN git clone https://github.com/input-output-hk/cardano-node.git
WORKDIR cardano-node
RUN  git fetch --all --recurse-submodules --tags && git tag && git checkout tags/$VERSION_TAG
RUN cabal configure --with-compiler=ghc-8.10.2
RUN echo  "package cardano-crypto-praos" >>  cabal.project.local && echo "  flags: -external-libsodium-vrf" >>  cabal.project.local

RUN cabal build all

RUN cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-node-$VERSION_TAG/x/cardano-node/build/cardano-node/cardano-node /bin/cardano-node
RUN cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-cli-$VERSION_TAG/x/cardano-cli/build/cardano-cli/cardano-cli /bin/cardano-cli

RUN chmod +x /bin/cardano-cli
RUN /bin/cardano-cli --version



FROM ubuntu:latest

RUN apt update

COPY --from=cardano /usr/local/ /usr/local

ENV LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
ENV PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"


COPY --from=cardano /bin /usr/bin

RUN chmod +x /usr/bin/cardano-cli
RUN mkdir /database

COPY --from=cardano /usr/local/ /usr/local

WORKDIR /config

then i downloaded all the configuration files inside of a data folder

wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-config.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-byron-genesis.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-shelley-genesis.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-topology.json

Build the docker Image (this takes quite some time as it builds the cardano node from source)

docker build -t cardano-node-image:test .

run the docker container

docker container run -it -v $(pwd)/data/:/config cardano-node-image:test /bin/bash

Now inside of the docker container run the node:

./cardano-node run --topology testnet-topology.json --database-path db --socket-path db/node.socket --config testnet-config.json --port 3000 --host-addr 0.0.0.0

@weebl2000
Copy link

Are you sure it's complete? When running I get:

cabal: Cannot find the program 'ghc'. User-specified path 'ghc-8.10.4' does
not refer to an executable and the program is not on the system path.

@toky03
Copy link
Author

toky03 commented Jun 7, 2021

I just checked it and i had to use this command instead: /usr/bin/cardano-node run --topology testnet-topology.json --database-path db --socket-path db/node.socket --config testnet-config.json --port 3000 --host-addr 0.0.0.0
Did you try to build the Docker Image with the Dockerfile given above?
If you want to try it with a pre built Docker Image (then you do not have to build it again) you can use: docker pull toky03/cardano-node-image:test
and then run the container with: docker container run -it -v $(pwd)/data/:/config toky03/cardano-node-image:test /bin/bash
With this i always get the same result.
[85f5d4cc:cardano.node.DnsSubscription:Warning:55] [2021-06-07 13:17:30.89 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
Do you use another Docker Cardano Image which works?

@weebl2000
Copy link

I've tried with inputoutput/cardano-node image, which seems to work:

My /tmp/cardano just contains config and a "db" folder which is empty.

$ docker run -v /tmp/cardano:/tmp/cardano  inputoutput/cardano-node run --topology /tmp/cardano/mainnet-topology.json --database-path /tmp/cardano/db --socket-path /tmp/cardano/db/node.socket --host-addr 0.0.0.0 --port 3001 --config /tmp/cardano/mainnet-config.json
Listening on http://127.0.0.1:12798
[26221b64:cardano.node.networkMagic:Notice:5] [2021-06-07 13:26:50.90 UTC] NetworkMagic 764824073
[26221b64:cardano.node.basicInfo.protocol:Notice:5] [2021-06-07 13:26:50.90 UTC] Byron; Shelley
[26221b64:cardano.node.basicInfo.version:Notice:5] [2021-06-07 13:26:50.90 UTC] 1.27.0
[26221b64:cardano.node.basicInfo.commit:Notice:5] [2021-06-07 13:26:50.90 UTC] 8fe46140a52810b6ca456be01d652ca08fe730bf
[26221b64:cardano.node.basicInfo.nodeStartTime:Notice:5] [2021-06-07 13:26:50.90 UTC] 2021-06-07 13:26:50.900762829 UTC
[26221b64:cardano.node.basicInfo.systemStartTime:Notice:5] [2021-06-07 13:26:50.90 UTC] 2017-09-23 21:44:51 UTC
[26221b64:cardano.node.basicInfo.slotLengthByron:Notice:5] [2021-06-07 13:26:50.90 UTC] 20s
[26221b64:cardano.node.basicInfo.epochLengthByron:Notice:5] [2021-06-07 13:26:50.90 UTC] 21600
[26221b64:cardano.node.basicInfo.slotLengthShelley:Notice:5] [2021-06-07 13:26:50.90 UTC] 1s
[26221b64:cardano.node.basicInfo.epochLengthShelley:Notice:5] [2021-06-07 13:26:50.90 UTC] 432000
[26221b64:cardano.node.basicInfo.slotsPerKESPeriodShelley:Notice:5] [2021-06-07 13:26:50.90 UTC] 129600
[26221b64:cardano.node.basicInfo.slotLengthAllegra:Notice:5] [2021-06-07 13:26:50.90 UTC] 1s
[26221b64:cardano.node.basicInfo.epochLengthAllegra:Notice:5] [2021-06-07 13:26:50.90 UTC] 432000
[26221b64:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:5] [2021-06-07 13:26:50.90 UTC] 129600
[26221b64:cardano.node.basicInfo.slotLengthMary:Notice:5] [2021-06-07 13:26:50.90 UTC] 1s
[26221b64:cardano.node.basicInfo.epochLengthMary:Notice:5] [2021-06-07 13:26:50.90 UTC] 432000
[26221b64:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:5] [2021-06-07 13:26:50.90 UTC] 129600
[26221b64:cardano.node.addresses:Notice:5] [2021-06-07 13:26:50.90 UTC] [SocketInfo 0.0.0.0:3001]
[26221b64:cardano.node.diffusion-mode:Notice:5] [2021-06-07 13:26:50.90 UTC] InitiatorAndResponderDiffusionMode
[26221b64:cardano.node.dns-producers:Notice:5] [2021-06-07 13:26:50.90 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-mainnet.iohk.io", dstPort = 3001, dstValency = 2}]
[26221b64:cardano.node.ip-producers:Notice:5] [2021-06-07 13:26:50.90 UTC] IPSubscriptionTarget {ispIps = [], ispValency = 0}
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.16 UTC] Opened imm db with immutable tip at genesis (origin) and chunk 0
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.16 UTC] Opened vol db
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.16 UTC] Replaying ledger from genesis
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.29 UTC] Opened lgr db
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.29 UTC] Opened db with immutable tip at genesis (origin) and tip genesis (origin)
[26221b64:cardano.node.ChainDB:Notice:35] [2021-06-07 13:26:51.29 UTC] Chain extended, new tip: d2d90d10993b3eb309f5485a5413cb9c9fe4f088e79b7b4e815dbbe8fcb8e4b7 at slot 0
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] CreatingServerSocket 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] ConfiguringServerSocket 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] ListeningServerSocket 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] ServerSocketUp 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] RunServer 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] CreateSystemdSocketForSnocketPath "/tmp/cardano/db/node.socket"
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] CreatedLocalSocket "/tmp/cardano/db/node.socket"
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] ConfiguringLocalSocket "/tmp/cardano/db/node.socket" (FileDescriptor 25)
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] ListeningLocalSocket "/tmp/cardano/db/node.socket" (FileDescriptor 25)
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] LocalSocketUp "/tmp/cardano/db/node.socket" (FileDescriptor 25)
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] RunLocalServer (LocalAddress "/tmp/cardano/db/node.socket")
[26221b64:cardano.node.DnsSubscription:Notice:67] [2021-06-07 13:26:51.30 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 18.133.59.65:3001
[26221b64:cardano.node.DnsSubscription:Notice:67] [2021-06-07 13:26:51.32 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt End, destination 18.133.59.65:3001 outcome: ConnectSuccess
[26221b64:cardano.node.DnsSubscription:Notice:68] [2021-06-07 13:26:51.33 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 3.129.199.184:3001
[26221b64:cardano.node.DnsSubscription:Notice:82] [2021-06-07 13:26:51.35 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 18.159.65.82:3001
[26221b64:cardano.node.DnsSubscription:Notice:82] [2021-06-07 13:26:51.37 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt End, destination 18.159.65.82:3001 outcome: ConnectSuccessLast
[26221b64:cardano.node.ErrorPolicy:Notice:54] [2021-06-07 13:26:51.37 UTC] IP 3.129.199.184:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[26221b64:cardano.node.ChainDB:Info:35] [2021-06-07 13:26:51.50 UTC] Block fits onto some fork: 89d9b5a5b8ddc8d7e5a6795e9774d97faf1efea59b2caf7eaf9f8c5b32059df4 at slot 0
[26221b64:cardano.node.ChainDB:Info:35] [2021-06-07 13:26:51.53 UTC] Block fits onto some fork: f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f at slot 0
[26221b64:cardano.node.ChainDB:Notice:35] [2021-06-07 13:26:51.54 UTC] Switched to a fork, new tip: f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f at slot 0
[26221b64:cardano.node.ChainDB:Notice:35] [2021-06-07 13:26:51.54 UTC] Chain extended, new tip: 1dbc81e3196ba4ab9dcb07e1c37bb28ae1c289c0707061f28b567c2f48698d50 at slot 1

@toky03
Copy link
Author

toky03 commented Jun 8, 2021

It works with the official inputoutput image indeed.
i still have some issues with communicating via dns to my nodes via kubernetes service dns directly but i assume it is some misconficuration on my side.

Do you know by any chance how the official docker image is built differently? I am not familiar with the nix docker build inside the Repository

@weebl2000
Copy link

weebl2000 commented Jun 8, 2021

I've never used a nix docker build like that, I did try to run it yesterday but did not succeed and did not put more time into it.

I did make a cardano-node build inside an ubuntu 20.04 container yesterday and that all seemed to work fine.

You could try enabling more verbose tracing by changing the following in the config:

  "TraceDNSResolver": true,
  "TraceDNSSubscription": true,
  "TraceDiffusionInitialization": true,
  "TracingVerbosity": "MaximalVerbosity",
  "minSeverity": "Debug",

You might want to disable some other logging since it will probably output a lot.

@toky03
Copy link
Author

toky03 commented Jun 11, 2021

Thank you for the hint @weebl2000.
Unfortunately there is not nothing more which helps me to find the error:

[465fb394:cardano.node.DnsSubscription:Debug:55] [2021-06-11 09:23:27.59 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Starting Subscription Worker, valency 2
[465fb394:cardano.node.DnsSubscription:Warning:55] [2021-06-11 09:23:27.59 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[465fb394:cardano.node.DnsSubscription:Debug:55] [2021-06-11 09:23:28.60 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Restarting Subscription after 1.005226535s desired valency 2 current valency 0

Meanwhile i will use the official Cardano Image and try to figure out whats the Problem with the internal Kubernetes Service.

@toky03
Copy link
Author

toky03 commented Jun 21, 2021

I will close this issue as we can use the official cardano-node image which is working correctly.

@toky03 toky03 closed this as completed Jun 21, 2021
@Katochimoto
Copy link

It works for me, I hope it helps you too
smth like this in Dockerfile

RUN apt-get update && apt-get install -y netbase

From here
#953 (comment)

@DonaldJimenez
Copy link

Just had the same issue while trying to start a fresh node on Windows 10. My node couldn't get past this error for a full day.

Domain: "relays-new.cardano-mainnet.iohk.io" Failed to start all required subscriptions

I modified the topology file and replaced domain name with an IP address. It immediately helped and node started to download blockchain data.

@locallycompact
Copy link

Also getting this error trtying to connect to the testnet

@SebastienGllmt
Copy link
Contributor

I ran into this issue as well and it turns out the DNS lookup on my machine was having issues (running nslookup relays-new.cardano-mainnet.iohk.io would return "REFUSED")

Fixed this by changing my DNS settings on my machine

@TypicalHog
Copy link

TypicalHog commented Jul 23, 2022

I semi-fixed it by changing the mainnet-topology.json to this:

{
  "Producers": [
    {
      "addr": "3.127.133.53",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.194.25.1",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.159.63.239",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "3.22.52.164",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.133.108.207",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.180.136.78",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "54.176.88.11",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "54.238.30.120",
      "port": 3001,
      "valency": 2
    }
  ]
}

I got the IPs from nslookup relays-new.cardano-mainnet.iohk.io

EDIT: I do get errors every now and then tho, as some of the IPs seem to not work. log.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants