Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v20.10] Harbor docker-compose.yaml can't access external DNS after upgrade to Docker 20.10 #1164

Open
2 of 3 tasks
GJKrupa opened this issue Dec 10, 2020 · 5 comments
Open
2 of 3 tasks

Comments

@GJKrupa
Copy link

GJKrupa commented Dec 10, 2020

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Harbor starts up

Actual behavior

Harbor fails to start up due to DNS timeout errors resolving the external database address

Steps to reproduce the behavior

Install Docker 20.10 on Ubuntu 18.04
Download the Harbor installer package from https://github.com/goharbor/harbor/releases/download/v1.10.4/harbor-offline-installer-v1.10.4.tgz
Extract the package into /etc/harbor/habor
Customise the Harbor configuration in /etc/harbor/harbor/harbor.yml to use an external Postgresql database

external_database:
  harbor:
    host: ADDRESS
    port: 5432
    db_name: DB_NAME
    username: USERNAME
    password: PASSWORD
    ssl_mode: disable
    max_idle_conns: 2
    max_open_conns: 0

Run ./install.sh

Run docker-compose up -d

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:59:53 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:57:44 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1030-aws
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.487GiB
 Name: notary
 ID: ZBNK:F2O4:2RSN:5Z2B:RJNC:EZBD:27OF:LLEK:GCFE:Z35X:JZ3V:UBYF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support

Additional environment details (AWS, VirtualBox, physical, etc.)

  • We're running this in EC2 based off a standard Ubuntu 18.04 AMI.
  • Docker installed by adding the https://download.docker.com/linux/ubuntu repo to APT
  • The following additional packages are installed: bsdutils, linux-aws, python3-pip, curl, ca-certificates, docker-ce, docker-ce-cli, containerd.io, postgresql-client, jq
  • The following Python packages are installed using pip3: docker, docker-compose, awscli
  • We have confirmed that the external database address can be resolved and connected from the host
  • If docker-ce is downgraded to 5:19.03.143-0ubuntu-bionic then the same docker-compose works without issue.

The docker-compose.yaml generated by the installer for us looks as follows:

version: '2.3'
services:
  log:
    image: goharbor/harbor-log:v1.10.4
    container_name: harbor-log
    restart: always
    dns_search: .
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - DAC_OVERRIDE
      - SETGID
      - SETUID
    volumes:
      - /var/log/harbor/:/var/log/docker/:z
      - type: bind
        source: ./common/config/log/logrotate.conf
        target: /etc/logrotate.d/logrotate.conf
      - type: bind
        source: ./common/config/log/rsyslog_docker.conf
        target: /etc/rsyslog.d/rsyslog_docker.conf
    ports:
      - 127.0.0.1:1514:10514
    networks:
      - harbor
  registry:
    image: goharbor/registry-photon:v1.10.4
    container_name: registry
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    volumes:
      - /data/registry:/storage:z
      - ./common/config/registry/:/etc/registry/:z
      - type: bind
        source: /data/secret/registry/root.crt
        target: /etc/registry/root.crt
    networks:
      - harbor
    dns_search: .
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "registry"
  registryctl:
    image: goharbor/harbor-registryctl:v1.10.4
    container_name: registryctl
    env_file:
      - ./common/config/registryctl/env
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    volumes:
      - /data/registry:/storage:z
      - ./common/config/registry/:/etc/registry/:z
      - type: bind
        source: ./common/config/registryctl/config.yml
        target: /etc/registryctl/config.yml
    networks:
      - harbor
    dns_search: .
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "registryctl"
  core:
    image: goharbor/harbor-core:v1.10.4
    container_name: harbor-core
    env_file:
      - ./common/config/core/env
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - SETGID
      - SETUID
    volumes:
      - /data/ca_download/:/etc/core/ca/:z
      - /data/psc/:/etc/core/token/:z
      - /data/:/data/:z
      - ./common/config/core/certificates/:/etc/core/certificates/:z
      - type: bind
        source: ./common/config/core/app.conf
        target: /etc/core/app.conf
      - type: bind
        source: /data/secret/core/private_key.pem
        target: /etc/core/private_key.pem
      - type: bind
        source: /data/secret/keys/secretkey
        target: /etc/core/key
    networks:
      - harbor
    dns_search: .
    depends_on:
      - log
      - registry
      - redis
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "core"
  portal:
    image: goharbor/harbor-portal:v1.10.4
    container_name: harbor-portal
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
      - NET_BIND_SERVICE
    networks:
      - harbor
    dns_search: .
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "portal"

  jobservice:
    image: goharbor/harbor-jobservice:v1.10.4
    container_name: harbor-jobservice
    env_file:
      - ./common/config/jobservice/env
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    volumes:
      - /data/job_logs:/var/log/jobs:z
      - type: bind
        source: ./common/config/jobservice/config.yml
        target: /etc/jobservice/config.yml
    networks:
      - harbor
    dns_search: .
    depends_on:
      - core
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "jobservice"
  redis:
    image: goharbor/redis-photon:v1.10.4
    container_name: redis
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    volumes:
      - /data/redis:/var/lib/redis
    networks:
      harbor:
    dns_search: .
    depends_on:
      - log
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "redis"
  proxy:
    image: goharbor/nginx-photon:v1.10.4
    container_name: nginx
    restart: always
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
      - NET_BIND_SERVICE
    volumes:
      - ./common/config/nginx:/etc/nginx:z
      - /data/secret/cert:/etc/cert:z
    networks:
      - harbor
    dns_search: .
    ports:
      - 80:8080
      - 443:8443
    depends_on:
      - registry
      - core
      - portal
      - log
    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://127.0.0.1:1514"
        tag: "proxy"
networks:
  harbor:
    external: false
@GJKrupa
Copy link
Author

GJKrupa commented Dec 10, 2020

We're running the following docker-compose version installed though PyPi:

docker-compose version 1.27.4, build unknown

@sluetze
Copy link

sluetze commented Dec 15, 2020

goharbor/harbor#13718 seems to be the same issue. there are some workarounds like explicitly definining dns-servers in daemon.json

@thaJeztah
Copy link
Member

May be related to / same as moby/moby#41819

@thaJeztah thaJeztah changed the title Harbor docker-compose.yaml can't access external DNS after upgrade to Docker 20.10 [v20.10] Harbor docker-compose.yaml can't access external DNS after upgrade to Docker 20.10 Dec 23, 2020
@cbeley
Copy link

cbeley commented Jan 14, 2021

I noticed the issue when Trivy was unable to download its DB. When I manually logged into the container, I verified by using curl that external domains would not resolve, but that I did otherwise have access to the internet via IP's.

Interestingly, I just removed dns_search: . from every service and all is working fine so far (Including Trivy). I'm not sure if this change will have any unintended side effects though.

@wy65701436
Copy link

wy65701436 commented Sep 3, 2021

@cbeley you can use this workaround.
And to remove the dns_search . will may introduce this issue mentioned here.

wy65701436 added a commit to wy65701436/harbor that referenced this issue Sep 8, 2021
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If who encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Sep 8, 2021
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit to goharbor/harbor that referenced this issue Sep 8, 2021
For details, please refer to #14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by #6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
lindhe pushed a commit to lindhe/harbor that referenced this issue Sep 22, 2021
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
Signed-off-by: Andreas Lindhé <andreas@lindhe.io>
prahaladdarkin pushed a commit to prahaladdarkin/harbor that referenced this issue Nov 12, 2021
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
prahaladdarkin pushed a commit to prahaladdarkin/harbor that referenced this issue Mar 13, 2022
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants