Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Notary services do not resolve "mysql" alias correctly. #6031

Closed
dbarkelew opened this issue Oct 12, 2018 · 5 comments
Closed

Notary services do not resolve "mysql" alias correctly. #6031

dbarkelew opened this issue Oct 12, 2018 · 5 comments
Assignees
@wy65701436
Labels
kind/bug target/1.5.4 target/1.7.0 Work items that should be delivered in 1.7.0 timeframe.
Milestone
Sprint 45

Comments

@dbarkelew
Copy link

Harbor notary-server and notary-signer are resolving the "mysql" alias to the resolv.conf search path instead of to "mysql." for the notary-db bridge IP. Looks like a bug caused by libnetwork ignoring ndot:n.
moby/libnetwork#2212

Reproduction Steps

  1. Add a "mysql" record to the external DNS. In my case was "mysql.gsslabs.org"
  2. Deploy harbor as containers using defaults. I used the VIC Appliance for this.
  3. Notary-server logs reports
Oct 12 16:28:58 172.18.0.1 notary-server[457]: notaryserver database failed to come up within 30 seconds
Oct 12 16:29:02 172.18.0.1 notary-server[457]: waiting for mysql://server@tcp(mysql:3306)/notaryserver to come up.

Workaround
Found changing "mysql" to "mysql." for the db_url in the signer-config.json, server-config.json and migrate.sh fixes the issue. Look something like this.

        "storage": {
                "backend": "mysql",
                "db_url": "server@tcp(mysql.:3306)/notaryserver?parseTime=True"
        },

Requesting for the default "mysql" to be changed to "mysql." in the harbor configs.templates for the next release used in the VIC Appliance as a workaround.
https://github.com/goharbor/harbor/blob/master/make/common/templates/notary/server-config.json
https://github.com/goharbor/harbor/blob/master/make/common/templates/notary/signer-config.json
Only seeing the migrate.sh scripts in the binary package use in the make.

Bugzilla 2214121
@reasonerjt
Copy link

But why the it queries the external DNS to resolve mysql given that there is an alias in the docker network notary-mdb?
https://github.com/goharbor/harbor/blob/release-1.5.0/make/docker-compose.notary.tpl#L54

@reasonerjt
Copy link

Seems the queries were due to the search domain on resolv.conf in the container.
@wy65701436 I think we can set dns_search via docker-compose template to overwrite the value from docker host.
https://docs.docker.com/compose/compose-file/compose-file-v2/#dns_search

@ywk253100
Copy link
Contributor

The root cause is a bug in Go DNS resolver before golang 1.8 golang/go#15419, the Notary in Harbor is built with golang 1.7.3. So I think @wy65701436 upgrading the base image should fix this.

@reasonerjt reasonerjt added kind/bug target/1.5.4 target/1.7.0 Work items that should be delivered in 1.7.0 timeframe. labels Oct 15, 2018
@dbarkelew
Copy link
Author

Seems the queries were due to the search domain on resolv.conf in the container.
@wy65701436 I think we can set dns_search via docker-compose template to overwrite the value from docker host.
https://docs.docker.com/compose/compose-file/compose-file-v2/#dns_search

I have tried this and it does seem to work so far. I did not run very many tests to see if anything breaks from the change.

@wy65701436 wy65701436 added this to the Sprint 45 milestone Oct 16, 2018
wy65701436 added a commit to wy65701436/harbor that referenced this issue Oct 16, 2018
@wy65701436
Limit dns search in harbor containers
b2ba167 
This commit is to set dns search to null in the harbor containers,
that means the dns search domains of docker host doesn't impact
the network IO in the containers.

If do not set this, Harbor notary-server and notary-signer are resolving
the "mysql" alias to the resolv.conf search path instead of to "mysql."
for the notary-db bridge IP, see goharbor#6031.

Signed-off-by: wang yan <wangyan@vmware.com>
@wy65701436 wy65701436 mentioned this issue Oct 16, 2018
Merged
wy65701436 added a commit that referenced this issue Oct 16, 2018
@wy65701436
Limit dns search in harbor containers (#6057)
08ae5f2 
This commit is to set dns search to null in the harbor containers,
that means the dns search domains of docker host doesn't impact
the network IO in the containers.

If do not set this, Harbor notary-server and notary-signer are resolving
the "mysql" alias to the resolv.conf search path instead of to "mysql."
for the notary-db bridge IP, see #6031.

Signed-off-by: wang yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Oct 17, 2018
@wy65701436
Limit dns search in harbor containers
7cca20b 
This commit is to set dns search to null in the harbor containers,
that means the dns search domains of docker host doesn't impact
the network IO in the containers.

If do not set this, Harbor notary-server and notary-signer are resolving
the "mysql" alias to the resolv.conf search path instead of to "mysql."
for the notary-db bridge IP, see goharbor#6031.

According to docker official document, 'Use --dns-search=.' if you don't
wish to set the search domain.

https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/

Signed-off-by: wang yan <wangyan@vmware.com>
@wy65701436 wy65701436 mentioned this issue Oct 17, 2018
Merged
wy65701436 added a commit that referenced this issue Oct 17, 2018
@wy65701436
Limit dns search in harbor containers (#6058)
985b570 
This commit is to set dns search to null in the harbor containers,
that means the dns search domains of docker host doesn't impact
the network IO in the containers.

If do not set this, Harbor notary-server and notary-signer are resolving
the "mysql" alias to the resolv.conf search path instead of to "mysql."
for the notary-db bridge IP, see #6031.

According to docker official document, 'Use --dns-search=.' if you don't
wish to set the search domain.

https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/

Signed-off-by: wang yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Oct 17, 2018
@wy65701436
Upgrade notary complile golang version to 1.9.4
2d0eedb 
This commit is to upgrade the golang version to 1.9.4, it because a
bug of golang 17.3 could introduce one dns resolver issue for harbor
mentioned by goharbor#6031.

The bug of golang is golang/go#15419, it makes
harbor containers to lookup 'endpoint.' firstly which may cause network
issue.

Signed-off-by: wang yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Oct 17, 2018
@wy65701436
Upgrade notary complile golang version to 1.9.4
890ab40 
This commit is to upgrade the golang version to 1.9.4, it because a
bug of golang 17.3 could introduce one dns resolver issue for harbor
mentioned by goharbor#6031.

The bug of golang is golang/go#15419, it makes
harbor containers to lookup 'endpoint.' firstly which may cause network
issue.

Signed-off-by: wang yan <wangyan@vmware.com>
@wy65701436 wy65701436 mentioned this issue Oct 17, 2018
Merged
wy65701436 added a commit that referenced this issue Oct 17, 2018
@wy65701436
Upgrade notary complile golang version to 1.9.4 (#6064)
a5e7ac9 
This commit is to upgrade the golang version to 1.9.4, it because a
bug of golang 17.3 could introduce one dns resolver issue for harbor
mentioned by #6031.

The bug of golang is golang/go#15419, it makes
harbor containers to lookup 'endpoint.' firstly which may cause network
issue.

Signed-off-by: wang yan <wangyan@vmware.com>
@wy65701436
Copy link
Contributor

close it as fixed, will provide a new release on v1.5.0 for VIC.

ktbartholomew pushed a commit to rcbops/kubernetes-harbor that referenced this issue Nov 12, 2018
Merge upstream releases 1.5.3 and 1.5.4 (#10)
352da61 
* Update photon base images -- for 1.5 branch (goharbor#5376)

This commit update the base photon image from vmware/photon:1.0 to
photon:1.0 in 1.5 branch

* Bump up clair to v2.0.5 (goharbor#5787)

Signed-off-by: Daniel Jiang <jiangd@vmware.com>

* Update the base image in the offline installer (goharbor#5807)

Signed-off-by: wang yan <wangyan@vmware.com>

* Promote release version to v1.5.3 (goharbor#5811)

Signed-off-by: wang yan <wangyan@vmware.com>

* Update LICENSE and OSL (goharbor#5874)

Signed-off-by: Daniel Jiang <jiangd@vmware.com>

* Bump up Clair to 2.0.6 (goharbor#6016)

Signed-off-by: Daniel Jiang <jiangd@vmware.com>

* Update OSL and bump up version to 1.5.4

Signed-off-by: Daniel Jiang <jiangd@vmware.com>

* Limit dns search in harbor containers (goharbor#6058)

This commit is to set dns search to null in the harbor containers,
that means the dns search domains of docker host doesn't impact
the network IO in the containers.

If do not set this, Harbor notary-server and notary-signer are resolving
the "mysql" alias to the resolv.conf search path instead of to "mysql."
for the notary-db bridge IP, see goharbor#6031.

According to docker official document, 'Use --dns-search=.' if you don't
wish to set the search domain.

https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/

Signed-off-by: wang yan <wangyan@vmware.com>
wy65701436 added a commit to wy65701436/harbor that referenced this issue Sep 8, 2021
@wy65701436
deprecate dns search
80c26fc 
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If who encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
@wy65701436 wy65701436 mentioned this issue Sep 8, 2021
Merged
wy65701436 added a commit to wy65701436/harbor that referenced this issue Sep 8, 2021
@wy65701436
deprecate dns search
faf27c5 
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
wy65701436 added a commit that referenced this issue Sep 8, 2021
@wy65701436
deprecate dns search (#15557)
93a078d 
For details, please refer to #14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by #6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
lindhe pushed a commit to lindhe/harbor that referenced this issue Sep 22, 2021
@wy65701436 @lindhe
deprecate dns search (goharbor#15557)
8895a3b 
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
Signed-off-by: Andreas Lindhé <andreas@lindhe.io>
prahaladdarkin pushed a commit to prahaladdarkin/harbor that referenced this issue Nov 12, 2021
@wy65701436 @prahaladdarkin
deprecate dns search (goharbor#15557)
36c8e3a 
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
prahaladdarkin pushed a commit to prahaladdarkin/harbor that referenced this issue Mar 13, 2022
@wy65701436 @prahaladdarkin
deprecate dns search (goharbor#15557)
8715a4f 
For details, please refer to goharbor#14146 (comment)
and docker/for-linux#1164.

If anyone encounter the issue mentioned by goharbor#6031, add the dns_search: . to the releated container.

Signed-off-by: Wang Yan <wangyan@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wy65701436 wy65701436

Labels
kind/bug target/1.5.4 target/1.7.0 Work items that should be delivered in 1.7.0 timeframe.
Projects
None yet
Milestone
Sprint 45
Development

No branches or pull requests
4 participants
@reasonerjt @wy65701436 @ywk253100 @dbarkelew