Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After upgrade, docker login breaks #2744

Closed
carlsone opened this issue Jul 11, 2017 · 15 comments
Closed

After upgrade, docker login breaks #2744

carlsone opened this issue Jul 11, 2017 · 15 comments

Comments

@carlsone
Copy link

carlsone commented Jul 11, 2017 •
edited

I have been using Harbor version 1.1.1 ok. When I do an upgrade to version 1.1.2 (https://github.com/vmware/harbor/releases/tag/v1.1.2), it works ok too.

But when I try to upgrade to a newer version found on https://console.cloud.google.com/storage/browser/harbor-builds/ using the same upgrade steps, I am not able to do a docker login anymore, due to these errors:

$ docker login ${HARBOR_IP}:443
Username (admin):
Password:
Error response from daemon: Login: <html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.11.5</center>
</body>
</html>
 (Code: 404; Headers: map[Server:[nginx/1.11.5] Date:[Tue, 11 Jul 2017 22:19:19 GMT] Content-Type:[text/html] Content-Length:[169]])
docker login failed

I have done this with version 640 and 734. Both fail the same way. Logs and more info below:

If you are reporting a problem, please make sure the following information are provided:
1)Version of docker engine and docker-compose.

Docker version 1.13.1, build 092cba3
docker-compose version 1.11.1, build 7c5d5e4

2)Config files of harbor, you can get them by packaging "harbor.cfg" and files in the same directory, including subdirectory.

See harbor_logs.ZIP below (actually a .tgz file).

3)Log files, you can get them by package the /var/log/harbor/ .

harbor_logs.ZIP

  1. steps to reproduce the problem
    Download any of the v1.1.2 installers from https://console.cloud.google.com/storage/browser/harbor-builds/

Run install script (which I got from #2208 ):

tar -xzvf harbor-offline-installer-v1.1.2-734-g4fcbece.tgz
cp -v /etc/vmware/harbor/docker-compose.yml harbor/docker-compose.yml 
cp -v /data/harbor/harbor.cfg harbor/harbor.cfg 
# Shut down old harbor
pushd /etc/vmware/harbor
docker-compose down
rm -v /data/harbor/secretkey
popd
# Install new harbor
pushd harbor
./prepare
if [ -f /data/harbor/secretkey ]; then
   docker-compose up -d
fi
docker-compose ps
popd

See Output:

Generated and saved secret to file: /data/harbor/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/app.conf
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.
Creating network "harbor_harbor" with the default driver
Found orphan containers (notary-server, notary-signer, notary-db) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating harbor-log
Creating registry
Creating harbor-db
Creating harbor-adminserver
Creating harbor-ui
Creating harbor-jobservice
Creating nginx
stty: 'standard input': Inappropriate ioctl for device
       Name                     Command               State                          Ports
------------------------------------------------------------------------------------------------------------------
harbor-adminserver   /harbor/harbor_adminserver       Up
harbor-db            docker-entrypoint.sh mysqld      Up      3306/tcp
harbor-jobservice    /harbor/harbor_jobservice        Up
harbor-log           /bin/sh -c crond && rm -f  ...   Up      127.0.0.1:1514->514/tcp
harbor-ui            /harbor/harbor_ui                Up
nginx                nginx -g daemon off;             Up      0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 80/tcp
registry             /entrypoint.sh serve /etc/ ...   Up      5000/tcp
~
# exit

Now try to log into harbor after the install:

$ docker login ${HARBOR_VM}:443
Logging into harbor ...
Username (admin):
Password:
Error response from daemon: Login: <html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.11.5</center>
</body>
</html>
 (Code: 404; Headers: map[Server:[nginx/1.11.5] Date:[Tue, 11 Jul 2017 22:19:19 GMT] Content-Type:[text/html] Content-Length:[169]])
docker login failed
@reasonerjt
Copy link
Contributor

@carlsone Erin let me confirm, are you testing with the deployment with Harbor only? or with Harbor+Admiral?

@carlsone
Copy link
Author

Harbor only. But I think I tried it on a Harbor+Admiral VM and got the same error.

@wy65701436
Copy link
Contributor

Hi @carlsone,

For the latest version of harbor, the DB scheme changes. You need update DB firstly, and then to start latest harbor. I will let you know how to update DB, but it's still changing before FC.

-Yan

@carlsone
Copy link
Author

Is there a way to replace the database with the version included in the installer bundle?

I do not need to preserve or upgrade the old database. I am happy to replace it with a fresh db.

@reasonerjt
Copy link
Contributor

@carlsone You can clean up the directory /data/database/
And restart Harbor by running:

docker-compose down -v
docker-compose up -d

Note that in VIC product Harbor will be deployed with Notary and Clair, if you need that you need more options when deploying Harbor.
@wy65701436 can provide more details.

@carlsone
Copy link
Author

@reasonerjt, I will not need Nortary or Clair. I have tried cleaning-up the database directory, but get the same login error as before. Please let me know if you see a mistake in the following repro:

$ docker login viclient3.xxx.com:443
Username (admin): 
Password: 
Login Succeeded

Login works for 1.1. Now shut-down old harbor:

$ ssh root@viclient3.xxx.com
root@viclient3 [ ~ ]# cp -v /etc/vmware/harbor/docker-compose.yml harbor/docker-compose.yml 
'/etc/vmware/harbor/docker-compose.yml' -> 'harbor/docker-compose.yml'
root@viclient3 [ ~ ]# cp -v /data/harbor/harbor.cfg harbor/harbor.cfg 
'/data/harbor/harbor.cfg' -> 'harbor/harbor.cfg'
root@viclient3 [ ~ ]# pushd /etc/vmware/harbor
/etc/vmware/harbor ~
root@viclient3 [ /etc/vmware/harbor ]# docker-compose down -v
Stopping harbor-jobservice ... done
Stopping nginx ... done
Stopping harbor-ui ... done
Stopping harbor-db ... done
Stopping harbor-adminserver ... done
Stopping registry ... done
Stopping harbor-log ... done
WARNING: Found orphan containers (notary-server, notary-signer, notary-db) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Removing harbor-jobservice ... done
Removing nginx ... done
Removing harbor-ui ... done
Removing harbor-db ... done
Removing harbor-adminserver ... done
Removing registry ... done
Removing harbor-log ... done
Removing network harbor_harbor

(clean up database)

root@viclient3 [ /etc/vmware/harbor ]# rm -rf /data/harbor/database/
root@viclient3 [ /etc/vmware/harbor ]# ls /data/harbor/database
ls: cannot access '/data/harbor/database': No such file or directory
root@viclient3 [ /etc/vmware/harbor ]# mkdir /data/harbor/database
root@viclient3 [ /etc/vmware/harbor ]# rm -v /data/harbor/secretkey
removed '/data/harbor/secretkey'
root@viclient3 [ /etc/vmware/harbor ]# popd

(install new harbor)

root@viclient3 [ ~ ]# cd harbor
root@viclient3 [ ~/harbor ]# ls
LICENSE  NOTICE  common  docker-compose.notary.yml  docker-compose.yml  harbor.cfg  harbor.dev.tar.gz  harbor_1_1_0_template  install.sh  prepare  upgrade
root@viclient3 [ ~/harbor ]# ./prepare 
Generated and saved secret to file: /data/harbor/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/app.conf
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

(rebuild database?)

root@viclient3 [ ~/harbor ]# 
root@viclient3 [ ~/harbor ]# docker-compose up -d
Creating network "harbor_harbor" with the default driver
WARNING: Found orphan containers (notary-server, notary-signer, notary-db) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating harbor-log
Creating harbor-db
Creating registry
Creating harbor-adminserver
Creating harbor-ui
Creating harbor-jobservice
Creating nginx

(No errors)

root@viclient3 [ ~/harbor ]# 
root@viclient3 [ ~/harbor ]# exit
logout
Connection to viclient3.xxx.com closed.

(Try login)

$ docker login viclient3.xxx.com:443
Username (admin): 
Password: 
Error response from daemon: Login: <html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.11.5</center>
</body>
</html>
 (Code: 404; Headers: map[Server:[nginx/1.11.5] Date:[Mon, 17 Jul 2017 21:07:35 GMT] Content-Type:[text/html] Content-Length:[169]])

@wy65701436
Copy link
Contributor

hi @carlsone , can you use this build https://storage.googleapis.com/harbor-builds/harbor-offline-installer-v1.1.2-802-g2e3c257.tgz?

Please make sure to remove /data before using.

@carlsone
Copy link
Author

carlsone commented Jul 20, 2017 •
edited

Thank you @wy65701436, I have now tried version harbor-offline-installer-v1.1.2-802-g2e3c257.tgz

If I remove /data before using, I get this error when I run the prepare script:

# ./prepare
Traceback (most recent call last):
  File "./prepare", line 110, in <module>
    validate(rcp, args)
  File "./prepare", line 31, in validate
    raise Exception("Error: The path for certificate: %s is invalid" % cert_path)
Exception: Error: The path for certificate: /data/harbor/cert/server.crt is invalid

So I tried again. This time, I made a copy of /data/harbor/cert and /data/harbor/ca_download/, so I could put them back before running the prepare script:

~/harbor ~
Generated and saved secret to file: /data/harbor/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/app.conf
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.
Creating network "harbor_harbor" with the default driver
Found orphan containers (notary-server, notary-signer, notary-db) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating harbor-log
Creating registry
Creating harbor-db
Creating harbor-adminserver
Creating harbor-ui
Creating harbor-jobservice
Creating nginx
stty: 'standard input': Inappropriate ioctl for device
       Name                     Command               State                          Ports
------------------------------------------------------------------------------------------------------------------
harbor-adminserver   /harbor/harbor_adminserver       Up
harbor-db            docker-entrypoint.sh mysqld      Up      3306/tcp
harbor-jobservice    /harbor/harbor_jobservice        Up
harbor-log           /bin/sh -c crond && rm -f  ...   Up      127.0.0.1:1514->514/tcp
harbor-ui            /harbor/harbor_ui                Up
nginx                nginx -g daemon off;             Up      0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 80/tcp
registry             /entrypoint.sh serve /etc/ ...   Up      5000/tcp

But I still see the login error:

Logging into harbor ...
Username (admin):
Password:
Error response from daemon: Login: <html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.11.5</center>
</body>
</html>
 (Code: 404; Headers: map[Server:[nginx/1.11.5] Date:[Thu, 20 Jul 2017 20:33:08 GMT] Content-Type:[text/html] Content-Length:[169]])
docker login failed

Logs:
harbor802.zip

@wy65701436
Copy link
Contributor

May I have your env to have a debug?

@carlsone
Copy link
Author

May I have your env to have a debug?

Thank you for offering. I have sent you credentials over email.

Erin

@wy65701436
Copy link
Contributor

It's due to the upgrade process, not an issue of Harbor.

The predefined docker-compose.yml points to v1.1.1 Harbor images, it must align with offline installer version.

@carlsone
Copy link
Author

Thanks for your analysis @wy65701436. I will need additional help to align the v1.1.1 docker-compose.yml with the offline installer. I sent an email requesting more information. Thank you.

@carlsone
Copy link
Author

I stopped using the prepare script suggested in issue 2208, because it doesn't work for 1.2.0. Now I use the install.sh script. First, I had to modify docker-compose.yml to workaround issue 2134. Now I can do docker login.

@wy65701436
Copy link
Contributor

@carlsone, do you need further support for this issue. If not, can I close it?

@carlsone
Copy link
Author

You can close this bug. I would still like to see Issue 2134 fixed though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@reasonerjt @wy65701436 @carlsone