Upgrade to 8.8.0

[solidnerd]

I made a PR for it, but i guess it's not released yet only the tag was made. So we need to wait with this.

EDIT:
For everyone who wants to try it out. I have been created a Gist for all configurations that are needed.

[solidnerd]

In my opinon we could merge it. An official Announcement is published.

Ping @sameersbn

[sameersbn]

@solidnerd there are a few additional changes that need to be done. I am working on adding those. Will merge with the changes soon

[solidnerd]

Okay. It seems an admin bug exists currently. So we should wait until this MR is merged.

[sameersbn]

@solidnerd oh.. thanks for pointing it out. I was trying to debug that.

[solidnerd]

So any things left to do for the update ?

[sameersbn]

That seems to be some other issue.
I dont see the admin user (root) being created on first boot. Specifying GITLAB_ROOT_PASSWORD also does not do anything.

[sameersbn]

This is the one major issue I see making the release.
Other things are pretty basic like updating the gitlab.yml config and nginx configuration templates.

[solidnerd]

The admin bug will be fixed in 8.8.2 . It's mentioned in the Changelog .

[solidnerd]

@sameersbn I added all related configuration options for the container registry. Tomorrow i will look at the nginx configuration template. After this is finished I will try to create the admin user. If you found more related stuff for the release.

[aranw]

How will this work with GitLab Registry Container? Will there be examples on how to link docker registry and gitlab together?

[lenovouser]

Typo

+Default
-Defaults

or

+to
-is

[solidnerd]

@aranw Thanks for the question. I didn't test it completely. After the test is finished I will try to write a guide for that. I guess the best place to start will be the GitLab Container Registry Administartion Documentation .

The communication should be implemented with docker networks, because we have only a docker-compose.yml V2 and this uses docker networks. The second argument would that docker marks the --link API as deprecated since Docker 1.9 and I don't know when they drop it completely . In my opion the only way would be with docker networks.

[lenovouser]

Typo

+API
-Api

[vitorarins]

Hi, will this be merged after release 8.8.2 from Gitlab? Or will it be merged now and another PR will update to 8.8.2? I am sorry for my confusion, I am not sure how you guys are working with versions here. Thank you for the great work!

[sameersbn]

@vitorarins I think we'll directly release 8.8.2 or whichever version fixes the admin account creation.

[sameersbn]

@solidnerd do you see the admin account creation issue?

[solidnerd]

@sameersbn Yes. It's needed to be fixed. I guess we missunderstood us, because i mean with the admin bug another problem. My admin bug was when you enter the admin/application_settings you get an 500.

I have no idea if anyone has reported this issue to gitlab and if it's a real gitlab issue. I need some time to investigate this.

[sameersbn]

@solidnerd the admin account creation issue seems to be something that is specific to this image. Tried gitlab/gitlab-ce:8.8.1-ce.0 and had no problems. Will need some time to debug it.

[solidnerd]

@sameersbn Okay. In the meatime I will try to test the container registry feature completly to ensure that it works or problems are by gitlab itself.

[mgansler]

How will the registry work if gitlab itself is running behind a reverse proxy?

[solidnerd]

@mgansler At the moment I don't know. My production setup has a reverse proxy. After i finished all tests i will try to make a duplicate of my production system and test it with this setup. But this needs some time. I guess that I have finished the tests tomorrow in the afternoon and I will report you about my results.

[lenovouser]

@sameersbn v8.8.2 has been released by the way. So this should be good to merge, right?

[boonkerz]

i have no registry entry in my project settings
-e 'GITLAB_REGISTRY_ENABLED=true'
should be enough?

[solidnerd]

@lenovouser Currently an issue exist that you can't create an admin account with a fresh version of 8.8.0 this must be fixed before we can merge it.

@bboehmke You need to configure all related GITLAB_REGISTRY_* env's.

[boonkerz]

@solidnerd ok because but why the defaults?
it looks like they are preset :)

[solidnerd]

@boonkerz You are right. It's currently a bug that the variables aren't subsituted correctly. I will fix this now..

[blandes]

@sameersbn I know that I'm not a common contributor but I think that the best route would be option 1. The thing is, the container registry is awesome and I'm so happy that they brought it into GitLab.

But as a DevOps guy, that would mean that I would need to change my entire DevOps CI/CD cycles based on this registry. Like mentioned above, I want to be able to sync this to S3 just because of safety and so on. I would have to reconfigure every Developer's computer, or at least inform them, of the new migrations to the GitLab container registry. Let's pause on the registry inclusion and focus on the actual update. You guys are great at these images, and I know that you will make the right decisions but some of us need the new features that 8.8.0 - 8.8.2 bring now. It's not like I'm going to immediately switch my Dockerhub registry to this container registry without the proper safety measures anyways.

Regardless, thanks for all the hard work. You guys are the backbone.

[bharrisau]

Seems like GitLab has separated the registry from the http proxy. You should be able to disable the omnibus registry and point the url to another docker container. Looks like it is just the TLS setup between registry and gitlab that is tricky.

[solidnerd]

Okay I will throw my 2 cents in how we could proceed. I understand the point that the container registry could lead to instability of this image. The current state is that the container registry is disabled by default. So any user of this image should have no problems if he doesn't use this feature.

In the first point of view I like the idea from @mgansler to put in the binary in this image. This will make a easier setup, but this produce a high coupling of dependencies and in the future It will be harder to maintain it. A example of it why it's hard to support dependencies. Is NGINX in my opionion, because it's not easy to implement Let's encrypt directly in this image you need to do a lot of stuff to achieve this. The first target should be GitLab by itself and not any services that are needed for a GitLab feature. So In my opinion the current way is the best.

Most common problem of the container registry is the lack of documentation. It tooks me a day to find out that the official container documentation is not correct. Another problem of the container registry is that it's really new and have a lot of early adopting problems.

The best way how we could handle all this would be option 1 from @sameersbn upgrading to 8.8.X and do not any configuration changes except they are implemented before and treat this PR as feature branch for the container registry.

How we could make a image based on this feature so that early adopters could test it and report problems because every knows that with the time you will be routine-blinded. Another step is to document how to setup the container registry. We could use my Gist as starting point for the documentation.

@bharrisau It's the concept how the container registry works. To run it you need the docker registry for storing images and distribute images. The part of GitLab is to authenticate a user for doing this actions against the docker registry.

[mgansler]

@solidnerd regarding your Gist:
I don't think you actually need the tls (certificate/key) entries in your config.
According to https://docs.docker.com/registry/configuration/#tls they are optional - and indeed, the Gitlab Nginx is handling SSL Termination for the registry. It all depends how you expose the registry to the outside world.
And you have three choices here, directly (docker run -p 1.2.3.4:5000:5000 registry), via the Gitlab Reverse Proxy (docker run --net gitlab_default -p 172.x.y.1:5000:5000 registry) or have another Reverse Proxy in front of Gitlab AND the registry.
In my opinion, the tls setting is only necessary with the direct option.

Also, if I understood it correctly, the rootcertbundle should (must?) not be the same as the SSL certificate.

Looking at distribution/distribution#520, I get the feeling that it should be indeed different from the SSL cert.

[solidnerd]

@mgansler You are right. I didn't know that tls is optional this would make an integration a lot of easier. You could use the registry in the same docker-compose.yml for GitLab as a standalone services and configure all related stuff via environment variables.

To the rootcertbundle yes it's same like you said. So what i read from docker/distribution#520, it can be every certificate but it's must be a trusted certificate or you must add to the trust store of os.

Thanks for the clearification. I will wirite up a docker-compose.yml later that supports everything that is needed for it.

[mgansler]

@solidnerd I don't think it even has to be trusted. It's just a private/public key used to sign/verify the token.
I generated one myself and it works fine. I didn't add it to neither the registry nor the gitlab container.

I found these instructions somewhere (I can never remember how to create a certificate):

openssl req -new -newkey rsa:4096 > registry.csr
openssl rsa -in privkey.pem -out registry.key
openssl x509 -in registry.csr -out registry.crt -req -signkey registry.key -days 3650

docker-compose.yml:

services:
  gitlab:
    environment:      
      - GITLAB_REGISTRY_ENABLED=true
      - GITLAB_REGISTRY_HOST=registry.example.com
      - GITLAB_REGISTRY_PORT=443
      - GITLAB_REGISTRY_API_URL=http://registry:5000
      - GITLAB_REGISTRY_KEY_PATH=/certs/registry.key
      # next two lines are only there because there would be errors with nginx otherwise
      - SSL_REGISTRY_KEY_PATH=/certs/registry.key
      - SSL_REGISTRY_CERT_PATH=/certs/registry.crt
    volumes:
      - ./certs:/certs

  registry:
    volumes:
      - ./certs:/certs

config.yml

auth:
  token:
    realm: https://gitlab.example.com/jwt/auth
    service: container_registry
    issuer: gitlab-issuer
    rootcertbundle: /certs/registry.crt

[vitorarins]

The docker registry is a real pain to install. Even a really good documented application like Portus is difficult to set it up, depending where you want to install things. So I think we are going to wait forever until we make things "easy" to set up. I say we could release this, after all it is optional, and work on the documentation. The gist is a good starting point, I can help showing what I did with HAProxy and systemd units for a CoreOS install of a registry. I started some work here: https://github.com/NeowayLabs/deploy-portus but there is a lot to do.

[tkaefer]

Hi there - just my two Cents.
Let's Encrypt is like no option for people that use GitLab with a corporate and protected environment. And there are actually on nginx-reverse-proxy container around that could do this trick, like https://github.com/eforce21/letsencrypt-nginx-proxy.
I would vote to just upgrade this image to GitLab 8.8.x and see if some one is able to do all of the certificate magic for own hosted CAs (which is mostly a pain somewhere in the lower area).

Nonetheless, I am really looking forward to see the docker registry included here - which might come later and I am totally fine with this.

Cheers.

[lenovouser]

I would rather see you guys including the registry now so that people who really want to use it can do so - and work on refining the image and documentation later. I've been hogging this issue for the past days because I really want to use the registry 😄

Also, AFAIK is the whole certificate stuff for the registry unnecessary - since you proxy it with NGINX anyway like you do it with GitLab too.

[ShakataGaNai]

Personally, I really want the registry integration. That was supremely exciting to me. However if getting the registry out the door "today" means a possibly not great setup that could potentially break people and/or break on upgrade, then I'd say wait on that portion. So Option 1 seems like the logical choice.

[hanej]

In my view, the primary responsibility of GitLab, and by extension this Docker image, is to provide a stable environment for a centralized git repository. Anything else is extra. I would much rather have a reliable image of GitLab that works well than try to add something else in that may not be quite ready yet.

From what I've read, the GitLab Docker registry still needs some improvement as it requires Docker-in-Docker running in privileged mode. That's not the way I want to operate. As an administrator who needs GitLab to be reliable for the entire company's source code, I would much go with Option 1 and wait a little longer until the kinks get worked out.

[mgansler]

@hanej what exactly has to run in privileged mode?
AFAIK, you MAY need privileged mode for certain CI functionality (spawning and linking services).
But this is nothing new.

Also: http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ (written by the guy behind dind)

I just tested it: for docker build/tag/rmi/push there is no need for privileged mode at all. All you need is the docker binary and access to the docker socket.

[hanej]

@mgansler Your docker container has to run in privileged mode if you're using dind and GitLab runners

Notice that it's using the privileged mode to start the build and service containers. If you want to use docker-in-docker mode, you always have to use privileged = true in your Docker containers.

http://docs.gitlab.com/ce/ci/docker/using_docker_build.html

[hanej]

See also https://gitlab.com/gitlab-org/gitlab-ce/issues/17769

[mgansler]

@hanej thanks for the link to the issue, but I don't completely understand their argument.
docker build (and therefor docker run) works just fine, docker push as well. So no limitations as far as I can tell.

Security is a different Story - mounting /var/run/docker.sock basically exposes the entire Host to users of gitlab - they could mount all kinds of directories. It could be used for spam and whatnot. But the same applies to dind. Only it's limited to everything inside this docker container.
But isn't it a good Idea to have the runners running on isolated hosts (VM?) anyway?

[sameersbn]

I am going to cherry pick the commits from this PR and ready a release without the contaienr bits. This will be followed by the creation of a branch with the container specific stuff this PR add and will setup and automated build for it so that users who want to setup and use the container registry feature can do so.

Oh shit. just realized this PR makes one gaint commit. 😢

[solidnerd]

At all. I can say that the container registry is working now. I guess that I have found a bug in GitLab that causes the issues with the container registry but i have fixed it locally. My next plan was to test it all and after that i will cleanup my commits and rebase it.

@sameersbn Sorry. I guess we should close this PR and make a new one for the upgrade only or create a patch of it. I could do it if you want.

[sameersbn]

@solidnerd no worries.. I have the branch with your individual commits. Will be making the release a few minutes from now 😄

[solidnerd]

@sameersbn Do you upgrade directly to 8.8.2 ? And can you create the branch for the Container Registery so i can Start a new PR and working on it. After that we can close this pr here.

[sameersbn]

I am publishing all the intermediate releases too. I will create a branch with all your work and give you commit access to the repo 😉

[sameersbn]

release 8.8.0 thru 8.8.2 have been published sans container registry support 😢

[solidnerd]

For everyone who wants to try the registry check this #PR 708 out.

[sameersbn]

the builds are in queue.. may take a couple of minutes before you can pull them from dockerhub.

[sameersbn]

@solidnerd can you post the relavant links to the PR (like the gist your created)

[blandes]

Thank guys!

[solidnerd]

@sameersbn Yes. I see bug through the old commit and I guess it will currently not work. I have created a fix for that of with one my latest commits.