Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extended Registry API v2 Support (images details, labels support, tag list) #84

Merged
merged 2 commits into from Apr 11, 2016
Merged

Extended Registry API v2 Support (images details, labels support, tag list) #84

merged 2 commits into from Apr 11, 2016

Conversation

arthurdk
Copy link
Contributor

I added some features to this project which work with the current registry API (v2.0):

  • Re activated the image details view : both image ancestry & size are disabled but the size is kind of easy to implement (i.e. a head request on the blobs end point). Every other things are functional as they were in the v1 branch.
    snapshot6

  • Display Docker image Labels in the image details view

    (to do : display labels from the ancestry)

snapshot7

  • Display the same information in the tag list as it was in the v1 branch (that is to say with some more details & with a link that leads to the image details view but again image size is obviously not implemented).

snapshot5

By the way, awesome work with this project and the way you set up the developer environment. Its really easy to use & nice to work with !

@kwk
Copy link
Owner

kwk commented Jan 13, 2016

Hi @arthurdk . Thank you for this big addition to the project and your kind words. I really appreciate it. Let me have a look at the changes.

@kwk kwk self-assigned this Jan 13, 2016
/**
* Calculates the total download size for the image based on
* it's ancestry.
*/
/*
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you effectively delete code by commenting it out, please simply remove it instead of keeping it. I left it there to pick it up when I have time. Now that you've implemented it, there's now need to keep the old code around unless there's something you want to implement in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That(s because I already started to implement it but anyway there are a lot of comments to remove I agree.

@arthurdk
Copy link
Contributor Author

So I've made the changes but I kept the comment on the size divs as I'm working on it. (see registry-services.js and images-controller.js for more info on size computing progress).

If you want to, you can wait to merge the PR until I finish the work on the image size.

@arthurdk
Copy link
Contributor Author

I will do another PR for the size feature, as I don't have enough time to finish it until some days.

So if you don't find problems then this batch is ready to go. :)

@@ -20,11 +20,18 @@ angular.module('tag-controller', ['registry-services'])
$scope.repositoryName = $route.current.params.repositoryName;
$scope.repository = $scope.repositoryUser + '/' + $scope.repositoryName;
$scope.tagName = $route.current.params.tagName;

// How to query the tags
$scope.tagsDetails = [];
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tagsDetails looked suspicious to me since you fill the tags with all details below. By grepping through the code of yours I found that is not used any longer. There fore I suggest to remove this line.

@arthurdk
Copy link
Contributor Author

Your latest suggestions are implemented.

If v1Compatibility does not appear in the server response then it will cause no error.

@HadesArchitect
Copy link

Any news? Really looking for merge! :)

@kwk
Copy link
Owner

kwk commented Jan 20, 2016

@arthurdk Thank you for all your work.

I would really like to merge this PR but what's hindering me from doing so is the amount of manifest GET requests that are necessary to list the details (author, docker version, etc.) for all tags inside a repository. We have images with ~200 tags (don't ask why ;) ) and then 200 requests need to be made. This slow the the browser down and leads to an unusable UI.

Let's brainstorm what our options are, shall we?

I don't see a way to get rid of the manifest GET requests per tag but we could introduce pagination for listing tags. Then the amount of requests per image is determined by the number of tags you allow per page. At the time I tried to introduce pagination for tags, the URL paramters (n and last) were simply ignored by the registry. I hope this is fixed now.... double checking .... No, they are still not respected. Maybe we should open an issue on the docker distribution project.

@arthurdk
Copy link
Contributor Author

@kwk Yeah I commented out the code that calculate image/layer size as I still have problem to get it totally working. (it'll be implemented for later)

I agree with the facts that it generate a lot of requests.

If the last registry version is still ignoring with the parameters for pagination (I'll check this too) we can use a temporary solution: only display tag name.

I agree that its not optimal but right now it'll be better than nothing.

@arthurdk
Copy link
Contributor Author

Ok, so I managed to get the registry working with pagination (I'm talking about the tag GET request) but my implementation is far from being clean so I need some more time before I can PR it to docker/distribution.

So in the mean time we appear to have 2 choices :

  • simulate pagination
  • only display tag name and access details by following the link (i.e. wait for pagination)

@kwk
Copy link
Owner

kwk commented Jan 22, 2016

@arthurdk Can you please share the code changes you did by checking them into a branch of your fork of docker/distribution? It doesn't matter if it is not polished. You can always git push -f origin yourbranch to overwrite any code you're unhappy with.

Let's recap, we both think that pagination is one solution to the problem of too many requests per page of a tag listing, right?

Although I would normally vote against a simulation of pagination there are some benefits to it.

Pros of simulation:

  • We have full control of the pagination.
    • For instance, the pagination of repositories lacks previous page or jump to page x controls. Which we can provide.
    • We can provide the actual number of tags for an image. By looking at the implementation and API to pagination repositories this is something we cannot expect from the official API.
  • We can deliver something earlier than waiting for docker/distribution to catch up. I've tested the API for pagination of tags a long time (over half a year) ago.

Cons of simulation:

  • we have to implement pagination on our own,
  • we always have to query for all tags of an image on each page of a tag listing unless we cache it in local storage or so,
  • we (still) cannot sort tags by date or any other tag detail for that matter. Without pagination that would be possible. But it isn't possible even with the official pagination API.

I have to think about this. But I'm in favor of not building something that is broken by design.

@arthurdk
Copy link
Contributor Author

@kwk It's planned, I'll share the code on my fork as soon as possible. I managed to get an almost working implementation but it lacked the link header which permits correct pagination (see. the pagination documentation). I do not have this version any more (I didn't commit this one as it was just a "proof of concept") as I'm working on the proper implementation. So I think I'll have something to share this weekend.

I totally agree with your recap and simulation seems to be the right choice from an user point of view. But regarding the way repository pagination is implemented (i.e. it can list repositories up to a 100 at maximum) so if the same happen for tag pagination in the future, it'll break our simulation. Simulation of pagination requires to have the ability to fetch all tags at once and its something that might change.

In my opinion its something we also have to discuss on docker/distribution.

Even if this happen we'll still be able to fetch all tags with n requests (n depending on the maximum numbers of tags we can get in a response and the number of tags stored in the registry). But it will still be risky to send a lot a requests because if some of them fail, our pagination system might broke. So simulate the pagination will be doable but kind of risky and as you listed them in your pros it can bring features such as previous page etc.

I don't know if I'm thinking too far ahead of this problem or not. What do you think of it?

@arthurdk
Copy link
Contributor Author

arthurdk commented Feb 1, 2016

@kwk Sorry for the delay, I was a little busy.

I added a basic pagination system for tags and set the default number to 10 (see readme).

Configuration is also possible via the corresponding environment variable.

@kwk
Copy link
Owner

kwk commented Feb 1, 2016

Hi @arthurdk Wow, I wasn't aware you would actually go about and implement this. I try to check it out tomorrow.

@arthurdk
Copy link
Contributor Author

arthurdk commented Feb 1, 2016

I thought that having the possibility to fetch images details is a nice feature so I'm doing everything I can so its fully working. :)

But my solution is basic and as I was saying the API implementation might change. In the mean time its simple and working so in my opinion it's good to have it even if it may be changed in the future. (I'll look into that as I am following registry development).

@kwk kwk added ready and removed ready labels Feb 3, 2016
@arthurdk
Copy link
Contributor Author

arthurdk commented Feb 4, 2016

Hi @kwk , is the removal of the label an error or did you find any problems during your review ?

@tomsherrod
Copy link

Screenshot has machine information best not tossed out.

Steps:
Select a tag url.
In the browser bar, back/erase up to the tag/
It give gives the "Index of /tag" directory.

I've mitigated it by "touch index.html" in each of the directories in the
/app directory. Alternative may be disable index in the apache config?

Tom

On Thu, Mar 10, 2016 at 10:19 AM, Arthur De Kimpe notifications@github.com
wrote:

@tomsherrod https://github.com/tomsherrod Hi, thank you for your
interest. :)

Can you take a screenshot ? I'm failing to reproduce your issue.


Reply to this email directly or view it on GitHub
#84 (comment)
.

@arthurdk
Copy link
Contributor Author

@tomsherrod My bad, I was using the dev environment which uses grunt.

It is not really related to this PR though as the bug was already present. Anyway it's corrected, the apache config lacked Options -Indexes

@kwk I just figured out that the Dockerfile in /develop is based on ubuntu whereas the standart one use for release is based on debian jessie. Is it a mistake ? It would be better to have both based on the same parent image.

@tomsherrod
Copy link

@arthurdk An observation: Image id shown in the Image details ui is not matching up with the later docker clients.
When I pull with a 1.6.0 client, docker images listing of image id matches the registry ui.
When I pull with a 1.8.3 or 1.9.1, their docker image ids are different from the registry ui. (They do match each other.)

Is this because the image id is being pull from the v1Compatibility data?
(The image is built with 1.9.1 and pushed to a v2 registry.)

Am I missing something?
How is image id being calculated with the newer clients?

@arthurdk
Copy link
Contributor Author

@tomsherrod I think the issue you are having is a bit similar to this one (see also this one).

I've experienced the same thing when pulling from Docker Hub with two different host running different version of Docker: when performing docker images on both, the image id shown was indeed different.

@kwk
Copy link
Owner

kwk commented Mar 22, 2016

@arthurdk I can confirm that the tags pagination issue is gone.
I thought I already wrote about the other issue I see but apparently I didn't. :(
Here it goes: Can you implement changing the number of tags per page just like we have it for repositories (see drop down list at between "First page" and "Next page" button)? That would make things more consistent. Sorry for giving this feedback so late.

@arthurdk
Copy link
Contributor Author

@kwk It's done, the drop down list is now present and the previous button is replaced by a go to "first page" button.
Of course, it's still not implemented using the pagination API of the registry.

@FrontSide
Copy link

Hey devs,
What's the status here?

@kwk
Copy link
Owner

kwk commented Apr 7, 2016

Hey @arthurdk . I've manually tested your changes and would like to ask you to squash all your changes into a single commit and force push it to your v2 branch. I'd like to have the chance to revert a PR if something breaks for others and reverting one commit is far more easy than reverting 15 commits. Please make sure your changes are rebased onto the current master.

After that I think we can merge it.

@0x46616c6b
Copy link

github has now the possibility to squash when you merge it. You need to enable this in the repository settings. https://github.com/blog/2141-squash-your-commits

arthur added 2 commits April 7, 2016 22:44
Revert changes on default projet file

Add detailed information for tag listing

Removed useless comments, divs, etc -> refactoring

Prevent bugs from future changes of API

Add digest attribute to Manifest query response

Add basic pagination support to tag listing

Add environment variable for tags per page

Fix bug of image history without config + disabled parent id

Fix tags pagination system - Fetch infos for all pages

add missing comma

 Fetch infos for all pages -> simpler fix

Disable apache directory listing feature

Update tag pagination system to make it feel more like repository pagination system
@arthurdk
Copy link
Contributor Author

arthurdk commented Apr 7, 2016

Hi @kwk !
All my changes are now squashed in a single commit & the merge is done. I've tested it briefly and I did not notice a single problem. (it's normal because only changes from upstream were a downgrade of libs).

@nikolaykolev
Copy link

3 months for 1 pull request?!? -.-

@kwk kwk merged commit d5dd87d into kwk:v2 Apr 11, 2016
@Filirom1
Copy link
Contributor

Whouhou !!! 🍻

@nikolaykolev
Copy link

🍻🍻🍻

@eLvErDe
Copy link

eLvErDe commented Apr 11, 2016

We all would love to get the docker image updated now 👍

@flixr
Copy link

flixr commented Apr 11, 2016

It seems like the detailed views, etc are disabled because start_apache.sh still contains ENV_MODE_BROWSE_ONLY=true?

Although it would be nice to also show the details in browse mode...

@FrontSide
Copy link

it would be nice to also show the details in browse mode

Works for me. Or do you mean it's not visible when you set
ENV_MODE_BROWSE_ONLY=false ?

@flixr
Copy link

flixr commented Apr 11, 2016

@FrontSide are you using the develop mode?
At least when pulling the latest docker image (correctly shows git revision as d5dd87d) it is not showing any details.
And my guess is because ENV_MODE_BROWSE_ONLY if always forced to true in start_apache.sh

@kwk
Copy link
Owner

kwk commented Apr 11, 2016

Browse only mode shouldn't influence the details. It just disables showing buttons for tag creation an deletion.

@FrontSide
Copy link

@flixr
I'm setting ENV_MODE_BROWSE_ONLY=true which is Browse mode.

At least when pulling the latest docker image...

You need to pull the :v2 image.
I'm using the provided docker image as well.

@lukeshaughnessy
Copy link

I'm also not seeing the details. I have pulled the latest image as such:

sudo docker run -d -e ENV_DOCKER_REGISTRY_HOST=myhostfoo.com -e ENV_DOCKER_REGISTRY_PORT=443 -e ENV_DOCKER_REGISTRY_USE_SSL=1 -p 8080:80 konradkleine/docker-registry-frontend:v2

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0a0e9b915745 konradkleine/docker-registry-frontend:v2 "/bin/sh -c $START_SC" 7 minutes ago Up 7 minutes 443/tcp, 0.0.0.0:8080->80/tcp tiny_mcnulty
c1c3ee315a56 nginx:1.9 "nginx -g 'daemon off" 10 days ago Up 10 days 80/tcp, 0.0.0.0:443->443/tcp dockerregistry_nginx_1
1442048fab5b registry:2 "/bin/registry /etc/d" 10 days ago Up 10 days 127.0.0.1:5000->5000/tcp dockerregistry_registry_1

@flixr
Copy link

flixr commented Apr 11, 2016

Browse only mode shouldn't influence the details. It just disables showing buttons for tag creation an deletion.

That would be good, but in any case I don't see any detailed image info... and yes, I'm using the :v2 image

@lukeshaughnessy
Copy link

a note: I notice that the red "Select Repositories to Delete" button flashes briefly when opening the Repositories page, as if they are loading but are suppressed.

@flixr
Copy link

flixr commented Apr 11, 2016

Same here, although that shouldn't be the problem..

Screenshot of the latest image pushed to our own registry, browsing with the latest v2 image from docker hub:
selection_001

Any hints on howto debug this?

@arthurdk
Copy link
Contributor Author

I just tried the last v2 image pushed to the Docker Hub and it worked just fine on my hand.

The command I ran in order to test the detailed view :

docker run   -d -e ENV_DOCKER_REGISTRY_HOST=registry  -e ENV_DOCKER_REGISTRY_PORT=5000   -p 8080:80 --link registry:registry  konradkleine/docker-registry-frontend:v2
docker run -p 5000:5000  --name registry registry:2.3.1

Please be sure to use the last v2 image available.

Are you having errors on your browser console ?

@flixr The fact that the git revision is correct but not the view is weird... @kwk Is it possible that older image show the latest git revision ? ( I did not have time to check how the revision is stored/retrieved )

@kwk
Copy link
Owner

kwk commented Apr 12, 2016

@arthurdk no that is not possible. But since the version is loaded from a file it might be one of the few things that is not affected when the browser cache hasn't been flushed. Please always try to use a clean browser state when testing. I had trouble seeing changes I made just due to the cache.

@lukeshaughnessy
Copy link

Tried with fresh install of FF, details now display. Assuming Chrome what not clearing cache entirely.

@flixr
Copy link

flixr commented Apr 12, 2016

Ok, the problem was indeed the cache, specifically /tag/tag-list-directive.html (and also app-mode.json).

In the network monitor I could see that the Last-Modified in the Response header of tag-list-directive.html was old, e.g. the one in repository-detail.html not...
Reloading with "skip-cache" didn't work (also already tried that yesterday).
What did work is going to http://frontend.docker.foo/tag/tag-list-directive.html and reloading that page. After that it also showed it correctly in the repository view.

So is this really expected, or is there something not set up correctly so that it doesn't even attempt to get the latest tag-list-directive.html?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet