How to create a new collection or a new container using the API with curl?

[pini-gh]

I need a way to create new private / public collections / containers via the command line.
Thanks in advance.

[vsoch]

I had originally removed this because I was concerned about some kind of spammy operation, but with authentication it's reasonable. Would you care to open a PR to do it? I just started a new job so I won't have time for at least a few weeks.

[pini-gh]

Well, I'm all for doing it, but I'd need a few hints :/ If this was removed, maybe there is an old commit I could start with?

[vsoch]

The previous implementation used the (now mostly) deprecated original push/pull container API, which is replaced with the library API. The first thing I'd do is check with sylabs if their library API has support for this kind of creation, and if yes, add it to the library API. If not, then it can go as a separate API function that has the same authentication check, and some ratelimiting.

[pini-gh]

Yes, their API do have support for it. Actually I need to set up a separate repository from an existing Sylabs installation, and I think I just miss these two features to be able to go with sregistry.

[vsoch]

Ah, great! So just reproduce those exact endpoints. You'll likely want to add a new file for collections in this folder https://github.com/singularityhub/sregistry/tree/master/shub/apps/library/views and you can copy the style of functions from images.py, for example (to have the correct ratelimiting and authentication checks). What I did to design this API was basically trace / look at the Sylabs SCS Client, so if they have functions for a new collection it should be somewhere in https://github.com/sylabs/scs-library-client. You can ask them on slack if it's not easy to find (I'm not familiar with this endpoint via Singularity).

[pini-gh]

Is that what you're talking about?
https://github.com/sylabs/scs-library-client/blob/82d22e604229c73a27d339106ebcb2c6e137e03f/client/api.go#L78-L112

[vsoch]

yes that's perfect! In that function you can see what the data is expected to look like, and the response. When I was developing the original library API, what I found helpful was to compile a version of singularity with print statements so I could see what the responses were expected to look like (and do this first against the Sylabs cloud for a working operation). Then you basically just duplicate that for sregistry.

[pini-gh]

With the sregistry data model, is it possible to have several collections with the same name, provided they are associated with different entities? It is possible with Scilabs, but here it seems I can search a collection by its name only, and not by its name + entity.

[vsoch]

Since a collection can have multiple owners, and since for the filesystem storage model we need to ensure uniqueness, the collection names must be unique https://github.com/singularityhub/sregistry/blob/master/shub/apps/main/models/shared.py#L49. When you say entity, do you mean container? Different collections can indeed have containers with the same name.

[vsoch]

I usually use this function https://github.com/singularityhub/sregistry/blob/master/shub/apps/library/views/helpers.py#L299 to retrieve a container based on a URI, if that's what you need.

[pini-gh]

As I understand it, in the sylabs data model, containers are identified with their path:

/entity/collection/container:tag

There is no collection or container name uniqueness. Only path uniqueness.

An entity is a user. She can hold several collections. With Sylabs each entitiy is created with a default collection named 'default'
A collection is an arbitrary goup of containers.
A container is an image name, and can exists in different versions represented as tags.

[vsoch]

In your example, entity corresponds with a username. singularity Regostry server allows collections to have > 1 owner (entity) so this is not possible.

This was designed years before Sylabs had a library, so it has its own database model.

[pini-gh]

I now have a working POC for collection creation via the API. I propose I submit it as a PR so you can review it. What do you think?

[vsoch]

Awesome! Yes please open a PR, I should have some time this weekend to review. Make sure that you:

• edit the documentation to walk the user (who will be me when I am testing!) through using the new endpoint with the Singularity client, and since you are also using CURL and have figured that out, it wouldn't hurt to include that as an example too.
• bump the version in the VERSION file in the root of up the repo.
• add an entry to the changelog with what you did, along with the new version.

If you still don't have black don't worry about that - I can run it again for you! I can also help with buffing up the docs when you have a basic walk through (so don't worry if it's not perfect). Looking forward to it!

[vsoch]

Closed with #345