Support features with resources, e.x. shell scripts

[pgronkiewicz]

Sometimes a feature has some resources that would be copied to a docker image, e.x. official Golang docker image comes with go-wrapper:
https://github.com/docker-library/golang/tree/1eab0db63794152b4516dbcb70270eb9dced4cbd/1.5

[PetrGlad]

I have a very raw protoype at https://github.com/PetrGlad/pazuzu-registry/tree/32-attachments
Respective UI and CLI parts are missing at the moment.
It uses filesystem for storage. Potentially this can be used in it current form with root of file storage mapped to necessary backend (S3, Docker data volume).

[PetrGlad]

There is now working server-side implementation in https://github.com/PetrGlad/pazuzu-registry/tree/32-attachments. Missing piece is file checksums. See also https://github.com/zalando/pazuzu-registry/wiki/File-attachments for details.

[PetrGlad]

To complete implementation we need UI part (add/remove files, and highlighting in code editor for present/missing files in Docker instructions) and support from CLI side (fetch files from registry before build).

You can ask me if you have any questions regarding this.

[whiter4bbit]

@PetrGlad regarding API, File object has name which has unique constraint, this raises next questions:

• What name should be chosen by UI when uploading new files? (it should be unique and generated somehow)
• How it is possible to link already uploaded file with Feature? (From UI need somehow to figure out which file should be linked with particular feature - when names are uniquely - say randomly generated - user needs to find file that he uploaded)

[pgronkiewicz]

I think easiest solution would be to provide path, like:
https://pazuzu.io/registry/storage/featureX/fileY

[whiter4bbit]

@pgronkiewicz you mean provide this path as file name for attachment to be uploaded? I thought about this too (not full, url, but just feature-name/file-name), but current implementation not allows this:

java.lang.IllegalArgumentException: Nested paths are not supported.
        at org.zalando.pazuzu.feature.file.FileContentService.getRealPath(FileContentService.java:62) ~[pazuzu-registry.jar!/:na]
        at org.zalando.pazuzu.feature.file.FileContentService.putFile(FileContentService.java:46) ~[pazuzu-registry.jar!/:na]
        at org.zalando.pazuzu.feature.file.FileService.create(FileService.java:44) ~[pazuzu-registry.jar!/:na]

[pgronkiewicz]

Some changes in the code may be needed. I don't think it should be passed though, as from the source you have access to feature name you are creating, so it should create a path using that.

[whiter4bbit]

Ok, so current File REST API should be rewritten, currently it looks like:

• POST /api/file?name= create file
• GET/DELETE /api/file/id get/delete file

So, no relations to particular feature. Probably there should be any relation to particular feature:

• POST /api/feature/{feature:id}/file?name=
  etc.

But then need to know featureId should be created, so first need to store Feature, get id, then associate attachments with it.

[pgronkiewicz]

One thing we should keep in mind is that we should not expose those files until they are approved. Otherwise we risk having our repository as a storage for uknown and potentially illegal/harmful content.

[whiter4bbit]

Ok, then we can add endpoint for approval:

• PUT /api/feature/{feature:id}/file/{file:id} { approved: true }

Calls:

• GET /api/feature/{feature:id}/file can return list of all files (whether they approved or not)
• GET /api/feature/{feature:id}/file/{file:id} can return all files for oauth user with unapproved-files scope and 401 otherwise

[pgronkiewicz]

We should write some good tests about it, to make sure we don't expose unapproved resources.

[whiter4bbit]

Ok, I will handle this

[PetrGlad]

First of all, why do you need to strictly bind a file to a particular feature? What's wrong with sharing them?
Second, you can group files with "group/filename" just make sure it is not interpreted directly as a filename on storage - for security reasons. I Implemented file storage using filesystem, but now I think it's worthwhile to just use DB blobs for that - it will simplify many things. E.g. you can have /s in file name - and for pazuzu registry they will be just part of name without any special meaning.

Regarding filenames - they can be arbitrary. If UI needs to select one it can just start with feature name with a generated suffix, however I was expecting that user enters some meaningful name when file is added.

Current API binds existing files to feature automatically when feature is saved - any references found in Dockerfile are linked (and missing are unlinked).

You can read files attached to a feature with existing API, /api/features/{feature-name}/files

[PetrGlad]

And there's unclear purpose of this project. If it's intended for in-house use by other organizations then it requires one set of features, if it's intended as public service there's host of completely different requirements. I do not see where feature/files approval fits here. If it's necessary I still think that it's best implemented with tags which we already have.

[pgronkiewicz]

We should never create tooling just for one company :)

Approvals are needed to ensure quality and security of features available through the registry. You should always trust what you are getting back, so we cannot have unchecked features and resources inside. This issue is viable both for running this software inhouse (so Security team is happy) and as an open-source registry (so people don't use it for sharing torrents, etc)