Add Package Registry #16510

KN4CK3R · 2021-07-21T21:51:54Z

This PR adds support for multiple package registries.

Currently implemented are

Composer
Conan
Container (Docker / Helm)
Generic packages
Maven
npm
NuGet
PyPI
RubyGems

Every user/org has an own package registry for all different types of packages.

Screenshots

Package list:

Admin page:

Composer package:

Conan package:

Docker image:

Helm Chart:

Generic package:

NuGet:

npm:

Maven:

PyPI:

RubyGems:

Instructions for testing: #16510 (comment)

close #15706
close #2316

…-packages

modules/templates/helper.go

delvh

Skimmed through it, what I've seen seems pretty good.
My focus was on Maven as that's what I'm most comfortable with.
No doubt, there will be reviewers that actually read through all 5000 lines of code… 😉

delvh · 2021-07-21T22:53:41Z

modules/packages/maven/metadata.go

+		URL          string `xml:"url"`
+		Distribution string `xml:"distribution"`
+	} `xml:"licenses>license"`
+	Dependencies []struct {


I am not sure whether they are needed, but the Maven Language Server gives me a few more options to pass for each dependency.
They are:

optional (String)

type (String)

exclusions (a set of dependencies)

classifier (String)

But I totally understand it if you want that to be part of another PR, the basic functionality already stands.
It wouldn't surprise me, however, if the current implementation fails as soon as one of the advanced features is used, limiting the initial usability. The same is true for the missing fields mentioned below.

I'm no Maven expert but does the client really trust the server informations about the dependencies? I would expect the client downloads the package and checks the local pom file for all needed informations.

Edit: The client downloads the .pom asset with all original informations so all metadata is just used for the UI. I don't know if these are benefitical for a normal user.

As I already said, I am also unsure whether they are needed.
Potentially yes, potentially no.
It could, however, be that the package cannot be registered in case advanced properties are used in the pom file. That's what I'm uncertain about.
In the worst case, this will simply have to be added to the data model in another PR, after this PR has been merged and tested thoroughly.

The pom file is not generated but the original file is stored. So all metadata is only used for the Gitea UI. Currently I don't think there is a problem.

modules/packages/maven/metadata.go

+	Version    string `json:"version"`
+}
+
+type pomStruct struct {


options/locale/locale_en-US.ini

lunny · 2021-07-22T04:07:23Z

Since docker is also some kind of package, maybe we should consider what's relation between this one and #14919 . @a1012112796

models/package.go

lunny · 2021-07-22T04:31:10Z

Thanks for the fantastic PR! Two small things you can consider to add in this PR. One is add webhooks for packages, PackageUploaded, PackageUpdated, and PackageDeleted. Another is to add a package_log table to record package or files added, updated, deleted.

delvh

Holy, I think I've never been sitting so long on a single review as I have now.
About four hours, and I reviewed only the easy 60%...
But before I forget the rest or this PR gets merged, here are a few points I noticed.

docs/content/doc/packages/composer.en-us.md

docs/content/doc/advanced/config-cheat-sheet.en-us.md

custom/conf/app.example.ini

docs/content/doc/packages/conan.en-us.md

docs/content/doc/packages/generic.en-us.md

delvh · 2022-03-28T22:42:40Z

services/cron/tasks_basic.go

+	RegisterTaskFatal("cleanup_packages", &OlderThanConfig{
+		BaseConfig: BaseConfig{
+			Enabled:    true,
+			RunAtStart: true,


Wasn't there a setting for this? Is that (now) unused?

Are you talking about #19221? If yes, that's already changed.

Ah, that value is simply the default value if the setting is missing, correct?
In that case ignore my comment, I thought it would mean that it would always run at start.

options/locale/locale_en-US.ini

…-packages

tboerger

Minimal docs typos

docs/content/doc/packages/composer.en-us.md

tboerger · 2022-03-29T19:37:36Z

Beside the implemented providers it could also make sense to implement the Terraform registry api, Terraform module api and the Terraform remote state api.

lunny · 2022-03-30T02:56:39Z

Beside the implemented providers it could also make sense to implement the Terraform registry api, Terraform module api and the Terraform remote state api.

Good idea but let's merge it at first and send other PRs.

wxiaoguang · 2022-03-30T06:00:20Z

Last call for reviews. If no code change in one day, let's merge.

Co-authored-by: Thomas Boerger <thomas@webhippie.de>

docs/content/doc/packages/nuget.en-us.md

tboerger · 2022-03-30T07:43:45Z

@wxiaoguang hopefully my comments will be fixed before the merge ;)

wxiaoguang · 2022-03-30T08:05:05Z

@wxiaoguang hopefully my comments will be fixed before the merge ;)

Yup, these document fixes have been committed. Anything more to change?

tboerger · 2022-03-30T08:25:08Z

Not from my side, looks good. Even if I'm still against bloating the gitea core more and more with features, but this won't change without a plugin system.

wxiaoguang · 2022-03-30T08:34:08Z

Agree to be more careful about minority features that bloating the core code. Gitea should only introduce features which benefit most users.

Personally speaking, I am not optimistic about Gitea can have a plugin system in near future. Many simple and refactoring PRs just get delayed for long time, even months, I can not imagine how the plugin system could be done in a predictable future. And indeed it seems that a plugin system is not a high priority task at the moment.

lunny · 2022-03-30T08:43:28Z

Let's merge then send patches. It took so long for this PR.

wxiaoguang · 2022-03-30T08:46:00Z

3 hours ago Last call for reviews. If no code change in one day, let's merge.

One day comes a little early 😂 but it's not bad, I like it. Let's move on ~~~~~ 🎉 test and patch

* giteaoffical/main: (31 commits) Add Package Registry (go-gitea#16510) Show messages for users if the ROOT_URL is wrong, show JavaScript errors (go-gitea#18971) [skip ci] Updated translations via Crowdin Make git.OpenRepository accept Context (go-gitea#19260) Use full output of git show-ref --tags to get tags for PushUpdateAddTag (go-gitea#19235) When conflicts have been previously detected ensure that they can be resolved (go-gitea#19247) More commit info from API (go-gitea#19252) Move some issue methods as functions (go-gitea#19255) Move project files into models/project sub package (go-gitea#17704) Granular webhook events in editHook (go-gitea#19251) Provide configuration to allow camo-media proxying (go-gitea#12802) Move init repository related functions to modules (go-gitea#19159) Move organization related structs into sub package (go-gitea#18518) Refactor repo clone button and repo clone links, fix JS error on empty repo page (go-gitea#19208) Show last cron messages on monitor page (go-gitea#19223) Allow API to create file on empty repo (go-gitea#19224) Use goproxy.io instead of goproxy.cn (go-gitea#19242) New cron task: delete old system notices (go-gitea#19219) Let web and API routes have different auth methods group (go-gitea#19168) Only send webhook events to active system webhooks and only deliver to active hooks (go-gitea#19234) ...

KN4CK3R · 2022-03-30T18:25:47Z

A little bit rushed at the end but I'm happy it got merged after that long time working on it.

6543 · 2022-03-31T11:22:43Z

locked as new issues should be created & linked to this ... if something pops up

KN4CK3R added 21 commits July 11, 2021 19:02

Added package store settings.

51bc02e

Added models.

48c2faf

Added generic package registry.

3e801f9

Added tests.

3c86d2b

Added NuGet package registry.

8d922ff

Moved service index to api file.

2b9d96a

Added NPM package registry.

99298b7

Added Maven package registry.

c7fb932

Added PyPI package registry.

8aaf412

Summary is deprecated.

06790f9

Changed npm name.

338dee3

Sanitize project url.

0544b22

Allow only scoped packages.

5a3c561

Added user interface.

ea36a75

Changed method name.

8c6715b

Merge branch 'main' of https://github.com/go-gitea/gitea into feature…

ec05929

…-packages

Added missing migration file.

4a57e22

Set page info.

b8ac3e3

Added documentation.

1386b27

Merge branch 'main' of https://github.com/go-gitea/gitea into feature…

9e5b2c0

…-packages

Added documentation links.

8ba72d7

KN4CK3R added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Jul 21, 2021

KN4CK3R added this to the 1.16.0 milestone Jul 21, 2021

6543 reviewed Jul 21, 2021

View reviewed changes

modules/templates/helper.go Show resolved Hide resolved

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jul 21, 2021

delvh reviewed Jul 21, 2021

View reviewed changes

lunny reviewed Jul 22, 2021

View reviewed changes

models/package.go Outdated Show resolved Hide resolved

lafriks added the type/changelog Adds the changelog for a new Gitea version label Jul 22, 2021

KN4CK3R and others added 2 commits March 28, 2022 17:36

Changed temporary buffer filename.

10c5f03

Merge branch 'main' into feature-packages

5d08e80

delvh reviewed Mar 28, 2022

View reviewed changes

KN4CK3R and others added 3 commits March 29, 2022 14:32

Added suggestions.

dbbbfa4

Merge branch 'main' of https://github.com/go-gitea/gitea into feature…

7efd245

…-packages

Merge branch 'main' into feature-packages

f55dcbf

tboerger reviewed Mar 29, 2022

View reviewed changes

docs/content/doc/packages/composer.en-us.md Outdated Show resolved Hide resolved

docs/content/doc/packages/composer.en-us.md Outdated Show resolved Hide resolved

Apply suggestions from code review

1d4ce1a

Co-authored-by: Thomas Boerger <thomas@webhippie.de>

wxiaoguang reviewed Mar 30, 2022

View reviewed changes

docs/content/doc/packages/nuget.en-us.md Outdated Show resolved Hide resolved

wxiaoguang added 2 commits March 30, 2022 14:08

Update docs/content/doc/packages/nuget.en-us.md

84ff7b0

Merge branch 'main' into feature-packages

16f8940

lunny merged commit 1d33234 into go-gitea:main Mar 30, 2022

KN4CK3R mentioned this pull request Mar 30, 2022

Package Registry related ToDo list #19270

Open

26 tasks

wxiaoguang mentioned this pull request Mar 31, 2022

Adding support for markdown in checks description #17980

Closed

go-gitea locked as resolved and limited conversation to collaborators Mar 31, 2022

KN4CK3R deleted the feature-packages branch April 2, 2022 11:04

KN4CK3R added the topic/packages label May 5, 2022

delvh removed the type/changelog Adds the changelog for a new Gitea version label Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Package Registry #16510

Add Package Registry #16510

KN4CK3R commented Jul 21, 2021 •

edited

Loading

delvh left a comment

delvh Jul 21, 2021 •

edited

Loading

KN4CK3R Jul 22, 2021 •

edited

Loading

delvh Jul 22, 2021 •

edited

Loading

KN4CK3R Sep 2, 2021

This comment was marked as resolved.

lunny commented Jul 22, 2021

lunny commented Jul 22, 2021

delvh left a comment

delvh Mar 28, 2022

KN4CK3R Mar 29, 2022

delvh Mar 29, 2022

tboerger left a comment

tboerger commented Mar 29, 2022

lunny commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022

tboerger commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022

tboerger commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022 •

edited

Loading

lunny commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022

KN4CK3R commented Mar 30, 2022

6543 commented Mar 31, 2022 •

edited

Loading

Add Package Registry #16510

Add Package Registry #16510

Conversation

KN4CK3R commented Jul 21, 2021 • edited Loading

delvh left a comment

Choose a reason for hiding this comment

delvh Jul 21, 2021 • edited Loading

Choose a reason for hiding this comment

KN4CK3R Jul 22, 2021 • edited Loading

Choose a reason for hiding this comment

delvh Jul 22, 2021 • edited Loading

Choose a reason for hiding this comment

KN4CK3R Sep 2, 2021

Choose a reason for hiding this comment

This comment was marked as resolved.

lunny commented Jul 22, 2021

lunny commented Jul 22, 2021

delvh left a comment

Choose a reason for hiding this comment

delvh Mar 28, 2022

Choose a reason for hiding this comment

KN4CK3R Mar 29, 2022

Choose a reason for hiding this comment

delvh Mar 29, 2022

Choose a reason for hiding this comment

tboerger left a comment

Choose a reason for hiding this comment

tboerger commented Mar 29, 2022

lunny commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022

tboerger commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022

tboerger commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022 • edited Loading

lunny commented Mar 30, 2022

wxiaoguang commented Mar 30, 2022

KN4CK3R commented Mar 30, 2022

6543 commented Mar 31, 2022 • edited Loading

KN4CK3R commented Jul 21, 2021 •

edited

Loading

delvh Jul 21, 2021 •

edited

Loading

KN4CK3R Jul 22, 2021 •

edited

Loading

delvh Jul 22, 2021 •

edited

Loading

wxiaoguang commented Mar 30, 2022 •

edited

Loading

6543 commented Mar 31, 2022 •

edited

Loading