Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rubygems): support GitHub Packages #11107

Merged
merged 29 commits into from Aug 20, 2021
Merged

Conversation

qnighy
Copy link
Contributor

@qnighy qnighy commented Aug 5, 2021

GitHub Packages doesn't seem to expose either /api/v1/gems or /versions endpoints. It instead uses /api/v1/dependencies to answer queries from bundler.
See https://andre.arko.net/2014/03/28/the-new-rubygems-index-format/ for the differences between these APIs.

This PR adds fallback from /api/v1/gems/*.json to /api/v1/dependencies to support GitHub Packages.
It involves parsing Ruby's Marshal format, so I created @qnighy/marshal for the purpose.

Note on marshal: I found a clear bug in _getLength and thought I can build better libraries myself. However, when writing this PR, I revisited the code and found _getLength is used less often (there is a redundant correct implementation elsewhere) and the bug's effect is fairly limited. It turned out to be well-written than I expected. I still think mine can compete and at least have a TypeScript definition where marshal doesn't. If marshal is preferred, I'm willing to rewrite the code to use marshal.

Changes:

When updating RubyGems artifacts and if it uses registries other than rubygems.org, Renovate used to inquiry /api/v1/gems/*.json for the package info. After this change, it falls back to /api/v1/dependencies if the former API is not available (returns 404). As a result, Renovate will be able to update dependencies from GitHub Packages RubyGems registry.

Context:

Trying to use Renovate with private dependencies in GitHub Packages, but it could not update them.

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

GitHub Packages doesn't seem to expose neither /api/v1/gems or /versions
endpoints. It instead uses /api/v1/dependencies to answer queries from
bundler.
See https://andre.arko.net/2014/03/28/the-new-rubygems-index-format/ for
the differences between these APIs.

This commit adds fallback from /api/v1/gems/*.json to
/api/v1/dependencies to support GitHub Packages.
It involves parsing Ruby's Marshal format, so I created @qnighy/marshal
for the purpose.
@CLAassistant
Copy link

CLAassistant commented Aug 5, 2021

CLA assistant check
All committers have signed the CLA.

Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use official marshal package.

lib/datasource/rubygems/get.ts Outdated Show resolved Hide resolved
lib/util/http/index.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/index.spec.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/get.ts Outdated Show resolved Hide resolved
Co-authored-by: Michael Kriese <michael.kriese@visualon.de>
@qnighy
Copy link
Contributor Author

qnighy commented Aug 6, 2021

Use official marshal package.

Then I'll use marshal, but please note marshal is not official by any means. marshal and @qnighy/marshal are just two different implementations and both of them just mimic how Ruby's Marshal behaves.

@qnighy
Copy link
Contributor Author

qnighy commented Aug 6, 2021

Will work on marshal things next week

@qnighy
Copy link
Contributor Author

qnighy commented Aug 10, 2021

Moved to marshal.

@qnighy qnighy requested a review from viceice August 10, 2021 07:46
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs deconflicting

lib/datasource/rubygems/get.ts Outdated Show resolved Hide resolved
lib/types/marshal.d.ts Outdated Show resolved Hide resolved
lib/util/http/index.ts Outdated Show resolved Hide resolved
lib/util/http/index.ts Show resolved Hide resolved
lib/datasource/rubygems/index.spec.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/index.spec.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/get-github-packages.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/get-github-packages.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/get-github-packages.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/releases.ts Outdated Show resolved Hide resolved
lib/datasource/rubygems/releases.ts Outdated Show resolved Hide resolved
@viceice viceice requested a review from rarkins August 13, 2021 07:33
viceice
viceice previously approved these changes Aug 13, 2021
@qnighy
Copy link
Contributor Author

qnighy commented Aug 13, 2021

@qnighy Do you have a sample gem on github to test against?

Unfortunately, I don't know a public one...

viceice
viceice previously approved these changes Aug 13, 2021
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good to go now

lib/datasource/rubygems/get.ts Outdated Show resolved Hide resolved
Copy link
Collaborator

@rarkins rarkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no further comments

@viceice viceice dismissed stale reviews from themself via b9ade79 August 19, 2021 12:46
@viceice viceice requested a review from rarkins August 19, 2021 12:48
@rarkins rarkins enabled auto-merge (squash) August 20, 2021 04:19
@rarkins rarkins merged commit 48acb42 into renovatebot:main Aug 20, 2021
@renovate-release
Copy link
Collaborator

🎉 This PR is included in version 26.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@qnighy qnighy deleted the bundler-old-api branch August 20, 2021 05:24
@qnighy
Copy link
Contributor Author

qnighy commented Aug 20, 2021

Thanks a lot for the follow-ups!

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants