-
Notifications
You must be signed in to change notification settings - Fork 942
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Cargo private registries #8719
Merged
honeyankit
merged 31 commits into
dependabot:main
from
CodingAnarchy:cargo-private-registries
May 1, 2024
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
86f9699
Add support for Rust alternative registries
johnbatty 74cbb0e
Fix handling of missing source info in dependencies
johnbatty d32944e
Fixed calculation of crates dl URL
johnbatty 3b3ab3a
Minor cleanups
johnbatty ab1b6d9
Fixes for crates-ms
johnbatty 7a44ddd
chore: merge main
iajoiner db1a615
Chore: Merge branch 'main'
CodingAnarchy 9102658
Update to get interface with private registry (tested with Cloudsmith…
CodingAnarchy 7d1c9c8
Try to handle the case of updating versions for private registries
CodingAnarchy e519ab1
Fix up some tests for fetching Cargo config file
CodingAnarchy 8f9a93c
Sorbet typechecking adjustments
CodingAnarchy 3041c90
Refactor sparse registry details to method
CodingAnarchy f4212cf
Expect Cargo config file to be fetched in FileFetcher spec
CodingAnarchy 8273528
Revert changes to base Gitlab metadata finder made in feature base PR
CodingAnarchy b8637e9
Merge branch 'main' into cargo-private-registries
CodingAnarchy 0fbe61a
Enable using index URL to fetch crate metadata from private registry
CodingAnarchy fd4b7d4
Merge branch 'cargo-private-registries' of github.com:CodingAnarchy/d…
CodingAnarchy 6400612
Remove special case for Microsoft; we can/need to authenticate normally
CodingAnarchy 4bafd3b
Cargo prefers config.toml to have the extension
CodingAnarchy b51216b
Fix typo in method name
CodingAnarchy 61f72ae
Fix filename for fixture
CodingAnarchy fabfce1
Merge main into cargo-private-registries
RobJellinghaus 15968e2
Merge branch 'main' into user/rjelling/cargo-private-registries
RobJellinghaus f67e4ea
Add a bit more spam in the DEBUG_HELPERS case.
RobJellinghaus 08ad44e
Merge branch 'main' into user/rjelling/cargo-private-registries
RobJellinghaus e1490d3
Fix issues found in local testing, especially setting `CARGO_REGISTRY…
RobJellinghaus f8f60a2
Merge branch 'main' into cargo-private-registries
CodingAnarchy def5d8a
Style lint fixes
CodingAnarchy 2b5dbca
Merge branch 'cargo-private-registries' of github.com:CodingAnarchy/d…
CodingAnarchy 1a013c5
Simplify conditional logic to satisfy ABC linter
CodingAnarchy fcb86e1
Merge branch 'main' into cargo-private-registries
abdulapopoola File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# typed: true | ||
# frozen_string_literal: true | ||
|
||
require "yaml" | ||
|
||
module Dependabot | ||
module Cargo | ||
module Helpers | ||
def self.setup_credentials_in_environment(credentials) | ||
credentials.each do |cred| | ||
next if cred["type"] != "cargo_registry" | ||
|
||
# If there is a 'token' property, then apply it. | ||
# If there is not, it probably means we are running under dependabot-cli which stripped | ||
# all tokens. So in that case, we assume that the dependabot proxy will re-inject the | ||
# actual correct token, and we just use 'token' as a placeholder at this point. | ||
# (We must add these environment variables here, or 'cargo update' will not think it is | ||
# configured properly for the private registries.) | ||
|
||
token_env_var = "CARGO_REGISTRIES_#{cred['cargo_registry'].upcase.tr('-', '_')}_TOKEN" | ||
|
||
token = "placeholder_token" | ||
if cred["token"].nil? | ||
puts "Setting #{token_env_var} to 'placeholder_token' because dependabot-cli proxy will override it anyway" | ||
else | ||
token = cred["token"] | ||
puts "Setting #{token_env_var} to provided token value" | ||
end | ||
|
||
ENV[token_env_var] ||= token | ||
end | ||
|
||
# And set CARGO_REGISTRY_GLOBAL_CREDENTIAL_PROVIDERS here as well, so Cargo will expect tokens | ||
ENV["CARGO_REGISTRY_GLOBAL_CREDENTIAL_PROVIDERS"] ||= "cargo:token" | ||
end | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
# typed: true | ||
# frozen_string_literal: true | ||
|
||
require "dependabot/file_fetchers" | ||
require "dependabot/file_fetchers/base" | ||
|
||
module Dependabot | ||
module Cargo | ||
class RegistryFetcher < Dependabot::FileFetchers::Base | ||
def self.required_files_in?(filenames) | ||
filenames.include?("config.json") | ||
end | ||
|
||
def self.required_files_message | ||
"Repo must contain a config.json" | ||
end | ||
|
||
def dl | ||
parsed_config_json["dl"].chomp("/") | ||
end | ||
|
||
def api | ||
parsed_config_json["api"].chomp("/") | ||
end | ||
|
||
private | ||
|
||
def fetch_files | ||
fetched_files = [] | ||
fetched_files << config_json | ||
end | ||
|
||
def parsed_config_json | ||
@parsed_config_json ||= JSON.parse(config_json.content) | ||
end | ||
|
||
def config_json | ||
@config_json ||= fetch_file_from_host("config.json") | ||
end | ||
end | ||
end | ||
end |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally we want to avoid passing tokens in the Dependabot ruby Updater because they can show up in logs or 3rd party issue trackers.
Pub is the most recent ecosystem that we enabled private registries support, if you want to take a look at that implementation as an example, although it only supports git registries AFAIK https://github.com/dependabot/dependabot-core/blob/main/pub/lib/dependabot/pub/helpers.rb#L235
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm open to suggestions to avoid that here, but I'm not sure how else to fetch the metadata for the crates without reaching out to the index API that Cargo uses internally. I'm using the pattern in your reference link to have
cargo
do the actual updates of the dependencies, but is there another way you would recommend to reach out the API and get a JSON response that avoids this use of the token?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added support for sparse registries in our proxy: #3478 (comment)
We'd need to test this out using the CLI rather than the dry-run script as that will be a more accurate representation of how this will run in the GitHub hosted version. The differences from the dry-run script will be:
It would be OK to remove the token auth support from this code here unless you plan to run this in a standalone mode without the proxy.