Allows the default git/gitoxide configuration to be obtained from the ENV and config #13687

linyihai · 2024-04-02T07:12:55Z

What does this PR try to resolve?

The default git/gitoxide feautures config can be obtained througt -Zgit and -Zgitoxide.
However, it cannot be obtained from environment variables or configurations.
It's not very ergonomic.

How should we test and review this PR?

The previous commit explained the staus quo, the next commit addressed the problem.

Additional information

Inspired by #11813 (comment)
See also #13285 (comment)

Change of usage

Mirror Zgit/Zgitoxide when they parsed as string

Specify the feature you like

CARGO_UNSABLE_GIT='shallow-deps' cargo fetch
cargo fetch --config "unstable.git='shallow-deps'"
cargo fetch -Zgit="shallow-deps"

Specify partial fields when parsed as table

CARGO_UNSTABLE_GITOXIDE_FETCH=true cargo fetch

The rest fields will use Rust default value. That said, checkout and internal_use_git2 will be false.

Besides, you can pass true to the whole feature to specify the pre-defined features.

CARGO_UNSTABLE_GITOXIDE=true cargo fetch

rustbot · 2024-04-02T07:13:00Z

r? @ehuss

rustbot has assigned @ehuss.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

epage · 2024-04-02T16:07:29Z

I assume this is meant to resolve the immediate part of #13688 (leaving aside the more general problem). Is that right?

epage · 2024-04-02T16:13:08Z

src/cargo/core/features.rs

@@ -908,15 +914,19 @@ fn parse_git(it: impl Iterator<Item = impl AsRef<str>>) -> CargoResult<Option<Gi
 #[derive(Debug, Copy, Clone, Default, Deserialize)]
 pub struct GitoxideFeatures {
    /// All fetches are done with `gitoxide`, which includes git dependencies as well as the crates index.
+    #[serde(default = "default_true")]


I assume these were chosen to align with GitoxideFeatures::safe() and to mirror if you do -Zgitoxide

However, this breaks down if you do CARGO_UNSTABLE_GITOXIDE_FETCH=true, then almost everything else becomes enabled (we should have a test to show this case).

Some other challenges with these defaults

Unclear what the motivation is (could have comment pointing to GitoxideFeature::safe)

Challenge in keeping in sync with GitoxideFeature::safe

Should we instead just default everything to false?

What I would love to just work is CARGO_UNSTABLE_GITOXIDE=fetch,list_files and CARGO_UNSTABLE_GITOXIDE=true mirroring the CLI flag (which would also imply you could have unstable.gitoxide = "fetch,list_files" in the config file).

Yeah maybe we can make it a comma separated list to reduce the complexity.

Wonder do people really use individual gitoxide features? I always assume that people would just enable them as a whole, though gitoxide author recommends keeping them separately: #13252 (comment).

What I would love to just work is CARGO_UNSTABLE_GITOXIDE=fetch,list_files and CARGO_UNSTABLE_GITOXIDE=true mirroring the CLI flag (which would also imply you could have unstable.gitoxide = "fetch,list_files" in the config file).

So what we defined as table is going to be replaced with a string? It makes sense, I have no objection to this, but it is not clear that it will break the configuration of the previous version？

It's possible to support both, via #[serde(deserialize_with = ...)] and a custom deserializer that handles a string/list or a struct, like

cargo/src/cargo/util/context/mod.rs

Line 2673 in 88d55a8

fn progress_or_string<'de, D>(deserializer: D) -> Result<Option<ProgressConfig>, D::Error>

bors · 2024-04-04T03:55:24Z

☔ The latest upstream changes (presumably #13696) made this pull request unmergeable. Please resolve the merge conflicts.

ehuss · 2024-04-06T19:07:12Z

r? weihanglo

linyihai · 2024-04-08T08:47:05Z

I assume this is meant to resolve the immediate part of #13688 (leaving aside the more general problem). Is that right?

Yes，this tend to mirror the -Zgit/-Zgitoxide as possible.

weihanglo

Sorry for the late review 🙇🏾‍♂️

weihanglo · 2024-05-10T01:59:28Z

src/cargo/util/context/de.rs

+        // then run `CARGO_UNSTABLE_GIT_SHALLOW_INDEX cargo fetch`, here will return `missing config key unstable.git`.
+        // It seems that anything that starts with CARGO_UNSTABLE_GIT, even like CARGO_UNSTABLE_GITOXIDE can trigger this.
+        // This is a workaround for now, but should be fixed in the future.
+        visitor.visit_none()


While this seems fine for now, I am a bit worried about the change.

What about having an enum deriving serde?

#[derive(Debug, Copy, Clone, Default, Deserialize)] #[serde(untagged)] pub enum Gitoxide { #[default] All, Some(GitoxideFeatures), }

so that I assumed it would work well with

[unstable] gitoxide = true # or [unstable.gitoxide] checkout = true

As well as enviroment variables?

CARGO_UNSTABLE_GITOXIDE=true CARGO_UNSTABLE_GITOXIDE_CHECKOUT=true

It doesn't seem to be working. see gitoxide_features_as_table in the lastest commit.

I had replaced vistor.visit_none() with below,

// The effect here is the same as in [`deserialize_option`]. if self.gctx.has_key(&self.key, self.env_prefix_ok)? { return visitor.visit_some(self); }

So we can still specify a inner field of a struct in the ENV, like CARGO_UNSTABLE_GITOXIDE_FETCH, even that CARGO_UNSTABLE_GIOXIDE doesn't exist.

But that's not enough, It will still warning unstable.git missing key, Why is this happening? After some groping, Problems gradually surfaced, see https://github.com/rust-lang/cargo/blob/master/src/cargo/util/context/de.rs#L268-L269

if env_key.starts_with(field_key.as_env_key()) { fields.insert(KeyKind::Normal(field.to_string()));

here will treat CARGO_UNSTABLE_GIT as prefix of CARGO_UNSTABLE_GIOXIDE_FETCH, and the unstable.git and unsable.gitoxide will be pushed into fields. This is a problem, when the unstable fields is a struct and the fields implement their deserialize_any method. So any environment variable that starts with the XX character will have missing XX keys.
See test struct_with_overlapping_inner_struct_and_defaults for more details.

src/cargo/util/context/de.rs

bors · 2024-05-24T04:09:55Z

☔ The latest upstream changes (presumably #13956) made this pull request unmergeable. Please resolve the merge conflicts.

weihanglo

Looks reasonable. Just some minor suggestions.

src/cargo/core/features.rs

Fix: Skip deserialization of unrelated fields with overlapping name ### What does this PR try to resolve? Split from #13687 (comment) This fixes the overlap of environment variable names: > For example, when env_key is UNSTABLE_GITOXIDE_FETCH and field_key is UNSTABLE_GIT, the field shouldn't be added because `unstable.gitoxide.fetch` doesn't belong to `unstable.git` struct. ### How should we test and review this PR? Updates of test cases `struct_with_overlapping_inner_struct_and_defaults` can be used to compare changes before and after changes ### Additional information r? weihanglo and very appreciate your more optimized code

…e table

weihanglo

One minor error message regression left and we're ready to go

weihanglo · 2024-06-02T14:33:03Z

src/cargo/core/features.rs

+    fn expecting() -> String {
+        let fields = vec!["`fetch`", "`checkout`", "`internal-use-git2`"];
+        format!(
+            "unstable 'gitoxide' only takes {} as valid inputs, for shallow fetches see `shallow-index,shallow-deps`",


Suggested change

"unstable 'gitoxide' only takes {} as valid inputs, for shallow fetches see `shallow-index,shallow-deps`",

"unstable 'gitoxide' only takes {} as valid inputs, for shallow fetches see `-Zgit=shallow-index,shallow-deps`",

Shouldn't it be -Zgitoxide

Looks like the old tip was wrong, too. See https://doc.rust-lang.org/cargo/reference/unstable.html, let me correct it to -Zgitoxide also

Sorry, I misunderstand

- pass true to enable all pre-definded git/gitoxide features - support parse git/gitoxide as table in Config, if the field is tagged with #[serde(default)], then it can be skipped

weihanglo · 2024-06-03T13:26:03Z

Thanks!

@bors r+

bors · 2024-06-03T13:26:05Z

📌 Commit ee87c91 has been approved by weihanglo

It is now in the queue for this repository.

bors · 2024-06-03T13:27:24Z

⌛ Testing commit ee87c91 with merge 4219b66...

bors · 2024-06-03T13:57:45Z

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing 4219b66 to master...

Update cargo 9 commits in 7a6fad0984d28c8330974636972aa296b67c4513..34a6a87d8a2330d8c9d578f927489689328a652d 2024-05-31 22:26:03 +0000 to 2024-06-04 15:31:01 +0000 - Silence the warning about forgetting the vendoring (rust-lang/cargo#13886) - fix(vendor): Ensure sort happens for vendor (rust-lang/cargo#14004) - fix(add): Avoid escaping double-quotes by using string literals (rust-lang/cargo#14006) - refactor(source): Split `RecursivePathSource` out of `PathSource` (rust-lang/cargo#13993) - doc: Add README for resolver-tests (rust-lang/cargo#13977) - Allows the default git/gitoxide configuration to be obtained from the ENV and config (rust-lang/cargo#13687) - refactor: Transition direct assertions from cargo-test-support to snapbox (rust-lang/cargo#13980) - Fix: Skip deserialization of unrelated fields with overlapping name (rust-lang/cargo#14000) - chore(deps): update alpine docker tag to v3.20 (rust-lang/cargo#13996) r? ghost

rustbot assigned ehuss Apr 2, 2024

rustbot added A-unstable Area: nightly unstable support S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 2, 2024

linyihai changed the title ~~Allows the default git/gitoxide configuration to be obtained from the command line or from an environment variable~~ Allows the default git/gitoxide configuration to be obtained from the ENV and config Apr 2, 2024

Nemo157 mentioned this pull request Apr 2, 2024

Unhelpful error messages with env-vars that don't set all fields of a config struct #13688

Closed

epage reviewed Apr 2, 2024

View reviewed changes

rustbot assigned weihanglo and unassigned ehuss Apr 6, 2024

linyihai force-pushed the git-features-env branch 2 times, most recently from e852649 to b5fd826 Compare April 8, 2024 07:48

This comment was marked as outdated.

Sign in to view

weihanglo reviewed May 10, 2024

View reviewed changes

rustbot added the A-git Area: anything dealing with git label May 11, 2024

linyihai force-pushed the git-features-env branch 2 times, most recently from 76421f7 to b4f699e Compare May 21, 2024 06:59

linyihai commented May 21, 2024

View reviewed changes

src/cargo/util/context/de.rs Outdated Show resolved Hide resolved

weihanglo reviewed May 31, 2024

View reviewed changes

src/cargo/core/features.rs Outdated Show resolved Hide resolved

src/cargo/core/features.rs Outdated Show resolved Hide resolved

src/cargo/core/features.rs Outdated Show resolved Hide resolved

src/cargo/core/features.rs Outdated Show resolved Hide resolved

linyihai mentioned this pull request Jun 1, 2024

Fix: Skip deserialization of unrelated fields with overlapping name #14000

Merged

test: Added test cases about getting git/gitxide from Env and unstabl…

b176adb

…e table

linyihai force-pushed the git-features-env branch from b4f699e to dfc8e24 Compare June 2, 2024 06:48

weihanglo reviewed Jun 2, 2024

View reviewed changes

linyihai force-pushed the git-features-env branch from dfc8e24 to 22ef8cb Compare June 3, 2024 02:32

feat: Support parse string as git/gitoxide features from ENV and Config

ee87c91

- pass true to enable all pre-definded git/gitoxide features - support parse git/gitoxide as table in Config, if the field is tagged with #[serde(default)], then it can be skipped

linyihai force-pushed the git-features-env branch from 22ef8cb to ee87c91 Compare June 3, 2024 02:42

weihanglo approved these changes Jun 3, 2024

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 3, 2024

bors merged commit 4219b66 into rust-lang:master Jun 3, 2024
21 checks passed

weihanglo mentioned this pull request Jun 5, 2024

Update cargo rust-lang/rust#126005

Merged

rustbot added this to the 1.80.0 milestone Jun 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allows the default git/gitoxide configuration to be obtained from the ENV and config #13687

Allows the default git/gitoxide configuration to be obtained from the ENV and config #13687

linyihai commented Apr 2, 2024 •

edited

Loading

rustbot commented Apr 2, 2024

epage commented Apr 2, 2024

epage Apr 2, 2024

Nemo157 Apr 2, 2024 •

edited

Loading

weihanglo Apr 2, 2024

linyihai Apr 3, 2024

Nemo157 Apr 3, 2024

bors commented Apr 4, 2024

ehuss commented Apr 6, 2024

This comment was marked as outdated.

linyihai commented Apr 8, 2024

weihanglo left a comment

weihanglo May 10, 2024 •

edited

Loading

linyihai May 11, 2024

linyihai May 21, 2024 •

edited

Loading

bors commented May 24, 2024

weihanglo left a comment

weihanglo left a comment

weihanglo Jun 2, 2024

linyihai Jun 3, 2024

linyihai Jun 3, 2024

linyihai Jun 3, 2024

weihanglo commented Jun 3, 2024

bors commented Jun 3, 2024

bors commented Jun 3, 2024

bors commented Jun 3, 2024

	"unstable 'gitoxide' only takes {} as valid inputs, for shallow fetches see `shallow-index,shallow-deps`",
	"unstable 'gitoxide' only takes {} as valid inputs, for shallow fetches see `-Zgit=shallow-index,shallow-deps`",

Allows the default git/gitoxide configuration to be obtained from the ENV and config #13687

Allows the default git/gitoxide configuration to be obtained from the ENV and config #13687

Conversation

linyihai commented Apr 2, 2024 • edited Loading

What does this PR try to resolve?

How should we test and review this PR?

Additional information

Change of usage

rustbot commented Apr 2, 2024

epage commented Apr 2, 2024

Choose a reason for hiding this comment

Nemo157 Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Apr 4, 2024

ehuss commented Apr 6, 2024

This comment was marked as outdated.

linyihai commented Apr 8, 2024

weihanglo left a comment

Choose a reason for hiding this comment

weihanglo May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linyihai May 21, 2024 • edited Loading

Choose a reason for hiding this comment

bors commented May 24, 2024

weihanglo left a comment

Choose a reason for hiding this comment

weihanglo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weihanglo commented Jun 3, 2024

bors commented Jun 3, 2024

bors commented Jun 3, 2024

bors commented Jun 3, 2024

linyihai commented Apr 2, 2024 •

edited

Loading

Nemo157 Apr 2, 2024 •

edited

Loading

weihanglo May 10, 2024 •

edited

Loading

linyihai May 21, 2024 •

edited

Loading