Tracking issue for namespaced-features #5565
Comments
Indeed a tracking issue is useful! I think though this still has a large piece to implement which is to propagate namespaced features to the registry index |
Yes, I figured as much. So do you have any ideas/guidance about how to go about that? Are there established ideas on how to evolve the registry index? |
The evolution here is just a new key in the registry which we've done a bunch of times before, so it's fine to do whenever once there's an agreed up on design. Most of the time this is hard to undo though so we want to be pretty sure of the design before it lands. Technically it'll involve modifying crates.io as well as the publishing code in Cargo to send more data to the registry |
Can you be a bit more specific about which parts of the cargo and/or crates.io code need changing? |
You may want to hop onto the crates.io channel on discord as they may know more, but it's all around |
@alexcrichton okay, so, I'm still not entirely clear on what needs to be done on the crates.io side here. Is it just propagating the Either way, I'm not sure how the compatibility with old cargo is supposed to work, or who all needs to be on board with a supposed design. |
@djc there's a few things here I think that we'd need to consider:
Before we go much further though I think it'd be good to get weigh-in from @rust-lang/cargo because once we start changing the registry it's sort of the point of no return |
@alexcrichton So, this is exactly the kind of change that really motivated the idea of supporting Cargo version dependencies in a way that older versions of Cargo can handle. We can't make old versions of Cargo unable to parse new versions of the index; that's a serious problem. If we're going to do this, we need to version the index, such that older versions of Cargo don't see crates that use this feature at all. |
@joshtriplett indeed! To be clear though the main issue is that an older Cargo won't understand the newer index, but we get to control how it doesn't understand the new index as well (aka no features, a mapping of features, or something like that). Having a fully versioned index seems far off still so I'd be fine implementing this before that comes along. Now having a versioned index is a good idea though and we can start at any time to add safeguards into Cargo to avoid looking at future revisions! |
Is this going to move towards stabilisation soon? |
Probably not -- I'm still motivated to write code to move this forward, but I don't know exactly what needs to be done. |
Oh, I thought it was already implemented, just needed stabilising. |
The Cargo part is implemented -- it apparently needs complementary stuff in crates.io. |
Aha. What I still don't understand is how this issue relates to implicit features for dependencies, exactly. (I was linked here from that issue.) |
I agree that more could be said here. I find it odd how none of the examples I can locate for this feature show how it can be used to overcome the limitations of optional dependencies (which were its primary motivation)! See this page: And now let me try to put it in context: Suppose you have this: [dependencies]
rand = { version = "0.5.5", optional = true } Without [features]
rand = [] # implicitly generated, always ...and With [features]
rand = ["crate:rand"] # implicitly generated, unless you write your own rand feature ...and If you look closely, the actual namespacing of the dependencies does not, in itself, have anything to do with the issue of implicit features. It is the other changes enabled by the The justification for tying these changes together was given in #1286. Er... maybe. Actually, I can't really find it explicitly spelled out there. But the proposal is strictly more powerful than if we were to simply allow the implicit feature to be overridden with no other changes (because this also allows you to have a |
Is it possible to publish a crate that uses this feature? I'm getting:
|
@tikue I believe not currently. In addition to crates.io checks the registry index isn't ready for namespaced features |
@alexcrichton thanks, it was easy enough to work around, just had to name a serde feature serde1. |
@alexcrichton Does an issue for that exist on the crates.io repo? |
@alexreg I think that's this issue, the tracking issue for this feature. The feature needs to be designed for crates.io first. |
@alexcrichton Hmm? That's what I mean, we need to implement namespaced features on crates.io. There should be a separate issue for that over on that repo, I think, no? |
FWIW, I'd like to move this forward, but I'm currently having a hard time figuring out exactly what needs to be done about the index and crates.io. If someone can provide me with more guidance on that, that would be great. |
@djc I think that rust-lang/crates.io#1487 may be helpful in providing a rough template for how to make progress? |
Yup, see rust-lang/crates.io#2277. @ehuss why is this blocked on the new resolver? |
I'm not sure what you mean. Maybe because it is in the Project board? I probably did that just to organize things, not necessarily blocked. One concern is that if we are making backwards-incompatible changes to feature resolution, we may want to gate them all under a single flag. We'd like to avoid littering |
It would be nice if we can use the edition upgrade to flip defaults on some of these flags. |
Yep, we will likely do that. However, that might be a long ways away. |
Anyway, I'm fine with merging this into another flag if the timing lines up! |
One thing that has confused me about namespaced-features is why it really needs a special syntax. A concern is that the following might be confusing (especially for people already familiar with cargo): # …
namespaced-features = true
[dependencies]
foo = {version="1.0", optional=true}
[features]
myfeat = ["foo"] This doesn't enable the "foo" dependency, which it does today. It seems like it is unnecessary to prefix with As an alternative, why not allow features and optional dependencies to coexist? Like you can say: [dependencies]
url = "2.1.1"
serde = {version="1", optional=true}
[features]
serde = ["serde", "serde/derive", "url/serde"] And just allow that to work? I realize there were other things that |
My understanding is that this does enable the
This would introduce ambiguity. If we define another feature
what will this do? Will it enable the optional dependency |
What @smarnach said is also my understanding of the feature. I do think we should discuss the design before I address the issues you've filed. For those following along: #8044 Cargo cannot read index with namespaced features (I do think the index has to know about namespaced features, if the index metadata is used to discover which transitive dependencies are to be built.) #8046 namespaced-features panics on unknown feature name (This looks like a simple oversight that should error with a "non-existent feature" error.) #8047 namespaced-features allows implicitly named features with overlapping optional deps I think this is intentional. The base idea was that the dependency namespace and the feature namespace were to be cleanly separated. However, I left in one loophole to cut down on boilerplate, which is that optional dependencies for which no feature of the same name exists get an implicitly defined feature of that name. So: namespaced-features = true
[features]
<empty>
[dependencies]
serde = { version = "1.0", optional = true } is equivalent to namespaced-features = true
[features]
serde = ["crate:serde"]
[dependencies]
serde = { version = "1.0", optional = true } I'm open to having a discussion about the desirability of this, though! |
Running a test with the example above and
Both (kinda like it does today). Just a caveat, that's just a rough idea, I just want to brainstorm if there is a way to avoid adding whether or not something is namespaced into the index. It's OK if old cargo's ignore crates with
This doesn't seem to be working as I understand it should. Printing out the FeatureMap:
If I run with |
I seem to remember we had a plan to issue a warning for explicitly defined features of the same name as an optional dependency that did not include the
The conflicts in that case seem at odds with the original desire to undo the merging of the dependency and feature namespaces, which I (still) think would be a worthwhile goal that eventually makes everything easier to understand.
Yes, that sounds like a bug then. I'm guessing the feature is being spliced in too late in the process for the resolver to do the right thing. |
If an optional dependency implicitly creates a feature of the same name, then they aren't conceptually separate namespaces (all optional dependencies are also features of the same name). And adding a warning if you are missing I've been thinking about this for a while, and I think I've been mislead a little by the current implementation. If it is really intended that optional dependencies always create a Another question I had: The original issue mentioned changing If we can keep it backwards compatible (where old syntax is interpreted the same), then we wouldn't even need the |
It looks like there isn't a full consens yet what exactly this feature should look like. After reading #1286 I think this should go through a new, proper RFC process, and we should thoroughly the other options as well, including @sfackler's original proposal. I'm not very familiar with the RFC process, but I'd like to ask the Cargo team to discuss this in your next meeting. We discussed rust-lang/crates.io#2277 in the crates.io team meeting. Since this kind of change may be hard to revert for crates.io, we'd first like to finish the discussion on what exactly we want here before implementing this in crates.io. |
At a minimum this feature does need to go through some more work before making any changes on crates.io. We may want to consider an RFC, but beforehand we should at least:
I'd like to see an approach that doesn't require adding new fields to the index. I think that is feasible, we just need to avoid changing the meaning of existing, valid features. |
I don't think it necessarily makes sense to fix bugs and issues in the current implementation before we have an agreement on how the design should look -- otherwise I might have to change some things in the implementation that will then have to be changed again after we reach design consensus. At the very least, it might make sense to also post this to internals to get more exposure if we manage to reach consensus within this issue. So, mini-RFC: MotivationOptional dependencies implicitly masquerading as features causes two issues today, as described in #1286:
Guide-level explanationA feature is defined as This document proposes to make the difference between feature values targeting another feature and feature values targeting a dependency explicit by requiring a With this proposal, features that need to forward the feature can trivially do so: foo = ["crate:foo", "a-derive/foo"] By requiring explicit features to be defined in order to control dependencies, it will be obvious when a feature is being removed, thus decreasing the backwards compatibility hazard. Reference-level explanationI'll skip this for now, because the above seems to cover things to large extent. Please feel free to point out things that should be clarified here. One thing that continues to be unclear to me is the interactions between cargo and the index. As far as I understand the index currently does not use the features data that it gets from cargo, so it seems to me that the impact should be limited. Drawbacks
Rationale and alternativesThe current behavior of mixing features and optional dependencies is compact, but can be hard to completely understand for more complex scenarios. In particular, having to introduce alternative names for features to work around clashes between feature and dependency names causes ugly and harder to find feature names that do not comply with the C-FEATURE guideline. Making features and dependencies explicitly distinct namespaces seems to keep most of the same advantages although potentially at the cost of more boilerplate for simple scenarios. As an alternative, the original issue #1286 outlined a different way of handling this, where feature values are always dependencies or dependency features. If a feature wants to enable a different feature, a separate TOML table is specified for that feature, with fields for specifying its feature and dependency requirements. Prior artGentoo: I'm familiar with Gentoo Linux ebuilds USE flag feature, which bears some similarities to Cargo features. In Gentoo, dependencies are fully separate from USE flags. Dependency edges can specify USE flags, and dependency (edge) USE flags can depend on current ebuild USE flags. Unresolved questions
Future possibilitiesIf we introduce the |
Thanks for writing that up! To be explicit, here is my understanding of how namespaced-features would be different from today's stable Cargo:
If that is all that it entails, then I don't think there needs to be an opt-in flag. This can be turned on for everyone, since it looks to be backwards compatible. And, IIUC, older cargos will ignore entries in the index if they contain I also want to check, in the Unresolved questions, is this referring to if As for the "hiding" situation ("backwards compatibility hazard"), more concrete examples would be useful here, too. Like is the following a good example?
For that case, I can think of a few alternate solutions. One would be to ride on public/private dependencies. If an optional dependency is also private, then it does not automatically create a public feature of the same name. Another solution would be: if
|
One simple example, though not concrete: I optionally support serialization via The impls are gatable by Here's an informal (sorry, don't have the bandwidth to make it more formal) counter-proposal that doesn't cause the backwards-compatibility concerns: "Just" allow defining a feature This resolves motivation 1 the obvious way (concern: users can't X; solution: let them X), and motivation 2 is at least partially solved by hiding the implicit feature for private dependencies. |
Crates.io index stores only the defined features and not the implicit features. And because of that, I agree that this looks to be backwards compatible. What are the next steps for this? |
Probably fix the bugs identified so far. |
I have posted a PR at #8799 to propose the new behavior which I described above which should make it backwards compatible (avoiding the opt-in) and fix all the known issues. |
Regarding the Maybe it would be better to use |
New namespaced features implementation. This is a new implementation for namespaced features (#5565). See the `unstable.md` docs for a description of the new behavior. This is intended to fix several issues with the existing design, and to make it backwards compatible so that it does not require an opt-in flag. This also includes tangentially-related changes to the new feature resolver. The changes are: * `crate_name/feat_name` syntax will now always enable the feature `crate_name`, even if it is an inactive optional dependency (such as a dependency for another platform). The intent here is to have a certain amount of consistency whereby "features" are always activated, but individual crates will still not be activated. * `--all-features` will now enable features for inactive optional dependencies. This is more consistent with `--features foo` enabling the `foo` feature, even when the `foo` dep is not activated. I'm still very conflicted on how that should work, but I think it is better from a simplicity/consistency perspective. I still think it may be confusing if someone has a `cfg(some_dep)` in their code, and `some_dep` isn't built, it will error. The intent is that `cfg(accessible(some_dep))` will be the preferred method in the future, or using platform `cfg` expression like `cfg(windows)` to match whatever is in Cargo.toml. Closes #8044 Closes #8046 Closes #8047 Closes #8316 ## Questions - For various reasons, I changed the way dependency conflict errors are collected. One slightly negative consequence is that it will raise an error for the first problem it detects (like a "missing feature"). Previously it would collect a list of all missing features and display all of them in the error message. Let me know if I should retain the old behavior. I think it will make the code more complicated and brittle, but it shouldn't be too hard (essentially `Requirements` will need to collect a list of errors, and then `resolve_features` would need to check if the list is non-empty, and then aggregate the errors). - Should `cargo metadata` show the implicit features in the "features" table? Currently it does not, and I think that is probably best (it mirrors what is in `Cargo.toml`), but I could potentially see an argument to show how cargo sees the implicit features.
As discovered in #8832, this unfortunately breaks Cargo versions older than 1.19, even if they have a Cargo.lock file. We'll need to figure out how to address that concern before this can move forward. |
I forgot to post a followup here. #9161 implemented changes to support namespaced features in the index and avoid breaking older versions of cargo. There is some risk that versions older than 1.51 may resolve packages using namespaced features in such a way that those features don't get activated. I'll feel a little more comfortable once more time has passed so that 1.51 is at least a few months old. |
Docs: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#namespaced-features
Original issue: #1286
Implementation: #5300
Not sure what the process is for stabilizing experimental Cargo features, I figured a tracking issue might be useful? @alexcrichton any guide on what should be done to move this towards stabilization?
Unresolved issues
About tracking issues
Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.
The text was updated successfully, but these errors were encountered: