RFC: Syntax for embedding cargo-script manifests #3503

epage · 2023-09-26T19:40:34Z

This is for the T-lang side of #3502

Example:

#!/usr/bin/env cargo

---
[dependencies]
clap = { version = "4.2", features = ["derive"] }
---

use clap::Parser;

#[derive(Parser, Debug)]
#[clap(version)]
struct Args {
    #[clap(short, long, help = "Path to config")]
    config: Option<std::path::PathBuf>,
}

fn main() {
    let args = Args::parse();
    println!("{:?}", args);
}

FCP

thomcc · 2023-09-27T00:28:54Z

In PL/Rust (a Rust subset that works as a postgres procedural language handler) we use a somewhat hacky syntax like (see https://github.com/tcdi/plrust/blob/main/doc/src/dependencies.md for example source)

[dependencies]
rand = "0.8"

[code]
use rand::Rng;
Ok(Some(rand::thread_rng().gen()))

I greatly prefer the approach in this RFC (and will likely push PL/Rust transition to it if the RFC is accepted) but it's probably worth noting as prior art.

est31 · 2023-09-27T00:42:32Z

If you think this RFC a little bit further, then you could generalize this to make rustc ignore all ``` delimitered blocks, to say that rustc should just ignore all these sections and leave them to other tools similarly to how it does it already for the shebang. And then you'd see how similar triple backtick blocks are to /* */ comments. From there to just adding `/* */` comments to the AST it's only a small step.

IMO it's bad style to add cargo specific extensions to Rust's syntax. Just say that ``` delimitered blocks are ignored by rustc and that parser implementations are suggested to add them.

epage · 2023-09-27T00:42:42Z

In PL/Rust (a Rust subset that works as a postgres procedural language handler) we use a somewhat hacky syntax like (see https://github.com/tcdi/plrust/blob/main/doc/src/dependencies.md for example source)

Thanks!

I've added this as prior art and would love more interoperability (one of the stated motivations for the cargo script RFC)

epage · 2023-09-27T00:45:43Z

text/3503-frontmatter.md

+### Alternative 7: Extended Shebang
+
+````rust
+#!/usr/bin/env cargo
+# ```cargo
+# [dependencies]
+# foo = "1.2.3"
+# ```
+
+fn main() {}
+````


@bstrie (moved here for context / easier to follow)

I would like to consider alternative 7, the extended shebang. I don't think the backticks and the redundant cargo specifier should be necessary, producing this:

#!/usr/bin/env cargo # [dependencies] # foo = "1.2.3" fn main() {}

I think this looks quite good. It's fewer lines than the proposed syntax, and mirrors both the shebang syntax and the attribute syntax.

I understand that people might want this to be generalizable/extensible, but this could suffice for now and any discussion about how to generalize the "info" portion can be left for a future discussion. If people think that it's important to make it generalizable right now, then I'd be interested to hear some concrete use cases.

EDIT: although I suppose an unmentioned downside of this is that # [dependencies] might look a bit too much like an attribute.

As you mentioned, there is the syntax ambiguity and backticks let us "escape" this block of # lines.

Yes, the cargo specifier is redundant to the reader but each use of cargo is for a different purpose

One is for execution

One is for parsing

I've noted in the Future Possibilities that we could relax the requirement on having cargo in the infostring in the future. I would lean towards defaulting to cargo rather than parsing the shebang because shebang parsing is messy.

We also wanted to leave the door open (slightly) for adding additional frontmatter blocks, like if we decide to embed lockfiles.

epage · 2023-09-27T00:51:36Z

If you think this RFC a little bit further, then you could generalize this to make rustc ignore all ``` delimitered blocks, to say that rustc should just ignore all these sections and leave them to other tools similarly to how it does it already for the shebang. And then you'd see how similar triple backtick blocks are to /* */ comments. From there to just adding `/* */` comments to the AST it's only a small step.

Is this meant to positive suggest something or to point out a slipper slope? I'm not too sure the intent.

IMO it's bad style to add cargo specific extensions to Rust's syntax. Just say that ``` delimitered blocks are ignored by rustc and that parser implementations are suggested to add them.

I'm not seeing how this is different than the suggested future state. We are starting with it being locked to cargo initially as we work out the design / usage and then can remove that restriction which is noted Future Possibilities.

text/3503-frontmatter.md

jplatte · 2023-09-27T07:31:48Z

text/3503-frontmatter.md

+[drawbacks]: #drawbacks
+
+- A new concept for Rust syntax, adding to overall cognitive load
+- Requires people escape markdown code fences with an extra backtick which they are likely not used to doing (or aware even exists)


Does this refer to markdown inside multiline TOML strings? It's not really clear. Do any of Cargo's manifest fields even support markdown syntax?

It may be referring to general usage outside of Rust source files, for example if I wanted to express this syntax here in this comment I would have to escape the backticks somehow (or indent by four spaces, but that doesn't allow syntax highlighting).

Ooohh I see. In that case there's also the option of using the (older, I think?) syntax of prefixing the .rs snippet with four spaces instead of ````, though I'm not sure it it's as well-known as code fences, and it has the downside of not supporting language tags.

394387b adds some context. This is about sharing snippets over github or zulip. Since you are putting a code fence inside of a code fence, the outer one needs to use 4 backticks

bstrie · 2023-09-27T07:49:13Z

text/3503-frontmatter.md

+- Parsers are available to make this work (e.g. `syn`)
+
+Downsides
+- The `cargo` macro would need to come from somewhere (`std`?) which means it is taking on `cargo`-specific knowledge


For the macro approach, I don't think it would be necessary to embed any Cargo-specific knowledge in std. In every other approach the data here is stored in a glorified comment, which means we're fine if it gets thrown away as far as Rust is concerned. The macro here could simply expand to nothing, and trust that other tooling will parse the macro body as needed (which is easier than it sounds, since the macro body should just be treated as raw tokens rather than anything that needs parsing). Rather than calling it cargo, call it build! or meta! or something. Although I suppose the fact that it will still need to lex to Rust tokens might be limiting compared to a string or a comment, unless we want to make it magical.

I added mention of meta! in 42077b7. I don't bother exploring it due to the other problems with macros.

pksunkara · 2023-09-27T09:09:56Z

text/3503-frontmatter.md

+# Future possibilities
+[future-possibilities]: #future-possibilities
+
+- Treat `cargo` as the default infostring


What is the main reason for not including this in this proposal?

I clarified this a little in 842f722

Basically, we want to start with the absolute minimal approach and see what we feel needs to be relaxed from there (which is backwards compatible) rather than make a lot of assumptions and then regret them,

kurtbuilds · 2024-02-29T22:12:41Z

The question when adding something should always be, "why should it be included?", not "Why not include it?" In this case, the specific question is, why should it become Rust syntax?

I argue above, that it shouldn't. It becomes a 9th (I think?) comment syntax, leading to all kinds of downstream questions. Can I put it elsewhere? Does it work like docstrings? What's the single-line analog of it? All questions that add complexity to learning the language.

You raise an excellent point in your post, though. If there's comments around or before the frontmatter - what impact does that have. It would imply that the frontmatter handler has to know something about comment syntax to handle those cases correctly. At which point, I'd ask the question, why not just put the "frontmatter" block inside existing comment blocks? It doesn't need the full weight of a rust syntax parser, just enough to handle comment blocks - which seems like no more or no less machinery than a --- block parser.

EDIT: There's another prior art discussion that I don't think has been brought up before. There's large precedent for comments being used for somewhat-out-of-band tooling. Directives for formatters and linters is the main one. Putting a Cargo.toml within a plain old comment block fits very nicely within that pattern and mental model.

epage · 2024-03-18T19:29:16Z

@dlight

Could this become Rust syntax? Or at least, be recognized by rustc in order to give high quality error messages (or to just ignore it and assume it has been handled by an external tool).

That is what this proposal is about is making --- blocks Rust syntax.

(Sorry if this has been proposed already) I think that there should at least be an idiom to optionally wrap this into a Rust comment, like this:

See https://github.com/epage/rfcs/blob/frontmatter/text/3503-frontmatter.md#alternative-5-regular-comment

epage · 2024-03-18T19:34:57Z

@kurtbuilds

At which point, I'd ask the question, why not just put the "frontmatter" block inside existing comment blocks? It doesn't need the full weight of a rust syntax parser, just enough to handle comment blocks - which seems like no more or no less machinery than a --- block parser.

This does not solve any of problems laid out in the RFC

We are still adding structure to an unstructured syntax
At minimum, we then need to be able to handle the given types of Rust comments
- Unless we put further restrictions on the comment, we have to deal with not just comments but the full Rust grammar.
- If we put restrictions on the comment, we are taking something with few rules and added special rules to it, making it harder to intuit and work with as there is a gap in power between how the tool they are using can generally be used and how it can be used in this one case.

EDIT: There's another prior art discussion that I don't think has been brought up before. There's large precedent for comments being used for somewhat-out-of-band tooling. Directives for formatters and linters is the main one. Putting a Cargo.toml within a plain old comment block fits very nicely within that pattern and mental model.

Besides shebang, which has syntactic meaning, the Rust ecosystem doesn't rely on this precedence. For example, rustfmt and clippy use attributes over comments.

LunarLambda · 2024-04-11T08:44:26Z

A good argument against embedding the manifest in comments is that it will make it harder for tooling like treesitter (used in various text editors for syntax handling) to handle them properly.

For example, many editors have no issues highlighting code blocks in Markdown with the appropriate language, since code blocks are a separate syntactical element from the rest. However, as of today I think only a couple editors (I'm only aware of VS Code) highlights markdown syntax embedded in Rust doc-comments, which is harder to do since it's embedded inside of an existing syntactical element (comments), which are commonly line-based and not block-based as well. (How many people write /** doc comments?).

Additionally, changing and controlling (e.g. where it can appear, etc.) a separate syntax element on Rust's side of the equation is a lot easier than trying to do the same with comments.

Allowing tools to provide good integration with the feature without technical compromises is an important aspect of this feature's design after all.

nikomatsakis · 2024-04-11T11:18:30Z

@rfcbot reviewed -- thanks for your patience and hard work on this, @epage

rfcbot · 2024-04-11T11:18:35Z

🔔 This is now entering its final comment period, as per the review above. 🔔

rfcbot · 2024-04-21T11:30:44Z

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

jsgf · 2024-04-21T21:09:00Z

This all looks great and my initial questions and concerns were all addressed by the text of the rfc itself.

My principle concern is that we can support non cargo, both so that non cargo can ignore this and also make use of the machinery for its own purposes. I think the infostrings will be fine though I guess "missing" will default to cargo, or maybe depend on that the interpreter is.

For cargo, isn't config.toml also a plausible thing here? For example for target specific code, or use of special rustc flags?

kennytm · 2024-04-22T10:54:53Z

For cargo, isn't config.toml also a plausible thing here? For example for target specific code, or use of special rustc flags?

I suppose that has been addressed in the Future possibilities part.

#!/usr/bin/env -S cargo +nightly -Zscript
--- Cargo.toml
[dependencies]
serde = "1"
---

--- .cargo/config.toml
[profile.dev]
panic = "abort"
---

use serde::Deserialize;
...

epage · 2024-04-22T14:56:56Z

For cargo, isn't config.toml also a plausible thing here? For example for target specific code, or use of special rustc flags?

The important part for this RFC is that we leave room for future expansion if its needed. If nothing else, there is the chance of needing support for Cargo.lock (though I personally hope we can find ways to avoid that)

For what gets supported for embedding, the more relevant RFC is #3502. Specifically, the Cargo team wanted all of this to focus more on an MVP. We don't even support workspaces in the first iteration. On that RFC, configs are worthwhile to discuss as for how likely of a future possibility they are. My quick note on that though is they are unlikely to ever be supported. Configs are environmental and not project configuration. It doesn't make sense to embed them in the project and it mixes up their roles (which are already muddled enough as-is). Instead, the focus should be on rust-lang/cargo#12738. Again, if you want to discuss this further, #3502 is a better place for this.

jsgf · 2024-04-23T00:01:11Z

Really I was just commenting on the wording:

At least for cargo's use cases, the only other file that we would consider supporting is Cargo.lock...

"Only" seems pretty strong there. But you're right that's in the weeds as far as this RFC goes.

joshtriplett

This has passed FCP. Merging. Thank you!

jwnrt · 2024-05-04T17:48:02Z

I have a question about the alternatives that use macros (4 and 6). These are blocked on:

Either we expose syns lesser parse errors or we skip errors, deferring to rustc’s, but then have the wrong behavior on commands that don’t invoke rustc, like cargo metadata

I agree that syn isn't a good option here but using rustc as the parser seems ideal. Do you happen to know the reason that commands like cargo metadata cannot invoke rustc when they're used on a cargo script file? It seems reasonable to require rustc be installed.

I hope I'm understanding correctly that this would work similarly to how you can run rustc --print crate-name on a file containing ![crate_name = "foo"] and get foo back without rustc doing a full compilation. I thought maybe rustc defaulting to the 2015 edition would cause problems here, but that command works fine on files containing async functions (for example) and also worked on files using dependencies.

(aside: sorry if it's bad etiquette to discuss an RFC after it's accepted, I can take this elsewhere if necessary)

epage · 2024-05-04T19:37:51Z

Some approaches to parsing include

syn for which we have to decide between reporting poor errors or sometimes ignoring the manifest when rustc isn't invoked to error on our behalf
rustc_lexer (and maybe more parsing mechanisms) this would get in the way of this being focused on any editor tool being able to operate on
creating a new mechanism for dumping any attribute with rustc --print. This wasn't considered in the original design. An important aspect is that we be able to edit the embedded manifest. We'd have to write this to include span information of the attribute's string which is quite specialized (since this is a generic attribute dumping feature and attributes can come in many forms) and would require special handling for the print to generate parseable content for cargo.

(aside: sorry if it's bad etiquette to discuss an RFC after it's accepted, I can take this elsewhere if necessary)

In my experience, re-litigating RFCs without there being dramatically new information is not very productive.

jwnrt · 2024-05-04T20:08:01Z

We'd have to write this to include span information of the attribute's string which is quite specialized (since this is a generic attribute dumping feature and attributes can come in many forms) and would require special handling for the print to generate parseable content for cargo.

This is definitely an issue. The output of rustc --print would ideally be the "processed" string that doesn't include escaped characters etc, but the span would need to be the "raw" string which could even include things like concat!(...). It doesn't feel impossible to solve, though.

In my experience, re-litigating RFCs without there being dramatically new information is not very productive.

Understood, I really don't want to waste everyone's time. I don't have new information beyond the concerns raised previously and should have gotten involved before the FCP.

I'll stick my concern at the bottom of this comment but don't really want to restart discussion here. I suppose a new RFC would be needed to propose an alternative.

My concern is that this syntax feels unlike any existing concept in Rust. It conflicts with prior art, by which I mean other tools (rustfmt, clippy, rustdoc) that interface using attributes and other embedded languages (asm and markdown) which go into macros and comments/attributes respectively. Subjectively it "feels ugly" which I know doesn't carry much weight, but even in Markdown it feels like a bodge that breaks everywhere outside Zola and Hugo. I do feel that an attribute and a cargo! { ... } macro for convenience would be more explicit and integrate better with other Rust concepts like include_str.

Apologies if this is too critical of the proposal that I know you put a lot of time and work into.

epage · 2024-05-04T20:42:05Z

Have you used it in practice or is this from only reading the RFC?

jwnrt · 2024-05-04T20:45:20Z

Just the examples in the RFC. I'm enjoying cargo-script but haven't tried the frontmatter syntax.

epage · 2024-05-04T21:00:40Z

I'd recommend having first hand experience with it as well as making sure you are familiar with all of the background information in the section leading up to describing each of the options.

For context where I'm coming from, I used the doc comment syntax for about 6 months with 4 months for backtick and dash frontmatter each. Just over the last 8 months, I've written about 50 scripts from scratch as I use it to reproduce nearly every bug report I get.

fix(toml): Remove unstable rejrected frontmatter syntax for cargo script ### What does this PR try to resolve? With rust-lang/rfcs#3503 approved, we no longer need to allow easy, high fidelity experiments with alternative cargo script syntax. ### How should we test and review this PR? ### Additional information We still need to improve the experience for users writing bad syntax but that can come later.

epage added 3 commits September 26, 2023 11:24

feat: Add skeleton

ddcaeea

feat: Initial frontmatter RFC

290cb97

fix: Update cross-links to cargo-script

c088429

epage added the T-lang Relevant to the language team, which will review and decide on the RFC. label Sep 26, 2023

epage mentioned this pull request Sep 26, 2023

RFC: cargo-script #3502

Merged

epage added 7 commits September 26, 2023 14:42

fix: Update for RFC number

f644b5f

fix: Grammar

29839ae

fix: Add multi-frontmatter future possibility

d97eb11

styke: Formatting

f7b3f47

fix: Correct PEP 723 description

ea5d05e

fix: Remove some template boilerplate

8f144ee

fix: Note that cargo being the default infostring for future

a223b95

This comment was marked as resolved.

Sign in to view

fix: Add more prior art

1a243fd

fix: Call out why shebang-ext needs backticks

ddcca01

epage commented Sep 27, 2023

View reviewed changes

epage added 2 commits September 26, 2023 21:20

fix: Expand on a benefit

67114fe

fix: Adjust formatting

bcd9c6d

programmerjake reviewed Sep 27, 2023

View reviewed changes

text/3503-frontmatter.md Outdated Show resolved Hide resolved

jplatte reviewed Sep 27, 2023

View reviewed changes

bstrie reviewed Sep 27, 2023

View reviewed changes

pksunkara reviewed Sep 27, 2023

View reviewed changes

This comment was marked as resolved.

Sign in to view

epage added 3 commits September 27, 2023 08:48

fix: Remove redundant word

bb26c75

fix: Clarify cargo-script inside markdown

394387b

fix: Clarify why infostring default is deferred

842f722

rfcbot added the final-comment-period Will be merged/postponed/closed in ~10 calendar days unless new substational objections are raised. label Apr 11, 2024

rfcbot removed the proposed-final-comment-period Currently awaiting signoff of all team members in order to enter the final comment period. label Apr 11, 2024

rfcbot added finished-final-comment-period The final comment period is finished for this RFC. and removed final-comment-period Will be merged/postponed/closed in ~10 calendar days unless new substational objections are raised. labels Apr 21, 2024

rfcbot added the to-announce label Apr 21, 2024

joshtriplett approved these changes Apr 23, 2024

View reviewed changes

joshtriplett merged commit 130caeb into rust-lang:master Apr 23, 2024

epage deleted the frontmatter branch April 23, 2024 16:18

rami3l mentioned this pull request Apr 24, 2024

Decide on a better strategy for release notes/changelogs rust-lang/rustup#3784

Open

traviscross removed the I-lang-nominated Indicates that an issue has been nominated for prioritizing at the next lang team meeting. label May 1, 2024

epage mentioned this pull request May 2, 2024

Tracking Issue for cargo-script RFC 3424 rust-lang/cargo#12207

Open

31 tasks

epage mentioned this pull request May 4, 2024

fix(toml): Remove unstable rejrected frontmatter syntax for cargo script rust-lang/cargo#13861

Merged

RFC: Syntax for embedding cargo-script manifests #3503

RFC: Syntax for embedding cargo-script manifests #3503

Conversation

epage commented Sep 26, 2023 • edited by traviscross Loading

thomcc commented Sep 27, 2023

This comment was marked as resolved.

est31 commented Sep 27, 2023

epage commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epage commented Sep 27, 2023

jplatte Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bstrie Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

kurtbuilds commented Feb 29, 2024 • edited Loading

epage commented Mar 18, 2024

epage commented Mar 18, 2024

LunarLambda commented Apr 11, 2024

nikomatsakis commented Apr 11, 2024

rfcbot commented Apr 11, 2024

rfcbot commented Apr 21, 2024

jsgf commented Apr 21, 2024

kennytm commented Apr 22, 2024

epage commented Apr 22, 2024

jsgf commented Apr 23, 2024

joshtriplett left a comment

Choose a reason for hiding this comment

jwnrt commented May 4, 2024 • edited Loading

epage commented May 4, 2024

jwnrt commented May 4, 2024 • edited Loading

epage commented May 4, 2024

jwnrt commented May 4, 2024

epage commented May 4, 2024

epage commented Sep 26, 2023 •

edited by traviscross

Loading

jplatte Sep 27, 2023 •

edited

Loading

bstrie Sep 27, 2023 •

edited

Loading

kurtbuilds commented Feb 29, 2024 •

edited

Loading

jwnrt commented May 4, 2024 •

edited

Loading

jwnrt commented May 4, 2024 •

edited

Loading