Add parser values

Cargo.toml

@@ -33,6 +33,7 @@ debug = [] # Non-stable, for internal use only
 proc-macro2 = { version = "1.0.93" }
 syn = { version = "2.0.107", default-features = false, features = ["parsing", "derive", "printing", "clone-impls", "full"] }
 quote = { version = "1.0.38", default-features = false }
+slotmap = { version = "1.0.7" }
 
 [dev-dependencies]
 trybuild = { version = "1.0.110", features = ["diff"] }

plans/TODO.md

@@ -166,29 +166,49 @@ These are things we definitely want to do:
 First, read the @./2025-11-vision.md
 
 - [x] We store input in the interpreter
-- [ ] Create new `Parser` value kind
-- [ ] Create (temporary) `parse X as Y { }` expression
+- [x] Create new `Parser` value kind
+- [x] Add ParserHandle to `InputHandler` and use some generational map to store ParseStacks (or import slotmap)
+  - [x] Look at https://donsz.nl/blog/arenas/
+  - [ ] If using slotmap / generational-arena, replace the arena implementation too
+- [x] Create (temporary) `parse X => |Y| { }` expression
+- [x] Bind `input` to `Parser` at the start of each parse expression
 - [ ] Create `consume X @[ .. ]` expression
-- [ ] Bind `input` to `Parser` at the start of each parse expression
 - [ ] Move transform logic from transformers onto `Parser`, and delete the transformers
-- [ ] Rename the transform stream to `ParseModeStream`
-- [ ] Reversion works in attempt blocks, via forking and committing or rolling back
-      the fork, fix `TODO[parser-input-in-interpreter]`
-- [ ] Remove all remaining parsers.
+- [ ] Remove all remaining transformers.
 - [ ] Remove parsing in a stream pattern - instead we just support a literal
+- [ ] Rename the transform stream to `ParseModeStream`
+- [ ] Reversion works in attempt blocks, via forking and committing or rolling back the fork, fix `TODO[parser-input-in-interpreter]`
 - [ ] Address any remaining `TODO[parser-no-output]` and `TODO[parsers]`
+- [ ] Add tests for all the methods on Parser, and for nested parse statements
 
 `Parser` methods:
-* `ident()`, `is_ident()`
-* `literal()`, `is_literal()`
-* `integer()`, `is_integer()`
-* `float()`, `is_float()`
-* `char()`, `is_char()`
-* `string()`, `is_string()`
-* `error()` etc
-* `end()`, `is_end()`
-* `token_tree()`
-* `span()` or `cursor()` -- maybe? outputs a token with a span for outputting errors. If at end of an inner stream, it outputs the ident `END` with the span of the closing bracket.
+- [x] `ident()`, `is_ident()`
+- [x] `literal()`, `is_literal()`
+- [x] `integer()`, `is_integer()`
+- [x] `float()`, `is_float()`
+- [x] `char()`, `is_char()`
+- [x] `string()`, `is_string()`
+- [x] `end()`, `is_end()`
+- [ ] `rest()`
+- [ ] `error()` etc
+- [ ] `token_tree()`
+- [ ] `span()` or `cursor()` -- maybe? outputs a token with a span for outputting errors. If at end of an inner stream, it outputs the ident `END` with the span of the closing bracket.
+
+And all of these from normal macros:
+- [ ] block: a block (i.e. a block of statements and/or an expression, surrounded by braces)
+- [ ] expr: an expression
+- [ ] ident: an identifier (this includes keywords)
+- [ ] item: an item, like a function, struct, module, impl, etc.
+- [ ] lifetime: a lifetime (e.g. 'foo, 'static, …)
+- [ ] literal: a literal (e.g. "Hello World!", 3.14, '🦀', …)
+- [ ] meta: a meta item; the things that go inside the #[...] and #![...] attributes
+- [ ] pat: a pattern
+- [ ] path: a path (e.g. foo, ::std::mem::replace, transmute::<_, int>, …)
+- [ ] stmt: a statement
+- [ ] tt: a single token tree
+- [ ] ty: a type
+- [ ] vis: a possible empty visibility qualifier (e.g. pub, pub(in crate), …)
+```
 
 Consider if we want separate types for e.g.
 * `Span`
@@ -215,8 +235,7 @@ input.repeated(
 ```
 
 Later:
-- [ ] Support for starting to parse a `input.open('(')` in the left part of an attempt arm
-      and completing in the right arm `input.close(')')` - there needs to be some error checking in the parse stream stack. We probably can't allow closing in the LHS of an attempt arm. We should record a reason on the new parse buffer and raise if it doesn't match
+- [ ] Support for starting to parse a `input.open('(')` in the left part of an attempt arm and completing in the right arm `input.close(')')` - there needs to be some error checking in the parse stream stack. We probably can't allow closing in the LHS of an attempt arm. We should record a reason on the new parse buffer and raise if it doesn't match
 
 ## Methods and closures
 
@@ -351,6 +370,9 @@ preinterpret::run! {
 ## Optimizations 
 
 - [ ] Look at benchmarks and if anything should be sped up
+- [ ] Speeding up stream literal processing
+  - [ ] When interpreting a stream literal, we can avoid having to go through error handling pathways to get an `output` from the intepreter by storing a `OutputInterpreter<'a>` which wraps an `&mut OutputStream` and a pointer to an Intepreter, and can be converted back into/from an `Interpreter` easily
+  - [ ] Possibly similarly for an `InputInterpreter<'a>` when processing a `ConsumeStream`
 - [ ] Speeding up scopes at runtime:
   - [ ] In the interpreter, store a flattened stack of variable values
   - [ ] `no_mutation_above` can be a stack offset

src/expressions/array.rs

@@ -213,7 +213,7 @@ define_interface! {
                 Ok(())
             }
 
-            [context] fn to_stream_grouped(this: ArrayExpression) -> StreamOutput<impl StreamAppender> {
+            [context] fn to_stream_grouped(this: ArrayExpression) -> StreamOutput<impl StreamAppender> [ignore_type_assertion!] {
                 let error_span_range = context.span_range();
                 StreamOutput::new(move |stream| this.output_items_to(&mut ToStreamContext::new(stream, error_span_range), Grouping::Grouped))
             }

src/expressions/control_flow.rs

@@ -548,3 +548,75 @@ impl AttemptExpression {
         self.braces.control_flow_err("No attempt arm ran successfully. You may wish to add a fallback arm `{} => { None }` to ignore the error or to propogate a better message: `{} => { %[<tokens for error span>].error(\"Error message\") }`.")
     }
 }
+
+pub(crate) struct ParseExpression {
+    parse_ident: ParseKeyword,
+    input: Expression,
+    _fat_arrow: Unused<Token![=>]>,
+    _left_bar: Unused<Token![|]>,
+    parser_variable: VariableDefinition,
+    _right_bar: Unused<Token![|]>,
+    scope: ScopeId,
+    body: UnscopedBlock,
+}
+
+impl HasSpanRange for ParseExpression {
+    fn span_range(&self) -> SpanRange {
+        SpanRange::new_between(self.parse_ident.span(), self.body.span())
+    }
+}
+
+impl ParseSource for ParseExpression {
+    fn parse(input: SourceParser) -> ParseResult<Self> {
+        let parse_ident = input.parse()?;
+        let input_expression = input.parse()?;
+        let _fat_arrow = input.parse()?;
+        let _left_bar = input.parse()?;
+        let parser_variable = input.parse()?;
+        let _right_bar = input.parse()?;
+        let body = input.parse()?;
+        Ok(Self {
+            parse_ident,
+            input: input_expression,
+            _fat_arrow,
+            _left_bar,
+            parser_variable,
+            _right_bar,
+            scope: ScopeId::new_placeholder(),
+            body,
+        })
+    }
+
+    fn control_flow_pass(&mut self, context: FlowCapturer) -> ParseResult<()> {
+        context.register_scope(&mut self.scope);
+        self.input.control_flow_pass(context)?;
+        context.enter_scope(self.scope);
+        self.parser_variable.control_flow_pass(context)?;
+        self.body.control_flow_pass(context)?;
+        context.exit_scope(self.scope);
+        Ok(())
+    }
+}
+
+impl ParseExpression {
+    pub(crate) fn evaluate(
+        &self,
+        interpreter: &mut Interpreter,
+        ownership: RequestedValueOwnership,
+    ) -> ExecutionResult<EvaluationItem> {
+        let input = self
+            .input
+            .evaluate_owned(interpreter)?
+            .resolve_as("The input to a parse expression")?;
+
+        interpreter.enter_scope(self.scope);
+
+        let output = interpreter.start_parse(input, |interpreter, handle| {
+            self.parser_variable.define(interpreter, handle);
+            self.body.evaluate(interpreter, ownership)
+        })?;
+
+        interpreter.exit_scope(self.scope);
+        Ok(output)
+    }
+}

src/expressions/evaluation/control_flow_analysis.rs

@@ -185,6 +185,7 @@ impl Leaf {
             Leaf::AttemptExpression(attempt_expression) => {
                 attempt_expression.control_flow_pass(context)
             }
+            Leaf::ParseExpression(parse_expression) => parse_expression.control_flow_pass(context),
             Leaf::Discarded(_) => Ok(()),
             Leaf::Value(_) => Ok(()),
         }

src/expressions/evaluation/node_conversion.rs

@@ -64,6 +64,11 @@ impl ExpressionNode {
                         let item = attempt_expression.evaluate(context.interpreter(), ownership)?;
                         context.return_item(item)?
                     }
+                    Leaf::ParseExpression(parse_expression) => {
+                        let ownership = context.requested_ownership();
+                        let item = parse_expression.evaluate(context.interpreter(), ownership)?;
+                        context.return_item(item)?
+                    }
                 }
             }
             ExpressionNode::Grouped { delim_span, inner } => {

src/expressions/evaluation/value_frames.rs

@@ -752,8 +752,7 @@ impl EvaluationFrame for BinaryOperationBuilder {
                 // Try method resolution first (we already determined this during left evaluation)
                 if let Some(method) = method {
                     // TODO[operation-refactor]: Use proper span range from operation
-                    let span_range =
-                        SpanRange::new_between(left.span_range().start(), right.span_range().end());
+                    let span_range = SpanRange::new_between(left.span_range(), right.span_range());
 
                     let mut call_context = MethodCallContext {
                         output_span_range: span_range,

src/expressions/expression.rs

@@ -161,6 +161,7 @@ pub(super) enum Leaf {
     WhileExpression(Box<WhileExpression>),
     ForExpression(Box<ForExpression>),
     AttemptExpression(Box<AttemptExpression>),
+    ParseExpression(Box<ParseExpression>),
 }
 
 impl HasSpanRange for Leaf {
@@ -176,6 +177,7 @@ impl HasSpanRange for Leaf {
             Leaf::WhileExpression(expression) => expression.span_range(),
             Leaf::ForExpression(expression) => expression.span_range(),
             Leaf::AttemptExpression(expression) => expression.span_range(),
+            Leaf::ParseExpression(expression) => expression.span_range(),
         }
     }
 }
@@ -188,7 +190,8 @@ impl Leaf {
             | Leaf::LoopExpression(_)
             | Leaf::WhileExpression(_)
             | Leaf::ForExpression(_)
-            | Leaf::AttemptExpression(_) => true,
+            | Leaf::AttemptExpression(_)
+            | Leaf::ParseExpression(_) => true,
             Leaf::Variable(_) | Leaf::Discarded(_) | Leaf::Value(_) | Leaf::StreamLiteral(_) => {
                 false
             }

src/expressions/expression_parsing.rs

@@ -111,8 +111,10 @@ impl<'a> ExpressionParser<'a> {
 
                 if punct.as_char() == '.' {
                     UnaryAtom::Range(input.parse()?)
-                } else {
+                } else if punct.as_char() == '-' || punct.as_char() == '!' {
                     UnaryAtom::PrefixUnaryOperation(input.parse()?)
+                } else {
+                    return input.parse_err("Expected an expression");
                 }
             }
             SourcePeekMatch::Ident(ident) => {
@@ -129,6 +131,7 @@ impl<'a> ExpressionParser<'a> {
                     "while" => UnaryAtom::Leaf(Leaf::WhileExpression(Box::new(input.parse()?))),
                     "for" => UnaryAtom::Leaf(Leaf::ForExpression(Box::new(input.parse()?))),
                     "attempt" => UnaryAtom::Leaf(Leaf::AttemptExpression(Box::new(input.parse()?))),
+                    "parse" => return Ok(UnaryAtom::Leaf(Leaf::ParseExpression(Box::new(input.parse()?)))),
                     "None" => UnaryAtom::Leaf(Leaf::Value(SharedValue::new_from_owned(
                         ExpressionValue::None.into_owned(input.parse_any_ident()?.span_range()),
                     ))),

src/expressions/mod.rs

@@ -12,6 +12,7 @@ mod integer;
 mod iterator;
 mod object;
 mod operations;
+mod parser;
 mod range;
 mod statements;
 mod stream;
@@ -41,5 +42,6 @@ use character::*;
 use expression_parsing::*;
 use float::*;
 use integer::*;
+use parser::*;
 use range::*;
 use string::*;