Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Variable parsing rework (Array input) #1200

Open
wants to merge 64 commits into
base: master
Choose a base branch
from

Conversation

quinchs
Copy link
Member

@quinchs quinchs commented Jan 31, 2024

Summary

This PR reworks the variable input mechanism to use nom as a parser; allowing us to parse array inputs and other complex inputs.

Array Inputs

The array input works much like writing an array in EdgeQL, as proposed by Sully:

cli> select <array<int64>>$arg;
Parameter <array>$arg: [1, 3, 5]
{[1, 3, 5]}

and it also forces str to be quoted with either single quotes or double quotes in the input:

cli> select <array<str>>$arg;
Parameter <array>$arg: ["abc", 'def']
{['abc', 'def']}

Error reporting and hints

With the change to nom, errors and hints have changed too as a side effect, in the current state of this PR it dumps the parser errors pretty verbosely, I would like some feedback on how we should format these errors to be as concise as possible, some examples:
image
image
image
image

Things get pretty messy with some of the parsers though:
image

No context yet, so error information is not helpful, as well as some parsing is broken/major change
mostly done with quoted string parser, still a few gotcha cases like `'abc\'"', but it should be simple to fix that.
@quinchs quinchs linked an issue Jan 31, 2024 that may be closed by this pull request
@1st1 1st1 requested a review from fantix January 31, 2024 18:16
@1st1
Copy link
Member

1st1 commented Jan 31, 2024

Can you add a few examples of how successful input looks like? What if you have an array of tuples which have arrays in them?

@msullivan
Copy link
Member

We definitely shouldn't be including things like TakeWhileMN in error messages

@msullivan
Copy link
Member

We need to figure out how to make check warnings not appear once for each platform

Copy link
Member

@msullivan msullivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the overall approach is solid, but have a bunch of suggestions about various parsers.

Also, we need some tests

src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Show resolved Hide resolved
@msullivan
Copy link
Member

(Get rid of all the unused imports)

src/prompt/variable.rs Outdated Show resolved Hide resolved
@quinchs
Copy link
Member Author

quinchs commented Feb 7, 2024

Can you add a few examples of how successful input looks like? What if you have an array of tuples which have arrays in them?

Array of tuples

image

Array of tuples with arrays

image

Array of json

image

src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
src/cloud/ops.rs Outdated
@@ -85,19 +98,49 @@ pub struct Version {
pub version: String,
}

#[derive(Debug, serde::Deserialize)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm..., weird? why is github including this in the diff?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no clue

src/prompt/variable.rs Outdated Show resolved Hide resolved
Copy link
Member

@msullivan msullivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

I think we still need named tuples but that can be a follow-up PR if you want

src/prompt/variable.rs Show resolved Hide resolved
@quinchs
Copy link
Member Author

quinchs commented Feb 13, 2024

Added named tuples, might wanna gloss over it

@msullivan
Copy link
Member

Oh, clean up the unused imports please

@msullivan
Copy link
Member

Do the tuples and array support trailing commas? They should, and we should make sure to test it

src/prompt/variable.rs Outdated Show resolved Hide resolved
src/prompt/variable.rs Outdated Show resolved Hide resolved
@quinchs
Copy link
Member Author

quinchs commented Feb 13, 2024

Do the tuples and array support trailing commas?

No, any extra comma errors. I'll change that

@msullivan
Copy link
Member

Hm, actually now that I think about it more, maybe we should require the named tuples to be in order, also? That matches edgeql better (where the order is important), though (for example) the Python bindings will accept dictionaries and namedtuples without caring about order. @elprans thoughts on this?

We should probably also support writing non-named tuples when a named tuple is expected also, which I now realize is what @elprans was probably saying earlier

@msullivan
Copy link
Member

Oh, and leading spaces after an opening ( or [

@quinchs
Copy link
Member Author

quinchs commented Feb 13, 2024

We should probably also support writing non-named tuples when a named tuple is expected

this will be exclusive right? ex: cant have 1 named and 1 unnamed part; its either all named or no named?

@msullivan
Copy link
Member

We should probably also support writing non-named tuples when a named tuple is expected

this will be exclusive right? ex: cant have 1 named and 1 unnamed part; its either all named or no named?

Yeah

delimited(multispace0, f, multispace0)
}

fn space(i: &str) -> IResult<&str, &str, ParsingError> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same as multispace0, right?

Comment on lines 592 to 604
if result.len() < self.shape.elements.len() {
return Err(ParsingError::Incomplete {
hint: Some(
format!(
"Expecting one of the following element name(s): {}",
self.shape.elements.iter()
.filter(|v| !result.iter().any(|t| t.0 == v.name.as_str()))
.map(|v| v.name.clone())
.collect::<Vec<String>>()
.join(", ")
)
)
})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to handle the case with duplicate field names also

@msullivan
Copy link
Member

I just pushed commits that support trailing commas for tuples, named tuples, and arrays. Mostly I did this as a personal experiment with nom.

The combinator-y tuple that works by building up a new boxed parser isn't too bad. It took some fiddling because I'm not that good at rust anymore, but is basically reasonable, I think.

The array/namedtuple version I found very frustrating. I implemented a function trailing_separated_list0, with the intention that it behave like separated_list0 but allow trailing separators. This worked but I wasn't able to figure out a way to do it purely by composing combinators, and needed to invoke the parsers directly, which cuts against the benefit of parser combinators somewhat. The final result isn't actually that bad, though, just unsatisfying.

@aljazerzen
Copy link
Contributor

@quinchs what's the status of this?

@quinchs
Copy link
Member Author

quinchs commented Apr 18, 2024

@quinchs what's the status of this?

I've got some remaining changes to fix with it which sully has mentioned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support array input parameters
5 participants