Conservative variadic functions

[sgrif]

Attempts to add a proposal for variadic functions which require the
minimal possible commitment, and avoids introducing new syntax.

Rendered

[jonas-schievink]

IMO variadic generics (#376) seem like a cleaner solution, and they have the potential to solve other problems, too.

[cramertj]

Have you seen @eddyb's implementation of tuple-based variadics? Personally, I find this approach less magic than the conversion of foo(arg1, arg2, arg3) into foo(Cons(arg1, Cons(arg2, Cons(arg3)))). It would also allow implementations of the Cons and Split traits for existing types, rather than requiring that everything variadic-like be converted to a the concrete struct Cons and struct Nil representation.

[sgrif]

I have. I've also spent a lot of time working with code that has to work with tuples of various sizes. Ultimately these types are completely opaque to the caller, and for the callee tuples are harder to work with than an hlist is. Adding traits to make a tuple pretend to be an hlist just complicates things and is a big part of why I think previous proposals for this have fallen over.

[cramertj]

tuples are harder to work with than an hlist is

Compare the RFC example to the rough equivalent under the trait-based version:

impl<T> ToSql for T where
    T: Split,
    T::Head: ToSql,
    T::Tail: ToSql,
{
    fn to_sql(self, types: &mut [Type], out: &mut Vec<u8>) -> Result<(), Error> {
        let (head, tail) = self.unpack();
        head.to_sql(types, out)?;
        tail.to_sql(types, out)?;
        Ok(())
    }
}

impl ToSql for () {
    fn to_sql(self, _: &mut [Type], _: &mut Vec<u8>) -> Result<(), Error> {
        Ok(())
    }
}

The two seem fairly similar to me. In this case, the tuple being used is being consumed, but that can be fixed with a call to .ref_all() by the caller. I don't think this is any more opaque to the caller than fn foo<Args: ArgBound>(args: Args) where ArgBound is implement for Const<...> and Nil.

[cramertj]

Additionally, I don't think we're 'pretending' that a tuple is an HList. A tuple is a finite, ordered, heterogeneous collection.

[sgrif]

The two seem fairly similar to me.

They're not at all similar. impl<T> SomeTrait for T where T: SomeBounds is significantly harder to work with than having a concrete type, as it causes major coherence issues.

[cramertj]

@sgrif That's true! When I said "fairly similar" I was referring to the ease/readability of the implementation. But you're correct, coherence does cause a number of problems in these cases (I've been stumbling on this one recently myself).

I'd prefer to resolve those coherence issues rather than designing new language features to avoid them. However, your design for variadics would provide a more immediate solution for those cases, which is a plus.

[eddyb]

FWIW, we could probably allow impl<H, ...T> Foo for (H, ...T), which should be nicer.

[eddyb]

IOW, while the tuple trait is an interesting design, the compiler would always need an internal representation that allows (A, ...B, C) and it's just simpler to expose the sugary syntax to the user than simulate the correct trait behavior (when it matches and what associated types are).

[Ixrec]

The details of this proposal feel a lot like introducing new syntax, even if they technically aren't. I'm also not clear on how "variadic generics" is "out of scope", as the solution seems to involve and accept a lot of generics. Finally, the two case studies mentioned seem like they could both be solved using macros until we get real variadic generics, and in fact the diesel proposal you linked appears to be suggesting that diesel will do exactly that. So overall, I feel like I'm either misunderstanding something huge or this isn't really a minimally commital subset of what we eventually want to end up with.

I do like the dedicated hlist type instead of tuples (tbh I never understood what the advantage of using tuples for variadics was supposed to be), but that seems like it might as well be part of a proper variadic generics proposal.

[sgrif]

Finally, the two case studies mentioned seem like they could both be solved using macros until we get real variadic generics, and in fact the diesel proposal you linked appears to be suggesting that diesel will do exactly that.

Yes, that is 100% true. However, having a canonical representation in the language/standard library, and not forcing a macro invocation on our users is a huge win. (Forcing users to wrap multiple arguments in random_macro_from_this_lib!() is a much larger cost than forcing them to wrap something in a type from the standard library, e.g. a slice in terms of how approachable it makes the library)

[Ixrec]

I could definitely get behind a push to standardize a single hlist implementation (assuming "everyone" agrees that hlists are better than tuples for doing variadics).

The question then is whether a std hlist can be standardized without also settling on a design for the variadic generics feature that would eventually use it. That, I have no idea how to answer.

[withoutboats]

I think if we're going to standardize on something, I'd rather it be something like first class variadic generics (like the (H, ...T) that @eddyb mentioned), because that doesn't expose its mechanics to the user.

[ahicks92]

This looks incredibly unergonomic, without some sort of generic lambdas.

Maybe we get a crate that solves it, I don't know.

Possible alternative: work out scoped macros, allow #[scoped] or something on current macros to change the behavior, then library devs don't have to worry about clashing and can make all their variadic functions macros instead. I'd prefer this because it's a feature we sorely need anyway imho, and there's probably a much better long term design for variadic functions.

[arielb1]

@sgrif

Why the constraint on the last type parameter? Won't you just get type mismatches if it is not a type parameter and/or used in other places.

@camlorn

What's the point of generic lambdas?

[nikomatsakis]

So @cramertj has at some point written this gist which details an alternative approach to variadic generics:

https://github.com/cramertj/rfcs/blob/variadic-generics/text/0000-variadic-generics.md

@cramertj hope you don't mind me posting the link. =)

I personally am inclined to close this particular RFC but I'd like to see @sgrif, @cramertj, @eddyb, and others (@nrc also expressed interest) pursue a revised RFC along these more generic lines.

[sgrif]

Yes, I'm happy to close this in favor of a more full featured RFC. I just figured I'd throw out a "minimalist" form and see what people thought, as it seemed like the variadic generics had been proposed and shot down a few times.

[eddyb]

@sgrif I'm growing more and more tired of the "rust-call" hack and perhaps it's time to mentor someone to fix the implementation to have some form of variadic generics function signatures, at least.
But if we get the internals working, it's mostly a matter of bikeshedding surface syntax then.

[ahicks92]

@arielb1
What you often want to do with a variadic function is do something to all the arguments it gets passed without dynamic dispatch. It would be useful if you could just be like arg_pack.map(|x| x.foo()) or similar.

You can make everything else you want from this by implementing it as a state machine and iterating over the arguments in order.

I suppose this is sort of possible now if you're willing to write everything as a separate function and give an API which allows passing a second parameter, but that's annoying.

[eddyb]

@camlorn In one of the demos (which I don't have handy right now), I used .map(CloneFn) on a tuple of references to implement Clone (or a trait with the same signature, rather) for tuples, where CloneFn was an unit struct with a generic Fn impl, i.e. what could end up being for<T: Clone> |x: &T| x.clone().

[ahicks92]

@eddyb
Huh. I like that syntax for generic closures. I kept trying to come up with one but repeatedly failed.

[eddyb]

@camlorn Oops, I got it wrong, it's without for. I'm pretty much talking about #1650 FWIW.

[Ericson2314]

@withoutboats

because that doesn't expose its mechanics to the user.

Sugar is fine, but I'd like to just have sugar too with an underlying (perhaps unstable for longer like Fn*) trait so intermediate users don't need to learn new semantics / doubt whether there is new expressive power at all.

I know I, for one, immediately understood the current closures better than I had ever understood the purely-magic ones they replaced.

[cramertj]

@Ericson2314 One concern I have with implementing Tuple or similar via sugar from the beginning is that, ideally, any Tail: Tuple would implement for<Head: Sized> Cons<Head>. That is, for type Head: Sized and Tail: Tuple, (Head, ...Tail) is valid and equal to <Tail as Cons<Head>>::Out. I don't think this is currently possible to represent without type HRTBs or ATCs. With ATCs, you could even avoid a separate Cons trait entirely by making Cons an associated type constructor based on the type of Head, like (A, B, C)::Cons<T> = (T, A, B, C).

Edit: basically, I agree -- in the long run I'd like it to just be sugar. However, I don't want the feature to be blocked on type HTRBs/ATCs landing.

[eddyb]

(Head, ...Tail) is valid and equal to <Tail as Cons<Head>>::Out

Only outside of impl Self / trait parameters: there it has to mean both <Tail as Cons<Head>>::Out and any Split<Head=Head, Tail=Tail>, which makes it far more powerful.
That is, the compiler could both expand Tail in (Head, ...Tail) or match against it to extract Tail.

IOW, to get the same effect through desugaring you'd probably have to do this:

impl<Head, Tail: Tuple> Foo for (Head, ...Tail) {...}
// could desugar to:
impl<T, Head, Tail> Foo for T
    where T: Split<Head=Head, Tail=Tail>,
          Tail: Cons<Head, Out=T>
{...}

However, that encoding is suboptimal from an implementation point of view, and it only gets worse the more complex your tuple is, e.g. (A, B, C, ...D) requires 3 type parameters to repeatedly split off the front, or to prepend to D, one element at a time.
And (...A, B) is much harder to express (have to recurse through every element of A and even then I'm not sure coherence would play along), which is the asymmetry I disliked with previous cons-like proposals.

[sgrif]

The one thing that I think most people can agree on is that ultimately we want hlists for this or an hlist like structure. Most proposals and examples given in the past have either been explicitly focused around hlists, or trying to treat a tuple as an hlist example example example.

I do think it is important that the result be a concrete type. Having trait Tuple or similar would make it nearly impossible to work with from a coherence point of view, and often ignores the existence of the Nil case.

I have no issues with the idea of (A, ...B) (the intention was for this proposal to eventually lead to that sort of syntax regardless), but I do think it's important to be able to reason about what B could be, as you will basically always have to handle the nil case. Is nil () there? Often times rather than handling Cons<Head, Tail> and Nil, one instead has to write impls for Cons<Head, Cons<Head2, Tail>> and Cons<Head, Nil>. One would somewhat expect that to be written as (A, B, ...C) and (A,) but if () is nil then it's actually (A, ()).

That isn't to say that I don't think we should have the ... syntax, simply that I think that (A, ...B) needs to be a distinct type from (A, B, C, D) or what have you. Tuples work great for when you know that you have a fixed number of elements, and terribly when you need arbitrary length. I don't think sugar is the solution there, as it makes the case of 1 and 0 elements ambiguous.

The goal of this proposal was basically to come at the problem from a perspective of "if we know we need some kind of hlist, we could do this without affecting the type system or syntax". I figured giving that baseline (with the assumption that we may add syntax later, and affect the type system when we get to true variadic generics), but by skipping syntax and variadic generics the scope would be smaller. Diesel is looking at changing how we handle our workarounds for the lack of variadic functions, so having an idea of the direction the language will head with regards to that feature is important to us for being comfortable going 1.0.

That said, there's clearly a lot of opposition to this proposal, so I'm fine with closing it if there's no further discussion to be had on the minimalist form.

[eddyb]

I don't understand why you assume tuples to be a bad fit. The compiler can reason about variadic tuples exactly as well as it could reason about nominal hlists in terms of coherence.

[sgrif]

@eddyb I thought I laid that out clearly, sorry if I didn't. The main problem with tuples lies around the 1 element and 0 element case, which are very important when writing code that works with "an arbitrary number of elements of differing types which conform to this trait". If we want to be able to say where A: SomeTrait, ...B: SomeTrait, that requires that ...B have some representation for all numbers of elements, which would imply that for 0 elements it is (). But that would mean that (A, ...B) is (A, ()) when there is a single element, while one would expect to write (A,) to handle the single element case.

[cramertj]

@sgrif (A,) is (A, ...()), not (A, ()). Did you typo, or am I misunderstanding?

[sgrif]

@cramertj Ah, you're right. My mistake.

[cramertj]

Having trait Tuple or similar would make it nearly impossible to work with from a coherence point of view`

Did you read my proposal that Niko linked above? The idea is to mark Tuple as #[fundamental], which should help a lot with the coherence restrictions. The details on this are in the rebalancing coherence RFC.

[sgrif]

@cramertj Having a trait be #[fundamental] does not allow T: FundamentalTrait to be considered disjoint from T: SomeOtherTrait, which is important for the type of code variadics would be meant to allow. It also is impossible to implement for (), which means that ...Tail would not guarantee that Tail: Tuple

[eddyb]

@sgrif The trait can be implemented for all tuples (as it needs no associated items with the sugary syntax).
Also, how is your coherence example different for hlists at all?

[sgrif]

Implementing for (A, ...B) and () is not different from a coherence point of view. The point I was tryign to make by bringing up the trait is that having T: Tuple<Head=Whatever, Tail=Whatever> and () is quite different.

[eddyb]

T: Tuple<Head=Whatever, Tail=Whatever>

I've tried to avoid that syntax, it's the Split trait in some of my examples, that is not implemented for ().
Whereas the Tuple trait would be implemented for all tuples, including ().

Is the worry that an impl for (A, ...B) plus an impl for () doesn't count as an impl for all tuples?

[sgrif]

I've tried to avoid that syntax,

I wasn't referring to any specific example. Mostly trying to coordinate things across all examples.

Is the worry that an impl for (A, ...B) plus an impl for () doesn't count as an impl for all tuples?

No, writing those two impls should be equivalent to the hlist idea (assuming that impl<A, ...B> is reasonably similar to impl<Head, Tail> from the compiler's point of view).

I guess my main hesitation to tuples for this is that (A, B, C, D) being equivalent to (A, ...(B, ...(C, ...(D, ...())))) feels really off to me. My gut tells me there's a reason that the overlap will cause problems that I can't put my finger on, but it might just be me wanting to separate concerns. Having a specific type for the task just feels "cleaner" to me, but I admit that's a weak argument at best.

[Ixrec]

The last time variadics based on tuples was proposed, there was an enormous and iirc inconclusive tangent about tuple memory layout guarantees because in general an (a, b, c) value is not guaranteed to contain a value of type (b, c). This confused me at the time because I would've expected that the "parameter pack" (to borrow C++'s term) of a variadic generic function call is one of those things that exists only during monomorphization and vanishes entirely at codgen time/runtime, regardless of whether it's implemented as a tuple or something else.

Are we now assuming that both the tuple and the hlist options would be types that "disappear during monomorphization", so we don't need to worry about their layouts?

[eddyb]

I guess my main hesitation to tuples for this is that (A, B, C, D) being equivalent to (A, ...(B, ...(C, ...(D, ...()))))

To clarify, it's the other way around. Other examples are (A, ...(B, C), D) and (...(A, B, C), D).
It's technically possible to allow more than one repeat, but:

1. it wouldn't work in an impl's Self or trait parameters (i.e. "impl header")
2. it would be more expensive to represent in the compiler

[eddyb]

@Ixrec The problem was that people wanted to borrow tuple tails, i.e. get a &T out of (H, ...T).
I believe that to be unnecessary and we could simply have an intrinsic which transforms &(A, B, C) into (&A, &B, &C) to always "iterate" by-value.

[sgrif]

Are we now assuming that both the tuple and the hlist options would be types that "disappear during monomorphization", so we don't need to worry about their layouts?

No, this is part of why a separate type (which does have an explicit memory layout) comes to mind.

To clarify, it's the other way around. Other examples include (A, ...(B, C), D) and (...(A, B, C), D).

Do you think that there's value in simply disallowing repeats other than in the last position, at least initially? It drastically simplifies the feature to focus.

[sgrif]

I believe that to be an unnecessary problem and we could simply have an intrinsic to transform &(A, B, C) into (&A, &B, &C) to always "iterate" by-value.

Heh. Ironically I've wanted it the other way around in Diesel a few times.

[eddyb]

It drastically simplifies the feature to focus.

This isn't true IME, it only serves to make expanding the feature later tedious by having implicit limitations all over the compiler. I am even worried about a single repeat but less so because it's required sometimes.

[sgrif]

Even beginning to consider the cases that impl<...A, B> SomeTrait for (...A, B) adds compared to just last position variadics is huge from where I'm standing. I'm just going to close this. This doesn't feel like we're making any productive progress, and if a proposal for this has to consider variadics in any position to even be considered, I don't think I'm qualified to handle that.

[Ixrec]

@Ixrec The problem was that people wanted to borrow tuple tails, i.e. get a &T out of (H, ...T).
I believe that to be unnecessary and we could simply have an intrinsic which transforms &(A, B, C) into (&A, &B, &C) to always "iterate" by-value.

That explains a lot, thanks. I agree this sounds totally unnecessary and the Tuple::AsRef suggestion is a much better idea.

[eddyb]

@sgrif No worries! I hope we can continue the discussion when @cramertj opens a PR.

FWIW I think the repeat formulation is easiest to make sense in terms of appending tuples, as opposed to adding a single element at one end or another, i.e. more Prolog than LISP, like the rest of our type system.