add select_by #2367

clouds56 · 2020-04-18T05:25:52Z

fixes #2037

diesel/src/query_builder/select_by_statement/dsl_impls.rs

Mingun · 2020-04-18T06:31:44Z

diesel/src/deserialize.rs

+/// #     Ok(())
+/// # }
+/// ```
+pub trait TableQueryable {


I think, the best name will be SelectResult or even Selectable, because as you noted, implementors of this trait not necessary represents a table

I also think TableQueryable is not a good name.
I think Selectable is better, would there be any conflict (or conflict in the future)?

I agree that this should have a better name, Selectable is not bad, but I'm not 100% sold to it yet.

Changed to QueryableByColumn.
If there's some better name, welcome to propose.

I don't like that name because it's completely obscure. What do ByColumn mean? Looks as if usual Queryable doesn't query columns.

Maybe AutoQueryable? Auto means, that struct fields automatically maps to table columns. Although there may be other, unwanted connotations. Selectable in this sense a much better. It is similar to Queryable, but different.

One other variant is Mapped, that means that struct is mapped to some table or view.

Or even just TableOrView.

I do not like QueryableByColumn either, Selectable sounds better for me.

That said: This is a discussion that does not really matter till the actual feature implementation is finished.

It is not "Auto", the column name could be specified by #[column_name = "id"] while Mapped and TableOrView seems a little unrelated.
I think Queryable do not query columns, they query index. Queryable query index, QueryableByName query names, and QueryableByColumn query columns. (but we would use term "SelectExpression" instead of "Column", so it seems Selectable more close to SelectExpression)
While I worry about Selectable would be confused with SelectableExpression. And it might also have more opportunity to conflict with other top level ident.

Anyway, I'd like to change the trait name and change Columns to SelectExpression at the last second, I don't have a refactor tools and it is much easier to have QueryableByColumn to be sed than Selectable :)

diesel/src/query_builder/select_by_statement/mod.rs

weiznich

Thanks for working on this 👍

I've done a quick review of the change and added notes as a went through the code. Please do not consider this review to be fully complete in the sense that I will probably add comments in other places later as well.
Beside of that I also would like to see the following things added:

A bunch of test cases in diesel_tests, basically checking that all those QueryDsl methods are implemented correctly
A changelog entry
Improved/fixed documentation.

diesel/src/deserialize.rs

weiznich · 2020-04-18T21:06:19Z

diesel/src/deserialize.rs

+/// #     Ok(())
+/// # }
+/// ```
+pub trait TableQueryable {


I agree that this should have a better name, Selectable is not bad, but I'm not 100% sold to it yet.

diesel/src/deserialize.rs

diesel/src/query_builder/select_by_statement/mod.rs

diesel/src/query_dsl/mod.rs

diesel_derives/src/queryable_by_name.rs

clouds56 · 2020-04-19T05:39:30Z

Is there any suggestion for the trait/derive name instead of TableQueryable?
@Mingun suggests Selectable and SelectResult
I just comes around mind QueryableWith (and maybe we should use select_with), but it seems in rust with implies a closure RefCell::replace_with, or QueryableByStruct?

weiznich · 2020-08-07T22:18:57Z

With #2182 merged now we can avoid much of the code duplication here. The new basic idea is as following:

Selectable remains the same
We introduce a new select clause type, based on Selectable and implement the required traits. We are using a newly introduced type as Expression::SqlType. Somehow this type needs to reference the Selectable type. Let's name it SelectBy<Q>
That will require that we have some wild card impl FromSqlRow<SelectBy<Q>, DB> for Q where Q: Selectable + FromSqlRow<…>

This should reduce the complete code to something like

 pub trait Selectable {
     type Selection: crate::expression::Expression;
 }

 pub struct SelectBy<ST>(std::marker::PhantomData<ST>);

 impl<ST> TypedExpressionType for SelectBy<ST> {}

 impl<ST, DB> QueryMetadata<SelectBy<ST>> for DB
 where
     DB: Backend,
 {
     fn row_metadata(_: &Self::MetadataLookup) -> Option<Vec<Self::TypeMetadata>> {
         None
     }
 }

 impl<Q, DB, ST> FromSqlRow<SelectBy<Q>, DB> for Q
 where
     ST: crate::expression::Expression,
     DB: Backend,
     Q: Selectable<Selection = ST>,
     Q: FromSqlRow<SqlTypeOf<ST>, DB>,
 {
     fn build_from_row<'a>(row: &impl Row<'a, DB>) -> Result<Self> {
         <Self as FromSqlRow<SqlTypeOf<ST>, DB>>::build_from_row(row)
     }
 }

 use crate::expression::SelectableExpression;
use crate::query_builder::select_clause::*;

 impl<QS, S> SelectClauseExpression<QS> for SelectBy<S>
 where
     S: Selectable,
     S::Selection: SelectableExpression<QS>,
 {
     type Selection = S::Selection;
     type SelectClauseSqlType = SqlTypeOf<S::Selection>;
 }

Some code to expose a API to actually use this.

This should greatly simplify the implementation and remove the blockers from my list above.

clouds56 · 2020-08-08T10:56:49Z

Thanks for response and not forget about this PR.

What about GroupBy? I think the only blocker was that you said you'd like to rethink GroupBy.
I know we are here to make diesel better, but after several months, if someone has to read the latest inner API again, rewrite all the code and fix all comments of review, would he feel tired and hopeless especially no idea about when could it be merged?
From my point of view, code at last commit is ready for merge, the things about GroupBy could be postponed to future PR. You said you have not much time on this PR and doing a refactoring of code without any discussion (I think it would be better if I'm informed before writing the PR and take it into consideration when design select_by).
My feeling is that the feature would never be accepted and I have to do refactoring again and again to chase after master or any other new ideas. And I'm already lost all the context about how diesel internally works.

For my point of few the following points are should be checked again: (Most of them likely need some experimentation)

Is select_by:: really a good API for this or should we try to do something like .select(Type::some_static_method()) instead?

How does this interact with .group_by clauses? I think that should be fine, but someone should try to write at least a compile_fail tests for some trivial cases.

Can we avoid some of the code duplication by reusing SelectStatement more directly? For example by having another generic parameter?

These are still not answered.

clouds56 · 2020-08-08T10:59:00Z

I'm not saying I'm giving this PR up. But I'd like to do nothing until things went clear.

weiznich · 2020-08-08T11:31:33Z

What about GroupBy? I think the only blocker was that you said you'd like to rethink GroupBy.
I know we are here to make diesel better, but after several months, if someone has to read the latest inner API again, rewrite all the code and fix all comments of review, would he feel tired and hopeless especially no idea about when could it be merged?
From my point of view, code at last commit is ready for merge, the things about GroupBy could be postponed to future PR. You said you have not much time on this PR and doing a refactoring of code without any discussion (I think it would be better if I'm informed before writing the PR and take it into consideration when design select_by).
My feeling is that the feature would never be accepted and I have to do refactoring again and again to chase after master or any other new ideas. And I'm already lost all the context about how diesel internally works.

I can fully understand that it is really frustrating that this PR is not merged yet, but as this is a really large change I do want to make sure that we get it somewhat right before merging it. In the end it's up to us diesel maintainers to maintain and live with all code we have merged. So better make sure those code is fine, otherwise we run into large trouble pretty soon (and probably will lose any interest in continue maintaining diesel). And yes working on OpenSource can sometimes be really frustrating. (Not only as contributor, but also as maintainer. Just check the issue tracker about what some comments request from us, because someone cannot use diesel for their business case...)

Additionally this feature is not something I personally need, therefore it's something where I accept contributions but will not spend much time on my own to figure out remaining open points. In the end my day has only a fixed amount of time and I cannot do everything. (And I'm not willing to spend all my free time on maintaining a open source project for other people. It's more like I fix the things I personally care about and spend some time to help/guide other people to fix/implement their features.) Therefore the list of open questions, as nobody has spend time to work on those it seems like there is not that much interest in this feature.

As for the concrete open points:

Is select_by:: really a good API for this or should we try to do something like .select(Type::some_static_method()) instead?

This remains a open question. In the end it just needs someone who sits down tries out a few possibilities and provide a summary write up what worked well and what not. In the end this is a cosmetic small scale change, which would not change much of the underlying implementation. Just trying to get a nice API.

How does this interact with .group_by clauses? I think that should be fine, but someone should try to write at least a compile_fail tests for some trivial cases.

As mentioned here: I would like to see and compile_fail test for some trivial case. Those needs to be written.

Can we avoid some of the code duplication by reusing SelectStatement more directly? For example by having another generic parameter?

#2182 Allows to remove the code duplication. As mentioned here we can now implement this feature by "just" having a custom select clause instead. This should allow to remove basically all of diesel/src/query_builder/select_statement/select_by/.

This was not something that was clear in the beginning of this PR, but some of my motivation while implementing #2182 was to simplify approaches like this one and remove the code duplication here.

I'm not saying I'm giving this PR up. But I'd like to do nothing until things went clear.

To write that again: I'm willing to help here with advice + code review as far as I find some spare time, but I do not have the time capacity or motivation to work on this feature by myself. If someone is interested in getting this merged they need to spend time and address the points above.

clouds56 · 2020-09-20T02:54:05Z

I think we doesn't need impl<QS, S> SelectClauseExpression<QS> for SelectBy<S>.
I'm not sure what QueryMetadata is but provide it as is.
updated and plz review.

weiznich

Thanks for picking this up again. It looks now nearly finished. I left a bunch of small stylistic comments, otherwise this is fine from my side.
What I would like to see are 2 or 3 simple compile fail tests showing that this feature correctly rejects malformed select clauses (something like table appears not in from clause + maybe mixed aggregates).

diesel/src/deserialize.rs

diesel/src/expression/mod.rs

diesel/src/expression/nullable.rs

diesel/src/expression/select_by.rs

diesel/src/query_builder/select_statement/dsl_impls.rs

weiznich · 2020-09-22T14:01:48Z

diesel/src/query_builder/select_statement/dsl_impls.rs

+    SE: Expression,
+    Selection: Selectable<Expression = SE>,
+    Self: SelectDsl<SE>,
+    <Self as SelectDsl<SE>>::Output: SelectByQuery<Expression = SE>,


I'm not sure why the SelectByQuery bound here is required. Instead of that bound there should be some bounds along the lines of SelectBy<Selection>: SelectableExpression<F> to verify that the select clause is fine for the given from clause.

It's just what SelectDsl write.

diesel_derives/tests/selectable.rs

Co-authored-by: Georg Semmler <Georg_semmler_05@web.de>

clouds56 · 2020-09-23T07:00:40Z

No, it doesn't work.
I realize that impl<Q, DB, E> FromSqlRow<SelectBy<Q>, DB> for Q has no effect, since when we are doing .load::<Id>() but not .load::<SelectBy<Id>>(). In this example, we have

#[derive(Selectable)]
#[table_name = "users"]
struct Id {
  id: i32
}

Now the problem is, there's no way to make load::<U>() only accept Id.

use Expression other than SqlType in trait AsQuery and struct AnyExpression<SqlType>() for original .select() dsl.

impl<T: Query> AsQuery for T {
    type SqlType = <Self as Query>::SqlType;
    type Query = Self;

    fn as_query(self) -> Self::Query {
        self
    }
}

another api named .load_by
not implement Query for SelectStatement<SelectClause<SelectBy<T>>>
- would need carefully treatment and may introduce new select clause like SelectByClause
- might make inner join of SelectBy statement much more complex, since it might relies on AsQuery

weiznich · 2020-09-23T15:40:30Z

@clouds56 I've put together a minimal version that works here. That obviously needs some cleanup + refactoring + finishing the implementation, but I think the basic idea should be clear.

Edit: Technically the changes to Connection + LoadDsl (introduction of CompatibleType) is not required, but this enables using .load() without having to specify the type for U again there.

clouds56 · 2020-11-03T04:06:46Z

That's an interesting point to introduce constraints with CompatibleType, while we cannot handle things like (ST, Untyped) as required in example of diesel_dynamic_schema.
I tried to impl<A: CompatibleType, CompatibleType> CompatibleType for (A, B) via impl_tuples, but things doesn't goes well, any suggestions?
Still investigating.

weiznich · 2020-11-03T16:45:35Z

@clouds56 Thanks for updating the PR 👍 I'm quite occupied with work currently, but will try to find some time for looking into that.

to support mixed typed/untyped queries

weiznich · 2020-12-04T11:51:18Z

@clouds56 I've found some time to look at this. I think it should be fixed by the impl added with my last commit.

diesel/src/query_dsl/mod.rs

+    /// let all_users = users.load::<(i32, String)>(&connection)?;
+    /// assert_eq!(vec![(1, String::from("Sean")), (2, String::from("Tess"))], all_users);
+    ///
+    /// let all_names = users.select_by::<UserName>().load::<UserName>(&connection)?;


clouds56 added 2 commits April 18, 2020 13:08

add select_by_statement

1ee97e2

add LoadQuery

c1f8ceb