Fixed #29229 -- Fixed column mismatch crash when combining two annotated values_list() querysets with union(), difference(), or intersection().

[AmrAnwar]

https://code.djangoproject.com/ticket/29229

[charettes]

Hello @AmrAnwar, could you add a test for it in tests/queries/test_qs_combinators.py as well?

[AmrAnwar]

@charettes ok course, I didn't see that it need a test from the site here: https://code.djangoproject.com/ticket/29229
But I'll try now

[sauravjaiswalsj]

It is fixed i guess so.

[AmrAnwar]

@charettes hope you check it now

[timgraham]

Can the test check the results of the query rather than internal implementation details like .query.combined_queries[1].annotations?

Incidentally, is there a way to write your query without QuerySet.extra()? That's likely to be deprecated in the future once we can obsolete it.

[timgraham]

Relying on pk might be problematic since we shouldn't assume the values that the database might assign.

[felixxm]

Agreed. Maybe:

def test_union_with_two_annotated_values_list(self):
    qs1 = Number.objects.filter(num=1).annotate(
        count=Value(0, IntegerField()),
    ).values_list('num', 'count')
    qs2 = Number.objects.filter(num=2).values('pk').annotate(
        count=F('num'),
        num=Value(1, IntegerField()),
    ).values_list('num', 'count')
    self.assertCountEqual(qs1.union(qs2), [(1, 0), (2, 1)])

[felixxm]

We can fix this condition without nested parentheses, i.e.

if not compiler.query.values_select and not compiler.query.annotations and self.query.values_select:

[AmrAnwar]

I was going to push the code in the test function but your code is better

        qs1 = Number.objects.filter(num=1).annotate(
            count=Value(0, IntegerField())
        ).values('count', num2=F('num')).order_by()
        qs2 = Number.objects.filter(num=0).values('num').annotate(
            num2=F("num")).annotate(
            count=Value(0, IntegerField()),
        ).values_list('count', "num2")
        qs1_union_qs2 = qs2.union(qs1).order_by('num2')
        self.assertEqual(list(qs1_union_qs2), [(0, 0), (1, 0)])

[felixxm]

Agreed. Maybe:

def test_union_with_two_annotated_values_list(self):
    qs1 = Number.objects.filter(num=1).annotate(
        count=Value(0, IntegerField()),
    ).values_list('num', 'count')
    qs2 = Number.objects.filter(num=2).values('pk').annotate(
        count=F('num'),
        num=Value(1, IntegerField()),
    ).values_list('num', 'count')
    self.assertCountEqual(qs1.union(qs2), [(1, 0), (2, 1)])

[timgraham]

A couple builds failed, is the test non-deterministic?

[felixxm]

It should be deterministic, because the query is really simple:

SELECT "queries_number"."num", 0 AS "count"
FROM "queries_number"
WHERE "queries_number"."num" = 1
UNION 
SELECT "queries_number"."num" AS "count", 1 AS "num"
FROM "queries_number"
WHERE "queries_number"."num" = 2

The test failed only python 3.5, so maybe it revealed an unrelated bug 🤔. I will take a closer look at this.

[felixxm]

@timgraham Splitting annotations in the qs2 fixed tests failure. QuerySet.annotate() doesn't preserve ordering of kwargs in python 3.5.