Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Query: Second level expand not returning correct data #6366

Closed
vukasinspasic opened this issue Aug 19, 2016 · 6 comments
Closed

Query: Second level expand not returning correct data #6366

vukasinspasic opened this issue Aug 19, 2016 · 6 comments
Assignees
@maumar
Labels
closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. type-bug
Milestone
1.0.1

Comments

@vukasinspasic
Copy link

vukasinspasic commented Aug 19, 2016 •
edited by divega

Hello. I noticed bugs in library.

Steps to reproduce

When I try to make query on Posts DbSet with 2 expands of 2. level:

var query = dbContext.Posts.AsQueryable();
query = query
    .Include(x => x.User)
        .ThenInclude(x => x.Roles)
    .Include(x => x.VoteDefinition)
        .ThenInclude(x => x.PossibleAnswers)

Database model (some properties omitted for clarity):

    public class Post
    {
        public int Id { get; set; }
        public int UserId { get; set; }
        public User User { get; set; }
        public int? VoteDefinitionId { get; set; }
        public VoteDefinition VoteDefinition { get; set; }
    }
    public class User
    {
        public int Id { get; set; }
        public IList<Post> Posts { get; set; }
        public IList<Role> Roles { get; set; }
    }
    public class Role
    {
        public int Id { get; set; }
        public int UserId { get; set; }
    }
    public class VoteDefinition
    {
        public int Id { get; set; }
        [ForeignKey("Post")]
        public int PostId { get; set; }
        public Post Post { get; set; }
        public IList<VoteDefinitionPossibleAnswer> PossibleAnswers { get; set; }
    }
    public class VoteDefinitionPossibleAnswer
    {
        public int Id { get; set; }
        public int VoteDefinitionId { get; set; }
        public VoteDefinition VoteDefinition { get; set; }
    }

The issue

This generated bad SQL query resulting in User.Roles not expanded.

Further technical details

As you can see in generated SQL queries, in 2. SQL query, there is line
LEFT JOIN "VoteDefinitions" AS "v" ON "v"."PostId" = "f"."Id"
which should be only in 3. SQL query

SQL generated by this query:

info: Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommandBuilderFactory[1]
Executed DbCommand (55ms) [Parameters=[@__ToLower_0='?', @__p_2='?', @__p_1='?'], CommandType='Text', CommandTimeout='30']

SELECT "f"."Id", "f"."Cancelled", "f"."CategoryId", "f"."CreatedBy", "f"."CreatedByRole", "f"."CreatedOn", "f"."Draft", "f"."ModifiedBy", "f"."ModifiedByRole", "f"."ModifiedOn", "f"."RowVersion", "f"."Text", "f"."Title", "f"."Type", "f"."UserId", "f"."VoteDefinitionId", "v"."Id", "v"."Cancelled", "v"."CreatedBy", "v"."CreatedByRole", "v"."CreatedOn", "v"."ModifiedBy", "v"."ModifiedByRole", "v"."ModifiedOn", "v"."PostId", "v"."Question", "v"."RowVersion", "v"."Type", "u"."Id", "u"."Active", "u"."BirthDate", "u"."Cancelled", "u"."CreatedBy", "u"."CreatedByRole", "u"."CreatedOn", "u"."DocumentNumber", "u"."Email", "u"."FirstName", "u"."FullName", "u"."LastName", "u"."ModifiedBy", "u"."ModifiedByRole", "u"."ModifiedOn", "u"."Password", "u"."RowVersion", "p"."Id", "p"."Cancelled", "p"."CreatedBy", "p"."CreatedByRole", "p"."CreatedOn", "p"."ModifiedBy", "p"."ModifiedByRole", "p"."ModifiedOn", "p"."Name", "p"."RowVersion"
FROM "Posts" AS "f"
LEFT JOIN "VoteDefinitions" AS "v" ON "v"."PostId" = "f"."Id"
INNER JOIN "Users" AS "u" ON "f"."UserId" = "u"."Id"
LEFT JOIN "PostCategories" AS "p" ON "f"."CategoryId" = "p"."Id"
WHERE (("f"."Cancelled" = FALSE) AND LOWER("f"."Title") LIKE ((('%' || @__ToLower_0)) || '%')) AND ("f"."Draft" = FALSE)
ORDER BY "f"."Id", "v"."Id", "u"."Id"
LIMIT @__p_2 OFFSET @__p_1
info: Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommandBuilderFactory[1]
Executed DbCommand (34ms) [Parameters=[@__ToLower_0='?', @__p_2='?', @__p_1='?'], CommandType='Text', CommandTimeout='30']
SELECT "u0"."Id", "u0"."Cancelled", "u0"."CreatedBy", "u0"."CreatedByRole", "u0"."CreatedOn", "u0"."ModifiedBy", "u0"."ModifiedByRole", "u0"."ModifiedOn", "u0"."RoleName", "u0"."RowVersion", "u0"."UserId"
FROM "UserRoles" AS "u0"
INNER JOIN (
SELECT DISTINCT "t0".*
FROM (
SELECT "f"."Id", "v"."Id" AS "Id0", "u"."Id" AS "Id1"
FROM "Posts" AS "f"
**LEFT JOIN "VoteDefinitions" AS "v" ON "v"."PostId" = "f"."Id"**
INNER JOIN "Users" AS "u" ON "f"."UserId" = "u"."Id"
WHERE (("f"."Cancelled" = FALSE) AND LOWER("f"."Title") LIKE ((('%' || @__ToLower_0)) || '%')) AND ("f"."Draft" = FALSE)
ORDER BY "f"."Id", "v"."Id", "u"."Id"
LIMIT @__p_2 OFFSET @__p_1
) AS "t0"
) AS "u1" ON "u0"."UserId" = "u1"."Id"
ORDER BY "u1"."Id", "u1"."Id0", "u1"."Id1"
info: Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommandBuilderFactory[1]
Executed DbCommand (47ms) [Parameters=[@__ToLower_0='?', @__p_2='?', @__p_1='?'], CommandType='Text', CommandTimeout='30']
SELECT "v0"."Id", "v0"."Answer", "v0"."Cancelled", "v0"."CreatedBy", "v0"."CreatedByRole", "v0"."CreatedOn", "v0"."ModifiedBy", "v0"."ModifiedByRole", "v0"."ModifiedOn", "v0"."RowVersion", "v0"."VoteDefinitionId"
FROM "VoteDefinitionPossibleAnswers" AS "v0"
INNER JOIN (
SELECT DISTINCT "t".*
FROM (
SELECT "f"."Id", "v"."Id" AS "Id0"
FROM "Posts" AS "f"
LEFT JOIN "VoteDefinitions" AS "v" ON "v"."PostId" = "f"."Id"
WHERE (("f"."Cancelled" = FALSE) AND LOWER("f"."Title") LIKE ((('%' || @__ToLower_0)) || '%')) AND ("f"."Draft" = FALSE)
ORDER BY "f"."Id", "v"."Id"
LIMIT @__p_2 OFFSET @__p_1
) AS "t"
) AS "v1" ON "v0"."VoteDefinitionId" = "v1"."Id"
ORDER BY "v1"."Id", "v1"."Id0"

EF Core version: 1.0.0-preview2-final
Operating system: Windows 10 and also Linux Ubuntu
Visual Studio version: VS 2015
@roji roji mentioned this issue Aug 19, 2016
Closed
@divega divega added this to the 1.1.0 milestone Aug 19, 2016
@vukasinspasic
Copy link
Author

I also noticed that Roles are not expanded if ordering is by Id (primary key) in query:

var query = dbContext.Posts.AsQueryable();
query = query.Include(x => x.User).ThenInclude(x => x.Roles).OrderBy(x => x.Id);

but for example when ordering with another property, ex. OrderBy(x => x.CreatedOn), Roles are expanded correctly

@maumar
Copy link
Contributor

maumar commented Aug 20, 2016

The problem is not necessarily about the additional JOIN to VoteDefinitions in the second query, but the fact that we are choosing incorrect column to join on between Users and Roles:

SELECT [r].[Id], [r].[Name], [r].[UserId]
FROM [Roles] AS [r]
INNER JOIN (
    SELECT DISTINCT [t0].*
    FROM (
        SELECT [v].[Id], [u].[Id] AS [Id0]
        FROM [Posts] AS [x]
        LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
        INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
        ORDER BY [v].[Id], [u].[Id]
        OFFSET @__p_0 ROWS
    ) AS [t0]
) AS [u0] ON [r].[UserId] = [u0].[Id] -- this should be u0.Id0
ORDER BY [u0].[Id], [u0].[Id0]

This only seems to be happening for queries with Skip. Without it we produce the following, which works as expected:

SELECT [r].[Id], [r].[Name], [r].[UserId]
FROM [Roles] AS [r]
INNER JOIN (
    SELECT DISTINCT [v].[Id], [u].[Id] AS [Id0]
    FROM [Posts] AS [x]
    LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
    INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
) AS [u0] ON [r].[UserId] = [u0].[Id0]
ORDER BY [u0].[Id], [u0].[Id0]

@vukasinspasic
Copy link
Author

vukasinspasic commented Aug 21, 2016 •
edited

Yes, this situation is happening if Skip is used (even if Skip(0)). But in case OrderBy is by Id (primary key) then both Roles and PossibleAnswers are not expanded. In case OrderBy(x => X.CreatedOn) which is not a key property(column) then Roles are not expanded but PossibleAnswers are.

@maumar maumar removed this from the 1.1.0 milestone Aug 22, 2016
@maumar
Copy link
Contributor

maumar commented Aug 22, 2016

sending back to triage, as it's potentially a candidate for 1.0.1 - in some circumstances we could return invalid data i.e. navigations linked to wrong parent entities

@maumar maumar mentioned this issue Aug 22, 2016
Closed
maumar added a commit that referenced this issue Aug 22, 2016
@maumar
Fix to #6366 - 2 level expand not working correctly
2e8a387 
Problem happens for complex include scenarios like so:

ctx
    .Posts.Include(x => x.User).ThenInclude(x => x.Roles)
    .Include(x => x.VoteDefinition).ThenInclude(x => x.PossibleAnswers)
    .Skip(1)

We generate 3 queries for this case:

- first one gets Posts and 1:1 navigations (User and Vote Definition)

SELECT [x].[Id], [x].[Name], [x].[UserId], [x].[VoteDefinitionId], [v].[Id], [v].[PostId], [u].[Id], [u].[Name]
FROM [Posts] AS [x]
LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
ORDER BY [v].[Id], [u].[Id]
OFFSET @__p_0 ROWS',N'@__p_0 int

- second one copies the first one and joins it with Roles table to return associated Roles

SELECT [r].[Id], [r].[Name], [r].[UserId]
FROM [Roles] AS [r]
INNER JOIN (
    SELECT DISTINCT [t0].*
    FROM (
        SELECT [v].[Id], [u].[Id] AS [Id0]
        FROM [Posts] AS [x]
        LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
        INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
        ORDER BY [v].[Id], [u].[Id]
        OFFSET @__p_0 ROWS
    ) AS [t0]
) AS [u0] ON [r].[UserId] = [u0].[Id]
ORDER BY [u0].[Id], [u0].[Id0]',N'@__p_0 int

- third one is similar to the second one.

Problem is that in second query we are joining based on a wrong column (VoteDefinition's Id rather than User's Id). This is because we lose track of the information about which column is associated with which property.
Due to Skip() call in the original LINQ query, we generate SELECT * around the relevant column information. Later, when looking for which column to join, we peek into the SELECT expression and don't see any information there so we assume that we should join based on a property with the name we need. This works fine if there are no duplicate columns with the same name projected out (and uniquefied) from other joined tables.

Fix is, in case of a SELECT * query, to look for the relevant column information in the TablesExpressions/SelectExpressions that the SELECT * is projecting from.
maumar added a commit that referenced this issue Aug 23, 2016
@maumar
Fix to #6366 - 2 level expand not working correctly
f4f5b05 
Problem happens for complex include scenarios like so:

ctx
    .Posts.Include(x => x.User).ThenInclude(x => x.Roles)
    .Include(x => x.VoteDefinition).ThenInclude(x => x.PossibleAnswers)
    .Skip(1)

We generate 3 queries for this case:

- first one gets Posts and 1:1 navigations (User and Vote Definition)

SELECT [x].[Id], [x].[Name], [x].[UserId], [x].[VoteDefinitionId], [v].[Id], [v].[PostId], [u].[Id], [u].[Name]
FROM [Posts] AS [x]
LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
ORDER BY [v].[Id], [u].[Id]
OFFSET @__p_0 ROWS',N'@__p_0 int

- second one copies the first one and joins it with Roles table to return associated Roles

SELECT [r].[Id], [r].[Name], [r].[UserId]
FROM [Roles] AS [r]
INNER JOIN (
    SELECT DISTINCT [t0].*
    FROM (
        SELECT [v].[Id], [u].[Id] AS [Id0]
        FROM [Posts] AS [x]
        LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
        INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
        ORDER BY [v].[Id], [u].[Id]
        OFFSET @__p_0 ROWS
    ) AS [t0]
) AS [u0] ON [r].[UserId] = [u0].[Id]
ORDER BY [u0].[Id], [u0].[Id0]',N'@__p_0 int

- third one is similar to the second one.

Problem is that in second query we are joining based on a wrong column (VoteDefinition's Id rather than User's Id). This is because we lose track of the information about which column is associated with which property.
Due to Skip() call in the original LINQ query, we generate SELECT * around the relevant column information. Later, when looking for which column to join, we peek into the SELECT expression and don't see any information there so we assume that we should join based on a property with the name we need. This works fine if there are no duplicate columns with the same name projected out (and uniquefied) from other joined tables.

Fix is, in case of a SELECT * query, to look for the relevant column information in the TablesExpressions/SelectExpressions that the SELECT * is projecting from.
@divega divega added this to the 1.0.1 milestone Aug 23, 2016
@divega divega changed the title 2 level expand not working correctly Second level expand not working correctly Aug 23, 2016
@Eilon
Copy link
Member

Eilon commented Aug 26, 2016

Approved for 1.0.1.

maumar added a commit that referenced this issue Aug 27, 2016
@maumar
Fix to #6366 - 2 level expand not working correctly
bd45e7d 
Problem happens for complex include scenarios like so:

ctx
    .Posts.Include(x => x.User).ThenInclude(x => x.Roles)
    .Include(x => x.VoteDefinition).ThenInclude(x => x.PossibleAnswers)
    .Skip(1)

We generate 3 queries for this case:

first one gets Posts and 1:1 navigations (User and Vote Definition)
SELECT [x].[Id], [x].[Name], [x].[UserId], [x].[VoteDefinitionId], [v].[Id], [v].[PostId], [u].[Id], [u].[Name]
FROM [Posts] AS [x]
LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
ORDER BY [v].[Id], [u].[Id]
OFFSET @__p_0 ROWS',N'@__p_0 int
second one copies the first one and joins it with Roles table to return associated Roles
SELECT [r].[Id], [r].[Name], [r].[UserId]
FROM [Roles] AS [r]
INNER JOIN (
    SELECT DISTINCT [t0].*
    FROM (
        SELECT [v].[Id], [u].[Id] AS [Id0]
        FROM [Posts] AS [x]
        LEFT JOIN [VoteDefinitions] AS [v] ON [v].[PostId] = [x].[Id]
        INNER JOIN [Users] AS [u] ON [x].[UserId] = [u].[Id]
        ORDER BY [v].[Id], [u].[Id]
        OFFSET @__p_0 ROWS
    ) AS [t0]
) AS [u0] ON [r].[UserId] = [u0].[Id]
ORDER BY [u0].[Id], [u0].[Id0]',N'@__p_0 int

third one is similar to the second one.

Problem is that in second query we are joining based on a wrong column (VoteDefinition's Id rather than User's Id). This is because we lose track of the information about which column is associated with which property.
Due to Skip() call in the original LINQ query, we generate SELECT * around the relevant column information. Later, when looking for which column to join, we peek into the SELECT expression and don't see any information there so we assume that we should join based on a property with the name we need. This works fine if there are no duplicate columns with the same name projected out (and uniquefied) from other joined tables.

Fix is, in case of a SELECT * query, to look for the relevant column information in the TablesExpressions/SelectExpressions that the SELECT * is projecting from.
@maumar
Copy link
Contributor

maumar commented Aug 29, 2016

Fixed in bd45e7d (1.0.1) and 2dd1d0c (dev)

@maumar maumar closed this as completed Aug 29, 2016
@maumar maumar added the closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. label Aug 29, 2016
@rowanmiller rowanmiller changed the title Second level expand not working correctly Query: Second level expand not returning correct data Sep 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maumar maumar

Labels
closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. type-bug
Projects
None yet
Milestone
1.0.1
Development

No branches or pull requests
4 participants
@Eilon @divega @maumar @vukasinspasic