Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throw when using relational FromSqlRaw with Cosmos, or vice-versa #26502

Closed
JonPSmith opened this issue Nov 1, 2021 · 14 comments · Fixed by #26579
Closed

Throw when using relational FromSqlRaw with Cosmos, or vice-versa #26502

JonPSmith opened this issue Nov 1, 2021 · 14 comments · Fixed by #26579
Assignees
Labels
area-query breaking-change closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. customer-reported type-bug
Milestone

Comments

@JonPSmith
Copy link

In NET6 EF Core I get an compile error if I use the method FromSqlRaw \ FromSqlRawAsync when both of the following NuGet Packages are installed:

  • Microsoft.EntityFrameworkCore Version="6.0.0-rc.2.21480.5"
  • Microsoft.EntityFrameworkCore.Cosmos" Version="6.0.0-rc.2.21480.5"

The error is:

Code	CS0121	
The call is ambiguous between the following methods or properties: 'Microsoft.EntityFrameworkCore.RelationalQueryableExtensions.FromSqlRaw<TEntity>(Microsoft.EntityFrameworkCore.DbSet<TEntity>, string, params object[])' and 'Microsoft.EntityFrameworkCore.CosmosQueryableExtensions.FromSqlRaw<TEntity>(Microsoft.EntityFrameworkCore.DbSet<TEntity>, string, params object[])'

Example code

public void TestDemo()
{
    //SETUP
    var options = SqliteInMemory.CreateOptions<MyDbContext>();
    using var context = new MyDbContext(options);
    context.Database.EnsureCreated();

    //ATTEMPT
    var query = context.MyEntities.FromSqlRaw("SELECT * FROM MyEntities"); //THIS LINE HAS ERROR

    //... rest of code left out
}

Its easy to get around by adding the namespace before the FromSqlRaw, e.g.

var query = RelationalQueryableExtensions.FromSqlRaw(context.MyEntities, "SELECT * FROM MyEntities");

Include provider and version information

EF Core version: 6.0.0-rc.2.21480.5
Database provider: (various, Microsoft.EntityFrameworkCore.Sqlite)
Target framework: (e.g. .NET 6.0.rc2)
Operating system: Window
IDE: (e.g. Version 17.0.0 Preview 7.0)

@roji
Copy link
Member

roji commented Nov 1, 2021

Yeah, that's intentional - both Relational and Cosmos have their own versions of FromSqlRaw, and they don't recognize each other's version (so e.g. the relational FromSqlRaw wouldn't be recognized by Cosmos). Making them ambiguous as above forces the user to explicitly specify which one they want, which prevents confusion etc.

@JonPSmith
Copy link
Author

So are you saying that a relational database should use the RelationalQueryableExtensions version of FromSqlRaw and Cosmos version should use the CosmosQueryableExtensions version? If so I have another error as the following code fails, but if I use RelationalQueryableExtensions.FromSqlRaw it works.

var options = this.GetCosmosDbOptions<CosmosDbContext>();
using var context = new CosmosDbContext(options);
//... other code left out 

var list = await CosmosQueryableExtensions.FromSqlRaw(context.Books, 
    "SELECT * FROM c ORDER BY c.BookId").ToListAsync();

Stacktrace for the failed version

When using CosmosQueryableExtensions.FromSqlRaw I get this error.

    Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Microsoft.Azure.Cosmos.Query.Core.Monads.ExceptionWithStackTraceException: TryCatch resulted in an exception. ---> Microsoft.Azure.Cosmos.Query.Core.Monads.ExceptionWithStackTraceException: TryCatch resulted in an exception. ---> Microsoft.Azure.Cosmos.Query.Core.Exceptions.ExpectedQueryPartitionProviderException: {"errors":[{"severity":"Error","location":{"start":38,"end":55},"code":"SC2202","message":"'ORDER BY' is not supported in subqueries."}]}
 ---> System.Runtime.InteropServices.COMException (0x800A0B00): 0x800A0B00
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPartitionProvider.TryGetPartitionedQueryExecutionInfoInternal(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount)
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPartitionProvider.TryGetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount)
   at Microsoft.Azure.Cosmos.CosmosQueryClientCore.TryGetPartitionedQueryExecutionInfoAsync(SqlQuerySpec sqlQuerySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Threading.Tasks.UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessInnerTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessCompletedOuterTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.RunOrQueueCompletionAction(ITaskCompletionAction completionAction, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

   --- End of inner exception stack trace ---
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPartitionProvider.TryGetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount)
   at Microsoft.Azure.Cosmos.CosmosQueryClientCore.TryGetPartitionedQueryExecutionInfoAsync(SqlQuerySpec sqlQuerySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Threading.Tasks.UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessInnerTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessCompletedOuterTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.RunOrQueueCompletionAction(ITaskCompletionAction completionAction, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
);

  Stack Trace: 
QueryPartitionProvider.TryGetPartitionedQueryExecutionInfo(SqlQuerySpec querySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount)
CosmosQueryClientCore.TryGetPartitionedQueryExecutionInfoAsync(SqlQuerySpec sqlQuerySpec, PartitionKeyDefinition partitionKeyDefinition, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount, CancellationToken cancellationToken)
AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
AsyncStateMachineBox`1.MoveNext()
AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
Task.RunContinuations(Object continuationObject)
Task.FinishContinuations()
Task`1.TrySetResult(TResult result)
UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
UnwrapPromise`1.ProcessInnerTask(Task task)
UnwrapPromise`1.ProcessCompletedOuterTask(Task task)
UnwrapPromise`1.Invoke(Task completingTask)
Task.RunOrQueueCompletionAction(ITaskCompletionAction completionAction, Boolean allowInlining)
Task.RunContinuations(Object continuationObject)
Task.FinishContinuations()
Task.FinishStageThree()
Task.FinishStageTwo()
Task.FinishSlow(Boolean userDelegateExecute)
Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
Task.ExecuteFromThreadPool(Thread threadPoolThread)
ThreadPoolWorkQueue.Dispatch()
WorkerThread.WorkerThreadStart()
Thread.StartCallback()

The information is the same at the initial issue comment.

@roji
Copy link
Member

roji commented Nov 1, 2021

So are you saying that a relational database should use the RelationalQueryableExtensions version of FromSqlRaw and Cosmos version should use the CosmosQueryableExtensions version?

Yes.

If so I have another error as the following code fails, but if I use RelationalQueryableExtensions.FromSqlRaw it works.

Can you share your full model configuration (or ideally the project)?

@smitpatel
Copy link
Member

"SELECT * FROM c ORDER BY c.BookId"

Don't think that is valid SQL for cosmos unless the container name is "c". The error message also indicates that it was bad query sent to CosmosDb.

@JonPSmith
Copy link
Author

Thanks @smitpatel, I didn't look close enough into the error. I took out the 'ORDER BY c.BookIdand theSELECT * FROM c` query worked.

However an Cosmos DB query of SELECT * FROM c ORDER BY c.BookId is valid as that exact query works in the Azure Cosmos DB Emulator. So I used ToQueryString (great feature!) to look at produced SQL and the ORDER BY is in a subquery of the WHERE part which causes the error (see below).

    SELECT c
    FROM (
        SELECT * FROM c ORDER BY c.BookId
    ) c
    WHERE (c["Discriminator"] = "CosmosBook")

I then added the modelBuilder.Entity<CosmosBook>().HasNoDiscriminator() to the Cosmos configuration to remove the outer WHERE part, but it still failed because of FROM part.

    SELECT c
    FROM (
        SELECT * FROM c ORDER BY c.YearPublished
    ) c

One last thing: If I use RelationalQueryableExtensions.FromSqlRaw(context.Books, "SELECT * FROM c ORDER BY c.BookId") it runs successfully but the returned data hasn't been ordered. Perhaps the relational FromSqlRaw should throw an exception to say you shouldn't use the RelationalQueryableExtensions version of FromSqlRaw.

NOTE: You can find the unit test TestFromSqlRaw, with the RelationalQueryableExtensions.FromSqlRaw code commented out.

@smitpatel
Copy link
Member

QueryRootExpression queryRootExpression => CreateShapedQueryExpression(queryRootExpression.EntityType),

This line should do exact type match.

@ajcvickers
Copy link
Member

ajcvickers commented Nov 2, 2021

Note from triage: using the relational method should throw when used on Cosmos. However, not sure if this is safe to patch.

@ajcvickers ajcvickers changed the title ambiguous between the following methods: FromSqlRaw in RelationalQueryableExtensions and CosmosQueryableExtensions Throw when using relational FromSqlRaw with Cosmos, or vice-versa Nov 2, 2021
@smitpatel
Copy link
Member

This will be breaking change not just for this but anything which is custom query root and being used with diff provider. e.g. Using temporal AsOf operator in query doesn't throw currently if used on non-SqlServer provider.

@ajcvickers ajcvickers added this to the 7.0.0 milestone Nov 5, 2021
smitpatel added a commit that referenced this issue Nov 8, 2021
Query root design

Core exposes QueryRootExpression which is base class for any query root so we can access EntityType out of it. This is needed for nav expansion and any potential future use case.
Since any derived query root would derive from it, core assembly translates it only when type is exact match to QueryRootExpression.
Relational layer also processes QueryRootExpression when they are mapped to default SqlQuery, which also uses exact type match now.

Apart from QueryRootExpression, no other kind of query root which can appear in different providers should be derivable (else they need to use exact type match too).
This rule makes relational ones sealed class. In case any one needs to derive from it, they need to add additional processing anyway.

Provider specific derived query roots can be non-sealed. If anyone is deriving from it then they should be using their derived provider which process those nodes too and if the derived provider wasn't used and shipped provider is used then it is an error from user perspective. If derived query root is used on other provider (targeting diff database) then it will fail since even the base shipped query root is unknown.

Resolves #26502
@smitpatel smitpatel added the closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. label Nov 8, 2021
smitpatel added a commit that referenced this issue Nov 9, 2021
Query root design

Core exposes QueryRootExpression which is base class for any query root so we can access EntityType out of it. This is needed for nav expansion and any potential future use case.
Since any derived query root would derive from it, core assembly translates it only when type is exact match to QueryRootExpression.
Relational layer also processes QueryRootExpression when they are mapped to default SqlQuery, which also uses exact type match now.

Apart from QueryRootExpression, no other kind of query root which can appear in different providers should be derivable (else they need to use exact type match too).
This rule makes relational ones sealed class. In case any one needs to derive from it, they need to add additional processing anyway.

Provider specific derived query roots can be non-sealed. If anyone is deriving from it then they should be using their derived provider which process those nodes too and if the derived provider wasn't used and shipped provider is used then it is an error from user perspective. If derived query root is used on other provider (targeting diff database) then it will fail since even the base shipped query root is unknown.

Resolves #26502
smitpatel added a commit that referenced this issue Nov 9, 2021
Query root design

Core exposes QueryRootExpression which is base class for any query root so we can access EntityType out of it. This is needed for nav expansion and any potential future use case.
Since any derived query root would derive from it, core assembly translates it only when type is exact match to QueryRootExpression.
Relational layer also processes QueryRootExpression when they are mapped to default SqlQuery, which also uses exact type match now.

Apart from QueryRootExpression, no other kind of query root which can appear in different providers should be derivable (else they need to use exact type match too).
This rule makes relational ones sealed class. In case any one needs to derive from it, they need to add additional processing anyway.

Provider specific derived query roots can be non-sealed. If anyone is deriving from it then they should be using their derived provider which process those nodes too and if the derived provider wasn't used and shipped provider is used then it is an error from user perspective. If derived query root is used on other provider (targeting diff database) then it will fail since even the base shipped query root is unknown.

Resolves #26502
@JonPSmith
Copy link
Author

I would also suggest that using ORDER BY in a Cosmos FromSqlRaw has a problem, because the SQL is wrapped by a SELECT c FROM, which triggers a "ORDER BY is not supported in subqueries" exception. See my previous comment.

Not sure what you can do but the current Cosmos FromSqlRaw means you can't use ORDER BY, and maybe other SQL commands that are "not supported in subqueries".

smitpatel added a commit that referenced this issue Nov 19, 2021
Query root design

Core exposes QueryRootExpression which is base class for any query root so we can access EntityType out of it. This is needed for nav expansion and any potential future use case.
Since any derived query root would derive from it, core assembly translates it only when type is exact match to QueryRootExpression.
Relational layer also processes QueryRootExpression when they are mapped to default SqlQuery, which also uses exact type match now.

Apart from QueryRootExpression, no other kind of query root which can appear in different providers should be derivable (else they need to use exact type match too).
This rule makes relational ones sealed class. In case any one needs to derive from it, they need to add additional processing anyway.

Provider specific derived query roots can be non-sealed. If anyone is deriving from it then they should be using their derived provider which process those nodes too and if the derived provider wasn't used and shipped provider is used then it is an error from user perspective. If derived query root is used on other provider (targeting diff database) then it will fail since even the base shipped query root is unknown.

Resolves #26502
@markdav-is
Copy link

Yeah, that's intentional - both Relational and Cosmos have their own versions of FromSqlRaw, and they don't recognize each other's version (so e.g. the relational FromSqlRaw wouldn't be recognized by Cosmos). Making them ambiguous as above forces the user to explicitly specify which one they want, which prevents confusion etc.

How do I explicitly specify CosmosQueryableExtensions in my project that uses both the SQL Server provider and the CosmosDB provider in the same project. Is there something in my using clause I can do to make it pick the right one?

@roji
Copy link
Member

roji commented Dec 29, 2021

FromSql is an extension method in RelationalQueryableExtensions and CosmosQueryableExtensions, so it can be invoked explicitly as follows:

var result = RelationalQueryableExtensions.FromSqlRaw(ctx.Blogs, "SELECT ...").ToList();
var result = CosmosQueryableExtensions.FromSqlRaw(ctx.Blogs, "SELECT ...").ToList();

@markdav-is
Copy link

FromSql is an extension method in RelationalQueryableExtensions and CosmosQueryableExtensions, so it can be invoked explicitly as follows:

var result = RelationalQueryableExtensions.FromSqlRaw(ctx.Blogs, "SELECT ...").ToList();
var result = CosmosQueryableExtensions.FromSqlRaw(ctx.Blogs, "SELECT ...").ToList();

Thanks so much! I was getting tripped up on calling it from DBSet and forgot you can pass it into the menthod as well.

@ajcvickers ajcvickers added this to the 7.0.0 milestone Nov 5, 2022
@lavachakka

This comment was marked as off-topic.

@roji

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-query breaking-change closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. customer-reported type-bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants