Non-deterministic 'AccessViolationException' under specific conditions when using 'System.Data.OleDb' in an x64 process

[lauxjpn]

@roji As discussed yesterday in our call with @ajcvickers and the others, here the information about the AccessViolationException issue I was talking about.

The AccessViolationException issue has been previously discussed in detail in CirrusRedOrg/EntityFrameworkCore.Jet#43 (comment) (and I also referenced it before in #33899 (comment)).

It contains the specific conditions under which the issue appears and also contains the original reproduction code.

I updated the code (simplified it further) and also added projects for ODBC using ADO.NET and for directly using OLE DB via C++.

Both of those added projects work as expected, so this issue is either a System.Data.OleDb issue after all, or it is an OLE DB layer issue, but depends on specific operations that are being performed by ADO.NET, but not by my Working_Jet_OleDb_x64_Native C++ project.

---

The original post from CirrusRedOrg/EntityFrameworkCore.Jet#43 (comment):

I created the smallest possible sample to reproduce the AccessViolationException.

• The exception will only be thrown when run as an x64 process.
• The exception should be thrown at least once in every 5 app executions.
• I tested with Microsoft.ACE.OLEDB.16.0 and Microsoft.ACE.OLEDB.12.0. Both will throw the exception (v16 usually very early, when still below 10 iterations; v12 usually after around 50 iterations).
• The exception will only be thrown, when a UNION query as been executed and later a SELECT query over the same table as the UNION query is executed.
• This second SELECT query must return only scalar values (no column reference).
• It doesn't matter, what the scalar values are (e.g. 0, 1, NULL).
• The count of scalar values of the second SELECT query must be at least as high as the count of returned fields in the previous UNION query.
• It does not matter, if another query is executed in between.
• In case there has been no AccessViolationException within the first 100 interations, it is very unlikely that they will ever appear within the lifetime of the process. (Though sometimes one is thrown near the 50K mark.)

A couple of examples that will throw:

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, NULL
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, NULL
FROM [Customers] AS [c];

A couple of examples that will not throw:

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT [c].[CustomerID] /* <-- not a scalar value */
FROM [Customers] AS [c];

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'; /* <-- not a UNION */

SELECT 1
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1 /* <-- fewer scalar values than fields in UNION */
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, [c].[CustomerID] /* <-- not just scalar values */
FROM [Customers] AS [c];

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT *
FROM [Orders] AS [o];  /* <-- table differs from the one used in the UNION */

The following code can be used to reproduce the issue. The code should be run with the latest nightly-build of System.Data.OleDb:

using System;
using System.Data.OleDb;

namespace JetProviderExceptionTests
{
    internal static class Program
    {
        private static void Main()
        {
            Console.WriteLine($"Running as {(Environment.Is64BitProcess ? "x64" : "x86")} process.");
            
            var tagDBPARAMBINDINFOName = "tagDBPARAMBINDINFO" + (Environment.Is64BitProcess
                ? string.Empty
                : "_x86");
            
            // Is 2 on x86 and 8 on x64.
            Console.WriteLine($"{tagDBPARAMBINDINFOName} field alignment is {Type.GetType($"System.Data.OleDb.{tagDBPARAMBINDINFOName}, System.Data.OleDb").StructLayoutAttribute.Pack}.");
            
            // Is 8 on both x86 and x64.
            Console.WriteLine($"tagDBPARAMS field alignment is {Type.GetType("System.Data.OleDb.tagDBPARAMS, System.Data.OleDb").StructLayoutAttribute.Pack}.");

            Console.WriteLine();
            Console.WriteLine("Press any key to start...");
            Console.ReadKey();

            RunNorthwindTest();
        }

        private static void RunNorthwindTest()
        {
            try
            {
                using var connection = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.16.0;Data Source=Northwind.accdb");
                connection.Open();

                for (var i = 0; i < 1000; i++)
                {
                    Console.WriteLine($"{i:000}");

                    //
                    // Select_Union:
                    //

                    using (var command1 = connection.CreateCommand())
                    {
                        command1.CommandText = @"SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London'";

                        using (var dataReader1 = command1.ExecuteReader())
                        {
                            while (dataReader1.Read())
                            {
                            }
                        }
                    }

                    /*
                    using (var command15 = connection.CreateCommand())
                    {
                        command15.CommandText = @"SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Madrid'";

                        using (var dataReader15 = command15.ExecuteReader())
                        {
                            while (dataReader15.Read())
                            {
                            }
                        }
                    }
                    */
                    
                    //
                    // Select_bool_closure:
                    //

                    using (var command2 = connection.CreateCommand())
                    {
                        command2.CommandText = @"SELECT 1
FROM [Customers] AS [c]";

                        using (var dataReader2 = command2.ExecuteReader())
                        {
                            while (dataReader2.Read())
                            {
                            }
                        }
                    }
                }
            }
            catch (AccessViolationException e)
            {
                Console.WriteLine(e);
            }
        }
    }
}

The code uses the version of the Northwind database we are also now using in our functional tests:
Northwind.zip

There are no transactions and no multi-threading involved to trigger the exception.

[...]

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @roji, @ajcvickers
See info in area-owners.md if you want to be subscribed.

Issue Details

---

@roji As discussed yesterday in our call with @ajcvickers and the others, here the information about the AccessViolationException issue I was talking about.

The AccessViolationException issue has been previously discussed in detail in CirrusRedOrg/EntityFrameworkCore.Jet#43 (comment) (and I also referenced it before in #33899 (comment)).

It contains the specific conditions under which the issue appears and also contains the original reproduction code.

I updated the code (simplified it further) and also added projects for ODBC using ADO.NET and for directly using OLE DB via C++.

Both of those added projects work as expected, so this issue is either a System.Data.OleDb issue after all, or it is an OLE DB layer issue, but depends on specific operations that are being performed by ADO.NET, but not by my Working_Jet_OleDb_x64_Native C++ project.

---

The original post from CirrusRedOrg/EntityFrameworkCore.Jet#43 (comment):

I created the smallest possible sample to reproduce the AccessViolationException.

• The exception will only be thrown when run as an x64 process.
• The exception should be thrown at least once in every 5 app executions.
• I tested with Microsoft.ACE.OLEDB.16.0 and Microsoft.ACE.OLEDB.12.0. Both will throw the exception (v16 usually very early, when still below 10 iterations; v12 usually after around 50 iterations).
• The exception will only be thrown, when a UNION query as been executed and later a SELECT query over the same table as the UNION query is executed.
• This second SELECT query must return only scalar values (no column reference).
• It doesn't matter, what the scalar values are (e.g. 0, 1, NULL).
• The count of scalar values of the second SELECT query must be at least as high as the count of returned fields in the previous UNION query.
• It does not matter, if another query is executed in between.
• In case there has been no AccessViolationException within the first 100 interations, it is very unlikely that they will ever appear within the lifetime of the process. (Though sometimes one is thrown near the 50K mark.)

A couple of examples that will throw:

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, NULL
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, NULL
FROM [Customers] AS [c];

A couple of examples that will not throw:

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT [c].[CustomerID] /* <-- not a scalar value */
FROM [Customers] AS [c];

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'; /* <-- not a UNION */

SELECT 1
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1 /* <-- fewer scalar values than fields in UNION */
FROM [Customers] AS [c];

SELECT [c].[Address], [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address], [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT 1, [c].[CustomerID] /* <-- not just scalar values */
FROM [Customers] AS [c];

SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London';

SELECT *
FROM [Orders] AS [o];  /* <-- table differs from the one used in the UNION */

The following code can be used to reproduce the issue. The code should be run with the latest nightly-build of System.Data.OleDb:

using System;
using System.Data.OleDb;

namespace JetProviderExceptionTests
{
    internal static class Program
    {
        private static void Main()
        {
            Console.WriteLine($"Running as {(Environment.Is64BitProcess ? "x64" : "x86")} process.");
            
            var tagDBPARAMBINDINFOName = "tagDBPARAMBINDINFO" + (Environment.Is64BitProcess
                ? string.Empty
                : "_x86");
            
            // Is 2 on x86 and 8 on x64.
            Console.WriteLine($"{tagDBPARAMBINDINFOName} field alignment is {Type.GetType($"System.Data.OleDb.{tagDBPARAMBINDINFOName}, System.Data.OleDb").StructLayoutAttribute.Pack}.");
            
            // Is 8 on both x86 and x64.
            Console.WriteLine($"tagDBPARAMS field alignment is {Type.GetType("System.Data.OleDb.tagDBPARAMS, System.Data.OleDb").StructLayoutAttribute.Pack}.");

            Console.WriteLine();
            Console.WriteLine("Press any key to start...");
            Console.ReadKey();

            RunNorthwindTest();
        }

        private static void RunNorthwindTest()
        {
            try
            {
                using var connection = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.16.0;Data Source=Northwind.accdb");
                connection.Open();

                for (var i = 0; i < 1000; i++)
                {
                    Console.WriteLine($"{i:000}");

                    //
                    // Select_Union:
                    //

                    using (var command1 = connection.CreateCommand())
                    {
                        command1.CommandText = @"SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Berlin'
UNION
SELECT [c0].[Address]
FROM [Customers] AS [c0]
WHERE [c0].[City] = 'London'";

                        using (var dataReader1 = command1.ExecuteReader())
                        {
                            while (dataReader1.Read())
                            {
                            }
                        }
                    }

                    /*
                    using (var command15 = connection.CreateCommand())
                    {
                        command15.CommandText = @"SELECT [c].[Address]
FROM [Customers] AS [c]
WHERE [c].[City] = 'Madrid'";

                        using (var dataReader15 = command15.ExecuteReader())
                        {
                            while (dataReader15.Read())
                            {
                            }
                        }
                    }
                    */
                    
                    //
                    // Select_bool_closure:
                    //

                    using (var command2 = connection.CreateCommand())
                    {
                        command2.CommandText = @"SELECT 1
FROM [Customers] AS [c]";

                        using (var dataReader2 = command2.ExecuteReader())
                        {
                            while (dataReader2.Read())
                            {
                            }
                        }
                    }
                }
            }
            catch (AccessViolationException e)
            {
                Console.WriteLine(e);
            }
        }
    }
}

The code uses the version of the Northwind database we are also now using in our functional tests:
Northwind.zip

There are no transactions and no multi-threading involved to trigger the exception.

[...]

Author:	lauxjpn
Assignees:	-
Labels:	area-System.Data.OleDB, untriaged
Milestone:	-

[ajcvickers]

Thanks @lauxjpn. We'll get back to this when people come back for the new year.

[lauxjpn]

I'll get the same exception, when I perform the same steps I do in the (working) C++ project, but in .NET. So when I manually manage all the COM interfaces of OLE DB in C#, I'll get an AccessViolationException around iteration 50. No System.Data.OleDb reference involved.

I'll update the reproduction project with the new project.

[ajcvickers]

Moving this to Future for now to follow up with the Access team, as per offline discussion with @lauxjpn.

[lauxjpn]

We are still getting lots of crashes in different Access Runtime versions and processor architectures.
So this is still an issue and it would still be great to get in contact with whoever is currently maintaining Microsoft Access, so that those memory access violations get fixed.

(Recent discussion in CirrusRedOrg/EntityFrameworkCore.Jet#152.)