Add provider specific column types #10

MargaretKrutikova · 2021-10-19T22:08:33Z

This PR attempts to fix #7 by adding a new attribute for fields that require db provider-specific type to be set on the corresponding command parameter. E.g. postgres requires NpgsqlDbType.Json and NpgsqlDbType.Jsonb to be set on the command parameter for columns of types json and jsonb correspondingly when using Npgsql data provider.

I am planning to add tests for the functionality after the initial review, and when I figure out how to add a new json column to the existing database used in tests 😄

MargaretKrutikova · 2021-10-19T22:12:01Z

src/SqlHydra.Query/SqlHydra.Query.fsproj

@@ -32,4 +32,8 @@
        <PackageReference Include="SqlKata" Version="2.3.7" />
    </ItemGroup>

+    <ItemGroup>
+      <ProjectReference Include="..\SqlHydra\SqlHydra.fsproj" />


This might be a wrong thing to do, I needed to reference the attribute I created inside SqlHydra. Does it mean the nuget package will get a dependency to SqlHydra?
Also had to change the target framework to be compatible with that project. Is it on purpose that this project is targeting netstandard2.0 to be compatible with the older versions of dotnet? Does it make sense to create a shared project that will be referenced from both projects instead?

Yes, SqlHydra.Query is netstandard2.0 to make it compatible with more project types. Since SqlHydra.Query doesn't use any new features, it would be nice to allow greater compatibility for a little while longer if possible.

I think you should be able to:

Change SqlHydra project to use netstandard2.0 so that it can be used in SqlHydra.Query

Add the following line into the SqlHydra.Query project so that it will copy the SqlHydra dll to nuget (this has already been done in the provider projects that reference SqlHydra.):

<TargetsForTfmSpecificBuildOutput>$(TargetsForTfmSpecificBuildOutput);CopyProjectReferencesToPackage</TargetsForTfmSpecificBuildOutput>

It seems like SqlHydra.fsproj references package Microsoft.Build version 16.11.0 which gives warning that it may not be fully compatible with netstandard 2.0. Not sure whether this can seriously break things somewhere.

SqlHydra.Query really doesn't need to reference Microsoft.Build or anything like that since most of that stuff is only used by the generator projects.

So we will have to do the following:

Create a new netstandard2.0 project, SqlHydra.Domain.

Move Domain.fs and the new attribute to the new project.

In Domain.fs, open GlobExpressions and applyFilters function need to be moved back to the SqlHydra project; they can be added to a new file Filters.fs (since this is only used by the generators).

JordanMarr

This looks really good so far! 😀

FYI, when testing code generation changes, there are .bat files in the Tests.fsproj folder that will regenerate the AdventureWorks.fs file for each provider so you can quickly iterate.

JordanMarr · 2021-10-20T00:46:34Z

src/SqlHydra.Npgsql/NpgsqlSchemaProvider.fs

+            | "json" -> { TypeName = nameof NpgsqlDbType; TypeValue = nameof NpgsqlDbType.Json } |> Some
+            | "jsonb" -> { TypeName = nameof NpgsqlDbType; TypeValue = nameof NpgsqlDbType.Jsonb } |> Some
+            | _ -> None
+


Rather than defining a lookup function, I think that this mapping should be added as a string option field on the supportedTypeMappings for each of the providers (NpgsqlDataTypes.fs, SqliteDataTypes.fs, and SqlServerDataTypes.fs. Most of them can default to None.

This value can then be moved from the Column record to to the TypeMapping record.
The TypeMapping record already has a DbType property that holds the System.Data.DbType enum.
Since each provider has its own more specific enumeration (i.e. NpgsqlDbType for Postgres, SqlDbType for SQL Server -- and I don't think SQLite has a more specific enum), I suppose the new property should be ProviderDbType.

JordanMarr · 2021-10-20T00:52:03Z

src/SqlHydra.Npgsql/NpgsqlSchemaProvider.fs

@@ -87,6 +93,7 @@ let getSchema (cfg: Config) : Schema =
                    |> Option.map (fun typeMapping ->
                        { 
                            Column.Name = col.ColumnName
+                            Column.DbColumnType = getDbColumnType col.ProviderTypeName


As stated above, move this property to TypeMapping and rename to ProviderDbType.

JordanMarr · 2021-10-20T01:02:28Z

src/SqlHydra.Query/QueryContext.fs

+            let value = System.Enum.Parse(property.PropertyType, type'.TypeValue)
+            dbTypeSetter.Invoke(param, [|value|]) |> ignore
+        | _ -> ()    
+


It would be nice to add a handler here for SQL Server as well:

| Some type', :? SqlKata.Compilers.SqlServerCompiler when type'.TypeName = "SqlDbType" -> let property = param.GetType().GetProperty("SqlDbType") ...

I had TypeName and TypeValue in the attribute before, but I figured you meant to just have a string there so I stored only the enum value of the provider type without the type name (like NpgsqlDbType.Json became just Json). So now I can't really do the check on type'.TypeName = "SqlDbType". What are you thoughts on this? Do we need to store the full type like NpgsqlDbType.Json or SqlDbType.Whatever?

What you have in NpgsqlDataTypes.fs and the attribute looks perfect. I think that it will be safe enough to assume that if they are using a given SqlKata compiler that they should be passing in types with the appropriate attributes.

So this should work:

let setParameterDbType (param: DbParameter) (qp: QueryParameter) = match qp.ProviderDbType, compiler with | Some dbType, :? SqlKata.Compilers.PostgresCompiler -> let property = param.GetType().GetProperty("NpgsqlDbType") let dbTypeSetter = property.GetSetMethod() let value = System.Enum.Parse(property.PropertyType, dbType) dbTypeSetter.Invoke(param, [|value|]) |> ignore | Some dbType, :? SqlKata.Compilers.SqlServerCompiler -> let property = param.GetType().GetProperty("SqlDbType") let dbTypeSetter = property.GetSetMethod() let value = System.Enum.Parse(property.PropertyType, dbType) dbTypeSetter.Invoke(param, [|value|]) |> ignore | _ -> ()

Yeah, the code now looks almost as yours, just with a helper function to hide the reflection ugliness 🙂

MargaretKrutikova · 2021-10-21T19:51:09Z

src/SqlHydra/SchemaGenerator.fs

@@ -631,7 +660,10 @@ let substitutions =
    [
        /// Reader classes at top of namespace
        "open Substitute.Extensions",
-        """type Column(reader: System.Data.IDataReader, getOrdinal: string -> int, column) =
+        $"""
+open {nameof SqlHydra}.{nameof ProviderDbTypeAttribute}


Is this ugly? Is there a better way of doing it? Like maybe specifying the full name of the attribute so that there is no need to import any modules? Smth like [<SqlHydra.ProviderDbType(...)>]? I couldn't really figure out how do it with SynAttribute.

It's no uglier than any of the other substitutions there. 😅
But I think I would prefer using fully qualified attributes to reduce the number of ugly substitutions.

But this just sparked an idea:
We shouldn't need a shared project with an attribute because we could just recreate an attribute of the same name in the SqlHydra.Query project! That would simplify things a bit because we wouldn't need a shared project at all, so better separation.

hm... Are there any downsides of having a shared project?

No downsides per se, other than the added complexity of having another project and copying project references to NuGet. But that's already done anyway, so I suppose it can stay as-is. Maybe it could come in handy at some point.

Oh right, didn't think about copying it to nuget. It is easy to revert with git, so I don't really mind, if you think it is going to be better without it 🙂

It really seems like 50/50 - points could be made for either way. So let's just keep it as-is for now. 🙂

MargaretKrutikova · 2021-10-22T11:58:55Z

I would like to add a field of type json and jsonb to the postgres test database so that I can test inserts and updates for those fields in QueryIntegrationTests, what is the right way to do it?

JordanMarr · 2021-10-22T14:36:29Z

I would like to add a field of type json and jsonb to the postgres test database so that I can test inserts and updates for those fields in QueryIntegrationTests, what is the right way to do it?

You can just add a test that drops and then creates a new table as is done in Dapper.FSharp.

MargaretKrutikova · 2021-10-24T17:03:21Z

src/SqlHydra/SchemaGenerator.fs

+let createProviderDbTypeAttribute (mapping: TypeMapping) =
+    mapping.ProviderDbType
+    |> Option.map (fun type' ->
+    let attributeFullName = typeof<ProviderDbTypeAttribute>.FullName


In order to avoid adding open SqlHydra.... to the string code block below I am using the full name of the attribute split by dot, and it seems like LongIdentWithDots.Create manages to properly handle it.
If we were to duplicate the attribute here without having a shared project, we wouldn't be able to do this I think, and would have to hardcode the full name of the attribute from SqlHydra.Query since it is parsed there.

MargaretKrutikova · 2021-10-24T17:07:42Z

src/Tests/Npgsql/Generation.fs

@@ -74,8 +74,16 @@ let tests =

            let pks = allColumns |> List.filter (fun c -> c.IsPK)

-            Expect.equal schema.Tables.Length 68 ""
+            Expect.equal schema.Tables.Length 69 ""


Here I had to increase the number since I have a test that creates one more table to test the provider specific types (json and jsonb), however the test that creates that table (here) is not guaranteed to run before this one, so it seems very wrong to do this kind of thing 🙂 Is there a way to add this new table to the database with all the other tables that were already there (person, sales etc)? Do I need to add an sql script that creates this table under /src/AdventureWorks/SalesLT/Tables?

One option would be to add it to the Docker postgres install.sql.

Another option might be to create the table setup test according to the Expecto Setup/Teardown example for a setup test, which I think would guarantee that it would always run first in the postrges tests.

This approach seems to ensure that the setup fn is always executed before the tests:

let generateProviderDbTestTable () = dropProviderDbTestTable () use ctx = openContext() let createProviderDbTestTableCmd = ctx.Connection.CreateCommand () createProviderDbTestTableCmd.CommandText <- "create schema provider_test; create table provider_test.test(id serial, json_field json not null, jsonb_field jsonb not null);" createProviderDbTestTableCmd.ExecuteNonQuery () |> ignore [<Tests>] let tests = generateProviderDbTestTable () categoryList "Npgsql" "Query Integration Tests" [ testTask "Where city Starts With S" { use ctx = openContext() let addresses = select { for a in addressTable do where (a.city |=| [ "Seattle"; "Santa Cruz" ]) } |> ctx.Read HydraReader.Read gt0 addresses Expect.isTrue (addresses |> Seq.forall (fun a -> a.city = "Seattle" || a.city = "Santa Cruz")) "Expected only 'Seattle' or 'Santa Cruz'." } ...

Then you can remove generateProviderDbTestTable () from line 476.

MargaretKrutikova · 2021-10-24T17:14:23Z

src/SqlHydra.Npgsql/NpgsqlSchemaProvider.fs

@@ -52,12 +51,14 @@ let getSchema (cfg: Config) : Schema =
                TableName = col.["TABLE_NAME"] :?> string
                ColumnName = col.["COLUMN_NAME"] :?> string
                ProviderTypeName = col.["DATA_TYPE"] :?> string
+                OrdinalPosition = col.["ORDINAL_POSITION"] :?> int


I noticed that every time you run the generator the order of the fields changes, and then one of the tests for insert query failed because the order of the fields has changed (see the test here) so I thought I would add sorting by ordinal position (I assume it is the order columns were added to the table) to avoid random order after every re-generation.

That's a good idea, but I'm not sure that "ORDINAL_POSITION" is given for the other database providers.
If not, then I think a reasonable alternative would be to sort by column name.

MargaretKrutikova · 2021-10-24T17:20:17Z

I have added some tests but stumbled into a couple of problems:

It doesn't seem like a good solution to generate a table inside a test since other tests rely on it. I need to regenerate AdventureWorks to be able to test the functionality I added so the new table has to permanently stay in the database and other test rely on the fact that there are certain number of tables present. See comment.
I can't run any mssql integration tests with the error

 A network-related or instance-specific error occurred while establishing a connection to SQL Server. 
The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is 
configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)

even though it is running in docker. Could it be smth OS-specific? I am running mac, but I assume you are developing on windows (from the bat scripts and windows-like paths)?

JordanMarr · 2021-10-24T19:49:22Z

I have added some tests but stumbled into a couple of problems:

It doesn't seem like a good solution to generate a table inside a test since other tests rely on it. I need to regenerate AdventureWorks to be able to test the functionality I added so the new table has to permanently stay in the database and other test rely on the fact that there are certain number of tables present. See comment.

See response above.

I can't run any mssql integration tests with the error
 A network-related or instance-specific error occurred while establishing a connection to SQL Server. 
The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is 
configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)
even though it is running in docker. Could it be smth OS-specific? I am running mac, but I assume you are developing on windows (from the bat scripts and windows-like paths)?

Assuming the mssql instance is running properly, it should run on standard 1433 port if you are running vs code with the remote-containers extension.
But if you are manually spinning up from docker-compose then it's on 12019. I created a "DebugLocal" config that switches the test connection strings which changes the connection string to allow me to run tests from my localhost using Visual Studio (since I do not personally use vs-code).

(Maybe I should add this detail to the Contributing section.)

JordanMarr · 2021-10-27T22:32:02Z

How's it going? Is this ready to merge or still 🚧🚧? No rush, of course; just wanted to make sure you weren't waiting on me.

MargaretKrutikova · 2021-10-29T20:05:38Z

Was away for a couple of days 🙂 Just ran the tests and all of them passed. Will you be able to review the code one last final time? I saw there was another PR fixing the order of the columns, but I think it is only for SqlServer, and I only touched Npgsql.

JordanMarr · 2021-11-01T02:42:43Z

For some reason it is not adding the new table in postgres. I think it may have something to do with with the files on the local volume not be removed, so I tried docker-compose down -v to remove them but no luck.
Do you have any ideas?

MargaretKrutikova · 2021-11-01T22:45:09Z

I think I had to remove and re-create postgres service from docker. You can also try and run docker-compose up --build --force-recreate --no-deps -d postgresql to just demolish the service all together and rebuild it 🙂

JordanMarr · 2021-11-02T03:15:57Z

Fantastic work, Margarita. Thank you! 🎉

MargaretKrutikova · 2021-11-02T08:11:41Z

Thanks a lot for your help! 😄

Margarita Krutikova added 6 commits October 20, 2021 00:00

Add column type attribute in schema generator

c656d78

Fix schema generation with attribute

cf557ea

Add npgsql db column type mapping

a0f01b3

Add command parameter type to box db column type value

ba702b7

Set db type when building command in query context

525a05c

Add open sqlhydra in generated file

b2aecb9

MargaretKrutikova commented Oct 19, 2021

View reviewed changes

JordanMarr reviewed Oct 20, 2021

View reviewed changes

Margarita Krutikova added 3 commits October 21, 2021 21:11

Create domain project

bc80625

Fix schema generation, add sqldbtype handling

c08a55d

Cleanup

268bb58

MargaretKrutikova commented Oct 21, 2021

View reviewed changes

Margarita Krutikova added 7 commits October 23, 2021 20:31

Use full name for provider db type attribute

3b32f80

Add integration tests for npgsql provider specific db types

602d507

Fix updates for provider specific values

e646095

Fix test to not depend on json formatting in postgres

4e4db59

Add tests for provider type attribute generation

a88f445

Sort columns in ordinal position for postgres

f38df21

Fix number of table npgsql test

f1d2d95

MargaretKrutikova commented Oct 24, 2021

View reviewed changes

Remove unused open

fd0e9a6

MargaretKrutikova commented Oct 24, 2021

View reviewed changes

Add provider db test to postgres install script in dev container

feaf2ef

JordanMarr merged commit 8cc3461 into JordanMarr:main Nov 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add provider specific column types #10

Add provider specific column types #10

MargaretKrutikova commented Oct 19, 2021

MargaretKrutikova Oct 19, 2021

JordanMarr Oct 19, 2021

MargaretKrutikova Oct 20, 2021

JordanMarr Oct 20, 2021 •

edited

JordanMarr left a comment

JordanMarr Oct 20, 2021

JordanMarr Oct 20, 2021

JordanMarr Oct 20, 2021

MargaretKrutikova Oct 21, 2021

JordanMarr Oct 21, 2021

MargaretKrutikova Oct 24, 2021

MargaretKrutikova Oct 21, 2021 •

edited

JordanMarr Oct 21, 2021

MargaretKrutikova Oct 22, 2021

JordanMarr Oct 22, 2021

MargaretKrutikova Oct 23, 2021

JordanMarr Oct 23, 2021

MargaretKrutikova commented Oct 22, 2021

JordanMarr commented Oct 22, 2021

MargaretKrutikova Oct 24, 2021

MargaretKrutikova Oct 24, 2021 •

edited

JordanMarr Oct 24, 2021 •

edited

JordanMarr Oct 24, 2021

MargaretKrutikova Oct 24, 2021

JordanMarr Oct 24, 2021

MargaretKrutikova commented Oct 24, 2021 •

edited

JordanMarr commented Oct 24, 2021 •

edited

JordanMarr commented Oct 27, 2021

MargaretKrutikova commented Oct 29, 2021

JordanMarr commented Nov 1, 2021

MargaretKrutikova commented Nov 1, 2021

JordanMarr commented Nov 2, 2021

MargaretKrutikova commented Nov 2, 2021

Add provider specific column types #10

Add provider specific column types #10

Conversation

MargaretKrutikova commented Oct 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JordanMarr Oct 20, 2021 • edited

Choose a reason for hiding this comment

JordanMarr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MargaretKrutikova Oct 21, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MargaretKrutikova commented Oct 22, 2021

JordanMarr commented Oct 22, 2021

Choose a reason for hiding this comment

MargaretKrutikova Oct 24, 2021 • edited

Choose a reason for hiding this comment

JordanMarr Oct 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MargaretKrutikova commented Oct 24, 2021 • edited

JordanMarr commented Oct 24, 2021 • edited

JordanMarr commented Oct 27, 2021

MargaretKrutikova commented Oct 29, 2021

JordanMarr commented Nov 1, 2021

MargaretKrutikova commented Nov 1, 2021

JordanMarr commented Nov 2, 2021

MargaretKrutikova commented Nov 2, 2021

JordanMarr Oct 20, 2021 •

edited

MargaretKrutikova Oct 21, 2021 •

edited

MargaretKrutikova Oct 24, 2021 •

edited

JordanMarr Oct 24, 2021 •

edited

MargaretKrutikova commented Oct 24, 2021 •

edited

JordanMarr commented Oct 24, 2021 •

edited