New scaffolding framework

[MaceWindu]

Fix #1383
Fix #1586 (proper fix instead of old hack)
Fix #1825
Fix #1897 (wasn't actually fixed before)
Fix #1971
Fix #2168 (proper fix instead of old hack)
Fix #2515
Fix #2793
Fix #3345

Goal

Provide new code-generation library and command-line tool for database model scaffolding to be used by:

• LinqPAD integration
• users

Currently we use:

1. T4 templates
2. Copy of T4 code in LinqPAD integration

Issues with current approach:

• code duplication between T4 and LinqPAD, issues with changes sync between them
• T4 has very poor support outside of Visual Studio and unusable on most of 3rd party implementations due to lack of features (like MSBuild directives support)
• Issues with different target frameworks used to run T4: VS use .net framework, Rider use .net core
• poor support for T4 development, basically you work with text file
• current implementation contains of two parts with mixed responsibilities: schema API in linq2db with unrelated functionality like C# identifiers and types generation and T4 code that mix schema translation and code generation without responsibilities separation
• target codegeneration supports only one language (C#)
• a lot of known and unknown (e.g. discovered during development of current feature) issues with code generation, especially with code correctness of generated code (like invalid identifiers, naming conflicts)

Architecture

Instead of semi-monolitic approach, here we try to separate code generation into several distinctive steps:

• database schema load in form of new schema API, provided by linq2db that will not include unrelated functionality and bring more flexibility to this API. NOTE: initial releases will not have this API and we will have wrapper over current schema API instead
• in-memory data model build, that includes generation of descriptors of data context, entities, stored procedure/function helpers with names, that will be used in code, data model metadata (that will be converted to code model on next step using different metadata generators, e.g. attribute-based or fluent mapping mappings generators)
• conversion of data model to code model (AST), normalization of AST (e.g. fixing naming according to target language naming rules and resolving duplicate name conflicts)
• conversion of code AST to target language. Initial releases will have C# code generator, but later we plan to add at least VB.NET and F# generators as languages that were requested by users before

Tasks

• packaging: decide on release names for codegen library and commandline tool
• packaging: nuget packaging for codegeneration library
• packaging: nuget packaging for commandline tool
• packaging: deprecate T4 packages and stop automatic publishing on release
• cli: parametrization using command line and configuration file (json)
• cli: T4 migration preset to use options, similar to T4 defaults to simplify migration to new tool for existing T4 users
• api: finalize schema API
• api: finalize data model build API
• api: finalize code generation API
• api: finalize C# code generator
• extensibility interfaces to extend data model generation on all steps
• cli: decide and implement which approach to use for scaffolding customization: no customization for cli, loadable C# file, loadable assembly?
• [v.next] VB.NET code generator
• [v.next] F# code generator. See [T4 template] F# generation recods.  #1553
• [v.next] TypeScript code generator. See Extracting T4 renderer into separate class #1233
• [v.next] Fluent mapping generator. See [Scaffold] Generate mappings using fluent mapper #945
• linqpad: migration to new library
• documentation
• xml-docs in code
• tests
• [v.next] new schema api and deprecation of old API
• [v.next] new schema api: avoid table function/procedure execution to retrieve schema when supported by database (e.g. postgresql table functions)
• [v.next] new schema api: sequence name support. See Add SequenceNameAttribute generation to T4 #2766
• [v.next] Generation of data model in multiple classes/file-per-class mode. See How T4 generates multiple files #1825
• option to configure which .net type to use for specific database type instead of current implementation where type is hardcoded
• more pre-defined customization options. E.g. see Is there any way to change access modifier for scaffolded properties? #1257
• migrate some existing non-db T4 templates: Equatable.ttinclude
• decide what to do with WPF-related T4 templates. Discard and unrelared to linq2db or release as separate T4 package?
• improve association properties/extensions naming. See Generate user-friendly association names for MS Access relations without user-defined name #2331 (probably should be fixed on schema api side to return empty FK name for such Access keys)
• new FindQuery extension generation. See question T4 how to custom add FindQuery() without custom LinqToDB.ttinclude #2793
• more built-in methods for association property/extension naming. See Association Property names #2803
• [v.next] option to generate DTO classes. See DTO Objects #2850
• [v.next] option to generate mappings using C# records (requires fluent mapping support)
• verify that all applicable T4 logic replicated in new code

There are also a bunch of schema/T4 issues that could be addressed during current feature implmentation or later, which are not mentioned as they probably will be done separately.

---

Implemented changes

Main feature

• implemented scaffolding framework in linq2db.Tools project:
  • Source\LinqToDB.Tools\CodeModel: code model and code generation (currently C#-only)
  • Source\LinqToDB.Tools\DataModel: cotext data model to code AST conversion
  • Source\LinqToDB.Tools\Metadata: context metadata model and generation (currently attribute-based only)
  • Source\LinqToDB.Tools\Naming: model pluralization and identifier normalization
  • Source\LinqToDB.Tools\Scaffold: database schema to context data model conversion
  • Source\LinqToDB.Tools\Schema: database schema API (currently wrapper over existing schema API)
• added scaffolding CLI tool 'Source\LinqToDB.CLI' that use new framework
• added cli tool packaging as dotnet tool. Planned nuget name: linq2db.cli, dotnet tool name: dotnet-linq2db. Target framework: netcoreapp3.1 (with plan to switch to net6 after 3.1 EOL)

Various scaffolding fixes/improvements

• [Scaffold] Partial fix MS SQL Temproray tables in procedures. How to build #1468 - added new scaffold option to ignore schema load errors and generate procedure mappings without return table. Could be enough if procedure doesn't return table or user is not interested in it or plan to add table mapping manually.
• [CodeGeneration] Implement conflict detection for generated identifiers. Properly fix Wrong T4 generation for ColumnAttribute when table name is "DataType"  #2168, Table named "Relationships" #1586
• [CodeGeneration] Implement generated C# identifier validation/normalization to produce valid identifier name
• [Scaffold] enable @return parameter support for SQL Server stored procedures. Fix [Request, T4 template] Support for ParameterDirection.ReturnValue #1897 (previous fix wasn't included into T4 by mistake)
• Fix MS Access does not work with VS2022 64 BIT #3345. Implemented --architecture cli tool switch to start it as x64 or x86 process. This is needed for unmanaged providers, especially Access. VS2019- always used x86 T4 host and VS2022 use x64 test host, which made it hard to use with such databases.
• [Access] New tool supports schema load for Access using two connections (OLE DB + ODBC). This allows us to load more precise database schema. Both providers return incomplete schema information and with this feature we combine information from both providers.
• [Sybase] set ColumnType for procedure schema in schema provider
• [Access] Update provider detection logic to not identify connections with DataAccess in name as Access connection (as DataAccess is a part of oracle managed provider name)
• [Scaffold] support generation of async stored procedure mappings
• [Scaffold] update Find extensions generation (Fix question T4 how to custom add FindQuery() without custom LinqToDB.ttinclude #2793)
  • FindAsync method generation
  • FindQuery method generation
  • generation of extensions on of context class (previously only on ITable<T>)
  • generation of extensions with entity object as parameter instead of separate parameter for each primary key column
  • (results in 12 variations of Find generation)
• [Scaffold] interceptors mechanism to cusomize scaffolding using T4 template or pre-built interceptor assembly

Other fixes

• add new connection interceptor method AfterExecuteReader to handle user-requested case with oracle reader configuration. Proposed workaround was unusable due to use of internal API
• obsoleted nonfunctional AssociationAttribute properties (KeyName, BackReferenceName, IsBackReference, Relationship) and Relationship enum as they were never used
• obsoleted DataConnection.OnTrace static property
• replace use of StringComparison.Invariant* to StringComparison.Ordinal* enum values

Infrastructure changes

• Code sharing: shared code to enable recent C# features for older frameworks moved to shared folder Source\Shared from LinqToDB project and included to all projects (see Build/linq2db.Default.props)
• disabled nuget publishing for T4 templates. Current plan is to obsolete them and publish them only explicitly when we fix something.

InternalsVisibleTo removal

This attribute used by linq2db to open some internal stuff to tests which had several negative consequences:

• we had some API intended for use by users as marked internal (see below)
• [netfx-only] attribute requires strong name so we signed our tests assembly, which prevented it from references to unsigned assemblies. E.g. as dotMorten.Microsoft.SqlServer.Types is not signed - we used custom signed build for it instead of nuget

To solve it I've:

• removed attribute usages
• removed tests signing
• opened "internal" API, that became unavailable to tests

Opened API:

• LinqToDB.Common.Compilation.CompileExpression methods exposed for linq2db.Tools and tests, but still marked as internal API. It's needed mostly for Tools and we can remove use of it from tests
• LinqToDB.Common.EnumerableHelper.Batch methods exposed (were covered by tests)
• InvariantCultureRegion made public (for tests)
• MappingSchema.CombineSchemas method made public (for tests)
• LinqExtensions.Updatable made public (for tests)
• Query.Queries property, Query<T> class (for method ClarCache, property CacheMissCount and method GetQuery) and QueryInfo class made public for tests
• InternalExtensions class, methods Unwrap, IsSameGenericMethod, EvaluateExpression (for tests)
• TypeWrapperNameAttribute, ValueTaskToTaskMapper, CustomMapperAttribute and ICustomMapper made public (should've been public from start)
• ReflectionExtensions.IsSubClassOf method (for linq2db.Tools)
• SqlServerTransientExceptionDetector.IsHandled method (for linq2db.Tools)
• SqlServerTools.QuoteIdentifier method (for tests)
• PostgreSQLDataProvider.HasMacAddr8 property (for tests)
• DataContext methods GetDataConnection, ReleaseQuery, ReleaseQueryAsync (for tests)

I plan to review it later and probably hide some API back (requires tests refactoring). Also some tests are commented due to internal APIs - will address it too

[MaceWindu]

If anyone interested in testing and giving feedback for current dev. build, he can try it using attached nuget (unzip first) linq2db.cli.0.0.52.zip
Or for manual build:

• checkout PR branch
• build in release mode
• run NuGet\TestToolLocal.cmd X to build and install nuget, where x is a version number (increase it for each new build)

What is missing from this version:

• ability to customize generation using custom code (working on it)
• CLI options defaults not finalized yet
• T4 compatibility mode not configured yet (this is mode when tool use options defaults that generate code matching old T4 generation for more easy migration)

Check nuget readme for starters: https://github.com/linq2db/linq2db/blob/685a75093832e430e41411d72da64f5b40f62051/NuGet/CLI/README.md

[MaceWindu]

/azp run test-all

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MaceWindu]

/azp run test-all

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MaceWindu]

/azp run test-all

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[linq2dbot]

Test baselines changed by this PR. Don't forget to merge/close baselines PR after this pr merged/closed.