Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move all FluentMigration.Runner code to FluentMigrator.Runner.Core #1600

Merged
merged 4 commits into from
Jun 7, 2022

Conversation

mattbrailsford
Copy link
Contributor

@mattbrailsford mattbrailsford commented May 5, 2022

This is a re-implementation of PR #1198 but does not remove any legacy / obsolete code. This should result in the same outcome, that FluentMigrator.Runner is now just a meta package with a single extensions method for adding all database types to the IMigrationRunnerBuilder but all other code has moved to FluentMigrator.Runner.Core thus meaning folks shouldn't need to install all database providers.

The only real changes I've had to make are

  1. The AddAllDatabases extension method in FluentMigrator.Runner has changed from being a IServiceCollection extension method to an IFluentMigratorRunnerBuilder extension and has been made public.
  2. The FluentMigratorServiceCollectionExtensions.CreateServices method now no longer calls AddAllDatabases and instead accepts an optional Action<IMigrationRunnerBuilder> configureRunner to allow external configuration and to call the AddAllDatabases() extension
  3. TaskExecutor now accepts an optional Action<IMigrationRunnerBuilder> configureRunner that is passed through to the CreateServices method above.
  4. All AddDbType methods have been updated to register the provider factory with the MigrationProcessorFactoryProvider rather than the MigrationProcessorFactoryProvider being responsible for adding them all.

I guess something to clarify here is that I've only made the change such that it's possible to only depend on FluentMigrator.Runner.Core and the DB provider you need, but I haven't changed anything in the runner implementations (CLI, Console, etc) to allow you to only use what you need from there. To be honest, I don't know what these actually do / how they are used as I'm primarily using FluentMigrator as an in-app process and so I run the migrations via C# code so what I have implemented here is enough for my needs.

Maybe this PR can be accepted as a stepping stone and then review what can be done to the runner implementations such that those can also be configured to only require the dependencies they need.

…cept for a single extension method for adding all databases
@jzabroski
Copy link
Collaborator

jzabroski commented May 5, 2022

This seems nice, based on your comments/description of the changes. I have not looked at the code changes yet.

I'm in the process of updating the example Runner. - In particular, I am cherry-picking 369037c from the develop branch as part of #1348 to incorporate @fubar-coder removal of Obsolete tagged code, and he also made changes to make the Runner independent of the various processors.

Ideally, we also tackle #1006 , in particular #1208 - See my comments here: #1208 (comment)

Fully closing #1006 would probably entail smoothing out how we package FluentMigrator.MSBuild, as the way it is bundled today is incomplete and there should probably be a second "batteries included" version of FluentMigrator.MSBuild package.

I think we may want to obsolete remove the legacy Console as well for either 4.0 or 5.0, but that would currently affect FluentMigrator.MSBuild I think (sorry for cryptic comment - just thinking out loud notes for me to check on).

@jzabroski
Copy link
Collaborator

jzabroski commented May 5, 2022

I think we should update:

https://github.com/fluentmigrator/fluentmigrator/blob/master/src/FluentMigrator.Console/Program.cs#L32

to include a warning message (in yellow ANSI Color code) at the entry point:

System.Console.WriteLine($"WARN: FluentMigrator.Console is obsolete, and will be going away in 5.0.0 release.{System.Environment.NewLine}Please check the upgrade guide http://fluentmigrator.github.io/....");

Something like that.

That can go in a separate ticket.

@jzabroski
Copy link
Collaborator

Additional Notes to self: Pull over my comments from #1198

The economic value it adds is:

  1. Ability to write Roslyn Analyzers that suggest database-specific best practices based on which Runners the users import.
  2. Ability to track via nuget.org the popularity of database-specific runners and therefore where to focus efforts of FluentMigrator improvements by popularity (not that popularity is the only consideration, as die hard/passionate users are often the best ways to build community and improve projects - @fubar-coder , @eloekset and myself fall in the die hard camp and that's how we all got involved - we just couldn't stand to see the project not be actively maintained)
  3. Ability to split out unit tests per runner - I started doing this as part of Delete SAP SqlAnywhere #1218 - with the major advantage being that if some vendors/projects fall behind in supporting the .NET releases, the core branch isn't over-complicated with #if NETFRAMEWORK and other pre-processor directives and so on.
    1. My thought is that I could then generate a report for which runners cover which "test drivers", and have an "architecture verification test" for saying things like "SQLite doesn't support partitions, so ignore tests related to partitions and partitioned indexes" and "SQL Server seems to be missing tests for partitioned indexes but has create/alter partition tests."

Previously, nobody could give me great reason to do this other than "I hate files in my directory".

Remaining proof-of-concept is re-bundling FluentMigrator.Console and verifying it ships all runners still. I am assuming you expect FluentMigrator.DotNet.Cli to also ship everything, and that this change is mainly to support using the In-Process Runner?

and

what happens if you upgrade to FluentMigrator 3.2.8 locally but your build server is on 3.2.1. I believe the behavior will depend on RollForward behavior, and that any runtime dependency mismatches will not get caught until you go to production. Do you have any suggestions on how we can detect such mismatches and avoid them? See #1211 for an example where a customer was bitten by this.

along with:

The downside I see to hot loading types in RunWithServices is that you move away from deps.json auto-resolving the dependency, so you can end up in situations where a developer has upgraded to a newer version of fluentmigrator, but the build ci/deploy cd servers are on older versions, and thus you get a runtime error due to not resolving dependencies.

I gave this a lot of thought over the years. Perhaps for 4.0 we not address it. Ideally we can analyze the deps. I believe .NET 6.0 has APIs to do this.

@mattbrailsford
Copy link
Contributor Author

mattbrailsford commented May 5, 2022

Not sure if there is anything you want me to action in there, or if it's just notes for your own review, but let me know if there is anything you need me to implement, otherwise I'll wait for a code review.

If based on your comments though this is already being tackled in another area, it would be good if some info could be placed somewhere as it's hard to track the status of the project and where to direct efforts to assist.

I wonder if some kanban board or something is needed to know the status of the next release and what is still required to be done and how anyone can help, and if anything is "up for grabs".

@jzabroski
Copy link
Collaborator

These are just notes for me to move into documentation/roadmap pages.

If based on your comments though this is already being tackled in another area, it would be good if some info could be placed somewhere as it's hard to track the status of the project and where to direct efforts to assist.

No, this PR is needed. The notes are just the "economic benefits" and what this would enable long-term, plus a few minor dependency issues I am aware of that are tangentially related.

@jzabroski
Copy link
Collaborator

jzabroski commented May 5, 2022

I wonder if some kanban board or something is needed to know the status of the next release and what is still required to be done and how anyone can help, and if anything is "up for grabs".

Something that would be "up for grabs" is looking at #1348 , in particular the spreadsheet I compiled, and taking all the Snowflake commits from the develop branch and getting them into master for 4.0. It's mostly busy work, but I have my hands full with work, running a company, two open source projects, kid on the way, etc. In most cases I know exactly what should be done with this project, it's just finding time to go do it.

@jzabroski jzabroski added this to the 4.0.0 milestone May 5, 2022
@jzabroski
Copy link
Collaborator

Additional note:

If we're going to break this apart, we need document how to load the SNI (Server Name Indication) package, or people are probably going to file time consuming bugs.

@jzabroski
Copy link
Collaborator

Spoke to Simon Cropp. He told me a lot of popular nuget packages, like Newtonsoft JSON and Serilog, lock the assembly version at the major version number, even though the package might use servicing releases in the SemVer. This will address #1211 mentioned above in this comment: #1600 (comment)

@fubar-coder Thoughts on this particular solution to the "reflection can't find MigrationAttribute" issue when dynamically loading assemblies via out-of-process runners?

@fubar-coder
Copy link
Member

@jzabroski This problem usually happens when you're referencing (and using) two different versions of the assembly containing the attribute, which means that the type information instance is different even though it's the same type/code (but from 2 different assemblies).

There are two possible workarounds:

  • Avoid loading both assembly assemblies by overriding the assembly resolver
  • Compare by type name (and namespace)

@jzabroski
Copy link
Collaborator

jzabroski commented May 9, 2022

@mattbrailsford So, here are my takeaways from the PR (for release notes, and possibly questions/remarks):

The following things are moving from FluentMigrator.Runner to FluentMigrator.Runner.Core

  1. Announcers

  2. ConnectionlessVersionLoader

  3. Migration Constraint API

  4. DefaultConventionSet

  5. DefaultMigrationInformationLoader

  6. Exceptions

    1. Exceptions/InvalidMigrationException
    2. Exceptions/MissingMigrationsException
    3. Exceptions/VersionOrderInvalidException
    4. Extensions/TagsExtensions
    5. Extensions/TypeExtensions
  7. Generators/MigrationGeneratorFactory

  8. IMaintenanceLoader

  9. IMigrationInformationLoader

  10. IMigrationRunner

  11. IMigrationScope

  12. IMigrationScopeManager

  13. IMigrationScopeStarter

  14. IProfileLoader

  15. AssemblyLoader/AssemblyLoaderFactory

  16. DefaultConnectionStringProvider

  17. Initialization Logic

    1. Initialization/IScanIn
    2. Initialization/IScanInBuilder
    3. Initialization/IScanInForBuilder
    4. Initialization/NetFramework/Initialization/AppConfigConnectionStringAccessorOptions
    5. Initialization/NetFramework/ConnectionStringManager
    6. Initialization/NetFramework/INetConfigManager
    7. Initialization/RunnerContext
    8. Initialization/TaskExecutor
  18. LegacyExtensions -> I'm not sure where this code is coming from.

  19. Logging

    1. Logging/FluentMigratorConsoleLogger
    2. Logging/FluentMigratorConsoleLoggerProvider
    3. Logging/IPasswordMaskUtility
    4. Logging/LegacyFluentMigratorLoggerProvider
    5. Logging/PasswordMaskUtility
  20. MaintenanceLoader

  21. MigrationConventions

  22. MigrationRunner

  23. MigrationRunnerBuilderExtensions

  24. MigrationProcessorFactoryProvider

  25. ProfilerLoader

  26. StopWatch

  27. TrackingMigrationScope

  28. TransactionalMigrationScope

  29. DefaultVersionTableMetaData

  30. Versioning API

    1. IVersionLoader
    2. VersionLoader
    3. Versioning/IVersionInfo
    4. Versioning/VersionInfo
    5. Versioning/VersionMigration

I'm not finished reviewing even the first commit yet, but these are my notes so I can tackle it piecemeal.

@jzabroski
Copy link
Collaborator

Should wrap up reviewing this week. Got buried with life and work.

@@ -45,6 +46,9 @@ public static IMigrationRunnerBuilder AddSqlServer(this IMigrationRunnerBuilder
.AddScoped<IMigrationProcessor>(sp => sp.GetRequiredService<SqlServer2016Processor>())
.AddScoped<SqlServer2016Generator>()
.AddScoped<IMigrationGenerator>(sp => sp.GetRequiredService<SqlServer2016Generator>());

MigrationProcessorFactoryProvider.Register(new SqlServerProcessorFactory());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these should be unneeded. MigrationProcessorFactoryProvider has been marked as obsolete for some time now, and it really duplicates what DI can do.

@@ -44,38 +32,7 @@ public class MigrationProcessorFactoryProvider

[Obsolete]
static MigrationProcessorFactoryProvider()
{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should just delete this class and ConnectionlessProcessorFactoryProvider - need to see what that impacts.

/// <param name="runnerConventions">The runner conventions used to identify a version table metadata type</param>
/// <param name="runnerContext">The runner context defining the search boundaries for the custom version table metadata type</param>
/// <returns>A custom or the default version table metadata instance</returns>
public static Type GetVersionTableMetaDataType(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these "LegacyExtensions"? Shouldn't there be a section explaining what to use? Shouldn't this be marked as Obsolete if its Legacy?

/// <param name="assemblies">The assembly collection</param>
/// <param name="serviceProvider">The service provider to get all the required services from</param>
/// <returns>A custom or the default version table metadata instance</returns>
public static Type GetVersionTableMetaDataType(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these "LegacyExtensions"? Shouldn't there be a section explaining what to use? Shouldn't this be marked as Obsolete if its Legacy?

/// <param name="connectionStringProvider">The connection string provider</param>
/// <param name="runnerContext">The runner context</param>
/// <returns>The found connection string</returns>
public static string LoadConnectionString(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these "LegacyExtensions"? Shouldn't there be a section explaining what to use? Shouldn't this be marked as Obsolete if its Legacy?

/// <param name="loaderFactory">The factory to create an <see cref="IAssemblyLoader"/> for a given assembly (file) name</param>
/// <param name="assemblyNames">The assembly (file) names</param>
/// <returns>The collection of assemblies that could be loaded</returns>
public static IEnumerable<Assembly> GetTargetAssemblies(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these "LegacyExtensions"? Shouldn't there be a section explaining what to use? Shouldn't this be marked as Obsolete if its Legacy?

/// </summary>
/// <param name="assemblies">The assemblies to get the exported types from</param>
/// <returns>The exported types</returns>
public static IReadOnlyCollection<Type> GetExportedTypes(this IEnumerable<Assembly> assemblies)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these "LegacyExtensions"? Shouldn't there be a section explaining what to use? Shouldn't this be marked as Obsolete if its Legacy?

@jzabroski
Copy link
Collaborator

@mattbrailsford OK, I think my review is pretty much done. Please take a look - it's pretty minor stuff I think, but just "fit and finish" polish before we release.

@mattbrailsford
Copy link
Contributor Author

@jzabroski just a couple of things

  1. Most of your review is really code that existed prior to this PR (I just moved it) so by changing that stuff it kinda expands the scope of this PR. I understand those things could do with resolving, but isn't that more of an overall project/v4 concern than a blocker for this PR?

  2. If we do address these issues, how far down that rabbit hole do you want to go? as removing MigrationProcessorFactoryProvider effectively means removing the whole legacy API no? If that is the case, again, that feels out of scope for this PR, and personally I'm not sure I know the project well enough to be confortable removing all that code as I'm not that sure where / how it is all used.

@jzabroski
Copy link
Collaborator

@mattbrailsford I see your point of view, and acknowledge your POV makes sense.

@jzabroski jzabroski merged commit e40e221 into fluentmigrator:master Jun 7, 2022
@jzabroski
Copy link
Collaborator

@mattbrailsford I'll start drafting the release notes and will target 4.0 release this weekend. I don't want to drag it out seeking perfection. Plus, decoupling the runner is a big accomplishment

@danielkraut
Copy link

@mattbrailsford I'll start drafting the release notes and will target 4.0 release this weekend. I don't want to drag it out seeking perfection. Plus, decoupling the runner is a big accomplishment

Hi, what is the current state and do you have a plan when we can expect the 4.0 release? I know you are probably busy but the latest comment about the release is from June 7th.

@jzabroski
Copy link
Collaborator

I need to release it. I just spent last weekend fixing fluentmigrator/documentation in prep for 4.0 release.

@mazuryv
Copy link

mazuryv commented Jun 1, 2023

Hello.
What is the current state and do you have a plan when we can expect the 4.0 release?
Probably, will you add these changes to the next release with version 3.X?
This is strongly required to use FluentMigrator.Runner without all DB dependencies.
Thank you.

@orientalpers
Copy link

I am waiting too

@robertcoltheart
Copy link

I got tired of waiting and made this: https://www.nuget.org/packages/FluentMigratorBridge which lets you use just a single database runner instead of referencing them all.

@dariogriffo
Copy link

I got tired of waiting and made this: https://www.nuget.org/packages/FluentMigratorBridge which lets you use just a single database runner instead of referencing them all.

Is this a fork of 3.3.2?

@robertcoltheart
Copy link

No, but it is compatible with FluentMigrator 3.3.2. It's a library that allows you to create migrations and a runner with just a single database, instead of pulling in all the libraries for all databases. Instructions are on the github repo.

@dariogriffo
Copy link

No, but it is compatible with FluentMigrator 3.3.2. It's a library that allows you to create migrations and a runner with just a single database, instead of pulling in all the libraries for all databases. Instructions are on the github repo.

Starred the repo, amazing job @robertcoltheart thank you so much. Worked like a charmed.
Below my git diff, didn't change a single line of code
image

@jzabroski
Copy link
Collaborator

I should be pushing 4.0.0 shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants