NuGet publishing

[Piedone]

Publish our OSS packages to NuGet. Individual issues for the same: Lombiq/Helpful-Extensions#34, Lombiq/Orchard-Azure-Application-Insights#15, Lombiq/UI-Testing-Toolbox#98

First steps:

• Set up the NuGet publishing with GitHub Actions of https://github.com/Lombiq/Helpful-Libraries and https://github.com/Lombiq/Helpful-Extensions. These will then at a later point serve as models for the rest.
• These two projects are special, and thus cover all important use cases, because Helpful Extensions depends on Helpful Libraries, and Helpful Libraries contains three csprojs that all need to be published (either as separate packages or one; perhaps the former because other projects may not depend on all of them).
• We're developing such OSS projects as part of this OSOCE solution. In other projects of ours we use them as submodules too, so consuming and improving them is easier. We'll keep this, thus we need some way for this to not clash with NuGet references.

Some criteria to keep in mind:

• We can handle the whole thing from a build in OSOCE too somehow, it needn't be for each project, if it makes things easier.
• Currently, we have about 30 projects that we'd like to publish to NuGet. This already necessitates the ability to manage NuGet publishing somehow centrally in a DRY manner (apart from credentials that can be stored in GitHub secrets) but we'll likely have many more. So, a solution where we need to maintain a pipeline in each of these repositories would be an issue

[Piedone]

The projects used for this proof of concept are Helpful Libraries (HL), and Helpful Extensions (HE) which depends on Helpful Libraries.

So one of the questions is, how can we make HE use HL from NuGet when it itself is published as NuGet but from source if both projects are used in source form as git submodules in a solution. I think what I have here will work: #24 (note the HL and HE PRs too).

1. Enable "Shape Tracing Helpful Extensions - Lombiq Helpful Extensions".
2. Note that the helpfullibrariesstatus response header has the value "From source." which is coming from HL here: https://github.com/Lombiq/Helpful-Libraries/pull/94/files#diff-c30eac4ab06ff10218da9878abfc536aeb35b8f659a21c0bdfaf3dc5df21f303R18

Why this is curious is because HE only references a Release DLL of HL (i.e. not the source), to simulate it coming from a NuGet package, which would write "From DLL". This is what happens when we remove all HL-referencing project references from the root web project, i.e. the DLL that gets built into the output folder is the one directly contained by HE. See here: https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/compare/task/nuget-reference-override-poc...task/nuget-reference-override-poc-he-only?expand=1

Conclusion: We can make the projects use the local source version of their dependencies if they're available (and built; for which we'd probably need to add HL references to the web project too) instead of NuGet references.

[deanmarcussen]

This works! some/most of the time.

It looks like it ends up depending on the build order, if one project references the nuget version and another is still referencing the source (using OSOCE). This would probably be resolved by referencing from the Web project, as you say, as it should be the last in the chain.
The eventual reference is important when packing as when it goes wrong the nuspec file ends up referencing an incorrect version.

This works also

  <ItemGroup Condition="'$(NuGetBuild)' != 'True'">
    <ProjectReference
      Include="..\..\Libraries\Lombiq.HelpfulLibraries\Lombiq.HelpfulLibraries\Lombiq.HelpfulLibraries.csproj"
      Condition="Exists('..\..\Libraries\Lombiq.HelpfulLibraries\Lombiq.HelpfulLibraries\Lombiq.HelpfulLibraries.csproj')" />
    <ProjectReference
      Include="..\..\Libraries\Lombiq.HelpfulLibraries\Lombiq.HelpfulLibraries.csproj"
      Condition="Exists('..\..\Libraries\Lombiq.HelpfulLibraries\Lombiq.HelpfulLibraries.csproj')" />
  </ItemGroup>

  <ItemGroup Condition="'$(NuGetBuild)' == 'True'">
    <PackageReference Include="Lombiq.HelpfulLibraries" Version="1.0.0" />
  </ItemGroup>

where the pack command is specified as dotnet pack /p:NuGetBuild=True -o ./artifacts

That maybe a better option, for clarity, and prevent an issue when releasing new versions, where the former would still try to restore the new version, (even though it would eventually, later in the build process, not use that version) which may not yet be published.

It will ultimately depend on which repository these are published / packed from.

It will probably be better to publish from each individuals repository, as sequencing could be an issue if they were published from OSOCE, the publishing / building of HE could have to wait until HL's new NuGet version had been published, and indexed by NuGet, thereby being available for download, during the build of HE. (This may also be resolvable, will only know with some test publishing).
That being said, it depends on whether you want to maintain different versions of them, or treat every NuGet package you have as being on the same version (i.e. in the way Orchard Core always produces a new version for everything). Perhaps it is simpler to have them all on the same version?
That would potentially mean bringing more projects/repos into OSOCE, as you said you have 30+ and that sln doesn't that many? Unless I misunderstood something there.

Either or will work, as they are very similar, so let me know your preference.

It would be helpful if you were able to create a cloudsmith account @Piedone for testing - I would prefer not to create one myself and accidently end up the owner of it.

The next step is to get some actual actions running across HL and HE and see what we can do to share the actions code.

Assuming I can get some shared actions, would HL be the appropriate repository to host those actions, or would you create another? It looks like it will need to be a public repository, as private action repositories are not yet supported by github.

I may need to add a .sln file to HL to have it automatically build all 3 projects, otherwise the build actions will end up being specific to those projects.

We could add a meta project to HL to reference them all, but the individual projects will still need to be published as well, so the meta has the correct references to them.

[0liver]

Exciting work! Keep it up 😃

[Piedone]

I like the conditional reference idea; I've understood it differently from your e-mail.

The OSOCE repo references about 20 repos (but those would be around 25 packages I think, based on the csprojs), the ones at least somehow related to Orchard, but we have our Hastlayer product with many more (actually, that would be more like 30 packages in itself). There are a couple of other independent projects too but those are less important for now.

I don't think we can realistically tie the release schedule of all projects used from OSOCE together since the level of activity is very different in them. Also, we might just want to publish quick bugfix release for a given project and don't want to basically re-publish the prior version for everything else. So this wouldn't work if publishing to NuGet from OSOCE would mean that we have to publish all projects at once. However, would that be really necessary? We'd version each project individually, so it might be simple to handle the case when e.g. we're about to publish v1.3 of HE when that already exists on NuGet. Publishing from OSOCE has the benefit that it can offer additional checks (like UI testing). (Later, for Hastlayer, the packages would actually have the same release cycle.)

I'd put common actions into a separate repository. I've created one, please fork it too: https://github.com/Lombiq/NuGet-Publishing-GitHub-Actions

Do you see any issue with publishing HL as three separate NuGet packages, and doing this at the same time if any of those changes? We have a similar release cycle mismatch there as what I've explained above.

I created two Cloudsmith repos and invited you: https://cloudsmith.io/~lombiq/repos/Helpful-Libraries/ and https://cloudsmith.io/~lombiq/repos/Helpful-Extensions/. The OSS rules require attribution, so please add this in your PRs (though we'll use them only temporarily): https://help.cloudsmith.io/docs/open-source-hosting-policy#attribution

[deanmarcussen]

I like the conditional reference idea; I've understood it differently from your e-mail.
👍

I don't think we can realistically tie the release schedule of all projects used from OSOCE together since the level of activity is very different in them. Also, we might just want to publish quick bugfix release for a given project

Cool, I suspected that to be the case. I'm going to suggest not using OSOCE at the moment for publishing, and publishing seperately. This is something that could change in the future, but for now makes it simpler not to go down that route.
In terms of build/test/publish, I tend to do them seperately, so build/test is triggered on prs, and pushes to main, publish triggered on tag push. So OSOCE can still be a basis for testing. You just choose not to push the tag on the submodule repo until the parents tests have passed.

Do you see any issue with publishing HL as three separate NuGet packages, and doing this at the same time if any of those changes? We have a similar release cycle mismatch there as what I've explained above.

They're going to need to be seperate anyway. NuGet packages tend to be on a per project basis, so to do it any other way requires hacking around in the .nuspec file to include multiple dlls. Event a .targets style .All metapackage is still just a reference to the 3 individual packages.

I'll try a couple of things, but here's a few ways I see this possibly working.

1. Tags. Tag based publishing is what I tend to use, so a tag with v prefixed means build and push to nuget (and pack with the v1.xx version number). So for HL, it could be prefix vmain-1.0.0 where main is the main library, vlinq-1.0.0 is the other etc, vall-1.0.0 could publish them all. (more room for confusion, but maybe better)
2. Version number commited to git. Where each project has it's version number held in the csproj, and those are incremented manually as required to publish seperately. Doing this we rely would build and push all, and rely on NuGet to fail the push when the version number has not changed. As NuGet is quite specific about not allowing a push when the version number hasn't changed. Still tag triggered, Think this might get more confusing than the value it offers.
3. Tags, v1.0.0, push all, increment all, even if code has not changed (which is what Orchard does). This is the simplest, and an option I like
4. Do the above (3), create a meta package, i.e. a csproj that references them all, and when consuming, just always consume that. So you're not mucking around figuring out which is the latest version all the time, just increment always. So you'd actually publish 4 packages, but consume one.

I created two Cloudsmith repos and invited you: https://cloudsmith.io/~lombiq/repos/Helpful-Libraries/ and https://cloudsmith.io/~lombiq/repos/Helpful-Extensions/. The OSS rules require attribution, so please add this in your PRs (though we'll use them only temporarily): https://help.cloudsmith.io/docs/open-source-hosting-policy#attribution
👍

[Piedone]

1. Only publishing a NuGet package on tag push is a great idea, let's do that then. We'll continue to use dev as the main development branch, so let's publish from there, but we may choose to await multiple dev commits until we want to release it on NuGet.
2. For HL publishing, and then later for similar "multiple csprojs/packages for the same logical project and repo" scenario let's use option 4, looks like the best one.

[deanmarcussen]

Initial prs with hopefully enough comments to make it clear on them.

Only suggestion is to just have one CloudSmith repo (and have a think about whether that should be a parameter the consumer is foreced to pass in, or just hardcoded in the actions...)

It's building on pr, push to dev atm (you may prefer it doesn't, but the builds are so quick it just seems worth it, to catch any unexpected errors).

Publish is on tag push (or github release tab will push a tag if you chose to use it)

[Piedone]

Great, thank you!

Renamed the HL repo on Cloudsmith: https://cloudsmith.io/~lombiq/repos/open-source-orchard-core-extensions Deleted the HE one.

[Piedone]

Let me know when you're done and ready for a new review.

[deanmarcussen]

I think done.

[Piedone]

Do I understand correctly that you're done?

[deanmarcussen]

Think I covered the last of your feedback this morning, so yes think ready for review again

[Piedone]

This shouldn't have closed itself really, keeping it open until we verified that everything works and we have all projects published.

[scleaver]

Will Lombiq.VueJs and Gulp.Extensions be added soon?

[Piedone]

These two are the trickiest (and the other projects using them). I'm not yet sure how and if we'll handle them because the NPM packages referenced in them need to be restored too. Loading something from NuGet that has a package.json which in

[deanmarcussen]

You could import the package.json as part of the nuget, as a content spec.

not sure how / if you can kick of an npm install during the nuget install process though.

[Piedone]

Yep, having the file there is doable but the rest seems tricky and/or really hackish. Gulp Extensions is pretty much an NPM package as well, so perhaps it should be as such too but I really don't want to go into that (perhaps only some local NPM package coming from NuGet-like trick).

[Piedone]

Apparently I didn't finish my reply to Seth. I wanted to add, that flat local files need to be referenced from Gulp Extensions from JS modules within the consumer project (namely, form a Gulpfile.js). I don't know how this can work well, I think you can only make those files end up in the build output folder (since you have to use the from JS, you can't used embedded files and such).

[deanmarcussen]

While it is possible, it's not ideal. Here's a blog post https://devblogs.microsoft.com/nuget/nuget-contentfiles-demystified/

The problem is that using content files in a nuspec means they are just copied in during the install process.
They are not maintained by NuGet afterwards. So you can delete / modify them etc

They do get recopied during an upgrade.

Not sure how you can do them without writing your own nuspec, i.e. from a csproj generated nuspec. Probably possible, but the last time I had to do this, was before we had csproj style nuspecs. Used to do it to copy linker .cs files in from Xamarin, which often needed further modification afterwards.

[Piedone]

Hmm, I used contentFiles before (Lombiq.Tests.UI has them) and to me it seems that while they appear as local files in the project from the Solution Explorer of VS, they are actually referencing the files in the package. I.e. if you want a path to them from outside of your .NET project (where you can reference them with a relative path from C#) you'll need to use the absolute path to the package in the NuGet package cache. They can be made copied to the output folder but then you have to reference the output folder which isn't really a good solution either.

[Piedone]

Found my soulmate: https://stackoverflow.com/questions/63262269/copy-files-into-project-on-nuget-package-install No solution, sadly.

[Piedone]

We can work around by committing the wwwroot folder to the repo to be able to publish projects that utilize Gulp Extensions. However, GE itself and the Vue module won't be usable from NuGet.

[Piedone]

This, and thus the first, important phase of NuGet publishing is now done. Publishing Gulp (or generally, Node)-using projects is not yet possible. We'll see what we can do there: #48

See here for the announcement: https://lombiq.com/blog/open-source-lombiq-projects-now-published-on-nuget