-
Notifications
You must be signed in to change notification settings - Fork 23
Changing the Umbraco Package format. #22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This paragraph exists (and belongs) in the Umbraco Schema section
Feature should be more fleshed out if we don't want any feedback on it. We can add ideas Out of Scope section that go beyond what we want to achieve in the first iteration.
Package format review
This paragraph exists (and belongs) in the Umbraco Schema section
Feature should be more fleshed out if we don't want any feedback on it. We can add ideas Out of Scope section that go beyond what we want to achieve in the first iteration.
…into package-format
Co-Authored-By: Shannon Deminick <sdeminick@gmail.com>
I think the default option will not be to upload to Our. The default option will be: where can we find your package on nuget.org. I don't see a big problem for CI / projects, custom feeds are a healthy and relatively common practice for many companies. |
@nul800sebastiaan it means back office will have to look into 2 feeds, which could be painful from perspective of waiting for an answer from each NuGet feed.
I don't really see any good reasons for that, but maybe there is something, but here is a question if HQ has resources to review all uploaded packages? As of NuGet.Or is public and everyone is aware it could contain packages which you should probably review first when Umbraco one will be read as a safer one and people will install packages without even thinking if they should review package.
Each Custom feed will be asked for a package - what actually slows down restoring packages |
As far as I understand the backoffice only gets data from the Our feed, which might proxy data from nuget.org.
That's not the point at the moment, the point is that we can remove unwanted packages from the backoffice feed (spam, security issues, etc). |
@nul800sebastiaan it is why I was flagging that it should be just aggregation based on reverse dependency, which could be easily filtered out instead of having own feed which will have to be review from the perspective of things which are automatically checked on nuget.org and only thing which we would have to do on Our is made aggregation with queue to approve to avoid spam packages, with proxy we could have similar approach, but if we are making own feed we have that review fully on HQ and maybe some community members. |
Just to note, Azure Devops feeds add upstream sources by default to minimize the lookup costs of multiple feeds. |
Didn't know that, but Visual studio or RIder don't do that anyway :) |
One comment about a custom feed on Our - in some cases may be more friendly to community developer members on Our that do not have or want an account on Nuget as they wouldn't require to setup a separate Nuget account in order to publish an Umbraco package. Whether that is relevant or not, I have no idea since I'm unsure how many of those people there might be but thought it was worth adding this note. |
From a business POV, there is a massive monetization opportunity for Umbraco to move to an "app store" for plugins. A store might bring conflict with a NuGet package solution for plugins. Is there a plan to move to a store in the future ? |
I don't think so it will be a problem, as NuGet packages are licenced on whatever you want, also look on WordPress in official store you could have only "Free" packages, but somehow people figure out how to sell packages in GPL madness. I think in case of Nuget system if it would be aggregation would be still possible to make store with free/premium packages.
@Shazwazza In theory, we could create "Umbraco Community" account and allow people to deploy using our as deployment system, but this way it will block the possibility of CI, but also I would still ask why I should trust Umbraco Feed when I couldn't trust official NuGet feed? |
It does matter because of the responsibility involved. If it's a simple feed where everybody push packages and is responsible for their own package then it's fine. HQ doesn't have to run tests for packages and people will figure out ways to monetize. If it's a store then it might affect the architecture & HQ has a responsibility to test the packages I guess ? Wordress is a good example. They do review the plugins: https://developer.wordpress.org/plugins/wordpress-org/plugin-developer-faq/ - I would say it's a store. I am not a lawyer but I would believe that the responsibility is shared between HQ and the developer / team in charge of the plugin. Also they look at the source code. Same for App store and Google Play. So, is the umbraco package uploaded to a store or to a simple feed ? |
@giuunit I think you made little confusion here:
They do but I would actually argue about the quality of that review, as I was working 8 years with Wordpress and most of the plugins could cause performance degradation, security issue etc. And Wordpress HQ not getting responsibility at all for plugins in their store(Check that point of guideline: Here) And actually plugin store as you said it is actually simple feed not real app store :).
They actually don't look into source code, as I saw already packages with obfuscation of code and they pass review, which shouldn't happen. |
I see. I think it's best if I leave it to the pro :) best of luck with that feature. |
Going over the RFC I really question if using NuGet packages for package distribution outside the context of a development environment is a good idea. I think some of the previous posts already touch on some of the troubles we could be expecting and introducing in the package installation process. I believe we should keep relying on the developers to manage NuGet (package) instead of trying to pull it into Umbraco. I think time used on "getting out of compatibility hell" is time wasted which could be used to make Umbraco's package ecosystem better. As a package developer myself I feel that, while it's nice that I only have to create one type of package file, implementing the suggested changes only feels like a very minor convienence gain for the package creator and none at all to the package consumer: there are currently already possibilities to generate the Umbraco package.zip file in an automated manner. I believe the focus should be on enhancing the package creation-and consumption process and focus less on the underlying technology. I think it is true to say most developers (including me) prefer and default to NuGet packages for distribution, but I never really considered creating a package.zip file for our.umbraco a big issue. It's true that there is currently no way to create the 'vanilla' Umbraco zip package file easily outside of Umbraco, but this is only useful if you don't really have any Umbraco content/media/etc to include (otherwise why not just use the interface in the Umbraco installation). As far as I know the package.zip file is mainly used to upload your package to our.umbraco? If this is the case why not resolve the issue of building the package in/on our.umbraco? The vision of the future for Umbraco packages as I see it would be something like the following: Umbraco's package creation process should focus completely on the "front-end" part of the packaging process (much like it's already doing now): including the required plugin(s), and make sure the required content/doc types/data types/content apps/sections get included. This package should not contain binary files. If the package has no back-end code, this package should already be "complete" and ready to upload to our.umbraco.org. Also as far as the package itself goes, I believe the use of a zip file to include everything is more than sufficient. This way you could also simply unzip the package in your website so it would work right off the bat, whilst also being able to just upload it to our.umbraco. On the other hand I don't feel like adding the zip directly in your own installation doesn't really add much as I would encourage distribution through our.umbraco instead (if you didn't want to or couldn't unload the zip in your website folder). If the package also uses binary files, it should not be more difficult than simply adding the NuGet package alias and version number to your package. The NuGet package could then contain just the binary files. And if you want to have the complete installation experience from NuGet? Then simply add the package file generated by Umbraco and have the NuGet package place it in a dedicated /app_data/packages/... folder so it can get picked up and installed by Umbraco. No hassle of having to figure out how to install the content for your package and writing tons of code.| Uploading a package to our.umbraco should simply be a matter of uploading the front-end component of your package. Including back end as well should be a matter of referencing a NuGet package. Perhaps the upload of a zip file could be supported for those that don't want to (or know how to) create a NuGet Package, but in this case each zip file could only target one specific version of Umbraco. If any NuGet package was referenced in the Umbraco package. our.umbraco could simply figure out the version of Umbraco you are running and checking dependencies for the Umbraco package to figure out if something will work or not. As Shannon mentioned, this might require NuGet package to always have Umbraco package dependencies to figure out compatibility. Although it could also be said that if there are no Umbraco dependencies in a package, there can be no Umbraco conflicts, thus making it a non-issue. I realise the following goes beyond the bounds of this RFC, but I think is should be a consideration as well: Keeping the front-end/Umbraco parts of the package together would also open the possibility of being able to preview packages (with obvious limitations) in your Umbraco installation without any risk of messing up your installation. As front-end packages are seperated from their back-end component, it should be easy for Umbraco to load up the "front-end" part of the package and give the user quick, easy live "preview" of what the package would feel like in their installation. Lastly to adress the uniformity of the installation experience: I believe the upload of a package file directly into an Umbraco installation (especially in a production environment) should be questioned, and I would dare say even discouraged as the installation can get really broken. Having the uploaded package be a NuGet package file would simply increase the potential troubles by that much more. I think Umbraco should encourage the installation of packages via the our.umbraco route in the backoffice, through NuGet (by developers), or perhaps simply being able to throw the package file generated by Umbraco into your website (for example a specific /app_data/ package folder, or unzipping if it's a zip file). |
I would say only smart idea is to use one package format, as it removes the complexity of solution as NuGet expose big part of Apis which will simplify the implementation of the package system in Umbraco
Why the change in package system should have an influence on package consumer? we changing something that is underhood not something that is visible for end user.
I think it should be a separate discussion as that rfc is about the logic behind, I would also make a lot of comments how to make consuming packages better
Actually I am seeing that as big issue, as has a lot limitation. Sure we could implement those features, but in my opinion, it is a waste of time to implement something that we could easily get from a switch of package format
But what if your package uses files of another package?
Umbraco sadly is not really easy to extend using only front-end, it means a big part of packages actually contains binary files, and here is an issue with lack of dependency resolving, if my package is dependent on the package I have to attach files or force a user to install that another package. Your proposition to maintain zip file system is actually adding a lot complexity to the system. If we want to use NuGet on Our to solve binary packages that means we have 2 separate package systems in one cms.
I would say option of creation packages trough backoffice should be actually totally removed, it should be replaced with Content Importer and Exporter and should be possible to use that from package system, but that should be a separate module. |
Hey bielu, Thanks for your feedback! I agree that figuring out and creating Umbraco package.zip files is a big hassle. Not having to go through all that saves package developers a lot of headaches and most importantly time. So I'm certainly not advocating for keeping .zip file over just creating a single NuGet package file :) You mentioned the resolution of dependencies, in the case I described it would work exactly the same: package dependencies would be declared in the NuGet package file. The package restoration also feels shady since you can't really know the NuGet package dependencies the current installation has. And what about package install/uninstall scripts (for example the inclusion of powershell scripts)? And as mentioned in the RFC, once a package is to be uninstalled again, it's impossible to roll back to the original files again (especially if multiple packages reference the same packages). Lastly the RFC states that developers would have the option to manually uninstall packages. How exactly would that take place? The situation with forcing the use of NuGet packages into Umbraco still feels like trying to force a square peg into a round hole. |
Here we are going to point which I was repeating long time for WordPress and now for Umbraco, you should block the installation of packages on a solution where you are using CI/CD :) |
I agree it doesn't make any sense to allow for installation of a package in such a scenario. Although in that case the Umbraco installation should (in theory) know which (binary) files are present in the website (the UmbracoCms NuGet Package + installed Umbraco packages). In that case, shouldn't it suffice for the website to send a package download request to our.umbraco which includes the Umbraco version, the package that is to be installed and other currently installed Umbraco packages? our.umbraco could then simply serve exact files as required, sending the package files and missing dependency files. This would also eliminate the issues related to having uninstallation leftovers as was mentioned in the RFC, since it's known when a file has no dependencies. The only thing still on my mind is why the burden of figuring out package requirements should be placed on the installation itself? As you stated if package installation is only allowed in a non Ci/CD scenario, the only way to install these packages is through our.umbraco. And our.umbraco has exact knowledge on the NuGet package and it's dependencies. If Umbraco ever changed it's mind on the package format they would only have to make changes in our.umbraco instead of having to consider backward-compatibility and having a lot of sites out there that are trying crawl a NuGet feed for (binary) files. I think leaving the Umbraco installation agnostic on the package format for developers is an important consideration. |
Just a quick comment (link, really) regarding schema. :) Would be really nice to do as linked below, with a GUI to opt in changes like described in the RFC. |
In case you hadn't heard already... the Packages team is hosting a virtual discussion to discuss this RFC on Wednesday 18th March at 12:00 GMT / 13:00 CET If you would like to join and haven't signed up already then please do so here: |
I will add a few additional comments here:
|
After looking at the RFC I feel that there are some things that I totally agree on and some things that would benefit from some tweaks. First of all: The idea of having one package that would work both for installation in the backoffice and as NuGet-reference in the VS-Projects is the dream scenario that we should aim for and it would be a great improvement. I can see that this comes with challenges, mostly around dependencies and specially around different versions of dependencies (ie: Would the backoffice-installer create "dependentAssembly-entries in web.config if needed?"). This is a big challenge and I think that this is because we're aiming to support two very different kind of usage patterns: 1. Backoffice installations for someone that is maybe not that technical and 2. NuGet installs for a developer in VS. Maybe we need to think more about questions like "Who is installing the package?" and "Who do we want to be ablet o install packages?". A experienced developer or someone trying to get started with Umbraco? What experience level are this person expected to have? In my case we would never have a editor install a package or install it in a stage/prod environment. We would always prefer NuGet over installing in the backoffice. But I can also remember "back in the days" when I tried to install different blog/e-commerce-packages etc. that created lots of doc types, data types etc. But at the end of the day, maybe this is different things? Maybe there is different approaches to installing packages that handle binary dependencies and other packages that "just" creates schema/content? A package that changes dependencies might contain the same schema/content-stuff but would be installed in a "safer" way ie. only via Nuget? This way we would not risk "crashing" a working site by putting dlls in the bin that are not compatible with existing ddls/configuration. Reg. the installation-process of the package the RFC states that the installation should be completed by "looking up" the package-section and continue the installation, I would vote for something more "intrusive" here ie. some sort of dialog the first time the site starts after a new package install (only if needed, some package might not need this). This dialog would benefit from a strict process where each "action/thing/change/migration-step" or what we call it should be visible to the end user, maybe some steps might be optional but I think that there is a clear need for mandatory steps during this process. Here we also have the opportunity to show very fine-grained process information and also clean error messages ie. if some permissions-settings are wrong or if something goes wrong with creating db-tables. I think that his part is where we can add lots of value by making "package migrations apis" that are very easy to implement and that also provides good visibility for to installing part about that is happening inside the solution, ie. uSync does this is a good way when importing changes between environments. I guess that no matter if you install the package from NuGet or via the Backoffice the process should look the same and be triggered by ie. a restart of the site. I guess this would be by placing this "meta data file" from the nuget-package somewhere in a "pending-package-install"-folder and trigger these during the startup. So from my point of view: NuGet package-format with some sort of meta-data about Umbraco-specific things and a very clean/strict/extensible installation pipeline that is also visible during installation (and logged) is a great improvement. Another thing also worth mentioning is that I think this "new package format"-process needs to be done "right" rather than fast, the current way to install packages actually works and the risk with putting something out that is not really well crafted would be that people skip to use the new APIs and just go with the old migrations, run code on startup etc. I guess that this project going forward would benefit from some sort of "compilation" of what needs most packages have today and what kind schema/content etc. that they need to be able to create during the installation and from there draw some lines reg. what a package is and is not. Also: If there should be a possibility to include stuff to a Package in the backoffice one should be able to choose what to include. But here I’m a little bit hesitant, are there any ”good” or ”used” packages created using the backoffice today? Automated building is use by lost popular packages. Its a great thing To have it in the backoffice but does it really promote quality packages? |
By popular demand we held a virtual discussion for this RFC last week. I took some notes throughout that I will share here, and you are also very welcome to watch the video recording RFC discussionParticipants
Topic: Do we need to unify the package format at all?
Topic: Installing from the backoffice
Topic: Uninstalling a pkg
Topic: Package migrations
Topic: Informing of new package updates
Thanks to all who participated! |
Just adding a thought that fell out of the notes. :P IMO, our / the backoffice could keep a list of nuget IDs instead of the full package schema thing that's there today. That would make it possible to make a "dependencyless" and "dll-less" package as a nuget package without even depending on Umbraco. Upload it to nuget, and add the ID to the "our" list of compatible packages. For private feeds, the ID could come from a similar table in the Umbraco installation. (db, json, whatever) That way, the process to "upload" something to the package repo is "just" adding the ID. It might even trigger some moderation process. For each ID, our/Umbraco could fetch the metadata and figure the type of package like discussed for the alerts needed for installing in the backoffice. When installing from the backoffice, the "engine" could look for a package.config in the file system hierarchy. If it's there, the installer could ask if you want to include the package as a dependency. When "only" a website with "dependencyless" packages - like non-dev cloud sites - I think there would need be some other mechanism to keep a record of which packages are installed. Maybe a package.config file that's stored somewhere else than root. However, this whole thing needs to be considered anew when moving to DotNet Core since package.config is gone and we're now working with PackageReference elements in the csproj files. However, that could be subject of the same discovery / approval mentioned, and would even be easier since it's now in the website folder and not the package.config file at root. #KUTGW #H5YR |
It could be easily handled by us... We should have information about installed packages it means we will be aware which version of newtonsoft json we are using and we could give to user choose install and use Redirecting Assembly Versions to solved that with newest version and it should be responsibility of User not Umbraco to decide....
Package system doesn't have righ to exists in CI/CD enviroments as all setup will go from local->something->something-> production, where you don't want check all envs if something new was added it means should be possibility for disable of package system. For Umbraco cloud everything is git based it means it should be fine to work like that:
But what about case 1 and 2 if packages using different versions of same JS library etc? This is why responsibility should be on user which install, exactly same thing Wordpress is doing and somehow it works...
AGAIN PACKAGE CONFIG is dead shouldn't be used, I think I have idea for that solving issue with packages dependencies but need find time to look into that. |
@bielu I'm not sure, but it seems you are missing the point about installing via backoffice for users who don't have a csproj file, and don't use Visual Studio / CLI tools. You are of course mostly right for developers who use a "proper development flow". |
It shouldn't be done for v8... it is huge breaking change
I am not missing that I am almost sure you could move package references to separate file and I have to test :) |
@bielu Mind that the most basic install of Umbraco does not have a csproj file nor any build process at all. Package dlls are just copied into the bin folder on install. That's the reason for the "simple package / no dependencies" thing. |
Thanks for all the input on this topic, I don't think it's an overstatement to say this RFC has grown a bit stale 😉 Thanks again H5YR 🙌 |
This RFC is for updating the package format in Umbraco to be centered around NuGet.
Read the RFC here and discuss by adding comments below 🙌