Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple the bundled modules to seperate Repo's #1979

Closed
kilasuit opened this issue Aug 20, 2016 · 30 comments
Closed

Decouple the bundled modules to seperate Repo's #1979

kilasuit opened this issue Aug 20, 2016 · 30 comments

Comments

@kilasuit
Copy link
Contributor

@kilasuit kilasuit commented Aug 20, 2016

Now that we have PowerShell Open Sourced we should really move to decouple all of the modules that are shipped "inbox" into their own repos so that updates to these can be make separately to the core engine and can also be pulled from the PowerShell Gallery

However this brings a number of additional challenges that warrants a detailed discussion and a plan of action as not to break the current build process.

Modules that should be decoupled and pulled in as submodules

  • - PowerShellGet Priority
  • - PackageManagement (this may already be out there in the OneGet Repo)
  • - Microsoft.PowerShell.ODataUtils
  • - PSReadline (already decoupled just needs to become a submodule)
  • - PowerShell.Archive (as above)
  • - Pester (as above but submodule to point to Pester not current submodule)
  • - CimCmdlets
  • - Microsoft.PowerShell.Diagnostics
  • - Microsoft.PowerShell.LocalAccounts
  • - Microsoft.PowerShell.Management
  • - Microsoft.PowerShell.Security
  • - Microsoft.WSMan.Management
  • - PowerShell.Utility
  • - PSDiagnostics
  • - PSScheduledJob
  • - PSWorkflow
  • - PSWorkflowUtility

This is especially more important as in the differing subfolders in https://github.com/PowerShell/PowerShell/tree/master/src/Modules there are a number of code duplications due to the modules residing in multiple folders which is rather unnecessary and adds unneeded complexity for management of this repository going forward.

@lzybkr

This comment has been minimized.

Copy link
Member

@lzybkr lzybkr commented Aug 20, 2016

We have definite plans to do this for some of the modules like PackageManagement, PowerShellGet, Pester, and PSReadline.

Utility and Management might not be worth the hassle because of implicit or explicit circular dependencies.

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Aug 20, 2016

I thought it may have been "on the plans" but couldn't see an issue for it so felt it best to raise one for visibility

@Carringguns

This comment has been minimized.

Copy link

@Carringguns Carringguns commented Aug 20, 2016

Thanks im not free and clear yet. Little cleanup still. But want go to the source and offer you the data. and its running state im keeping in a loop.

@Carringguns

This comment has been minimized.

Copy link

@Carringguns Carringguns commented Aug 20, 2016

Its design to eexplore when mission complete. Love this one. Better than most ive seen

@vors

This comment has been minimized.

Copy link
Collaborator

@vors vors commented Sep 2, 2016

The current proposition is the following:
Make sure that upstream version is suitable as a drop-in replacement.
If not, massage it to the point, when it is.

Replace code by a git submodule.

cc @andschwa @lzybkr
cc @dlwyatt (for Pester)

@andschwa

This comment has been minimized.

Copy link
Member

@andschwa andschwa commented Sep 2, 2016

Oh yeah. I can help save the history for PSReadLine (and fix incoming changes on the upstream repo).

@vors

This comment has been minimized.

Copy link
Collaborator

@vors vors commented Sep 2, 2016

@andschwa sweet, then I will assume you will take care about PSReadLine

@andschwa

This comment has been minimized.

Copy link
Member

@andschwa andschwa commented Sep 2, 2016

Okay, sounds good.

@SteveL-MSFT

This comment has been minimized.

Copy link
Member

@SteveL-MSFT SteveL-MSFT commented Sep 7, 2016

ODataUtils should be split out as well

@dragonwolf83

This comment has been minimized.

Copy link

@dragonwolf83 dragonwolf83 commented Sep 8, 2016

It is probably a bit late to change it, but Is there a better way to organize all the repos under https://github.com/PowerShell to handle splitting the code base more?

It makes perfect since to break out the modules, but it is a bit messy right now with all of the individual DSC repos + Documentation + RFC. There are 6 pages of repos already. I can imagine a dozen more springing up to clean up the code base and dozens more for new things.

Something to keep in mind for those of us watching the Repos! A landing page dedicated to source would probably be enough of it is self-maintaining (or just maintained).

@vors vors modified the milestone: 6.0.0 Sep 8, 2016
@RamblingCookieMonster

This comment has been minimized.

Copy link

@RamblingCookieMonster RamblingCookieMonster commented Sep 10, 2016

@dragonwolf83 - Agreed, probably worth opening an issue for this, although not sure where you would do that, given that it's an org-level issue : )

Could go as simple as a readme that points to the various repos, with a little organization... or even something like GitHub Pages. Ideally, the community could submit PRs and issues, so IMHO a Microsoft hosted site (unless fed by CI/CD from GitHub) would be less appropriate.

Cheers!

@vors

This comment has been minimized.

Copy link
Collaborator

@vors vors commented Sep 10, 2016

A landing page dedicated to source would probably be enough of it is self-maintaining (or just maintained).

I don't think it's worth the effort to be honest.

This is a generically applicable statement for any big github organization.
Take https://github.com/Microsoft or https://github.com/google as examples.

How people are finding repos in these cases?
Trying to list all the repos is not practical. There are projects that only a handful of people are interested in. With modern search engines (and also GitHub search) it's easy to find a particular project or explore interesting projects.

Also https://github.com/dotnet has a github-pages hosted site for the org, but there is no list of all projects there.

@vors

This comment has been minimized.

Copy link
Collaborator

@vors vors commented Sep 10, 2016

Talking about lists, there are a lot of lists under "awesome" umbrella.

https://awesome.re
https://github.com/alebcay/awesome-shell

Maybe awesome-powershell would be something that community is interested in?

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Sep 12, 2016

I'll update the Issue to include a list of modules that I think should be decoupled where reasonably possible and then if we raise a new issue for each module then we can track it properly as well with tasklists to see the progress

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Sep 12, 2016

@vors @lzybkr @andschwa @SteveL-MSFT - Please see the list of modules I think that need to be decoupled from this repository is now updated at the top of this issue in my initial comment

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Sep 29, 2016

@vors @lzybkr @andschwa @SteveL-MSFT - any further thoughts/comments to the suggested list at all?

@SteveL-MSFT

This comment has been minimized.

Copy link
Member

@SteveL-MSFT SteveL-MSFT commented Sep 29, 2016

@kilasuit thanks for taking the lead on this. the list looks good

@vors

This comment has been minimized.

Copy link
Collaborator

@vors vors commented Sep 29, 2016

I think we should be clear about the trade-offs and goals here.
Things like Pester must be separate because it's a separate project, it just happened to be very central for out testing and we are using our fork temporarely.

So essentially for anything that is makes sense to be shipped on the gallery and updated from the gallery, we can do the decoupling. I.e. PSReadLine.

Decoupling can reduce complexity, but it can also create it.
I don't see value in separating core modules like Microsoft.PowerShell.Management: would it be intended to update from the gallery? If I need to fix a bug that spans across SMA and Microsoft.PowerShell.Management in this decoupling, it would be way harder.

I think these module should stay in the repo:

  • PowerShell.Utility
  • Microsoft.PowerShell.Diagnostics
  • Microsoft.PowerShell.Management
  • Microsoft.PowerShell.Security
  • Microsoft.WSMan.Management
  • PSDiagnostics
  • CimCmdlets

These modules are for the Full CLR builds and hence again should stay in the repo

  • PSScheduledJob
  • PSWorkflow
  • PSWorkflowUtility
@SteveL-MSFT

This comment has been minimized.

Copy link
Member

@SteveL-MSFT SteveL-MSFT commented Sep 29, 2016

Microsoft.WSMan.Management and CimCmdlets should be separate since those two are related and likely used together should probably be in same repo.

@andschwa

This comment has been minimized.

Copy link
Member

@andschwa andschwa commented Oct 6, 2016

@vors you mention:

Replace code by a git submodule.

But we're currently under the impression of bringing most of these pieces in via packages (PSReadLine, PowerShellGet, and Pester at least). We want to reduce submodules. Is that okay with you?

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Oct 6, 2016

@andschwa as @lzybkr mentioned in #2405 pulling them in via packages is the right way long term as this will allow for a much better side by side story & could also make way for a better downlevel story too

@KirkMunro

This comment has been minimized.

Copy link
Contributor

@KirkMunro KirkMunro commented Oct 7, 2016

I generally agree with @vors about which modules should stay in the PowerShell/PowerShell repo.

My line in the sand to decide what stays vs what is submoduled would be whether or not module updates are already or may become later available to downlevel versions of PowerShell. If module updates do apply to downlevel versions of PowerShell, then use a submodule so that the module can be maintained independently as one that works for multiple versions of PowerShell, allowing for updates to be leveraged by all supported versions. If not, then keep it in the PowerShell/PowerShell repo because it is bound to that project.

PowerShellGet already has downlevel version support (although I don't think the way downlevel support is done in that module right now is the way to go, so there's some work to do there). PSScriptAnalyzer has downlevel version support. Microsoft.PowerShell.Archive is outside of PowerShell/PowerShell already, and currently requires version 5+, but it will have downlevel version support shortly once I submit my pull request. Other core modules listed above in @vors post should stay in the main project IMHO.

@lzybkr

This comment has been minimized.

Copy link
Member

@lzybkr lzybkr commented Oct 7, 2016

I see some value in breaking out all modules - e.g. the recent improvement to Join-Path in Microsoft.PowerShell.Management and Get-Credential in Microsoft.PowerShell.Security are both useful and shouldn't require updating to a newer version of PowerShell.

That said, decoupling a couple of the core modules has some risk, so I'd consider that lower priority.

@vors vors removed their assignment Nov 16, 2016
@vors

This comment has been minimized.

Copy link
Collaborator

@vors vors commented Nov 20, 2016

PSGet and OneGet are separated in #2711

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Mar 5, 2018

@joeyaiello - please have a look over this issue from 2016.

Is there any chance could we get some update/traction on this to make things easier/better going forward PowerShell 6.1 onwards

@iSazonov

This comment has been minimized.

Copy link
Collaborator

@iSazonov iSazonov commented Mar 6, 2018

The only thing left to complete is PSReadline.
I agree with @vors and don't see any benefit in moving out the rest of the modules. No one is actively developing these modules now. What prevents? Does someone have a big additions to (ex.) Odatautil in their repositories? Why can not commit here?

I would have seen more benefits in moving very popular cmdlets to this repository after they became stable.

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Jun 14, 2018

Bumping this for a review as PSReadline was recently decoupled and there is still a checklist (checked off PSReadline just now) that needs to be confirmed/denied if/when there are plans to do so

@iSazonov

This comment has been minimized.

Copy link
Collaborator

@iSazonov iSazonov commented Jun 14, 2018

ODataUtils and Pester is in separate repos.

@kilasuit

This comment has been minimized.

Copy link
Contributor Author

@kilasuit kilasuit commented Jun 14, 2018

updated the checkboxes - thanks @iSazonov

@iSazonov

This comment has been minimized.

Copy link
Collaborator

@iSazonov iSazonov commented Nov 21, 2018

I think we can close the issue. All works is done. The remaining modules will still be in this repository as discussed.
/cc @SteveL-MSFT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.