New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Eliminate plugins submodules #6605
Comments
+1 for this. We can then manage all Piwik plugins (Core + Premium) in unified way. |
+1 Instead of using composer we could also create a new console command to auto install all maintained plugins (which might contain premium plugins, that are ignored if there is no access given) |
+1 for composer. It's tool which is using in many PHP projects so I think that in someway becoming standard in PHP world. |
@sgiehl - on the other hand we don't want just anybody to download premium plugins I think. Also I rather ment those complete files rather for internal use of refactoring, testing, etc. |
Can you explain step by step what is to be done and which commands are to be executed when having to use a branch for plugins? And maybe also what is to be done when merging? To see whether it makes it actually easier to work with branches... |
Is there a reason why plugins in submodules are not in this repository? I get that they are optional plugins, but we could have them in piwik/piwik disabled by default. |
we don't need to bundle those plugins with core because they are optional. they are available via the Marketplace. Ultimately we would like to move more plugins to the Marketplace to keep core smaller, eg. #5341 |
Then why not removing the submodules and let developers checkout those plugins manually (or with a command or script)? PhpStorm can support all git repositories inside the project (e.g. I update with PhpStorm and it updates every plugin). The submodules brings no value except the initial installation. |
it will be error prone to clone all the repos.... maybe we could make it automatic? eg. maybe a new console command could init all repositories that the developer has access to and that we need to maintain as part of core? |
For me removing submodules is a high priority because it's pissing me off every single day. Every time I pull master submodules are in detached state and I have to re-checkout master on most of them so that PhpStorm works again. It also messes up git rebase, and sometimes when I switch branch I end up committing submodule updates when I should not. If we need to clone 8 repositories, that's doable. Here is a script:
There would still be left ui-tests and the php tracker but at least that's a lot of time saved already. And if we need to make sure those plugins are not broken, we don't need to check them out locally: we should use CI. #6542 Sounds like the solution. |
feeling the pain 👍
maybe we have to clone using SSH not the HTTPS otherwise it won't use the SSH key (when two factor enabled?) |
|
maybe we can create some command like |
The reason why we had choosen submodules was to automatically include those optional plugins in every dev environment. I'm still thinking that it is necessary to do that. Otherwise no one would include them manually, even if it would be easy doable with console command. Another question why did we start to move optional and required parts into submodules? I don't see so many problems with submodules that you probably do. Small hint: To get all submodules to latest master I simply use: |
for anyone who uses the Phpstorm then it can be configured to "git pull" all the repositories that Phpstorm finds in the project. So everyone will use the master branch of all plugins. If they don't use Phpstorm, maybe they could use the console
only the PiwikTracker submodule is actually required... there was no other easy way to stay BC while moving it to its own repository. See discussion in #6870 and matomo-org/matomo-php-tracker#1
definitely we'll do this whenever possible
it's true that the FAQ about Git deploy mentions |
We have to keep in mind there are "contributors" and "contributors". People working everyday on Piwik will check out those plugins, as well as other plugins too. |
Hm. I'm not sure about that. |
We can still have some sort of version dependencies for plugins (e.g. using composer). I have to agree with what is said above that submodules cause more pain than good. |
Can we use composer to require optional plugins? I don't think that is possible... |
Following a discussion today, here is a suggestion: replacing submodules with git subtrees (≠ the whole process described in #6757) We would still keep the current advantages of submodules:
We would gain:
Cons:
To sum up, it would be just like if those plugins are inside Thoughts? |
+1 for using git subtrees for open source plugins that are released in the marketplace. I would git-tree split all maybe others have more thoughts? |
I have done some tests with git subtrees and the script I mentioned, it worked as expected. You can see an example here:
I'm suggesting to try that with one of our plugins that we want always installed, for example To make this work we would need a server to setup a github hook that would sync the subtree on each push: @mattab I guess that's possible? We would also need to update the package script to remove the subtrees when doing a release. That's doable too. The only problem I've identified for now is that branches in piwik/piwik would be synced to plugins, but never deleted when they are deleted from piwik/piwik. This is an issue identified in dflydev/git-subsplit#6 and I'm sure we could implement that. All good for everybody? |
Oh one thing I just think about now: that would also synchronize the versions. So plugins setup as git subtrees would have the same versioning as Piwik itself :-/ I don't know if that's a good thing… |
We have this already for git diff emails, so no problem
That's not really a good thing since we want to manage and release them independently, there are less often actually changes so version number would less often increase but sometimes we want to release a new version before Piwik etc. |
A little summary:
|
We cannot get rid of submodules at this stage. We're considering automating tracking of submodules to latest commit, see issue #8425 |
Maybe we could get rid of submodules using the upcoming open source tool described here: https://speakerdeck.com/fabpot/a-monorepo-vs-manyrepos ? |
We likely won't get rid of submodules and if/when we do, we can reopen this issue or a new one |
The goal of this issue is to investigate our uses of sub-modules and ultimately get rid of them to use a better way instead.
What do we use submodules for?
SM or submodules are a feature of git that we use at Piwik to keep track of
Why getting rid of SM is important?
Submodules are quite painful to use by themselves and especially with branches. This conflicts with our project to use a always-green
master
branch and adevelop
branch #5954 - because we need a master green at all times,Idea: replacing submodules with Composer
What if we would simply use Composer tool instead of submodules?
As a developer, maybe Composer could help me checkout all the core plugin repositories (they can be put in the
composer.json
sectionrequire-dev
). Each of those plugins eg. inplugins/CustomAlerts
would be a clone of the git repo (see this SO answer).Managing branches across many repositories
Using phpstorm I can easily add the repos to Settings>Git and then:
Other ideas/brain dump
tests/PHPUnit/UI
submodule in another issue (out of scope?)What do you think of this?
RFC @tsteur @diosmosis @mgazdzik @mnapoli @vox3r @czolnowski @sgiehl @julienmoumne @halfdan @quba @tzi
The text was updated successfully, but these errors were encountered: