Shell we split the repo? #646
Comments
|
I don't mind splitting the repo between core INDI + 3rd party. It is going to make a few folks angry who currently depends on this structure to do everything (like build scripts). I say let's wait for more feedback from the community this is a huge undertaking. |
|
Exactly - this is just a first step and I am not quite sure it is a right step (vide Linux kernel). That is why I started the discussion. |
|
Splitting makes absolute sense. But I would go even further, since there are some drivers that belong to the core - like focuslynx - and others are 3rdparty. Does it really make sense having manufacturer specific drivers inside the core? |
|
In my opinion not really @sterne-jaeger. The change will help 3rd party driver authors if we can make the core stable and tightly controlled. I would be careful with that process - it puts some additional load on our BD (Benevolent Dictator @knro ) while removing some (maintaining some drivers). So I think it is roughly workload neutral for core maintainers. The huge is version skew and abandonware by the manufacturers. Monolithic structure is somewhat better in this respect - see, again, example of linux kernel drivers included in the mainline kernel tend to be better maintained. OTOH: we do not need to have it exclusive-or style. We can just create infrastructure for out-of-core drivers to enable authors to use additional stage of development. We kind-of have something like that already - the drivers which are not build by default. Maybe we should try to migrate those in the first place. Additionally, it works as an incentive for authors to make the drivers better. |
|
We had this with ASCOM a few versions ago and moved all the drivers out of the core platform build. I would definitely move the drivers to 3rdparty, could this be done piecemeal, as a driver is being actively maintained? It doesn't look like a massive job for an individual driver, using the support files for another driver as a pattern. But I'm approaching this as very much a newcomer to the indi/linux world and one thing I've learned is that the less you know of something the more you underestimate its difficulty. Chris |
|
Recent discussion from HN: |
|
Ok now that 1.7.6 is out of the way, I propose we split the repo into two:
This repo (indilib/indi) should contain libindi, while we move 3rdparty to its own indilib/3rdparty Only common part is cmake_modules, and we cannot use submodules because that breaks all the launchpad builds. So either duplicate them or some other solution. |
|
@knro I am starting work on the repo splitting. I'll try to do it as smoothly as possible but be prepared for at least some minor disturbances. |
|
@knro : I have a small issue with the name. 3rdparty seems to be too generic and disconnected from indi. Maybe indi-3rdparty would be better? I would appreciate your and others input on this. |
|
regarding cmake_modules: Would it be possible to insert subdir from other repo into the build on LP ? I thougt we have done it this way previously? On the other hand submodules are considered problematic anyway so I am thinking of using git-subtree for maintaining cmake_modules and maybe other parts of the repo. The main advantage is that it requires zero knowledge and effort from the developers down (or is it up?) the tree (single driver dev etc) and quite a little bookkeeping from the core developers. |
|
Absolutely, indi-3rdparty is perfectly fine. Regarding cmake_modules, I think each repo should have the ability to be built separately without required the existence of the other repo. So definitely not submodules, but no sure what else would be a good solution. Remember there would be a master.zip file for each of them so someone should be able to download the zip and get building right away. |
|
Ok. subtrees seems to be perfect for this since they are simply subdirs with full code residing inside the main repo. I will build everything in my fork (you can monitor the progress in project linked above).
|
|
Pawel, that's perfect!! No comments from me at this stage! |
|
No promisses! These are my goals. I think this can by done but only doing it will show if it works as intended. |
|
I think that I have a candidate for our setup: https://github.com/ingydotnet/git-subrepo |
|
My opinion: I don't like to learn a new tool and another dependency for my current workflow. We should have a standard git structure using regular git tool. |
|
@knro: fully agree, let’s stay with standard GIT. It’s already complex enough. |
|
@knro @sterne-jaeger : This is standard git. The repo will stay the same (except for 3rdparty moved to the separate repository). git-subrepo is my proposal to handle common code between the two (cmake_modules, maybe: CI and docker setup). Nobody will be forced to install or learn anything to maintain or use base indi or indi-3rdparty. The only time when someone will need to use anything except standard git is on setup (I will handle that) and when we want to push some change in cmake_modules/CI setup to common repo and pull back to 3rdparty. And this is just:
And this needs to be done by one person - for everybody else it is just a normal commit. Otherwise, we will end up with version skew and other problems with common code. I rejected submodules and subtree as candidate solutions exactly because they were too intrusive and complicated. |
|
I think the cmake modules are different for libindi and 3rdparty.. and I think they would have both different CIs as well, so maybe it won't be much code duplication for these two? |
|
I was busy last few days so I had to pause the work. The current state is fairly functional two repo system with working (and simplified) CI. There are two shared subrepos - cmake_modules and CI system. I would be all for simplifing it even further and subrepo system is extremely easy to use. And we can drop it at any moment without any side-effects since the subrepo is just a part of normal git tree which is also maintained outside - all it takes is to delete .gitrepo file from the directory (I have tested it). |
|
Ok, I guess we're down to two solutions:
I think it's easy to divide cmake_modules since we know what libindi requires and we can just migrate the rest to the 3rdparty's cmake_modules. Regarding MacOS, hopefully @rlancaste can share his opinion on this. I'll ask the other developers to voice their opinion on this before we move forward since it affects all. Thanks again for spending the time on this Pawel! |
|
@not7cd @TallFurryMan @mikefulb @rlancaste @sterne-jaeger @Chris-Rowland @jpaana @iliaplatone @d33psky Please share your thoughts on this before we make a final decision. |
|
I will still suggest going with 1) here - without any obligation for anyone (even @knro !) to use it. I will be doing initial sync and maintainment for this and for all the rest it is just part of the tree with one extra file inside. The person wishing to transfer some code between repos can do it manually or through the subrepo system - the logs will be different but the tree will be the same. It is important that the external cmake_modules repo is just a "meeting point" and it is not used in any moment during normal development. Both sub-trees are modified rather infrequently and CI/docker is mainly my responsibility anyway ;). So this is really up to cmake_modules - think about it @knro. But this turn out to be a minor issue - a realy important thing is our policy regarding driver development target. We should think about it carefully and make the decision public in the developers manual. |
|
Is release of mainstream branch dependent on 3rdparty? Because 3rdparty build is dependent on mainstream. If 3rdparty will be built against current mainstream, some errors might be introduced in 3rdparty build. |
|
We are talking about code duplication in the build code, right? If that’s the case I would opt for 2). I understand, 1) isn’t that complicated. Nevertheless it adds one small bit of complexity that I would avoid. But I could also live with 1) |
|
@not7cd: the infrastructure for unit testing (gtest/gmock) is included in the images. Its use depends only on driver developers. |
|
I don't have a strong opinion on this, the current repo works fine for me and I compile the whole thing so I'd need them all anyway. But having two places for drivers is a bit confusing and having one of them inside the library directory even more so. So I'd probably merge the drivers to one tree, maybe keeping the current split still. Division between 3rd-party and base drivers is a bit vague to me, though some that need a 3rd-party library are clear, but many of them don't seem to be any different to the ones under libindi. Having each driver in their own directory would be clear, but would also explode the amount of debian files if packaged separately. Which is probably why I have contributed my drivers under the libindi directory to avoid having to write them :) |
|
I've always disliked that in order to build all 3rd party drivers, you need to run cmake and make install twice. This repo restructure probably won't fix that but I hope it does. I agree we should improve the 3rdparty part of the tree. I do not know if moving it to its own repo is the answer, but it may very well be. On the other hand I fear that moving 3rdparty to its own repo will not help stabilizing things there as it may get even less attention. To me the choice whether a driver lives in base or in 3rdparty is confusing. Is it that these drivers require additional libraries ? Or is it a build order thing ? Or is it only a maintenance responsibility thing ? I do not like the latter. Though fancy I do not like to force people into using the git-subrepo system but if I read things right this is not really an issue because it's only on initial setup and on syncing up cmake_modules/CI which happens rarely and which jochym will handle and which can still be done manually if needed. And I'm of A split of cmake_modules also seems a fair choice to me. It might be incomplete with some code duplication which I think is no big deal. I have no idea if CI can be split as well. |
|
Ok, I don't want this to turn into what 3rd party drivers are for, we're way past that. But the original intentions for 3rd party drivers is to keep libindi as thin as possible and separate drivers that A. require external dependencies and/or B. require separate release cycle and/or C. has different license than libindi to be in their own semi-isolated place. For me, I don't have issues with either 1 or 2. A deal breaker would be if this breaks Launchpad, but if I understand this correctly, it shouldn't. |
|
@knro: I think @d33psky rose good points. Maybe we should take this opportunity to rethink the tree even deeper? Why we have two driver directories? Wouldn't it be better to have lib and drivers? This seems logical. From my point of view (CI/building and few drivers dev) the whole thing was always a bit too complicated and difficult to build. And it takes long time to build it all. For the single driver dev it makes the CI almost useless (that is why I developed single-driver compile system for CI). You know much better than me if it is possible to restructure the whole thing into lib+server and drivers. It seems logical and should make your life easier as well. How often you modify the libindi and indiserver over a single release cycle? My feeling is that we can make it into stable foundation against which all drivers are developed. This would define a stable API with some compatibility assurances for driver developers. What do you think? |
|
I seem that I am almost alone regarding the use of subrepo for the maintenance of the common directories. So I guess we will not make it into official workflow. I understand your points. I still disagree but that is fine. Using additional tools has some drawbacks. The problem is that it is the only solution for this problem without major deficiencies. If we go with some split I will probably ask for the management files in the tree (small .gitrepo files - one per subtree) for my convenience and stop talking about it for your convenience ;). It is probably possible to keep this local to just my fork, but I haven't tested it. |
|
I don't mind how the repo is organized, into one, two, or whatever. My biggest concern is that it is easy for a script or a user to build everything. I can always edit my various build scripts for OS X and Raspberry Pis etc if you guys decide a reorganization is in order. But whatever you decide, it should be logical and easy to understand. I recently made some craft recipes to build INDI and the 3rd Party drivers using craft. The truth as I found out rather quickly is that the INDI repo is very large and the way it is organized really affects how it builds in craft. d33psky mentioned that you have to build and install twice to get them to work. Yes that is very true because the libraries for the drivers need to be installed before the drivers get installed. I was having some trouble getting this to work nicely in craft so I tried creating recipes for each of the libraries (libsbig, libapogee, etc) so I could build them before I built 3rd Party. That wasn't actually too hard because each one was mostly the same except for a little edit on which folder it should build from. It also seemed logical because somebody might want to install the library but not install the INDI Driver. Who knows. But while this seemed like a good solution at first, I very quickly realized that something like 12-24 different recipes was very unwieldy, and then I realized that even worse, craft had to download INDI for every recipe which meant a lot of hard disk space and redundancy. So after talking with Hannah and Jasem, we came up with a better solution. I modified the 3rd Party cmakelists earlier this year so that you can use a build option to build the 3rd Party libraries and then you can later build the 3rd party drivers. This greatly improved things because then I just needed 3 recipes instead of dozens. So yeah, my main opinion is that whatever changes get made, please make them with this in mind. If the drivers get split into a whole bunch of different repos, then there needs to be an easy way to build them all together and each repo should not be too big with too many redundancies. If the drivers get split into just two repos INDI, and INDI3rdParty, then there needs to still be a 3rd Party build option to build the libraries like the one I added earlier this year, since the libraries are prerequisites for the 3rd Party drivers and it needs to be easy to tell it to build the libraries vs. building the drivers. My preference would be to have two repos, since that wouldn't really change my current build scripts at all, plus that would save on hard disk space because INDI wouldn't be duplicated in the download of the other repo. |
|
Excellent feedback everyone! If this is going to be broken down further, it could be as following:
This is going to increase the complexity a bit, but libindi does not get updated as often. |
|
The original reason that this was raised was because the repo was getting too big. But why? What is it that makes it imperative to build everything every time? Is it that the normal user install is to install everything, the core and all the drivers? My thought was that the indi core would be built separately but, at the interface level at least, would then change infrequently. Each driver would be built against the indi core interface and installed separately. Users would only install the drivers they needed, not everything. The 3rdParty section seems to indicate this is at least partly in place. A full build of the core and all drivers would be an infrequent thing, only needed if there was a breaking change to the core. Normally an individual driver would be modified, built and deployed independently of the core and independently of all the other drivers. But I suspect that I've misunderstood what is wanted, or what can be achieved. In practice the core and the drivers are so closely integrated that it's impractical to build one without building everything. Chris |
|
Well the reason to build everything at once in a monolithic unit is for distribution purposes. For OS X, we build everything for INDI and then bundle it into the app. So we want all of the INDI drivers, all of the 3rd Party drivers (that are supported on OS X), and just want to build all of it and bundle it up. Users expect that when they try to connect a device with INDI it will just work. They cannot build the different drivers they need independently because the end users are not building drivers at all. There is no method for us to build or install the different drivers independently on OS X unless we create some sort of installer. There is not a package manager like apt on OS X. There are things like homebrew and macports, but most end users will not have that installed. I have also written some Linux install scripts for Raspberry Pis etc. There again, we just tend to install all of INDI, not just the drivers they need so that it will work with whatever device they want to connect that is INDI supported. It is also important here because a number of the people who have started using Raspberry Pis to run their equipment are new to Linux (coming from either Windows or Mac) and don't know how to do a whole lot on the system. So running apt-get install for a bunch of drivers they need might be challenging for them coming from no command line experience. They just want it all to be installed. If the device is INDI supported, they expect it to work. |
|
I concur with @rlancaste, it's already a LOT of work to get things packaged and distributed to end users who just expect things to work and to have all their equipment supported. While it is possible to turn every core driver currently into its own driver (like 3rd party), it's a maintenance nightmare. Breaking this to libindi and 3rdparty should actually facilitate the build & distribution process. It can be further broken into libindi (library + server + tools) + indi-core drivers + indi-3rdparty drivers, but that's as low as it can get without creating an un-maintainable situation. I vote for libindi + 3rdparty unless folks have very strong opinion and want to further break it to libindi + indi-core + indi-3rdparty |
|
Either of the two Jasem just said are fine. The first one he said is simpler for the scripting and distribution because it isn’t much different from what it is now. The second option is fine too though. There is a third possibility that has not been mentioned yet that would actually make it simpler to install but possibly a little harder to maintain. Libindi in one repo, the INDI 3rd party libraries in another, and the INDI 3rd party drivers in a third. This is simpler to install/distribute because then all that code about installing twice or building the libraries first or building the libraries with a special build option could be deleted. You could just check if the libraries are installed and if not tell the user to install that package first. On the other hand it would make things harder to maintain because then the code for sbig or apogee would be in two different repos. I mean right now they are actually in two different folders in third party, but this puts them in a whole different repo. So I guess any of those 3 options are really fine with me, but I did want to throw that third one out there because some people are still confused about installing 3rd party twice |
|
I'm fine with a split into libindi + indi-3rdparty. About indi-3rdparty, we should put something like this in https://indilib.org/develop/developer-manual/100-3rd-party-drivers.html to replace the 3 points that are there now at the top of the page :
Some further remarks:
|
|
@jochym I think we're good to proceed with libindi + indi-3rdparty using either method. |
|
I have a small question about LX200 telescopes. I see there are variations of them, and that the structure of the driver forces a new implementation to be located in libindi/drivers/telescope. Any opinion on if new instances of drivers using lx200driver should go to a 3rd party repo or remain in core? |
|
@TallFurryMan They can be located either way, it mostly depends on other external libraries and if the maintainer wants them in libindi and ind-3rdparty @jochym Any update? |
|
Ok I'll go ahead and release 1.7.7 now so that this migration won't affect this release. |
|
Ok. Unfortunately I was unable to work on this. Sorry. Life... I'll try to
do it soon. It is better to release and then split, anyway.
P.
niedz., 14 kwi 2019, 17:42 użytkownik Jasem Mutlaq <notifications@github.com>
napisał:
… Ok I'll go ahead and release 1.7.7 now so that this migration won't affect
this release.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#646 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AFtzzgRsTGKfInG0KI15MFn0CQ74AiJ9ks5vg0xsgaJpZM4VKmKX>
.
|
|
Hello, any update on this? |
|
I am still alive but until start of october have no time at all...
I still think that splitting will be a good idea.
P.
…On Sun, 15 Sep 2019 at 09:42, Jasem Mutlaq ***@***.***> wrote:
Hello, any update on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#646?email_source=notifications&email_token=ABNXHTQ4QZTPTXIB3PRHOPLQJXRM7A5CNFSM4FJKMKL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XLGHY#issuecomment-531542815>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABNXHTQMZACUPNB5QQERAM3QJXRM7ANCNFSM4FJKMKLQ>
.
--
Paweł T. Jochym
|
|
Repo is now split. Travis CI is removed. I asked Circle-CI to grant us access to MacOS builds. They would be able to provide us 500 mins / month if I understand correctly. Circle CI scripts needs to be revised. I've hacked them a bit but if @jochym can take a look that would be great. |
|
Thanks Jasem. Now it's scramble time ;) |
|
Ok, so we got approval for the MacOS builds in Circle CI but I have no idea how to proceed. Also, I don't know how to make indi-3rdparty Circle-CI stuff work as well. Not sure how we can tell Circle CI to install libindi-dev from the repo (nightly), or if it can build it from Git before building the 3rdparty code itself. |
|
@knro I am really sorry but I have an extremely busy time at least to the end of the month. Thus I have literally no time to look at it. Life, unfortunately, and 24h/day ;) I'll try to find some time, but cannot promise anything at the moment. Keep me informed, please. I am still very much interested in the project. |
|
@jochym That's perfectly fine I understand. For now I've disabled the 3rdparty CI until we figure a solution out. |
|
Ok split is complete, so we should open another issue in the 3rdparty repo to fix the CI. |
The repo is going larger and larger with every single driver added. Is it time to split it the way other such projects are structured? I.E. core library/drivers or core library/core drivers/3rd party drivers. This would have multiple benefits. It is already structured that way. The build runs that way, the directories are constructed that way, CI runs that way. We just keep everything in one huge bag, and it starts to become a bit tangled. It makes CI less practical (20 min build time!) and the driver development is more difficult then it could (and should be). Lets face it. INDI has grown and what worked years ago is not guaranteed to work in the future - we need, as John Cleese said in Monty Python "Adopt, Adapt and Improve".
I am putting this as a place to discuss and, ultimately decide (I hope) what we should do. This is not a simple task and it will need changes, cooperation and will. I believe it is worth it and I am willing to help. Let me state the benefits I can see:
There are, naturally, some drawbacks/counter arguments:
This is prompted by problems the webcam driver (#639) is facing now, but I think this was quite common in the past - drivers got broken and slowed down development and complicated the testing. I am not sure this is a correct solution, I just think we need to think and talk about it. Discuss... ;)
The text was updated successfully, but these errors were encountered: