Proper tag support #53
Comments
|
With regard to the consequences and other things you wrote. The jetty.xml.production.patch does still need patching. Go to tag 10.0.0 and you will see that it's still there. Onlyoffice was introduced in Zimbra 10. None of their branches includes this because it's just their commits related to Network Edition. Most likely will get worse in the future as well and no doubt making building FOSS versions a nightmare. Patches for Alma are still needed because the issue is still outstanding on zm-build as are the others to fix security issues that Zimbra haven't commited to their repos. Also, as previously mentioned, once you have to make those kind of changes to remove Network Edition stuff, you are no longer the same version as their tag. So why is it so important? Please I'd really like you to explain this to me how we can claim to build the same tag, when it's not the same thing anymore? That just doesn't make sense. So what do you propose as a fix then? zm-build needs to be checked out, theoretically using a tag that may or may not exist, so then that is going to need some way of identifying whether the clone was successful or not. Also, not including having to assume that:
Up until now, all changes Zimbra did were commited to develop branch pretty much at the same time they then tagged everything. If you look at the repos you see a lot of other branches where all the work happens before they are merged into the develop branch. Pretty much almost effectively saying develop is their stable branch prior to it being tagged and released. Not easy, totally messy, and a lot of work, which I may or may not find time to complete it. You could always fork the project, provide your solution if you have one, to resolving this and make a PR to commit the changes back here? Feel free to offer that back to the Zimbra community, just like when someone made a PR to allow building Zimbra with Docker/Podman. I disagree with building something to a tag version when it's not the same as the tag being released by Zimbra. You can also disagree with that of course, but do feel free to fork the repository if that is something that you would like to see added. |
|
Here is a really good example of why offering proper tag support is a waste of time on my part until it is addressed elsewhere. See here: https://forums.zimbra.org/viewtopic.php?t=72204 According to this post, Zimbra 8.8.15 Patch 42 fixed the CVE issues. Now, if you see this repository: https://github.com/zimbra/zm-web-client there is no tag for Zimbra 8.8.15.P42. If we check out 8.8.15.P43 - the files supposedly removed in Patch 42 are still present in that tag. In fact, they are only removed in tag 8.8.15.P44. So, assuming that my scripts offered the ability to build from proper tags, it would claim to be 8.8.15P42 and yet, the files are still there which are removed in the patch that Zimbra made available for Network Edition users. You would assume that my scripts are building 8.8.15P42 and that it has addressed those CVE issues - which as you can see based on Zimbra's repositories, that is not the case. Those problems would only be addressed in 8.8.15P44 - two tags later. Even a CVE fix for 8.8.15P43 was only removed in 8.8.15P44. The onus on offering proper tag support needs to be addressed properly on Zimbra's side by making the correct commits and tags. Also what they should be doing is correcting their build process to supply only one tag to the build script so that it will build that version - not by providing dozens or more tags on the command line because they haven't pushed all of their repos to include all of the tags. Therefore, all repositories that are used for building Zimbra should all be tagged to the version that is being built. Which at present, that is not the case. Eg: Zimbra 10.0.5 exists, and yet some repositories still only have a tag of 10.0.1. The process they list in their documentation is pretty much a hack at best rather than address their build/commit processes in the first place. It would be very simple for them to:
As already mentioned, the "develop" branch only has the commits pushed to it before they make a tag release. Therefore, the branch develop isn't as unstable as you make it out to be in your post above. In fact, it's probably more secure, since that develop branch would have had all the fixes I've just mentioned above for a tag release that didn't have it. Just by adding functionality to my script to claim that we are building a proper tag release is misguided at best, especially considering what I wrote above about version 8.8.15P42/43/44. The Zimbra code repository linked and tags are evidence that backs up my reasoning behind it. My scripts have been created to simplify the Zimbra build process, and it does just that. It also addressed the CVE issues by patching before even Zimbra released those fixes and tagged on their repository. That is by far more important than claiming falsely that a certain tag release has been built. |
I wasn't aware of this.
Ok.
You are just claiming that you are using an specific Zimbra repos tag and applying your fixes there. What I'm saying is that you are not supposed to claim you are supporting a given patch... just that whatever Zimbra puts in a tag you are going to have it. If Zimbra messes up when a tag is published then it is Zimbra fault, not yours.
What you actually do is clone the repo by defaulting to develop branch, then you fetch the remote tags, sort them out and figure out either the tag which it's equal to given one or the previous one, finally you checkout on that tag.
Yeah, if it does not work... it's the problem of the user.
That's it.
So that's about right for Zimbra 10, isn't it? But what about Zimbra 9? How are saying that you are building Zimbra 9 if commits are being fetched from develop branches all around?
Yeah, I can't contribute to this in the short term but I will for sure implement this feature if I finally use it on my custom builds. |
That's not your problem. That's a Zimbra bug that should be reported so that they don't mess around again with this in the future. You know, now it's probably late, but I think that we should have been doing Zimbra 8.8.15 FOSS builds from the community, if what you get on an official binary Zimbra 8.8.15 installed system is different than what your own custom build generates then it's a bug and it should be fixed. I mean... we can always resort to say to Zimbra:
It's not easy to do the same thing for Zimbra 9.X and Zimbra 10.X because there are not official Zimbra 10.X OSE binaries.
I agree with you. In any case this documentation on how to build an specific tag should have been public much earlier than it was. As I said I might push a pull request about this in the long term.
Well, I guess those are different ways of handling tags on a Git repo. I have no idea which one is better or not. I also happen to work with onlyoffice repos which have a similar build system to this one (fetching several git repos) and they indeed push a tag on each one of the repos when a new release happens. It's quite a mess having so many tags there if I'm being honest but it's true it simplifies build system.
Ok.
See my comment above.
Well, it's not just about having a proper Zimbra 9 build or being it stable or not... is about being able to build a 6-month old tag version in 12 months from now. We should always get the same result too.
When an specific tag is given you are building tags, not releases. That's how I understand it.
I thank you for your all your work so far. It's great. |
|
Going probably offtopic here. BTW,... there is an unofficial announcement (without any concrete date) about a new free version of Zimbra: Extending Support for Zimbra 9 - BarryDeGraaf comment (September 24, 2023 at 10:40 PM) . I don't like the wording here because it does not say open source but free. It might be free as freedom but also free as in free beer which would mean that... you wouldn't have the source code for that specific version. |
|
There were chats in an internal Zimbra channel, that myself and some others involved with Zimbra for a long time now about Zimbra potentially providing a free version but limited inasmuch as let's say 25 user accounts. So fully functional, but limited in that way. Whether they intend to do that, I don't know yet, but the idea was floated. I would expect it would actually be Network Edition but with the limitations. Maybe more functionality than just mailbox limit. Regarding the post on the forum, not sure if that is related to that or not, or whether they are intending to build something else that's free. I expect it's more likely to be inline with mentioned above. The way I build right now is I build a version, be it 9.0.0 or 10.0.5 and I release that quarterly. Assuming that issues have been addressed by Zimbra themselves, or by patching to fix issues that haven't been commited. I don't claim to be a patch release, more building a version and that is it. I do however do the extra work to ensure certain issues, especially CVE's are addressed if it is possible for me to do so. Incidently I removed the ZM_BRANCH_TAG from the script anyway, because in reality it doesn't really do anything - especially considering the amended docs from Zimbra's side on how tags should be built. |
|
I'm going to leave the issue open (well, re-opened it), since there are already closed issues where people have asked about this. So if they see an open issue, then they will just add here than open a new one. I cannot guarantee that I will do this, maybe, maybe not, depending on how much time it will take to integrate and ensure it's working. In the beginning this build script was just to build the latest Zimbra be it 9.x and 10.x and that was it. But I do understand that people may want to build older versions and build from tags. For me, I'm not so bothered about it, because I'm only interested in building the latest to upgrade an existing install or whatever and ensure it's secure by providing whatever patches to fix issues like the CVE ones that appeared. Especially since this was not possible without intervention - the fixes were only pushed later to the repos (P44), than when they actually should have been to (P42). That's out of my control though. Also with that highlighted tag issue I mentioned, and you are right, that's not my fault if it's not included. I have had issues opened in the past from people saying that such-and-such doesn't work - which had nothing to do with my script but the usage of Zimbra. So I can also foresee people saying "your script doesn't build 8.8.15p42 properly because the files are still there" (as an example). People theoretically should know though that the script just builds Zimbra with whatever is available from the Zimbra repos so I'm not responsible if something isn't fixed. Assuming I do add this, I'm pretty sure I will see opened issues in relation to that and people potentially blaming my scripts for not building it right. Anyway, I'll leave it open, people can thumbs up or add comments saying they want that function or whatever - at least it'll be a way to gauge how much interest there actually is in this. But I cannot give a timescale for if/when it will be done. Until then, it's building the absolute latest 9.x or 10.x of Zimbra. |
|
Been doing some checking today, and Zimbra 9 is going to be a nightmare, purely because they list tags like 9.0.0p24.1 and 9.0.0.p24. It's not just that one, there is also p6.1 and p6.. Now none of those are mentioned in the list of tags on zm-build repository which means they are going to be tags somewhere across the 80-150 repositories that are used to build Zimbra with. Checking each of those manually to find tags like this is going to be pretty much impossible. Without being able to obtain a complete list published by Zimbra, that's impossible to integrate with the scripts. Had they all been listed as tags on the zm-build repository, that could then be cloned singularly and pull out all of the tags to make a list. But the fact they don't tag all of them on zm-build and have admitted that on a forum post I read a while back, that's juts not doable in it's current form. If we were to take 9.0.0.p37 as the version, and then use the script to parse and decrease the patch number, it would be easy enough to create a list of patches, p37, p36, p35 and so on. However to then attempt to find extra patch point releases without knowing about them and the lack of consistency here on Zimbra side is a nightmare. I'm certainly not going to trail over 80-150 repositories to try and build a list of tags. It's going to be pretty impossible as well to pull those tags from 80-150 repositories which looping over would be way too time consuming and not the correct way to do it. Zimbra 10 however at least in it's current form is more doable (although even that cannot be automated due to one of their tags being 10.0.0-GA). Assuming of course Zimbra actually continually adhere to standard tags without introducing extra point releases or other weird tag names. I was going to attempt this today with loops, etc, or to even put the list of tags in the script to update manually when a new tag comes out, but the lack of inconsistency on the Zimbra side makes offering something like building tag releases simply futile. Zimbra would need to publish a full list of their tags and the order they need to be applied, or at least adhere to some tag standards to allow this to be easily automated. |
|
Regarding the tags I have asked Zimbra themselves if they have a tool or helper for this same tag connundrum. Build Release 10.0.2.GA.0001 myself - some questions about upgrading and security - Tag helper comment. |
|
If Zimbra themselves don't have such a tool I am envisioning a This would be run once per day thanks to a Github action. Then from your helper you could fetch from that repo the complete tags lists So, in other words, the full list of tags will not be generated by the final user but by an external repo. |
|
He mentions about 10.0.2 being a tag on this link: https://github.com/Zimbra/zm-mailbox/tags as they don't have it as a tag on zm-build(since they don't update that repo as often or even when new tags are released) - so assuming they did make all tag updates here on zm-mailbox, then it would just mean obtaining a list of the tags here, and sorting. If they don't update that repo each and every tag release, then that could also be problematic. Just need confirmation from Zimbra a guaranteed place where all tags are always created and exist. |
|
Well using: on the zm-mailbox repo, shows that 9.0.0.p6.1 isn't on the list, so if that is a tag that exists (as used in the examples in their README), then that would confirm that zm-mailbox isn't possible to use as a full source of tags. I was intending to do: which would then have given me a variable to work with for then splitting the tags, sorting, etc. So we're still at a point where there is no single place to obtain a definitive list of tags that includes everything. I'm not even sure of the consequences if we build a version missing those 9.0.0.p6.1 and 9.0.0.p24.1 tags. |
|
I have finally found some time for developing zimbra-tag-helper. As you can see the current output is similar to: . I am planning on setting up an automatic github workflow that detects tag changes and updates a file which can be downloaded thanks to wget with something similar to: instead of the final build args so that it's easier to parse and consume from other scripts. The automatic github workflow will trigger daily and would wait for the latest tag associated tag date to be at least 1 day old to avoid inconsistencies. Can you give me some feedback? I mean... Now that you see how it can be done...
Thank you! |
|
I noticed in your 9.0.0 tag list that it's missing these tags: And just to clarify, I don't have a problem with doing it, provided that the source of the tag list is actually definitive and correct. Building with those two missed tags above could mean potentially unstable builds with missing functionality, features or fixes. In comparison, building from develop I have all the changes regardless of the tags plus any additional fixes at the time the build is made. I would therefore say that builds from develop would be more stable than a build with omitted tags. That is why the list of tags needs to be 100% correct, from wherever it's being obtained, be it Zimbra's repo's, or from the zm-build README.md assuming it's been updated every time a tag is published/released. Do I think Zimbra should publish it? They should since it is their product, but I doubt very much that they will. They aren't going to make it easy for us. |
One of the steps for gathering all of the tags is checking zm-build's So, yes, what you are seeing is an outcome of me using the latest commit on the zm-build develop branch. Let's try with zm-build's tags_for_9.0.0.p25.txt contents are: . Some comments about the list of tags:
I admit that You might ask what about |
|
I have rewritten zimbra-tag-helper in such a way that it loops inside of the different In theory given a tag we should search for the most recent (but also older) tag from zm-build but I think that searching for all of them should be the same thing in the end.
. As you might see there is no |
You can check how the tag list is handled in: https://github.com/Zimbra/zm-build/blob/2cf4095a9a2095215cb150d3769a4ccdb3c2d692/build.pl#L812-L831 . The code loops the different tags on the list from the start to the end. Basically once a I mean, yes you are right with that sentence (that's why I'm developing this tool) but many times using an incomplete tag list wouldn't matter so much. |
|
Even if the code remains quite dirty I have made some improvements on it. Now thanks to the And the You can check the current README.md with all of the examples. |
|
@adriangibanelbtactic is there any difference on these tags that have an uppercase P? |
So if you run some of my scripts modifying them so that they do not delete temporary files and then you check those temporary files you can find out that these specific tags are from |
|
@adriangibanelbtactic has done a fantastic job here: https://github.com/maldua/zimbra-foss-builder/ for building from tags. At the current point in time, if anyone is looking to build from tags I would suggest using Adrian's repositories that enable you to do just that. More info on his work can be found here: https://forums.zimbra.org/viewtopic.php?t=72645 and here: https://forums.zimbra.org/viewtopic.php?t=72655 I don't currently have the time required to integrate a solution with my build scripts, so they will continue for now as they always have by building from the |
Good luck dealing with 10.1.x stuff that has been added to the develop branch. You might need to disable the 10.1.x apt/yum repos among other things. |
|
It's not a problem for me. I've been doing that up until now anyway. The 10.0.6 builds I made had some 10.1 stuff as far as I remember, yet they still work. I will see how the 10.0.7 builds look when I've built/checked/tested to see if I need to do anything. But thanks anyway. |
I have read old topics from this subject: How to build a specific version and specially the Zimbra forums thread Upgrade the 9 patch 34.
I understand your rant about this tags problem on: #19 (comment) .
So... what I understand from all of this is that given a tag you cannot just use the tag as-is but you need to specify all of the tags previous to it till the original GA release of either 9 or 10.
That's what's explained on: official documentation on Building section.
Here:
https://github.com/Zimbra/zm-build/blob/d788f6a33960257cded5bd9ff27d74c149dfd414/README.md?plain=1#L89-L107
10.0.2 patch build example:
9.0.0.p25 patch build example:
Your current build uses
git clone -b ${ZM_BRANCH_TAG} https://github.com/zimbra/zm-buildto specify an specific branch but it has never used that--git-default-tagswitch forbuild.pl.Once you prove that this is the right way to build an specific tag you will probably have to deal with tag related improvements:
9.0.0.p34tag if9.0.0.p34tag does not exist inzm-buildrepo one needs to use:9.0.0.p33tag instead and so on.9.0.0.p25,9.0.0.p24.1,9.0.0.p24,9.0.0.p23,9.0.0.p22,9.0.0.p21,9.0.0.p20,9.0.0.p19,9.0.0.p18,9.0.0.p17,9.0.0.p16,9.0.0.p15,9.0.0.p14,9.0.0.p13,9.0.0.p12,9.0.0.p11,9.0.0.p10,9.0.0.p9,9.0.0.p8,9.0.0.p7,9.0.0.p6.1,9.0.0.p6,9.0.0.p5,9.0.0.p4,9.0.0.p3,9.0.0.p2,9.0.0.p1,9.0.0so that it can be used in--git-default-tag.--build-release-novalue based on the tag.And also with some other consequences:
1000to90in the repo configurations because it's using the proper tag and not develop one.jetty.xml.production.patch.Finally:
If this was my own project once the build based on tag works as expected... I would always use this feature to build my own tgz files because you never know when anyone from Zimbra might push any commit to any Zimbra repo.
The text was updated successfully, but these errors were encountered: