-
Notifications
You must be signed in to change notification settings - Fork 42
SEP 26: Package Salt with Tiamat #34
SEP 26: Package Salt with Tiamat #34
Conversation
Say I need to patch a bug in salt, is this still possible via some other method after this is packaged in to a frozen binary? I would love to say I don't do this but I've had multiple patches for every version of salt that we've used in prod so far. In the current system I just use salt's own |
That's exactly what you would need to do; fork -> apply patch -> repackage I'm currently building a docker container for tiamat that uses a very old glibc for maximum portability (and a second container that builds with wine for easy windows .exe creation) |
@raddessi the other option is to deploy this single binary as a "onedir" build - salt+all deps get extracted to one directory, so you would be able to patch those files. |
I heard that tiamat onedir only had pyc files, no py files. I checked https://artifactory.saltstack.net/ui/repos/tree/General/open-rpm-staging%2Fcentos%2F7%2Fx86_64%2Fsalt-3001-1.el7.x86_64.rpm and there are py files in there. We should get some clarification on this. |
oh onedir builds can have py files, it's the single binary that you can't change |
Hey nice to hear! I do like options. That's right I remember Tom talking about the "onedir" builds somewhere before.. maybe saltair? OK, both of those are viable (assuming |
The SEP suggests that the single binary will have all required Python Libraries and any C Library deps bundled. Looking at the pyinstaller docs, I'm not totally sure this is correct - I don't think it goes as far as bundling glibc for example does it? Might be good to clarify. I also think the proposal needs to cover what the process will be for tracking what libraries at what version are bundled and how Saltstack will handle the issue of a security vulnerability in a bundled library. Is there a way for a tiamat binary to show what libraries (and their versions) are bundled? @Akm0d - the work you're doing with trying to build stuff "with a very old glibc" sounds somewhat concerning. What distro are you using and how old is it? I assume it's something that's still getting security updates - otherwise, how are you ensuring you don't end up bundling something with a known vulnerability? Fundamentally, I think if you're proposing to move to a model where you are asking people to install software with a number of libraries bundled in, in a way that is opaque and not patchable in using normal OS procedures, then it's got to be clear that there's strong processes in place around security. If you go do this road you're going from being responsible for providing updates for vulnerabilities in Salt to being responsible for providing updates for vulnerabilities in Salt and all it's dependencies. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- It would be a learning curve for people to understand how to add python libraries to the new environment.
We could expand here that we will likely be providing a pip
command to install additional packages, so, the process will be almost the same as pip install package
- Adding python libraries to the new environment might be harder in an air gapped network.
A previous pip download package
on a similar system(because of the arch) using the same python version as the tiamat salt package would facilitate this scenario.
Then the downloaded packages could be brought into the air gapped environment for install. Since the tiamat salt package will include a pip
command, the pip download package
could potentially be done by a salt tiamat package with the same version as that in the air gapped system in order not to worry about arch/python version compatibility.
- People would have to re-install their python packages potentially if we updated the python in the tiamat environment on upgrades.
We could potentially have a directory which could keep the downloaded wheels/sdist since we will control how things get installed and use that on upgrades; If we can have the tiamat packaged salt detect that it's being upgraded, or by a special command that would reinstall everything on demand.
For the salt tiamat package, all shipped dependencies will be locked to a version(and the dependencies of a dependcy too).
Since the salt tiamat package will likely have a
That's correct, and a |
A couple of questions:
|
@Akm0d I also have this question. A quite common way to work around Salt bugs (and lack of frequent minor bugfix releases) is to self-patch it using the |
Just on a personal opinion, one of the most annoying things as a linux admin is projects that bundle all their own dependencies, instead of working properly with the OS package system. I guess the tide is turning against this (for desktop users anyway) with e.g. snaps. I am very sceptical that the salt team is capable of maintaining security patches for all targets in all the dependencies. Most distros are patching something at least every week, with thousands of people supporting it. Meanwhile we constantly hear from salt how it's not possible to do even minor releases more frequently than every couple of months. This proposal increases the maintenance burden by a huge amount. I need to see a believable commitment to do this properly, and not just vague promises. How (for example) are you going to get patches for undisclosed major CVEs, the way the distributions do? |
@OrangeDog this makes it easier on us to package more often because to install newer dependencies, we just upgrade via pip as part of a pipeline. The old way was really hard to update any dependencies that we added to the repo (take a look at amazon linux's dependencies in the salt repo) because we have to go find updated source rpms or source debs for each of those and then create the package. With all those rpm's and deb's, which we often had to create ourselves, it added a lot of manual effort. Windows and mac we already bundle everything and we can usually create those packages and get them tested in a few hours. The linux ones take several days. Hopefully this helps ease your concern. |
This issue affects regular users, formulas, and custom modules. Many modules require an additional python dependency in order to work. How is this going to work now? |
@max-arnold - I think #34 (comment) answers at least a few of your points. With a single binary style deployment it's static (though custom _modules & the like would still work -- I can't see why they wouldn't), so patching a state would not be possible (without overriding it with a custom _state, say). However, with the onedir style deployment (IIUC it would go under IIUC it's possible to take the same build and either execute it or have it unpackage itself into a onedir style deployment, but I might be misunderstanding that.
With the Tiamat builds, IIUC, the 3rd party libs would be included - though I'm not sure about optional deps. @Akm0d if this isn't documented in the SEP or documented elsewhere and linked from here, it should be. For source-only deps there shouldn't be an issue, but there may be issues with compiled deps. For sure there will be if the ABI has any changes, but there might not be otherwise.
It should reduce the burden of releasing Salt, since vagaries in dependencies would be eliminated - we'll just ship the deps that Salt needs. Obviously that means that we would have to ensure that we're keeping up with security advisories and cutting new releases when dependencies have issues. That should definitely be addressed in this SEP.
I believe that Pedro's answer covers this
We do already branch if we need to make a .2 release (or worse). Our existing approach to other releases should not have to change.
(see above)
single binary startup time definitely suffers, for the obvious reason of extracting the binary. I'm not aware of any other known issues. I seem to recall hearing something about improvements in the one-dir approach over existing Salt, but I don't remember the specifics.
That should definitely be documented and either added to the SEP or linked from the SEP
For a single binary option it should definitely be possible. I mean, heck, you can already do that now with Salt if you install to a virtual environment, or otherwise munge your PATH. |
I'm not sure I understand the question. https://gitlab.com/saltstack/pop/tiamat/-/blob/master/docs/topics/quickstart.rst might help. |
@waynew that's the probably the same question as what you said about ABI issues. |
@s0undt3ch - that sounds great and like you're covering the issues raised. Would it be possible to get that documented in the SEP as part of the detailed design? I think it's a fundamental part of what's being proposed so it would be good to have it covered in the SEP. |
Is it possible to quantify this? If calling |
@barneysowood the extraction is transparent to the user, aside from the extra time - a couple of seconds on a reasonably modern system. But if that's unacceptable, that's why onedir exists 🙃 |
@waynew are there any Tiamat builds of 3001.1 available to test with currently? |
I think that's a question for @bryceml. I know it's possible to build your own packages with Tiamat, and at least beardedeagle (on Slack) has been making their own internal Tiamat Salt builds. I don't know if there are instructions on the Tiamat page on how to build Salt, but the quickstart guide did look pretty straight forward (linked above in one of my other comments). |
Doesn't look like it. It should be as easy as tagging the salt-pkg repo and they should show up on the artifactory staging repos. We can probably do that without much difficulty next week. Also, if it's not clear yet, all the rpms and debs will contain the onedir format with no/negligible performance hit. The single binary will be for things like salt-ssh and testing type scenarios with a slight performance hit because it has to extract it. I don't imagine it's much different than salt-ssh's existing performance. |
Is tiamat going to bundle non-python dependencies or not? For example libgit2 and libzmq. None of the answers have made reference to them either way. If they're not bundled then I don't see how the dependency/testing problems are solved, and if they are then this need to talk about how they're upgraded, or new ones added, and whether the licensing is valid. |
Yes and No. As for libgit2, we're currently not including it in the tiamat package. If we were and linking was not enough, ie, it actually needed libgit2 available, we would include it in the tiamat package.
How come? I don't understand the problem you're trying to point out.
@thatch45 has already addressed the licensing concerns. Are there any more concerns regarding licensing? As for how and when they are upgraded, at the lack of a written plan, at least so far, I'd say we upgrade to latest releases every time we get out of code freeze to start working on the new release, or if/when needed because of a security release. |
Meeting notes from the Open Hour on 2020-NOV-19 discussion and communication of the poll in the above comment can be seen here: https://github.com/saltstack/community/wiki/Open-Hour-Agendas-and-Notes:-2020-11#existing-seps |
Just an update here from #34 (comment). We have a very rough and broken prototype of a bundled version of Salt, as described before. The source code is living here:
The targets for the different OSs are living in subprojects:
For example, this is for CentOS:
The only required package is
Some notes of this exercise:
There is another nice technical thing, that I do not know how Tiamat can address. There is this subtle problem with Python bindings that comes from system libraries. For example We addressed this problem in OBS importing the pristine |
Just wanted to give everyone a heads up on some recent changes you may or may not have seen related to the Salt bootstrap script or the Salt 3005 documentation. The change in question was to rename the wording around the new onedir packages that are being built for 3005 using the Tiamat tool. The core team discussed this topic at length and decided that it was more accurate and descriptive to refer to these packages as onedir rather than referring them to as tiamatpackages. This gives us a clear distinction between the new onedir packages and the classic packages by focusing on the concept used to build them rather than the tool used to build them. |
No description provided.