RFE: Guide for bootstrapping Conary on Fedora

[ncoghlan]

My goal: use conary to build a Kallithea Docker image atop the Fedora cloud image

I've had a fair bit of trouble figuring out how to do this. In #2, @mikjo suggested using http://sassoftware.github.io/appengine/ but the docs for that are all about setting up a full image builder, and I don't want that - I just want to use conary to manage some add-on packages which I'm going to install atop an otherwise RPM managed system. I'm pretty sure I don't need a huge VM for that, but I also don't know which of the SAS AppEngine setup instructions I can safely ignore for my use case.

Unfortunately the docs on the old Conary wiki for both conary and rMake both assume you already have a pre-existing instance of conary available.

To help resolve that, I'll use this issue to make notes on the dependencies I needed to install to get checkouts of both conary and rmake building locally on Fedora.

[ncoghlan]

Building conary on Fedora 22:

sudo dnf install python-kid elfutils-libelf-devel
git clone https://github.com/sassoftware/conary.git
cd conary
make

(This system is already set up to build CPython, so any shared deps between Conary and CPython would already be installed)

[ncoghlan]

Revised goal: getting a Fedora system/container image to a point where it's possible to run "python testsuite.py" in the conary checkout to run the conary integration tests. So far, that's added to the dep list:

• cloning https://github.com/sassoftware/testutils locally
• creating a virtualenv that can see the system site packages, and adding a "conary.pth" file for the conary checkout and the testutils checkout
• installing m2crypto, gunicorn, nginx and webob system wide using dnf
• installing webunit into the virtualenv with pip

I got several failures due to a SystemExit when attempting to cook recipes in the trove tests, as well as a large number of gateway errors. Currently at 332 failures out of the first ~1130 tests.

[ncoghlan]

OK, I'm definitely still missing some dependencies - out of 2809 integration tests, I had 1645 failures and 125 skips.

One of the earliest failures was an attempt to write to "/var" rather than to a user level directory, so I suspect the tests may be expecting to have complete control of the system, rather than being designed to be runnable purely in user space.

[conarysync]

I suggested the appengine merely as the easiest way to become familiar with the technology starting with something tested to work, not as your end goal.

The test suite has only really been developed on a system with conary already bootstrapped onto it, which is why you are seeing those failures. That doesn't mean that it's the only way that it could be useful, it just explains why it would have lots of failures when run in a context in which it hasn't been normally run in the past.

I'm now testing on F22 with native package, not using virtualenv. I'm ignoring all the webunit tests; they are about the web interface to the Conary repository and I don't think that matters to you, at least at this phase of your investigation. All the webunit-related tests should be properly skipped in the absence of webunit. I confirm many failures.

You will need conary-policy as well: https://github.com/sassoftware/conary-policy — that will, I think, resolve some of the SystemExit failures when attempting to cook recipes in trove tests. The conary-policy package was separated from Conary so that it could be forked separately to be adjusted to different distributions; an early attempt to generalize Conary.

I am running tests conary tests from within the conary clone, and with conary-policy cloned, as follows:

CONARY_POLICY_PATH=/path/to/conary-policy/policy PYTHONPATH=../testutils ./testsuite.py

I confirm that there are many errors in my run, and that at least plenty of them are real bugs that just didn't show up in previous contact for whatever reason, probably primarily related to the test suite running on conary-hosted systems, or newer versions of dependencies that require adjustments.

I would not expect the conary tests to write to /var; that would be a bug in the tests or a configuration issue. There are plenty of attempts to read from /var/lib/conarydb based on the assumption that it's running on a system packaged with Conary. Some of those tests don't really need a working conary database, and there we can work around it. Others really assume that there is a meaningful system database; those tests we could perhaps decorate with a SkipTestException if the database doesn't actually exist. Many of the failures might be indirectly due to this.

There are other places in the test suite that test against an installed conary; that's on purpose in order to be able to test an installed conary as well as run out of the checkout. This was done for CI of the entire app engine (rBuilder) but normal developer workflow is testing the checkout, not installed conary.

We have some documentation internally on developing for appengine but it assumes that you are doing the whole thing rather than trying to run, say, the conary test suite, it assumes you are working on a system with conary already installed, and it's a bit out of date, so I'd rather start by just working on making things run on F22 well enough to evaluate whether Conary would be useful in a Fedora context.

Digging into these errors is probably not the easier or most productive introduction to Conary. So I think that the next progress belongs to us.

[conarysync]

The SystemExit errors are from not finding conary-policy. I updated my previous comment to correctly reference the CONARY_POLICY_PATH variable and set it to an absolute path, rather than a relative path, which fixes those particular errors.

I have found one incompatibility with python 2.7 in the test suite so far (looks due to an old editing mistake that happened to not break on older python).

You also need to install the glibc-static package, as many of the tests actually build static binaries; otherwise, you will see something like:

RuntimeError: Shell command "cd '/tmp/conarytest-vr_tfh/build/testcase/testcase-1.0/'; CFLAGS="-O2 -g" CXXFLAGS="-O2 -g " CPPFLAGS="" CLASSPATH=""  LDFLAGS="-g" CC=gcc CXX=g++ LDFLAGS="-static" make    hello" exited with exit code 2

Many more of the errors are because there is no test repository to work against because the local gunicorn instance can't find conary.server.wsgi_hooks which means that I would expect vast numbers of tests to fail—as they do. More investigation on that next...

[mikjo]

My failure to find conary.server.wsgi_hooks was due to a local typo in my PYTHONPATH.

export PYTHONPATH=/path/to/conary:/path/to/testutils

That resolves many of the remaining errors.

In general, the

OpenError: Unable to open database /var/lib/conarydb: cannot create directory /var/lib/conarydb

errors are for tests for which we should probably just SkipTestException if /var/lib/conarydb doesn't exist. The cannot create directory is just because of code for bootstrapping conary onto a system and doesn't not mean that conary is expecting to be tested only as root. It can't be merely protected under an "if current user is root" because a normal use case of conary is to install as non-root under a prefix where it is expected to bootstrap its database.

[mikjo]

dnf install memcached will fix a test, but that could also maybe be changed to a dynamic SkipTestException if memcached is not installed.

[mikjo]

dnf install popt-devel will fix another test.

NormalizeInterpreterPathsTest needs to be adjusted for usrmove

[mikjo]

With those changes, I had 33 failures, most of which fit into the "port to F22" bucket.

[ncoghlan]

Thanks!

I've installed memcached, popt-devel and glibc-static, cloned conary-policy, and updated the main conary repo, and am now running the test suite with this command rather than using virtualenv:

CONARY_POLICY_PATH=~/fedoradevel/conary-policy/policy PYTHONPATH=~/fedoradevel/conary:~/fedoradevel/testutils ./testsuite.py

The tests take a while on my laptop, but they're definitely looking much happier now - I'll post the updated stats once the run finishes.

Hopefully we'll get to a point where I can put a conary, conary-policy and sas-testrunner repos up on https://copr.fedoraproject.org/ to make it easier for other folks to experiment.

[mikjo]

Glad to hear it! Commits 0286da4 and cdf9b4a fix several failures for this case. For me, that leaves 24 failures that deserve deeper investigation.

[mikjo]

On a 16GB i7-5600U laptop with SSD and a tmpfs /tmp, it takes about 40 minutes to run the test suite. I recall tmpfs making a substantial difference when I first started using it; haven't tested without it for many years so I don't know whether that's currently the case.

[ncoghlan]

My results: Fail: 47 Skip: 139 - 36:40/36:40 - 2810/2810 (IIRC, F22 uses tmpfs for /tmp by default, so my local setup sounds pretty similar to yours)

Several of those were due to the plain "patch" utility not being installed (I can't recall the last time I used anything other than git or hg to apply a patch to a tree).

I've installed that and pulled your updates, so I'll kick off another test run.

[ncoghlan]

The final test run output is short enough now that I'm prepared to post it inline. The full output of a subsequent --rerun-failed run is at http://paste.fedoraproject.org/252511/91708314/

Fail: 27 Skip: 148 - 36:18/36:18 - 2810/2810                                   
----------------------------------------------------------------------
Ran 2810 tests in 2178.233s
Failed tests:
conary_test.flavortest.FlavorTest.testDependencies
conary_test.libtest.openpgptest.OpenPGPMessageTest.testKeyWithExtraSigsOnSubkey
conary_test.clienttest.updatetest.ClientUpdateTest.testPartialUpdates
conary_test.cvctest.vercontroltest.CvsTest.testVersions
conary_test.cvctest.vercontroltest.SvnTest.testVersions
conary_test.cvctest.buildtest.sourcetest.SourceTest.testEmptyDeb
conary_test.conarycfgtest.EntitlementTest.testParsing
conary_test.cmdlinetest.CmdLineTest.testLocalOperationWithBadConaryrc
conary_test.showchangesettest.ShowChangesetTest.testChangedFiles
conary_test.deptest.DepTest.testBreakOnUpdate
conary_test.deptest.DepTest.testCrossFlavorDeps
conary_test.deptest.DepTest.testDependencies
conary_test.deptest.DepTest.testFileDependency
conary_test.filestest.FilesTest.testBlock
conary_test.filestest.FilesTest.testChar
conary_test.filestest.FilesTest.testOwnership
conary_test.libtest.cfgtest.ConfigTest.testIncludeConfigFileFails
conary_test.libtest.utiltest.UrlTests.testUrlSplitUnsplit
conary_test.libtest.magictest.MagicTest.testCarArchive
conary_test.libtest.magictest.MagicTest.testGzipCompressedTarArchives
conary_test.clienttest.resolvetest.ClientResolveTest.testResolveAgainstDownRepository
conary_test.cvctest.sourcetest.SourceTest.testMetaDataConflicts
conary_test.cvctest.buildtest.buildtest.TestReportMissingBuildReqs.testActionSuggestsBuildReqs
conary_test.cvctest.buildtest.packagepolicytest.RequiresTest.testRequiresMissingPerlModule
conary_test.cvctest.buildtest.packagepolicytest.RequiresTest.testRequiresPerlNoBootstrapPerl
conary_test.cvctest.buildtest.packagepolicytest.RequiresTest.testRequiresTest1
conary_test.cvctest.buildtest.destdirpolicytest.NormalizeInterpreterPathsTest.testNormalizeInterpreterPathsTest1
(Rerun w/ --rerun-failed to rerun only failed tests)
Exception TypeError: "'NoneType' object is not callable" in <bound method SqliteServer.__del__ of <testutils.sqlharness.SqliteServer object at 0x7fba552e7e10>> ignored

[mikjo]

I am testing fixes for several more errors; I'm down to 5 failures that I haven't diagnosed.

Patches to conary were done years ago for python 2.7 support, but the test suite was not completely modified to work with python 2.7. Many of the remaining failures were due to unintended sensitivity to /usrmove. There were forgotten assumptions that "uucp" and "daemon" users not only exist but have specific IDs; because the assumptions were correct previously, incomplete mocking of the IDs wasn't noticed.

Perl dependency discovery is broken. This probably requires updating the embedded ScanDeps module to handle newer perl. I have not yet investigated that. There are a few changes that I think are related to py27 but I'm not sure yet.

[mikjo]

Updating to current ScanDeps does not fix the perl dependency discovery problem.

[mikjo]

perl problems were due to a change in perl-CGI; CGI::Apache no longer exists; switching to using CGI::Util which does fixes the perl errors. I'm down to three failing tests.

[mikjo]

I have a patch waiting for review that resolves all test failures. It's currently +158, -94 so requires careful review first.

[mikjo]

While I wait for review of my comprehensive patch, it probably makes sense to step back and review path to progress on your original goal.

Like RPM, Conary manages dependencies. Its concept of dependency is substantially finer-grained and more complete than RPM's. That means that RPM dependencies cannot be mechanically extended to provide everything that a Conary package would require. Therefore, to use Conary to lay kallithea on top of a docker image would require disabling expressing requirements while building that package or those packages. (Not that hard; the magic invocation would be: r.Requires(exceptions='.*') That will reduce the value of Conary, of course; part of its value is that the fine-grained dependencies can include paths that distinguish otherwise similar dependencies; for example, the dependency space for an alternate python interpreter does not pollute the dependency space for the system default python intepreter(s).

The obvious (to me) question is then what specific benefits you are exploring for using Conary in this context. Since you are building docker images, differential update is probably not a benefit you are looking for. If you can enumerate at least some of the things you are interested in exploring (versus just installing it via an RPM) I can avoid giving you suggestions that would negate the value you are looking for.

Separately, Conary has not been ported to py3. There are undoubtedly a substantial quantity of changes that would be required (a lot of Conary development was done against Python 2.4), but there's something more than changing Conary itself; the recipes that Conary uses to build packages are Python modules checked into a repository and have at least one Python2 characteristic: octal numbers for file modes. Since you have no installed base to worry about, perhaps a solution there would be that if py3 is being used to build, it insists that modes are specified as strings and the strings are converted to octal explicitly. (It was a mistake to use octal numbers in the first place; I should have used strings instead.)

[ncoghlan]

We actually had some really interesting discussions around this at Flock last week, and while things are still evolving, it's starting to look like my main interest in all of conda, nix, and conary will be "provide a range of tools that allow folks to explore different ways of developing software atop Fedora".

Development of Fedora, by contrast, would stick with good old SRPM and RPM (and the supporting tools like COPR, dist-git and koji), together with the Source-to-image technology for defining a container image from a base language runtime and a git repo.

Getting stuff from other packaging ecosystems into that pipeline would then involve tools for automated conversion to either SRPM or a source-to-image repo.

[mikjo]

While I was the FPL for Fedora Core 1 and for the past couple of years have again been a Fedora user, I'm not a current Fedora developer, so I am not familiar with all current jargon. I don't know whether "Source-to-image" is a generic description or some specific implementation.

It seems like you are saying that you are shifting what you are trying to do away from building an image with kallithea in it, but from what you say it's not clear to me how that's not an example use case to prove out the idea. If it's not a good fit, can you give a different concrete example to help me understand what you are trying to accomplish? (No point in wasting our time sending you down a path that is poorly aligned with your goals!)

[ncoghlan]

Source-to-image is a tool the OpenShift community created to automate Docker container builds from a git repo containing the relevant source code and a base image containing a suitable language runtime: https://github.com/openshift/source-to-image

So rather than trying to build an image with Conary directly (which is what I was originally thinking of doing), my question is now whether or not Conary has a potential role to play in helping application developers to define and maintain a git repo that consists of:

• their own software
• bundled upstream software (perhaps with applied patches?)
• the necessary hooks to tell S2I how to turn that repo into a container image

Unfortunately, that means I'm now in a situation where I don't know either Conary or S2I well enough to answer my own question.

[mikjo]

So sti doesn't build RPMs at all, it just runs an "assemble" script, if I read the documentation correctly. So how to "assemble" is left as an exercise to the developer, but in any case is completely divorced from RPM? Would you then expect Conary to build artifacts that are committed as built artifacts into the git repository that the assemble script deploys into the image?

In any case, it seems that before we go much further, getting sti users/developers in the openshift community involved probably makes sense.

[ncoghlan]

Yeah, while I started working on this in preparation for Flock, discussions at Flock moved it down the list to "still an interesting idea, but not a priority".

At the moment, we're still not entirely clear on what the pipeline is going to look like even when just using upstream packaging tools directly.

[mikjo]

There were a few good review comments, so I had some improvements to make before publishing my changeset, but a5ae006 appears to resolve all test failures on my f22 test system.

I suggest that when it's more of a priority or others are interesting in joining the conversation, follow up in this issue with either more questions or more details about what is desired.