Async interface to Koji, using Twisted
Access Koji's XML-RPC API asynchronously (non-blocking) using the Twisted framework.
This supports the GSSAPI or Client SSL login methods.
Simple Example: Fetching a user's name
from txkoji import Connection from twisted.internet import defer from twisted.internet.task import react @defer.inlineCallbacks def example(reactor): koji = Connection('brew') # Fetch a user. # You may pass an ID or a krb principal here user = yield koji.getUser(3595) # user is a Munch (dict-like) object. print(user.name) if __name__ == '__main__': react(example)
Connecting to a Koji Hub
To connect to a Koji hub, create a new
You must pass a string to the constructor. This string is a profile name. For
example, if you call
Connector('mykoji'), then txkoji will search
/etc/koji.conf.d/*.conf for the
[mykoji] config section. This matches what the regular Koji client code
Making XML-RPC calls
Koji Hub is an XML-RPC server. You can call any method on the
class instance and txkoji will treat it as an XML-RPC call to the hub. For
example, this Twisted
inlineCallbacks-style code looks up information about
a given task ID and tag ID:
@defer.inlineCallbacks def example(reactor): koji = Connection('mykoji') task = yield koji.getTaskInfo(10000) print(task.method) # "createImage" tag = yield koji.getTag(2000) print(tag.name) # "foo-build"
To learn the full Koji XML-RPC API:
You can also read the koji source code to find out details about how each method works.
Your Koji hub must support GSSAPI or Client SSL authentication. You must have a valid Kerberos ticket or SSL keypair.
@defer.inlineCallbacks def example(reactor): koji = Connection('mykoji') result = yield login() print(result) # "True" print('session-id: %s' % koji.session_id) # "Who am I?" user = yield koji.getLoggedInUser() print(user)
Estimating build durations
txkoji.estimates module provides methods for estimating build times.
average_build_duration() method calls Koji's
getAverageBuildDuration RPC and gives you a
datetime.timedelta for a
package. For container packages, we do something similar client-side with the
average_last_builds() method, averaging the last five builds' durations.
Caching long-lived object names
Sometimes all you have is a user id number or tag id number, and you want the user's name or tag's name instead.
txkoji includes a read-through cache for obtaining the user name or tag name.
examples/cache.py for an example. txkoji's cache module stores its data
txkoji subdirectory of the location specified with the
$XDG_CACHE_HOME environment variable if that is set. It will fall back to
~/.cache/txkoji if the
$XDG_CACHE_HOME environment variable is
The following RPC methods will return special classes that inherit from the Munch class:
These classes have their own special helper methods to implement things I found interesting:
datetimeconversions for the start/completion timestamps,
urlproperties for representing the objects in Kojiweb,
- Unified property attributes across task methods, like
More special return values:
datetime.timedeltaobject instead of a raw float, because this is more useful to do time arithmetic.
task_idproperty is populated on OSBS's CG container builds (a workaround for https://pagure.io/koji/issue/215).
If you have to submit many RPCs to koji-hub at once, you can optimize this with "multicall".
Koji's XML-RPC implementation allows you to batch or "boxcar" many methods up into one single "multicall" RPC and send it to the server as one single HTTP request.
@defer.inlineCallbacks def example(reactor): koji = Connection('mykoji') multicall = koji.MultiCall() # Query the task information for several tasks in one shot: multicall.getTaskInfo(123) multicall.getTaskInfo(456) multicall.getTaskInfo(789) results = yield multicall() # results is a xmlrpc.client.MultiCallIterator for task in iter(results): print(task.id) # eg. "123" or "456" or "789" print(task.method) # eg. "tagBuild"
This is a bit similar to Twisted's
although it happens server-side instead of purely client-side.
If the hub returns an error for any of the calls within the multicall, the
iterator will raise
KojiException when iterating over the specific call
result that had the error.
Koji's messagebus plugin emits messages to an AMQP broker when certain events
txkoji.messages module has support for parsing these messages
into the relevant txkoji
- More KojiException subclasses for other possible XML-RPC faults?
- Implement krbV authentication (probably not unless there is an alternative to python-krbV).
- MikeM noted,
the callnum parameter will need special handling. We might need Twisted's
DeferredLockto ensure we only have one auth'd RPC in flight at a time. It's not really clear to me if we can actually hit a callnum error here. More integration testing needed for this.
- Ensure that Brew's "build time" equals the longest "buildArch" time for a task, and not something else, like the buildSRPMFromSCM time, nor even the overall build task's time. This has implications for estimating scratch builds. (comparing our tasks' times to getAverageBuildDuration)