Sometimes we need to perform a series of operations:
op[0], op[1], ... op[N]
These operations may allocate files, locks, connections, and op[K] may depend on result of op[K-1]. Each operation creates a context. When the operations are done, we want to destroy those contexts. Sometimes it's not feasible to use standard Python Context Manager Protocol, because the number of the resource involved in a transaction can be a variable. There is no way to use a variable number of the with statement, and contextlib.nested is being deprecated. Sometimes it's just too verbose to create standard Python Context Manager for each context.
This library implements a concise Context Manager and proposes an idiom for rollback.
RollbackContext is a context manager to be used in a with statement. Once instantiated, it creates a undo stack. It allows the programme to register undo function after each successful operation. Each new undo function is push to the top of the stack. When the the operations are successful and execution flow leaves the with statement, it tries to pop the undo functions and run them. In all, it runs the following operation series:
op[0], op[1], ... op[N], DONE, -op[N], ... , -op[1], -op[0]
If there is an exception from op[i], it aborts the execution of the op series and starts to run undos registered so far. If there is an exception from the undos, it ignores the exception temporary and continues to run the rest of the undos. At last, it re-raises the earliest exception it sees. This is because latter exceptions may be caused by an earlier exception, so the most helpful exception for diagnosing the problem is the earliest one. Meanwhile, it should execute all the undos to destroy all the contexts as much as possible.
- Run as superuser,
easy_install rollbackcontextorpip install rollbackcontext. - In Python,
from rollbackcontext import RollbackContext. - Use it in a
withstatement as following:
with RollbackContext() as rollback:
- Write code for the operations, after each operation success, register a reverse operation by calling the
pushmethod.
with RollbackContext() as rollback:
op0()
rollbcak.push(op0Reverse) # op0Reverse is a callable.
op1(args, ...)
rollbcak.push(op1Reverse, argsForOp1Reverse, ...)
The push method accepts callable and the optional arguments that would be passed to the callable. In the above case, RollbackContext runs op0(), then op1(args, ...), then op1Reverse(argsForOp1Reverse, ...), and at last op0Reverse().
A most simple example may be the following:
from sys import stdout
from rollbackcontext import RollbackContext
with RollbackContext() as rollback:
print "Op 0"
rollback.push(lambda: stdout.write("Undo 0\n"))
print "Op 1"
rollback.push(lambda: stdout.write("Undo 1\n"))
print "Op 2"
rollback.push(lambda: stdout.write("Undo 2\n"))
# Prints the following
# Op 0
# Op 1
# Op 2
# Undo 2
# Undo 1
# Undo 0
You can refer to unit test code to find many more examples.
Here are some examples from simplified production code:
def vm_lifecycle(...):
''' The task is to create the VM, perform some tests and
destroy the VM and related resources. '''
with RollbackContext() as rollback:
templates_create('testTemplate', ...)
rollback.push(template_delete, 'testTemplate')
vms_create('testVM', ...)
rollback.push(vm_delete, 'testVM')
vm_start('testVM', ...)
rollback.push(vm_stop, 'testVM')
# Do whatever with the VM
Another one:
def prepare(...):
''' The task is to detect if a NFS export could be mounted or not. '''
with RollbackContext() as rollback:
mnt_point = tempfile.mkdtemp(dir='/tmp')
rollback.push(os.rmdir, mnt_point)
mount_cmd = ["mount", ..., mnt_point]
try:
run_command(mount_cmd, 30) # Wait for 30 seconds
except TimeoutError:
return False
umount_cmd = ["umount", "-f", mnt_point]
rollback.push(run_command, umount_cmd)
# Do whatever with the mounted filesystem
return True
Yet another one:
def probe_user(self):
''' The task is to start a libvirt domain and detect the user id of the
VM process. '''
user = None
with RollbackContext() as rollback:
conn = libvirt.open('qemu:///system')
rollback.push(conn.close)
dom = conn.defineXML('...')
rollback.push(dom.undefine)
dom.create()
rollback.push(dom.destroy)
with open('/var/run/libvirt/qemu/%s.pid' % self.vm_name) as f:
pidStr = f.read()
p = psutil.Process(int(pidStr))
user = p.username
return user
The above code comes from project kimchi, a HTML5 based management tool for KVM.
Most of the time we need to run all the undos, but sometimes we want to cancel the undos if all operations are successful. In this case, call the commitAll method to cancel all the undos as following:
with RollbackContext() as rollback:
print 'Op 0'
rollback.push(op0Reverse)
print 'Op 1'
rollback.push(op1Reverse)
rollback.commitAll()
If the with statement ends successfully, commitAll() cancels all undo, so that the contexts created in the with statement will be left untouched for future use. If there is exception from the with statement, commitAll() will not be run, so it still runs all the undo functions. Sounds like "start transaction", "commit transaction" and "automatic rollback" in a database stored procedure, isn't it?
Sometimes we want to cancel a particular undo if all operations are successful. In this case, call the setAutoCommit method of the object returned from the push method.
with RollbackContext() as rollback: print 'Op 0' rollback.push(op0Reverse).setAutoCommit() print 'Op 1' rollback.push(op1Reverse)
If any exception would be raised within the with statement, op1Reverse() and op2Reverse() would be run. If the with statement was successful, only op1Reverse() would be run.
Normally the push method adds the undo function to the top of the undo stack. In case you want to insert undo function to the bottom of the undo stack, use the pushBottom method.
from sys import stdout
with RollbackContext() as rollback:
rollback.pushBottom(lambda: stdout.write("0\n"))
rollback.pushBottom(lambda: stdout.write("1\n"))
rollback.pushBottom(lambda: stdout.write("2\n"))
# Should print
# 0
# 1
# 2
Unfortunately, C programmers can not enjoy the delight from our RollbackContext, they have to detect error code of each operation and use goto out0, goto out1, and so on, to simulate our RollbackContext manually. The following function comes from Linux kernel source code:
static int __init init_nfs_fs(void)
{
int err;
err = register_pernet_subsys(&nfs_net_ops);
if (err < 0)
goto out9;
err = nfs_fscache_register();
if (err < 0)
goto out8;
err = nfsiod_start();
if (err)
goto out7;
err = nfs_fs_proc_init();
if (err)
goto out6;
err = nfs_init_nfspagecache();
if (err)
goto out5;
err = nfs_init_inodecache();
if (err)
goto out4;
err = nfs_init_readpagecache();
if (err)
goto out3;
err = nfs_init_writepagecache();
if (err)
goto out2;
err = nfs_init_directcache();
if (err)
goto out1;
#ifdef CONFIG_PROC_FS
rpc_proc_register(&init_net, &nfs_rpcstat);
#endif
if ((err = register_nfs_fs()) != 0)
goto out0;
return 0;
out0:
#ifdef CONFIG_PROC_FS
rpc_proc_unregister(&init_net, "nfs");
#endif
nfs_destroy_directcache();
out1:
nfs_destroy_writepagecache();
out2:
nfs_destroy_readpagecache();
out3:
nfs_destroy_inodecache();
out4:
nfs_destroy_nfspagecache();
out5:
nfs_fs_proc_exit();
out6:
nfsiod_stop();
out7:
nfs_fscache_unregister();
out8:
unregister_pernet_subsys(&nfs_net_ops);
out9:
return err;
}
If this function was to be written in Python (of course it never would), without RollbackContext, you have to write as the following:
def init_nfs_fs():
with op0() as context0: # Suppose you wrap opX into context managers
with op1() as context1:
# ...
for c in [contextN, ... , context1, context0]:
c.cancelDestroy()
Or the following:
def init_nfs_fs():
try:
op0()
try:
op1()
# ...
except Exception:
op1Reverse()
raise
except Exception:
op0Reverse()
raise
With the help of RollbackContext, we can re-structure it as the following:
def init_nfs_fs():
with RollbackContext() as rollback:
op0() # Suppose op0() raises exception when it fails
rollback.push(op0Reverse)
op1()
rollback.push(op1Reverse)
# ...
rollback.commitAll()
It's cleaner. Whenever you find yourself dealing with similar case in Python, nesting try...except...finally or with blocks, you might want to have a try on RollbackContext.
For more anti-pattern examples, you can just git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git, and git grep 'goto out5', git grep 'goto out6' and more. Currently the worst case is bfin_lq035q1_probe function in drivers/video/bfin-lq035q1-fb.c, it goto out10. Think of when a developer wants to add a new operation and cleanup code into the existing operation series, he has to manually change all the X in goto outX and outX:. 有多痛苦，你们感受一下 ;-).