b2luigi summarizes different topics to help you in your everyday task creation and processing. Most important is the b2luigi.process function, which lets you run arbitrary task graphs on the batch. It is very similar to luigi.build, but lets you hand in additional parameters for steering the batch execution.
b2luigi.process
If you want to use the default luigi.Task class or any derivative of it, you are totally fine. No need to change any of your scripts! But if you want to take advantage of some of the recipies we have developed to work with large luigi task sets, you can use the drop in replacements from the b2luigi package. All task classes (except the b2luigi.DispatchableTask, see below) are superclasses of a luigi class. As we import luigi into b2luigi, you just need to replace
import luigiwith
import b2luigi as luigiand you will have all the functionality of luigi and b2luigi without the need to change anything!
b2luigi.Task
b2luigi.ExternalTask
b2luigi.WrapperTask
b2luigi.dispatch
b2luigi.DispatchableTask
As b2luigi automatically also imports luigi, you can use all the parameters from luigi you know and love. We have just added a single new flag called hashed to the parameters constructor. Turning it to true (it is turned off by default) will make b2luigi use a hashed version of the parameters value, when constructing output or log file paths. This is especially useful if you have parameters, which may include "dangerous" characters, like "/" or "{" (e.g. when using list or dictionary parameters). See also one of our faq-label.
b2luigi.get_setting
b2luigi.set_setting
b2luigi.clear_setting
b2luigi.on_temporary_files
b2luigi.core.utils.product_dict
b2luigi.basf2_helper