Org-listcruncher is a planning tool. Planning using lists is a very natural approach, and in terms of a data structure it is similar to a mind map. Emacs Org-mode makes it very easy and efficient to work with lists, since it offers a lot of functionality for restructuring and modifying them.
Org-listcruncher provides a way to convert such Org-mode lists into a table structure following specific semantics. This tabular data structure can then be operated on by other code blocks, or it can just be exported. The list can retain all the comments, you can override values and keep the history of changes inside of it, while still being able to derive an easy data structure as an input for other planning tools further downstream.
I've used it for the initial stage of planning for bigger projects. It was ideal for producing the often complex tables required by formal project proposal templates, and at some point I just exported the basic structures as a good starting point for continuing with a typical project management software.
You can get the package from MELPA using emacs' package manager.
If you are using John Wiegley's use-package (which I recommend), just put the following line
into your ~/emacs.d/init.el (or ~/.emacs)
(use-package org-listcruncher)
Or more barebones, just require it.
(require 'org-listcruncher)
Write a planning list and give it a name using the appropriate Org
syntax (e.g. #+NAME: lstTest). Here is an example (look at the
raw form of this README.org to see the Org source buffer with all
the markup)
- item: item A modified by replacing values (amount: 15, responsible: Peter, end-year: 2020)
- modification of item A (amount: 20)
- another modification of item A introducing a column (newcol: 500)
- modification of the modification (newcol: 299)
- illustrating inheritance (responsible: Mary, end-year: 2024)
- item: item B. Some longer explanation that may run over multiple lines (amount: 10)
- item: item C (amount: 20)
- item: item D (amount: 30, newcol: 35)
- a modification to item C (amount: 25, responsible: Paul)
- item: item X modified by operations (amount: 50, responsible: Peter, end-year: 2026)
- modification by an operation (amount: +=50)
- modification by an operation (amount: *=1.5)
- item: item Y entered in scientific format (amount: 1e3, responsible: Mary, end-year: 2025)
- modification by an operation (amount: -=1e2)
- item: item Z illustrating += and -= with strings (amount: 1000, responsible: Peter, end-year: 2025)
- adding a string (responsible: +=Paul, end-year: +=1)
- adding a string (responsible: +=Mary, end-year: +=1)
- removing a string (responsible: -=Peter)
We can use org-listcruncher to convert this list into a table
(org-listcruncher-to-table listname)
| description | newcol | amount | responsible | end-year |
|---|---|---|---|---|
| item A modified by replacing values | 299 | 20 | Peter | 2020 |
| item B | 10 | Mary | 2024 | |
| item C | 20 | Mary | 2024 | |
| item D | 35 | 25 | Paul | 2024 |
| item X modified by operations | 150.0 | Peter | 2026 | |
| item Y entered in scientific format | 900.0 | Mary | 2025 | |
| item Z illustrating += and -= with strings | 1000 | Paul Mary | 2027 |
We can also provide an additional argument to affect the order of columns in which the table is produced.
(org-listcruncher-to-table listname :order '("description" "amount" "responsible"))
| description | amount | responsible | newcol | end-year |
|---|---|---|---|---|
| item A modified by replacing values | 20 | Peter | 299 | 2020 |
| item B | 10 | Mary | 2024 | |
| item C | 20 | Mary | 2024 | |
| item D | 25 | Paul | 35 | 2024 |
| item X modified by operations | 150.0 | Peter | 2026 | |
| item Y entered in scientific format | 900.0 | Mary | 2025 | |
| item Z illustrating += and -= with strings | 1000 | Paul Mary | 2027 |
It is also possible to directly obtain single table field values based on defining the row and column through the string corresponding to an item's description and its column name:
(org-listcruncher-get-field listname "item B" "amount")
10
The rules for writing such a planning list are
- Each line contains a tag defining whether the line will become a table row. For this example I defined this as the string "item:". Rows without such a tag just serve as metadata.
- A string following the output tag "item:" is taken as the description of the table row.
- Each line can contain any number of key/value pairs in parentheses in the form
(key1: val1, key2: val2, ...) - Lines of lower hierarchical order in the list inherit their default settings for key/values from the upper items.
- The key value of a higher order item can be overwritten by a new new value for the same key in a lower order line.
- If a given value is of the form +=10, -=10, /=10, *=10, i.e. an operator followed by a number, the operation is carried out on the previous value of the respective key. (Note: this changed in version 1.2, since the original use of "-10" did not allow differentiating between subtracting 10 or setting value to "-10". The old syntax is still allowed for all operators except "-")
- If a given value is of the form +=word then "word" is added to the previous string value for this key, using space as a separator. If -=word is used, then "word" is removed from the previous string value. This allows building lists of words.
You can define arbitrary parsing functions for the list items. They must obey the following API:
The function receives a list item (a string) as its single
argument. It must return a list (OUTP, DESCR, VARLST), where
OUTPis a boolean indicating whether this list item will become a table rowDESCRis the description string appearing in the table's "description" column (so this is only relevant for OUTP=True lines)VARLSTis the list of key/value pairs corresponding to the column name / values.
Simple example functions for this purpose can be generated using
the org-listcruncher-mk-parseitem-default generator function. It
allows modifying the tag that decides whether a list item will
become a table row. It also permits changing the description's
terminating tag and the brackets for the key/value pairs. E.g. if I
would like to match for "row:" instead for "item:", and if I would
like to use square brackets, I can obtain such a function by
executing.
(org-listcruncher-mk-parseitem-default :tag"\\*?row:\\*?" :bra "[" :ket "]")
Let's test it using this modified list:
- row: item A modified by replacing values [amount: 15, recurrence: 1, end-year: 2020].
- modification of item A [amount: 20]
- another modification of item A [newcol: 500]
- modification of the modification [newcol: 299]
- illustrating inheritance [recurrence: 2, end-year: 2024]
- row: item B. Some longer explanation that may run over multiple lines [amount: 10]
- row: item C [amount: 20]
- row: item D [amount: 30]
- a modification to item D [amount: 25, recurrence: 3]
- row: item X modified by operations [amount: 50, recurrence: 4, end-year: 2026]
- modification by an operation [amount: +50]
- modification by an operation [amount: *1.5]
- row: item Y entered in scientific format [amount: 1e3, recurrence: 3, end-year: 2025]
- modification by an operation [amount: -=1e2]
We invoke org-listcruncher with the above parsing function:
(org-listcruncher-to-table listname
:parsefn (org-listcruncher-mk-parseitem-default
:tag "\\*?row:\\*?"
:bra "["
:ket "]")
:order '("description" "amount" "recurrence"))
| description | amount | recurrence | newcol | end-year |
|---|---|---|---|---|
| item A modified by replacing values | 20 | 1 | 299 | 2020 |
| item B | 10 | 2 | 2024 | |
| item C | 20 | 2 | 2024 | |
| item D | 25 | 3 | 2024 | |
| item X modified by operations | 150.0 | 4 | 2026 | |
| item Y entered in scientific format | 900.0 | 3 | 2025 |
And another variant allowing to write the list with minimal markup for the tag: Here any line beginning with a bold markup string becomes a row with the description being taken as that string. I just define as tag/endtag the markup character "*".
- Defaults (color: white, form: cube, weight: 10)
- one item is heavy (weight: 20)
- another is lighter (weight: 5)
- it has other distinguishing features (color: green, form: disk)
- item three is the default
We invoke the parsing function:
(org-listcruncher-to-table listname
:parsefn (org-listcruncher-mk-parseitem-default
:tag "\\*"
:endtag "\\*"
:bra "("
:ket ")"))
| description | weight | color | form |
|---|---|---|---|
| one item is heavy | 20 | white | cube |
| another is lighter | 5 | green | disk |
| item three | 10 | white | cube |
The way that the table structure is created from the list can be customized by providing own implementations of the parsing function and of the consolidation function that combines the parsed key/value pairs into a table.
The current implementations are examples that are sufficient for the above use cases.
One can easily imagine much more sophisticated parsing functions which e.g. could be applied to a cooking recipe written with minimal concessions as to syntax. From such a recipe one could then derive a table of ingredients, their amounts, and cooking times; all ready for being displayed as a table, to calculate the adapted amounts according to the number of expected guests, and entering the items onto your shopping list.
I am planning to provide more sophisticated parsing and consolidation functions to choose from (and naturally would be happy to receive any additions from contributors).
The default functions that are used can be configured using the following customization variables.
-
org-listcruncher-parse-fn: This variable defines the default parsing function to use if you call the org-listcruncher functions without an explicit:parsefnkeyword agument. -
org-listcruncher-consolidate-fn: This variable defines the default function for consolidating all the values that a certain key got assigned for a list item. The function must accept two arguments: KEY and LIST. KEY is the key (i.e. column value) of the row that one is interested in. LIST contains all the values for the KEY in that row, i.e. it will contain any redefinitions of the key value in subitems of this list item. The consolidation function basically defines how these values get combined into the single value that we will assign to the column in this row. The default function either replaces the previous value or allows values with operators (e.g. +=10, *=0.5) to modify the previous value. Refer to the default functionorg-listcruncher-consolidate-defaultdocumentation.
The primary goal of org-listcruncher-to-table is to return a data structure
(an org table structure) that can be used for further processing by code, e.g.
in a babel block.
But often, one will be mainly interested in a fast way to produce an org table that one immediately wants to process with the standard org table functions, e.g. just summing up some columns. Listcruncher offers a fast way for these situations:
(princ (org-listcruncher-to-table listname :formula "@>$1=Total::@>$3=vsum(@I..@II)"))
| description | newcol | amount | responsible | end-year |
|---|---|---|---|---|
| item A modified by replacing values | 299 | 20 | Peter | 2020 |
| item B | 10 | Mary | 2024 | |
| item C | 20 | Mary | 2024 | |
| item D | 35 | 25 | Paul | 2024 |
| item X modified by operations | 150.0 | Peter | 2026 | |
| item Y entered in scientific format | 900.0 | Mary | 2025 | |
| item Z illustrating += and -= with strings | 1000 | Paul Mary | 2027 | |
| Total | 2125. |
Since when using formula the source block is not returning a Lisp
table data structure, but an already rendered org table string, one
needs to use :results output. Since we do not want the result to
be put into an org example block, we also need to add the raw
flag. In order to fill out the last row's description we just use
for the initial formula the string "@>$1=Total". So, the whole
org block now looks like this.
#+BEGIN_SRC elisp :results output raw :var listname="lstTest" :exports both
(princ (org-listcruncher-to-table listname :formula "@>$1=Total::@>$3=vsum(@I..@II)"))
#+END_SRC
Note: In an earlier version of this example I used an external
function lobPostAlignTables from my library of babel to calculate and iterate the
table with the formula in an org bable :post hook. This is no
longer necessary with the addition of the formula feature.
I apologize for a backwards incompatible API change for
org-listcruncher-to-table listname and
org-listcruncher-get-field listname, which now both accept
keyword parameters. This will make the functions more future proof
when further function arguments need to be introduced.
The original syntax of e.g. "-10" did not allow differentiating
between subtracting 10 or setting value to "-10". Therefore the
operator use is now defined using the operator followed by the
equal sign: -=, *=, etc. The old syntax is still
working to keep backward compatibility, but it is discouraged.
Org table formulas can be added to the resulting table and listcruncher will invoke the org spreadsheet functions to calculate and align the table.
If keg is available for installing the test environment the make targets will use it to install dependencies and run the commands.
Just run this inside of the git repository
make test
If you want to debug a single test, run
TESTNAME=parseitem-default3 make debug
The test (in this example parseitem-default3) will be run and you
will be dropped into a live Emacs session.