Org-mode and literate programmming

Org-mode

The first thing I will do is to show how an .org file looks in fundamental-mode. It’s actually nothing more than a text file. To know more about Org-mode visit the org-mode-site.

Content

Here are some of the basics we can have a look at:

Headings, how to create them and change them
What kinds of lists are supported
How tables look like
TODO:s

My favorite things

That it’s just plain text files
It’s automatic adjustments

Commands

Good to know commands in Org-mode

Command	Action
Tab heading	Demotes heading
Shift-Tab heading	Promote heading
C-Enter
M-Enter

Literate programming

Ex1 We are of course not only limited to perform simple operations like hello-world. Here is an example using the Eigen library and the c++17 compiler.

#include <iostream>
#include <Eigen/Dense>
using namespace Eigen;
using namespace std;
int main()
{
  Matrix3d m = Matrix3d::Random();
  m = (m + Matrix3d::Constant(1.2)) * 50;
  cout << "m =" << endl << m << endl;
  Vector3d v(1,2,3);

  cout << "m * v =" << endl << m * v << endl;
}

Chain

This is how to create a variable in bash and how to check it’s value:

MY_NAME="Niklas"
echo $MY_NAME

Ex2 Let’s create a variable

MY_NAME="Niklas"

And let’s print it’s value

echo $MY_NAME

Ex3 Let’s create a variable

MY_NAME="Niklas"

And let’s print it’s value

echo $MY_NAME

Example 4

Ex4 This works with other languages that supports interactive repls(read-eval-print-loop), like Python.

def calculate_sum(numbers):
    sum = 0
    for number in numbers:
       sum += number

    return sum

numbers = [x for x in range(20)]
print("The sum of the numbers is: " + str(calculate_sum(numbers)))

Compose

Chaining blocks through sessions is useful for some types of languages like python and shells. But that is not the only way. For me it can make sense to make the blocks as small as needed. That might differ depending on context. So we shall look more into how to compose different blocks. This is a feature called noweb.

Let’s take the previous python example.

Ex5 This is where we create our function

def calculate_sum(numbers):
    sum = 0
    for number in numbers:
       sum += number

    return sum

And here we create the data dn call the function

<<python-calc>>
numbers = [x for x in range(20)]
print("The sum of the numbers is: " + str(calculate_sum(numbers)))

Ex6 Why not mix languages as well, python and elisp working together

(/ (* 10 (+ 30 5 2)) 5)

<<python-calc>>
numbers = [x for x in range(<<elisp-max-numbers()>>)]
print("The sum of the numbers is: " + str(calculate_sum(numbers)))

Note taking

Ex7 Org-mode is an excellent tool for when studying, especially programming. Taking notes and experimenting with examples becomes very natural. Here is an example: An introduction to Elisp: The if Special Form

Our own image

I want to build my own Docker image and deploy an application on that image. These are the steps to make this happen:

sudo systemctl start docker

Build an image

I would like to build a docker image with Python3 installed so that I can run my python3 code. I got the inspiration from this minimal docker image example.

Requirements

Ex8 We need to specify the requirements that we are demanding:

Flask>=0.12,<0.13
flask-restplus>=0.9.2,<0.10
Flask-SSLify>=0.1.5,<0.2
Flask-Admin>=1.4.2,<1.5
gunicorn>=19,<20

Dockerfile

The Dockerfile specifying the commands for building the image looks like this:

FROM python:3.6-alpine

COPY requirements.txt /

RUN pip install -r /requirements.txt

COPY src/ /app
WORKDIR /app

CMD ["python", "app.py"]

Ex9 From create a layer from the python:3.6-alpine image:

FROM python:3.6-alpine

Copy adds files from your Docker client’s current directory

COPY requirements.txt /

Run builds your application with make

RUN pip install -r /requirements.txt

Cmd specifies what command to run within the container

CMD ["python", "app.py"]

The Dockerfile then becomes:

<<dockerfile-from>>

<<dockerfile-copy>>

<<dockerfile-run>>

COPY src/ /app
WORKDIR /app

<<dockerfile-cmd>>

The application

This is my application that I want to deploy in the docker

for i in range(20):
    print "i has the value " + str(i)

Create the image

docker build -t minimal_python .

First we verify that the image is now available among the docker images

docker images

Nice, let’s try the image

docker run minimal_python

Improve it

Ex10 Start our docker image:

docker run -it minimal_python /bin/sh

Verify that it runs:

docker ps

Ex11 I want to have a function which can return to me a TRAMP compliant path to a running docker image. Implementing that in Emacs lisp can look like this:

(defun docker-match (name-regexp)
  ;; return the name of the last docker image which matches the input
  ;; NAME-REGEXP
  (with-temp-buffer (shell-command "docker ps" t)
                    (goto-char (point-min))
                    (let ((name-match '()))
                      (while (not (eobp))
                        (let ((current-name (string-trim (thing-at-point 'line))))
                          (if (string-match name-regexp current-name)
                              (progn
                                (end-of-line)
                                (setq name-match (format "%s" (thing-at-point 'symbol))))))
                        (forward-line 1))
                      name-match)))

(defun docker-path (name-regexp  &optional extended-path)
  (if extended-path
      (format "/docker:%s:/%s" (docker-match name-regexp) extended-path)
    (format "/docker:%s:/" (docker-match name-regexp))))

(docker-path "minimal_python")

Ex12 Create a session which runs from inside the docker. Here we utilize the function we just added to set the :dir. This utilizes TRAMP to execute the block inside the Docker image.

ls

Ex13 We can now tangle the file directly into the running image

for i in range(20):
    print("i has the value " + str(i))

Ex14 Now we want to run a block in the previous session session-docker-agent. When I held this presentation there was a question on how to create a block which can automatically use the same session as the previous session in the file.

Now that was a really good question. The solution that I came up with later is worth showing here I think cause it also highlights the strengths of Emacs and lisp. I decided to create a function that can find the previous header argument

(defun org-babel-previous-session ()
  "Find the previous src code block which contains the session argument and
return it together with the language"
  (interactive)
  (save-excursion
    (let ((session nil)
          (language nil))
      (while (and (re-search-backward org-babel-src-block-regexp nil t) (not session))
        (goto-char (match-beginning 0))
        (let* ((block-info (org-babel-get-src-block-info))
               (block-lang (nth 0 block-info))
               (block-params (nth 2 block-info))
               (block-session (cdr (assoc :session block-params))))
          (unless (string= "none" block-session)
            (setq session block-session)
            (setq language block-lang))))
      (format "%s :session %s" language session))))

(org-babel-previous-session)

Ex15 Snippets in Emacs are very powerful:

# -*- mode: snippet -*-
# name: previous-session-block
# key: <ss
# --

$0

Performance analysis

Let’s make a fun little investigation. Assume we have some data of a signal and we want to make an algorithm that tracks the 1-d signal. This presentation is a little bit too short to make a real algorithm and data in so we will fake it instead

Synthesize the data

Ex16 There is not room here for creating our own algorithm, so instead we will fake the truth and estimates. To generate the true signal and the estimate we will use a block of elisp code.

(mapcar (lambda (i)
          (list i (+ (random 4) (- i 2))))
        (number-sequence 1 20))

Ex17 In order to get the table of data more comprehensive we utilize another block to post process the result. This block adds a header to the table.

(cons 'hline (cons '("Truth" "Estimate") (cons 'hline tbl)))

Visualize estimate and truth

Ex18 Here is some python code which can plot data using the matplotlib. So as a variable to this code block we will pass the numbers from the est-truth-data block.

import numpy as np
import matplotlib
matplotlib.use('Agg')
from matplotlib import pyplot as plt

# Convert list to numpy array
# The need for using 1 index here is because I added the box around truth and
# estimate in the table so it interprests the first value to be truth and estimate
truth = np.asarray(data)[1:, 0]
est = np.asarray(data)[1:, 1]
# Plot
fig=plt.figure()
plt.plot(truth, color="g", label="Truth")
plt.plot(est, marker="x", label="Estimate")
plt.legend(loc='upper left')
plt.xlabel("Sample")
plt.ylabel("Value")
plt.title("Tracking")
plt.savefig('.images/est_vs_truth.png')
'.images/est_vs_truth.png' # return this to org-mode

Cool, the performance of the fake algorithm is not that bad. I think we can be pretty happy with it. let’s see if we can gather some more information about it’s performance.

Ex19 It’s of course possible to use other languages for plotting like Octave

Performance numbers

A table can be a good way of displaying the data that we have and analyze the values. Tables in Emacs can use calc syntax org elisp code to make the table formulas.

Truth	Estimate	Error	Absolute error
1	2	-1	1
2	1	1	1
3	4	-1	1
4	5	-1	1
5	3	2	2
6	7	-1	1
7	8	-1	1
8	9	-1	1
9	7	2	2
10	9	1	1
11	11	0	0
12	13	-1	1
13	11	2	2
14	15	-1	1
15	13	2	2
16	15	1	1
17	17	0	0
18	18	0	0
19	19	0	0
20	19	1	1
Number of values	20
Mean error	0.2
RMSE	1.1832159566199232

In order to get the values from the other table I am using remote references. To refer to the values of the other table. The formulas that the table accepts should either follow the calc syntax or elisp.

Ex20 To calculate the mean value we can define a code block like this

And reference it in the table formula

Ex21 Any code block can of course be used, not only the ones of elisp. This is python block which calculates the rmse value:

Describe the flow

Ex22 I would like to describe the flow better. It would be great if we could visualize it, perhaps in a flow chart.

Emacs Calc

Now I thought that this article deserved some bonus information about Emacs calculator. I knew about it before but writing this part got me interested into knowing more. And I think that it was a good example of one of the aspects that I love about Org-mode which is to be able to experiment. I found some useful calc-babel-examples and org-examples which I took inspiration from.

Yes we can!

fsolve(x*2+x=4,x)

Ex23

Calc is available in tables as well

Expression	Derivative
sqrt(x)
cos(x)
x^2
1/x^2

Ex24 Another great feature in Emacs is that if you are not happy with something, change it. So I want a function that actually fix it at runtime.

(defun org-latex-export-to-pdf-and-open ()
"Export current buffer to LaTeX then process through to PDF and open the
resulting file"
  (interactive)
  (let* ((file-name (file-name-nondirectory buffer-file-name))
        (name (file-name-sans-extension file-name)))
  (org-latex-export-to-pdf)
  (find-file (concat name ".pdf"))))

Summary

Org-mode + literate programming

Context (connect everything needed)
Documentation (made easy)
Hackable (change what we don’t like)
Exports (replaces other programs)

Thanks for listening

The End? …

Blogging

Let’s turn something that we have made here into a blog post :) Org export pdf latex

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

preperations.org

preperations.org

Org-mode and literate programmming

Org-mode

Content

My favorite things

Commands

Literate programming

Chain

Example 4

Compose

Note taking

Our own image

Build an image

Requirements

Dockerfile

The application

Create the image

Improve it

Performance analysis

Synthesize the data

Visualize estimate and truth

Performance numbers

Describe the flow

Emacs Calc

Summary

Org-mode + literate programming

Thanks for listening

The End? …

Blogging

Files

preperations.org

Latest commit

History

preperations.org

File metadata and controls

Org-mode and literate programmming

Org-mode

Content

My favorite things

Commands

Literate programming

Chain

Example 4

Compose

Note taking

Our own image

Build an image

Requirements

Dockerfile

The application

Create the image

Improve it

Performance analysis

Synthesize the data

Visualize estimate and truth

Performance numbers

Describe the flow

Emacs Calc

Summary

Org-mode + literate programming

Thanks for listening

The End? …

Blogging