Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
11833 lines (9465 sloc) 405 KB

org-babel — facilitating communication between programming languages and people

Tasks [97/188]

Manual: explain that initial evaluation is needed to use :cache with export

  • State “TODO” from “” [2011-02-03 Thu 10:34]

code blemish in gnuplot

  • State “PROPOSED” from “” [2010-12-20 Mon 19:09] shouldn’t this be (eq result-type ‘output)

Revisit :file and :results file

  • State “DONE” from “PROPOSED” [2011-01-14 Fri 22:08]
  • State “PROPOSED” from “” [2010-12-17 Fri 10:51]

Here are some related development issues that all concern the details of returning files from code blocks.

Note that the behavior has changed slightly recently with commit 14f4dbd75f.

Email between Eric and Dan

Hey Dan,

I believe this has partially been subsumed by a piping discussion on the mailing list, so I’ll address a couple of dangling questions below and respond to the piping issue there.

Dan Davison <dandavison7@gmail.com> writes:

> Hi Eric, > > OK, so the following is an attempt at working out what I think. Some or > much of this may be off target. The below is almost consistent with your > proposal, except that mine would be > > -#+begin_src sh :results output :file current-time > +#+begin_src sh :results output file :file current-time > date > #+end_src > > > - “:results output file” should mean that stdout is sent to a file > > - Eventually that should be true for all languages that can output to > stdout (but I’m happy to start with shell) > > - “:results value file” should mean that the value is sent to a file > (verbatim from buffer i.e. in Org format? or as csv where appropriate?) > > - If either of the above are provided then the user is required to > supply :file in order to have a file name (or would we consider a > default output file name?) >

Agreed, generally we should allow :results file to save the results to a file. The question then is what do we do if no file name is specified? I’d say we could have a system of generating file names, either in the temp directory or based on the code block name. This should probably be implemented outside of the language specific handling and in ob.el for simplicity and consistency.

> > - “:file filename” on its own (without :results file) should mean that a > file link will be returned, but nothing else will be done (no > redirection of output, value or graphics to the file), leaving the > user’s code free to send content to that file. >

agreed

> > - Therefore R will need to be changed. Automagically sending R graphics > to :file was always a bit of a special-case hack. The above suggests > that this should be “:results graphics file :file xxx”. In any case I > seem to remember a bug report (Erik Iverson probably) pointing out > that the automatic graphic capture only works for “base” graphics and > not for other fancier form of R graphics). […] > (The patch was empty when it got to my gnus.) >

and I’ve since deleted it locally, so this piece of code will be lost…

[…] > > “Eric Schulte” <schulte.eric@gmail.com> writes: > >> Hey Dan, >> >> What do you think about the following change. I think it’s very slick, >> but it may not work on Windows systems (I’m not sure how their shell >> exec functions works). But then again, if it’s a shell code block, I >> don’t really know if those work on Windows machines currently. >> >> With the attached patch, any sh code block with a file specified with >> “:results output” and without a session will automatically re-direct its >> standard output to the specified file. e.g. >> >> #+begin_src sh :results output :file current-time >> date >> #+end_src >> >> What do you think? >> >> Maybe we could also intercept the results of the code block and dump >> them to the file. This would be more portable, but would required that >> the data pass through Emacs (although it wouldn’t need to be parsed and >> could simply be treated as a string). >> >> – Eric

(e.g.) R should create default file name for graphical output if none supplied

  • State “PROPOSED” from “” [2010-12-17 Fri 10:58] Requested by Stephen Eglen

Todo file-wide language-specific header arguments

  • State “TODO” from “” [2010-12-05 Sun 08:37]

Would be nice to be able to specify file-wide and language-specific header arguments. The syntax should read something like…

#+SH_Headers: results output

or

#+Babel_SH: results output

or something similar.

separators in the Error buffer

  • State “TODO” from “” [2010-11-26 Fri 13:36]
>> - add something visible after every execution, like a separator line or a
>>   ^L character (new page, than can easily be displayed as a rule)
>
> I like this one (above), especially if you could append the date+time to the
> formfeed and, ideally, some indication of the source code block responsible
> for what follows. The latter is important because I often have multiple
> source code blocks with the same language and it would be nice to
> distinguish between them in terms of error output. Having the srcname appear
> would be great, if it's possible.

:

I think that what'd be very useful is:

:

- some kind of separator
- some timestamp (à la Org?  with or without seconds)
- some language info?
- a referrer to the code block (like the =comments= in tangled files)

:

The separator could even be (why not?) a Org heading: the entire output buffer
could be an Org file, allowing for folding of less useful entries, or sparse
tree searches:

:

* 2010-11-23 Tue 22:29:11 sh block
/* [[file:~/src/test.org::*Commands][Commands:1]] */
Invalid command: ls\r
Invalid command: date\r
/* Commands:1 ends here */

:

Just a proposition example... For me, whatever the format...

Resolve references earlier and rewrite of variable assignments

  • State “STARTED” from “” [2010-10-16 Sat 15:40]

Newlines

  • State “TODO” from “” [2010-10-16 Sat 17:22] We may need to add “\n” on the end of body in ob-expand-body:generic?

python bug

  • State “TODO” from “” [2010-10-16 Sat 15:42]
return tab
row1ab
row1ab
row212

interactive versus non-interactive assignments

  • State “TODO” from “” [2010-10-16 Sat 17:39]

    Eric says: They differ in haskell and some lisps. (But prep-session and expand-body look pretty similar in ob-haskell?)

I don’t think ocaml prep-session loads the variables

  • State “TODO” from “” [2010-10-16 Sat 18:26]

Error in recent scheme change?

  • State “TODO” from “” [2010-10-16 Sat 18:40]

I changed org-babel-expand-body:scheme so that it applied the (mapcar #’cdr onto the (org-babel-get-header params :var). OK?

(What are those 6 spaces doing after the \n in org-babel-expand-body:scheme?)

Need to look at various calls to ob-get-src-block-info

  • State “TODO” from “” [2010-10-20 Wed 18:18] Which of them should use ‘light argument?

Dan needs to create rebased branch with clean commit logs

  • State “TODO” from “” [2010-10-20 Wed 18:19]

Create simple vectors in R?

  • State “STARTED” from “PROPOSED” [2010-10-09 Sat 13:01]
  • State “PROPOSED” from “” [2010-09-27 Mon 13:38]

Currently all variables are received in R as data frames: we cannot create simple vectors and we cannot create matrices (i.e. simple vectors with a dimension attribute).

Proposals

CurrentAB
Create data framesYesYesNo
Create R vectors?NoYesNo
Pass arbitrarily nested lists?NoNoYes

NA = no native representation

Current behaviour

OrgelispRpython
stringstringlength 1 character vectorstring
numbernumberlength 1 numeric vectornumber
NAlistdata framelist
tablelist of listsdata framelist of lists
NAother nested listERRORother nested list

Rows omitted from the proposals below are the same as in the current behaviour above.

Advantages
Disadvantages
Cannot create vectors
Cannot pass arbitrarily nested lisp lists

Proposal A: My initial patch in this thread

OrgelispR
NAlistvector
Advantages
Simple way to create vectors in R
Disadvantages
Cannot pass arbitrarily nested lisp lists

Proposal B: Using R lists

OrgelispR
NAlistlist
tablelist of listslist of lists
NAother nested listother nested list
Advantages
Conceptually simple

lisp lists == R lists

Appropriate

These are the analogous data structures in the two languages (mutable, mixed-type, arbitrarily nested lists). It solves the problem of sending nested lisp structures to R.

Disadvantages
Unnatural conversion to data frame
ab
12

is ‘((a b) (1 2))

which becomes list(list(“a”, “b”), list(1,2))

It would be natural to use data.frame(lapply(x, unlist)) but this would give us the transpose of what we want.

Cases

integer

print(str(x))

1D vector

print(str(x))

Column table

1
2
x
print(str(x))

Row table

12
x
print(str(x))

2D table

13
24
x
print(str(x))

wrap results in environment on export

  • State “TODO” from “” [2010-09-21 Tue 07:10]

Suggested by Sébastien Vauban (see gmane)

org-babel-lob-add-function

  • State “TODO” from “” [2010-09-19 Sun 21:43]

In the spirit of org-agenda-file-to-front add a new function for adding lob files, org-babel-lob-add. This should work similarly to the agenda-files function, and should not require any hand written customization files.

This will probably come in the form of org-babel-lob-add and org-babel-lob-remove functions, and a new org-babel-lob-files variable.

Now I just need to figure out how to load up these files when Org-mode is first initialized. The simplest solution may be to ask the user to place a call to org-babel-lob-initialize or somesuch in their .emacs file.

Thanks to Aidan Gauland and Jambunathan K. for the suggestion

a suggestion from Jambunathan

resolve references before cache calculation

  • State “TODO” from “” [2010-09-19 Sun 15:09]

In order to re-run cached code blocks when the values of their variables have changed we will need to resolve variables in ob.el before hash calculation. This could significantly cleanup the code by moving all variable resolution into ob.el and out of language-specific files.

This will require a change to org-babel-get-src-block-info, adding a new optional argument (or re-using HEADER-VARS-ONLY) to inhibit the full expansion of parameters and of variables in certain cases, e.g. during export when the code block exists in a temporary export buffer which may not contain the entire org-mode file.

integrate tangling with org export functionality

  • State “PROPOSED” from “” [2010-09-17 Fri 09:31]

Reasons for integrating tangling with org mode’s existing export functionality

  • tangling is a kind of export, to source code
  • ensures the original buffer remains unchanged through the temporary export buffer
  • makes expansion through tangling clear through setting the temporary export-target variable to tangle
  • would automatically expand #+INCLUDE: lines

results none header arg?

  • State “PROPOSED” from “” [2010-09-16 Thu 06:28]

I have some blocks which I’m executing in a remote session purely for their side effects (defining functions in the session). Rather than wait for the tramp machinery to transfer an empty results file back to local, I’m wondering whether a new header arg :results none might make sense.

This seems a near duplicate of the results silent header argument. Maybe none should just be an alias for silent. Or maybe silent should notice when it is used in remote execution and in such cases not bother transferring back a result?

Support ipython sessions?

Use new R-style session evaluation for other languages

Commit 08cdd0557967d69a093b1642a4467e800fe84c4b switched R to a different style of evaluation in the “:session :results value” case. It waits for the output file to be created, rather than for an end-of-output string. One advantage is a less cluttered session buffer. It may be worth investigating this switch for other languages. For example, here’s a quick go at moving python in that direction:

    Towards using ob-comint-eval-invisibly-and-wait-for-file for python

    python-send-region evaluates invisibly, so this commit results in no
    output to the python buffer. Also needs to be generalised to allow
    python-mode.

	Modified lisp/ob-python.el
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 1d2e8f5..a1e58ad 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -236,45 +236,34 @@ last statement in BODY, as elisp."
 If RESULT-TYPE equals 'output then return standard output as a
 string. If RESULT-TYPE equals 'value then return the value of the
 last statement in BODY, as elisp."
-  (flet ((dump-last-value
-	  (tmp-file pp)
-	  (mapc
-	   (lambda (statement) (insert statement) (comint-send-input))
-	   (if pp
-	       (list
-		"import pp"
-		(format "open('%s', 'w').write(pprint.pformat(_))" tmp-file))
-	     (list (format "open('%s', 'w').write(str(_))" tmp-file)))))
-	 (input-body (body)
-		     (mapc (lambda (statement) (insert statement) (comint-send-input))
-			   (split-string (org-babel-trim body) "[\r\n]+"))
-		     (comint-send-input) (comint-send-input)))
-    (case result-type
-      (output
-       (mapconcat
-	#'org-babel-trim
-	(butlast
-	 (org-babel-comint-with-output
-	     (session org-babel-python-eoe-indicator t body)
-	   (let ((comint-process-echoes nil))
-	     (input-body body)
-	     (insert org-babel-python-eoe-indicator)
-	     (comint-send-input))) 2) "\n"))
-      (value
-       ((lambda (results)
-	  (if (or (member "code" result-params) (member "pp" result-params))
-	      results
-	    (org-babel-python-table-or-string results)))
-	(let ((tmp-file (org-babel-temp-file "python-results-")))
-	  (org-babel-comint-with-output
-	      (session org-babel-python-eoe-indicator t body)
-	    (let ((comint-process-echoes nil))
-	      (input-body body)
-	      (dump-last-value tmp-file (member "pp" result-params))
-	      (comint-send-input) (comint-send-input)
-	      (insert org-babel-python-eoe-indicator)
-	      (comint-send-input)))
-	  (org-babel-eval-read-file tmp-file)))))))
+  (case result-type
+    (output
+     (mapconcat
+      #'org-babel-trim
+      (butlast
+       (org-babel-comint-with-output
+	   (session org-babel-python-eoe-indicator t body)
+	 (let ((comint-process-echoes nil))
+	   (input-body body)
+	   (insert org-babel-python-eoe-indicator)
+	   (comint-send-input))) 2) "\n"))
+    (value
+     ((lambda (results)
+	(if (or (member "code" result-params) (member "pp" result-params))
+	    results
+	  (org-babel-python-table-or-string results)))
+      (let ((tmp-file (org-babel-temp-file "python-results-")))
+	(with-temp-buffer
+	  (insert body)
+	  (let ((python-buffer session))
+	    (python-send-region (point-min) (point-max))))
+	(org-babel-comint-eval-invisibly-and-wait-for-file
+	 session tmp-file
+	 (if (member "pp" result-params)
+	     (format
+	      "import pp and open('%s', 'w').write(pprint.pformat(_))" tmp-file)
+	   (format "open('%s', 'w').write(str(_))" tmp-file)))
+	(org-babel-eval-read-file tmp-file))))))

remove existing results on nil result

  • State “DONE” from “PROPOSED” [2010-10-14 Thu 19:04]
  • State “PROPOSED” from “” [2010-09-03 Fri 15:38]

Currently when a code block returns nil Babel stops working, however it should probably remove any existing results from that block.

Resolution

This behaviour was changed at commit 1ebb9131ef0b0dbecdf08f88fdafbf6136952016a

explicit handling of header arguments without arguments

  • State “TODO” from “” [2010-09-01 Wed 22:11]

e.g.

  • “:noweb” should work as “:noweb yes”
  • when exporting blocks with :session arguments are not being evaluated in the same way as code blocks with named sessions

:exports results :results silent

  • State “PROPOSED” from “” [2010-09-01 Wed 18:16]

This should reliably evaluate the block on export, but not include it’s results in the exported file.

display results in an overlay?

  • State “PROPOSED” from “” [2010-08-31 Tue 17:25]

see Babel - display results in an overlay?

scheme support

  • State “DONE” from “” [2010-08-31 Tue 08:46]

Scheme code blocks are now supported. The scheme used for both external and interactive evaluation can be controlled through the org-babel-scheme-cmd variable and through the :scheme header argument.

The following demonstrates both regular and session based evaluation.

initial scheme support

#+source: numbers
#+begin_src scheme
  (map (lambda (el) (+ el 1)) '(1 2 3))
#+end_src

:

#+results:
| 2 | 3 | 4 |

:

#+begin_src scheme :var numbers=numbers
  (map (lambda (el) (- el 1)) numbers)
#+end_src

:

#+results:
| 1 | 2 | 3 |

scheme sessions

#+begin_src scheme :var number=9 :session *scheme* :scheme guile
  (+ number 0)
#+end_src

:

#+results:
: 9

:

#+begin_src scheme :var number=9 :session *scheme* :scheme racket
  (+ number 1)
#+end_src

:

#+results:
: 10

org-mode and code block major mode integration

  • State “DONE” from “PROPOSED” [2010-10-05 Tue 14:34]
  • State “PROPOSED” from “” [2010-08-29 Sun 23:36]

The theme of tighter integration between org-mode and the major modes of code blocks has appeared recently in a number of different forms.

  • major-mode syntax coloring of code blocks in org-mode files [1]
  • execution of code-specific major-mode functions from within org-mode files (e.g. indentation of code) [2]
  • execution of org-mode functions (e.g. tangling) from within Org-Edit-Src buffers [3]

It seems that all of these issue should be addressed in a unified manner, but it’s not clear exactly how…

Footnotes: [1] “Mode-specific fontification of babel source blocks” http://thread.gmane.org/gmane.emacs.orgmode/28307

[2] “Indent code with TAB in Org buffer” http://thread.gmane.org/gmane.emacs.orgmode/28790

[3] “Seemless editing of Babel Blocks” http://thread.gmane.org/gmane.emacs.orgmode/27788

Solution

See variables `org-src-fontify-natively’, `org-src-tab-acts-natively’ and functions `org-src-do-at-code-block’ and `org-babel-do-in-edit-buffer’.

integrate plantuml support

  • State “DONE” from “TODO” [2010-08-26 Thu 11:28]
  • State “TODO” from “” [2010-08-26 Thu 10:12]

Babel now support blocks of plantuml code. Thanks to Zhang Weize for adding this support.

Setup

  1. With the latest version of Org-mode setup consists of adding plantuml to `org-babel-load-languages’ with code like the following or through the customization interface.
    ;; active Org-babel languages
    (org-babel-do-load-languages
     'org-babel-load-languages
     '(;; other Babel languages
       (plantuml . t)))
        
  1. Then download the jar file save it somewhere on your system, set `org-plantuml-jar-path’ to point to this file.
    (setq org-plantuml-jar-path
          (expand-file-name "~/src/org/contrib/scripts/plantuml.jar"))
        

Usage

  • see http://plantuml.sourceforge.net/ for a variety of example usages, the following code block is an example of usage from within an Org-mode file.
    #+begin_src plantuml :file tryout.png
    ,  Alice -> Bob: synchronous call
    ,  Alice ->> Bob: asynchronous call
    #+end_src
    
    #+results:
    ,[[file:tryout.png]]
        

Visually cleaner session evaluation for R

  • State “STARTED” from “PROPOSED” [2010-08-25 Wed 10:53]
  • State “PROPOSED” from “” [2010-08-23 Mon 13:59]

    I’ve made progress on using low-level ess functions for the ‘:session :results value’ case. This results in much less noisy session evaluation, which is similar to the experience of sending R code from a .R buffer. This work is in branch ob-comint in git@github.com:dandavison/org-devel.git.

    Currently, with :results value, the babel session output is very noisy:

a <- 1
b <- 2
c <- 3
> a <- 1
b <- 2
c <- 3
write.table(.Last.value, file="/tmp/org-babel-R1486ljh", sep="\t", na="nil",row.names=FALSE, col.names=FALSE, quote=FALSE)
'org_babel_R_eoe'
a <- 1
> b <- 2
> c <- 3
> write.table(.Last.value, file="/tmp/org-babel-R1486ljh", sep="\t", na="nil",row.names=FALSE, col.names=FALSE, quote=FALSE)
> 'org_babel_R_eoe'
[1] "org_babel_R_eoe"
>

This contains a write.table command and the org_babel_eoe indicator and echos all the input. The write.table and eoe are internal implementation details which it would be nice to hide from the user.

Using the new R evaluation code (branch ob-comint) for the :results value :session case, currently results in the following output:

a <- 1
b <- 2
c <- 3
> a <- 1
> b <- 2
> c <- 3

> >

I currently have not managed to make the write.table command invisible without adding extra prompt characters. The invisible command is done by `org-babel-comint-eval-invisibly’. The extra prompt chars also occur with ess-eval-region when ess-eval-visibly-p is set to nil. The invisible evaluation currently blocks until the temporary transit file to come into existence rather than watching the comint buffer as is done by `org-babel-comint-with-output’.

Old task

re-implement R evaluation using ess-command or ess-execute

I don’t have any complaints with the current R evaluation code or behaviour, but I think it would be good to use the ESS functions from a political point of view. Plus of course it has the normal benefits of an API (insulates us from any underlying changes etc). [DED]

I’ll look into this. I believe that I looked at and rejected these functions initially but now I can’t remember why. I agree with your overall point about using API’s where available. I will take a look back at these and either switch to using the ess commands, or at least articulate under this TODO the reasons for using our custom R-interaction commands. [Eric]

ess-execute

Lets just replace org-babel-R-input-command with ess-execute.

I tried this, and although it works in some situations, I find that ess-command will often just hang indefinitely without returning results. Also ess-execute will occasionally hang, and pops up the buffer containing the results of the command’s execution, which is undesirable. For now these functions can not be used. Maybe someone more familiar with the ESS code can recommend proper usage of ess-command or some other lower-level function which could be used in place of org-babel-R-input-command.

ess functions

(ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)

Send the ESS process command COM and delete the output from the ESS process buffer. If an optional second argument BUF exists save the output in that buffer. BUF is erased before use. COM should have a terminating newline. Guarantees that the value of .Last.value will be preserved. When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))’ will be used in a few places where `a’ is proportional to `ess-cmd-delay’.

(ess-execute COMMAND &optional INVERT BUFF MESSAGE)

Send a command to the ESS process. A newline is automatically added to COMMAND. Prefix arg (or second arg INVERT) means invert the meaning of `ess-execute-in-process-buffer’. If INVERT is ‘buffer, output is forced to go to the process buffer. If the output is going to a buffer, name it BUFF. This buffer is erased before use. Optional fourth arg MESSAGE is text to print at the top of the buffer (defaults to the command if BUFF is not given.)

out current setup
  1. The body of the R source code block is wrapped in a function
  2. The function is called inside of a write.table function call

writing the results to a table

  1. The table is read using org-table-import

Interleaved results

  • State “TODO” from “” [2010-08-18 Wed 19:37]

See http://thread.gmane.org/gmane.emacs.orgmode/28989 a summary of which follows.

Interleaved input and output can be achieved easily in R :results output (no session) using the method below. So it might be nice to offer :results output echo, where “echo” means include the input commands. This requires:

  • implementation for :session

    (in principle sounds not hard, as it involves not doing something (i.e. not removing commands from comint output)).

  • implementation for other external interpreter processes

    For bash, it’s somewhat similar to ‘bash -x’. I looked at man python and didn’t see anything; perhaps it is accomplished by modifying the output stream when python is running rather than as a command line option to the interpreter. ruby? etc.

R :results output external interpreter process implementation:

(setq org-babel-R-command “R –silent –no-save”)

2+2
3+3

end results syntax for org formatted results

  • State “PROPOSED” from “” [2010-08-18 Wed 16:15]

This would allow :results org style results to be replaced in buffer. Thanks to David Hajage for suggesting this change.

Header argument to specify table delimiter

  • State “PROPOSED” from “” [2010-08-13 Fri 11:50]

In code blocks which don’t have a well defined table syntax (e.g. shell and ledger) it would be worthwhile to have a header argument which could be used to specify the delimiter (possibly to be passed as the optional argument to `org-table-convert-region’).

Review David O’Toole export patch

  • State “STARTED” from “TODO” [2010-08-29 Sun 18:50]
  • State “TODO” from “” [2010-08-15 Sun 13:31]

http://patchwork.newartisans.com/patch/187/

Allow contents of source blocks to be in external files

  • State “PROPOSED” from “” [2010-07-29 Thu 21:33]

Dan Davison <davison@stats.ox.ac.uk> writes:

> Hi Eric, > > I’d like to revisit the discussion that we had before about the > possibility of having src blocks in which the code is not actually > embedded in an org file. > > This is the only major obstacle I’ve encountered in using babel: what > started off as a project with tangled babel src blocks ended up as a > traditional non-babel project: the babel part died completely.

I’ve had similar issues, and see some more on the horizon.

> The reason was that some of the collaborators were non babel users, > and we needed a way for changes to the code to be propagated in all > directions among the authors. We were using git, and a related point > is that I am attracted to tracking code changes and org text changes > separately in my VCS (e.g. as would be done automatically if they > resided in separate files) > > The thread from the list discussion we had is > > http://thread.gmane.org/gmane.emacs.orgmode/24431 > > Do you have any updated thoughts on this topic? Is it relevant to babel?

I think that a general “remote block” solution would be best, maybe even a solution which allows remote tables (would include radio tables).

> > Should we be thinking in terms of org blocks or org links or both or > neither?? >

I’m not sure what the best implementation system would be. I’d check what the radio table implementation uses. Currently there is some minimal support for relating source code sections to org-mode documents through comments, however special comment insertion may not always be possible (thinking legacy code, or large projects). So ideally the solution should require no org-specific markup in the source files.

Definitely a challenging problem – I wonder if there is existing similar Emacs support for maintaining remote references to sections of files (e.g. TAGS).

Best – Eric

> > Dan

don’t add babel actions to comint input history

  • State “PROPOSED” from “” [2010-07-27 Tue 12:40]

This should be doable by protecting the value of the comint-input-ring from babel-evaluated lines.

Maybe a wrapper in org-babel-comint-with-output.

add org-mode lists as babel data types

  • State “PROPOSED” from “TODO” [2010-07-25 Sun 10:33]

from the mailing list

allow edit of header arguments from src-edit buffer

  • State “PROPOSED” from “” [2010-07-20 Tue 17:09]

Thanks to Jambunathan K for this suggestion

I like the idea of displaying the begin/end src lines as commented sections in the org-edit-special code buffer, and then potentially using them to change the values of the begin/end src lines in the original Org-mode buffer after exiting the src-edit buffer. It would be very cool to be able to edit header arguments while editing source code.

It may be hard to implement, but could be worth the effort.

library of babel for google API

  • State “STARTED” from “PROPOSED” [2010-07-20 Tue 16:53]
  • State “PROPOSED” from “” [2010-07-14 Wed 16:56]

see

Could be a number of generally useful functions here, including programatic access to google documents allowing syncing of portions of an org-mode document with google docs, useful for collaboration with people who can only use word.

Also, could be useful for syncing of org-mode tasks w/google calendar etc…

This has been started, see the two existing functions in the library-of-babel.org

function for jumping between tangled and org files

Thanks to Rainer Krug for suggesting this functionality.

If you are tangling with comments, then it should be possible to have a function called from a line in a tangled code file which when called

  1. finds it’s enclosing comments
  2. remembers it’s offset from the comments (which would then be it’s offset in the code block in the Org file)
  3. read the comment to learn which code block in which Org file it’s tangled from
  4. jump to the relevant line, in the relevant block, in the relevant file

Similarly when called form within a code block in an Org file the function could

  1. read it’s header argument to find the relevant tangled code file
  2. jump to that file
  3. use the comments in that file to move to the appropriate section of code and related line

I’ll think about such a function, and if it makes sense to implement it apart from a more general “activate org-mode links in comments” minor mode. Any ideas or suggestions would be welcome!

also,

(defun org-babel-tangle-jump-to-org ()
  "Jump from a tangled code file to the related Org-mode file."
  (interactive)
  (let ((mid (point))
        target-buffer target-char
        start end link path block-name)
    (save-window-excursion
      (save-excursion
        (unless (and (re-search-backward org-bracket-link-analytic-regexp nil t)
                     (setq start (point))
                     (setq link (match-string 0))
                     (setq path (match-string 3))
                     (setq block-name (match-string 5))
                     (re-search-forward (concat " " (regexp-quote block-name)
                                                " ends here[\n\r]") nil t)
                     (setq end (point))
                     (< start mid) (< mid end))
          (error "not in tangled code")))
      (when (string-match "::" path)
        (setq path (substring path 0 (match-beginning 0))))
      (find-file path) (setq target-buffer (current-buffer))
      (goto-char start) (org-open-link-from-string link)
      (if (string-match "[^ \t\n\r]:\\([[:digit:]]+\\)" block-name)
          (org-babel-next-src-block
           (string-to-int (match-string 1 block-name)))
        (org-babel-goto-named-src-block block-name))
      (setq target-char (point)))
    (pop-to-buffer target-buffer)
    (goto-char target-char)))

post-processing code block

  • State “PROPOSED” from “” [2010-07-12 Mon 21:59]

a new header argument which specifies a code block which accepts the output of the current block as input, and is run when the current block’s run completes.

document configuration changes for Babel integration

  • State “DONE” from “” [2010-07-05 Mon 11:03]

Babel took the integration into Org-mode as an opportunity to do some much needed house cleaning. Most importantly we have simplified the enabling of language support, and cleared out unnecessary configuration variables – which is great unless you already have a working configuration under the old model.

The most important changes regard the location and enabling of Babel (both core functionality and language specific support).

Babel
Babel is now part of the core of Org-mode, so it is now loaded along with the rest of Org-mode. That means that there is no configuration required to enable the main Babel functionality. For current users, this means that statements like
(require 'org-babel)
    

or

(require 'org-babel-init)
    

that may by lying around in your configuration must now be removed.

load path
Babel (including all language specific files – aside from those which are located in the contrib/ directory for reasons of licencing) now lives in the base of the Org-mode lisp directory, so no additional directories need to be added to your load path to use babel. For Babel users this means that statements adding babel-specific directories to your load-path should now be removed from your config.
language support
It is no longer necessary to require language specific support on a language-by-language basis. Specific language support should now be managed through the `org-babel-load-languages’ variable. This variable can be customized using the Emacs customization interface, or through the addition of something like the following to your configuration (note: any language not mentioned will not be enabled, aside from emacs-lisp which is enabled by default)
(org-babel-do-load-languages
 'org-babel-load-languages
 '((R . t)
   (ditaa . t)
   (dot . t)
   (emacs-lisp . t)
   (gnuplot . t)
   (haskell . nil)
   (ocaml . nil)
   (python . t)
   (ruby . t)
   (screen . nil)
   (sh . t)
   (sql . nil)
   (sqlite . t)))
    

Despite this change it is still possible to add language support through the use of require statements, however to conform to Emacs file-name regulations all Babel language files have changed prefix from org-babel-* to ob-*, so the require lines must also change e.g.

(require 'org-babel-R)
    

should be changed to

(require 'ob-R)
    

We have eliminated the org-babel-tangle-w-comments variable as well as the two main internal lists of languages, namely

  • org-babel-interpreters and
  • org-babel-tangle-langs

so any config lines which mention those variables, can/should be stripped out in their entirety. This includes any calls to the org-babl-add-interpreter function, whose sole purpose was to add languages to the org-babel-interpreters variable.

With those calls stripped out, we may still in some cases want to associate a file name extension with certain languages, for example we want all of our emacs-lisp files to end in a .el, we can do this will the org-babel-tangle-lang-exts variable. In general you shouldn’t need to touch this as it already has defaults for most common languages, and if a language is not present in org-babel-tangle-langs, then babel will just use the language name, so for example a file of c code will have a .c extension by default, shell-scripts (identified with sh) will have a .sh extension etc…

The configuration of shebang lines now lives in header arguments. So the shebang for a single file can be set at the code block level, e.g.

#+begin_src clojure :shebang #!/usr/bin/env clj
,  (println "with a shebang line, I can be run as a script!")
#+end_src

Note that whenever a file is tangled which includes a shebang line, Babel will make the file executable, so there is good reason to only add shebangs at the source-code block level. However if you’re sure that you want all of your code in some language (say shell scripts) to tangle out with shebang lines, then you can customize the default header arguments for that language, e.g.

;; ensure this variable is defined defined
(unless (boundp 'org-babel-default-header-args:sh)
  (setq org-babel-default-header-args:sh '()))

;; add a default shebang header argument
(add-to-list 'org-babel-default-header-args:sh
             '(:shebang . "#!/bin/bash"))

The final important change included in this release is the addition of new security measures into Babel. These measures are in place to protect users from the accidental or uninformed execution of code. Along these lines every execution of a code block will now require an explicit confirmation from the user. These confirmations can be stifled through customization of the `org-confirm-babel-evaluate’ variable, e.g.

;; I don't want to be prompted on every code block evaluation
(setq org-confirm-babel-evaluate nil)

In addition, it is now possible to remove code block evaluation form the C-c C-c keybinding. This can be done by setting the org-babel-no-eval-on-ctrl-c-ctrl-c variable to a non-nil value, e.g.

;; I don't want to execute code blocks with C-c C-c
(setq org-babel-no-eval-on-ctrl-c-ctrl-c t)

An additional keybinding has been added for code block evaluation, namely C-c C-v e.

Whew! that seems like a lot of effort for a simplification of configuration.

Improvements to error notification buffer

  • State “TODO” from “” [2010-07-13 Tue 10:21]

mention block name/number in error windows

  • State “PROPOSED” from “” [2010-06-30 Wed 09:30]

Thanks to Russel Adams for this idea.

This could be very useful both when evaluating an entire block or subtree, and also when old error messages have built up in the Error buffer.

python.el seems to make use of compile.el to provide links to errors. I wonder if there is any way we can do something similar.

Line 1632 in python.el

(with-current-buffer (process-buffer (python-proc))
  ;; Tell compile.el to redirect error locations in file `f' to
  ;; positions past marker `orig-start'.  It has to be done *after*
  ;; `python-send-command''s call to `compilation-forget-errors'.
  (compilation-fake-loc orig-start f))

q should bury the error buffer

  • State “TODO” from “” [2010-07-13 Tue 10:22]

`q’ should be bound to bury-buffer, or something similar

cache variables

  • State “PROPOSED” from “” [2010-06-27 Sun 11:39]
Working on the documentation of :cache it occurred to me that I rarely change source code, but frequently update tables passed in to the source block as variables. Since it now takes several minutes to export my adze web site, it would be nice to have a hash of the table variables cached as well and the code re-run only if one of more of the tables referenced by the variables had changed.

null values

  • State “DONE” from “” [2010-06-23 Wed 15:58]

it’s now possible to assign variable the value of nil

#+begin_src emacs-lisp :var notme='()
,  (when (null notme) (message "it's NULL"))
#+end_src

#+results:
,: it's NULL

babel bindings active in edit src buffers

  • State “PROPOSED” from “” [2010-06-21 Mon 11:21]

I often find myself switching from the source buffer, back to the org-mode buffer to tangle, this would be nice to be doable directly from the source buffer.

On a related note, I think a prefix argument to tangle having it only tangle the current block would be nice.

resolve remove links on export

  • State “PROPOSED” from “” [2010-06-17 Thu 14:27]

Thanks to Austin Frank for brining this up

this may be an org-mode issue

tangle entire org-mode file in comments

  • State “DONE” from “PROPOSED” [2010-09-04 Sat 08:47]
  • State “PROPOSED” from “” [2010-06-16 Wed 20:37]

The :comments header argument has been expanded to support the following options. These should implement at least a first pass at the functionality described below. The new header arguments are…

no
retains its behavior of not tangling any comments
yes
retains its behavior of wrapping the code in links back to the original org-mode file
link
is synonymous with “yes”
org
does not wrap the code in links back to the original org file, but does include preceding text from the org-mode file as a comment before the code block
both
turns on both the “link” and “org” options

examples are given of the effects of these header arguments (notice that the header arguments are contains as properties in the containing headline)

examples

tangling with no comments

This

**** tangling with no comments
   :PROPERTIES:
   :comments: no
   :tangle:   comments.el
   :END:
The top block
#+begin_src emacs-lisp
  (message "first block")
#+end_src

:

here's some text which won't be tangled

:

***** subheading
another block
| 1 | first  |
| 2 | second |
#+source: tangle-el-the-second
#+begin_src emacs-lisp
  (message "second")
#+end_src

:

and finally a block with a =:noweb= header argument
#+begin_src emacs-lisp :noweb yes
  (progn
    <<tangle-el-the-second>>)
#+end_src

results in

(message "first block")

(message "second")

(progn
  (message "second"))

tangling with yes or link comments

This

**** tangling with yes or link comments
   :PROPERTIES:
   :comments: yes
   :tangle:   comments.el
   :END:
The top block
#+begin_src emacs-lisp
  (message "first block")
#+end_src

:

here's some text which won't be tangled

:

***** subheading
another block
| 1 | first  |
| 2 | second |
#+source: tangle-el-the-second
#+begin_src emacs-lisp
  (message "second")
#+end_src

:

and finally a block with a =:noweb= header argument
#+begin_src emacs-lisp :noweb yes
  (progn
    <<tangle-el-the-second>>)
#+end_src

results in

;; [[file:~/src/babel-dev/development.org::*tangling%20with%20yes%20or%20link%20comments][tangling-with-yes-or-link-comments:1]]

(message "first block")

;; tangling-with-yes-or-link-comments:1 ends here

;; [[file:~/src/babel-dev/development.org::*subheading][tangle-el-the-second]]

(message "second")

;; tangle-el-the-second ends here

;; [[file:~/src/babel-dev/development.org::*subheading][subheading:2]]

(progn
  (message "second"))

;; subheading:2 ends here

tangling with org comments

This

**** tangling with org comments
   :PROPERTIES:
   :comments: org
   :tangle:   comments.el
   :END:
The top block
#+begin_src emacs-lisp
  (message "first block")
#+end_src

:

here's some text which won't be tangled

:

***** subheading
another block
| 1 | first  |
| 2 | second |
#+source: tangle-el-the-second
#+begin_src emacs-lisp
  (message "second")
#+end_src

:

and finally a block with a =:noweb= header argument
#+begin_src emacs-lisp :noweb yes
  (progn
    <<tangle-el-the-second>>)
#+end_src

results in

;; tangling with org comments
;;    :PROPERTIES:
;;    :comments: org
;;    :tangle:   comments.el
;;    :END:
;; The top block

(message "first block")

;; ***** subheading
;; another block
;; | 1 | first  |
;; | 2 | second |
;; #+source: tangle-el-the-second

(message "second")

;; and finally a block with a =:noweb= header argument

(progn
  (message "second"))

tangling with both comments

This

**** tangling with both comments
   :PROPERTIES:
   :comments: both
   :tangle:   comments.el
   :END:
The top block
#+begin_src emacs-lisp
  (message "first block")
#+end_src

:

here's some text which won't be tangled

:

***** subheading
another block
| 1 | first  |
| 2 | second |
#+source: tangle-el-the-second
#+begin_src emacs-lisp
  (message "second")
#+end_src

:

and finally a block with a =:noweb= header argument
#+begin_src emacs-lisp :noweb yes
  (progn
    <<tangle-el-the-second>>)
#+end_src

results in

;; tangling with both comments
;;    :PROPERTIES:
;;    :comments: both
;;    :tangle:   comments.el
;;    :END:
;; The top block

;; [[file:~/src/babel-dev/development.org::*tangling%20with%20both%20comments][tangling-with-both-comments:1]]

(message "first block")

;; tangling-with-both-comments:1 ends here

;; ***** subheading
;; another block
;; | 1 | first  |
;; | 2 | second |
;; #+source: tangle-el-the-second

;; [[file:~/src/babel-dev/development.org::*subheading][tangle-el-the-second]]

(message "second")

;; tangle-el-the-second ends here

;; and finally a block with a =:noweb= header argument

;; [[file:~/src/babel-dev/development.org::*subheading][subheading:2]]

(progn
  (message "second"))

;; subheading:2 ends here

preliminary discussion

TD notes that Babel already supports LP, which is designed to make excellent notes.

Belatedly documenting this issue from the mailing list. This issue has resurfaced on the mailing list, see

The main idea being that it would be nice to allow large chunks of an org-mode file to be exported as comments to tangled source code files. That way all the excellent documentation is not lost. The main questions seem to be about the best implementation. The entire thread is available on gmane at gmane.emacs.orgmode/25779.

some excerpts

Erik Iverson
Tom Dye

begin throwing errors on malformed header arguments

  • State “PROPOSED” from “” [2010-06-15 Tue 15:00]

It is easy to type in a bad header argument string and get confusing results. For example this

#+begin_src python results:output
,  print 'hello'
#+end_src

doesn’t return any value, while this does

#+begin_src python :results output
,  print 'hello'
#+end_src

#+results:
,: hello

Thanks to Ben Edwards for this suggestion.

set header argument values on a buffer bases

  • State “DONE” from “TODO” [2010-06-14 Mon 12:07]
  • State “TODO” from “” [2010-06-14 Mon 11:38]

This is currently implemented using a #+BABEL: line similar to the existing #+OPTIONS: style lines.

#+BABEL: :var number=8

#+begin_src ruby
,  number + 1
#+end_src

#+results:
,: 9

NOTE: currently once this value is read the list of header arguments is cached in a buffer local variable to avoid excessive load while exporting many source blocks. This means that if you make changes to the contents of the #+BABEL: line you may need to close and re-open the buffer for these changes to take place, or run something like the following which will clear out the current value of the local variable

any suggestions for a better alternative are welcome.

(defun babel/clear-buffer-header-arguments ()
  (interactive)
  (setq org-babel-current-buffer-properties nil))

Org-mode and Emacs integration

  • State “DONE” from “STARTED” [2010-06-23 Wed 13:25]
  • State “STARTED” from “TODO” [2010-06-11 Fri 15:25]
  • State “TODO” from “” [2010-06-11 Fri 11:08]

update the initialization infrastructure

  • State “DONE” from “STARTED” [2010-06-11 Fri 17:22]
  • State “STARTED” from “TODO” [2010-06-11 Fri 15:26]

This required changing the name of the require statements, so all users will have to switch from e.g.

(require 'org-babel-sh)

to

(require 'ob-sh)

All other options led to overly baroque and tortured load sequences.

documentation strings for Emacs integration

  • State “DONE” from “TODO” [2010-06-11 Fri 17:23]
  • State “TODO” from “TODO” [2010-06-11 Fri 11:09]

the first line of each docstring must be a complete, self-contained sentence.

ugh… complete

can’t use cl functions

  • State “DONE” from “TODO” [2010-06-23 Wed 13:25]
  1. Packages that are part of Emacs are not allowed to load cl at runtime. So you can use (eval-when-compile (require ‘cl)) in order to get the cl macros expanded, but you cannot use cl functions. Annoying yes - but this is how it is. Basically, any cl functions we want to use we have to find alternatives for it in normal emacs Lisp code, or write our own org-… version.

If the compiler is nice enough to complain about these at compile time, I guess we’ll just wait and let it sort this out for us.

see the compilation errors at compiler-warnings

unique ob-*.el file names

  • State “DONE” from “TODO” [2010-06-11 Fri 15:26]
  1. All file names in Emacs must be 8+3 unique, so the first 8 characters of a file name must make it unique. File names org-babel- something.el will not work.

move files over into the main lisp dir

  • State “DONE” from “TODO” [2010-06-11 Fri 15:14]
  • State “TODO” from “” [2010-06-11 Fri 15:07]

simple

Org-babel should work well when indented

  • State “DONE” from “TODO” [2010-06-10 Thu 15:48]
  • State “TODO” from “” [2010-06-10 Thu 10:13]

This should be fairly straightforward.

This actually turned into a fairly comprehensive reworking of info variable in Org-babel. It now carries indentation information, and most all org-babel functionality should now work when indented.

do we want to make the naming of code block optional?

  • State “PROPOSED” from “TODO” [2010-06-08 Tue 16:38]

Do we always want every named code block to export with a name?

It seems like this could be a useful place for a customization variable.

note that the actual exportation is handled in org-exp.el (see file:~/src/org/lisp/org-exp.el::format n s equiv n caption)

don’t require named arguments

  • State “TODO” from “TODO” [2010-06-08 Tue 11:52]
  • State “TODO” from “” [2010-06-08 Tue 11:49]

The requirement for named arguments is a little overmuch, when the argument can be inferred form the order in which they’re received.

output filter functions

  • State “TODO” from “” [2010-06-08 Tue 11:48]

Would be great to have functions which will automatically be called on the output of a code block every time it is run.

There should be a notation for specifying which of the arguments to the function is to come from the current code block – and a convention so that if the function only accepts a single argument, then there is no need to specify any arguments.

e.g. notice the :filter header argument in the second block

(+ input 1)
8

the above should output 9

evaluation of code block on export

  • State “DONE” from “TODO” [2010-06-06 Sun 19:10]
  • State “TODO” from “” [2010-06-06 Sun 15:00]

Currently when exporting a code block with the :exports code header argument, the code block is not evaluated. this can lead to unexpected behavior when there is a session argument, in situations like the following

** inline expressions
,   :PROPERTIES:
,   :session:  default
,   :END:

#+begin_src R :exports code :results silent
,  x<-8
#+end_src

,the sum of 1 and 4 is equal to src_R{x}

Lets try evaluating source code blocks when there is a session argument, and when the :exports code header argument is present.

There should however be a way to inhibit this evaluation. And now there is, through application of a :noeval header argument.

lob function for reading files

  • State “DONE” from “TODO” [2010-06-06 Sun 19:53]
  • State “TODO” from “” [2010-06-06 Sun 10:11]

This runs up against an increasingly common theme of tension between implementing functionality in org-babel vs. in host languages.

Nearly every language which would be used inside of a code block has it’s own mechanisms for reading the contents from a file, so to what extent do we want to implement this functionality inside of org-babel vs. leaving this up to the host language.

I think this may be a good place for a library of babel function.

Once we have an lob function, say read-file then we can place

:var path=”doi:10.1000/182” or :var path=”doi:10.1000/182”

in a header argument to pass a path to a code block, and we can place

:var path=read-file(path=”doi:10.1000/182”) or :var path=read-file(path=”doi:10.1000/182”)

in a header argument to pass the contents of a file to a code block.

Should the files be returned as a string, or as a list of lines? How about both.

read
reads the file and returns one large string
read-lines
reads the file and returns a list of lines

(see the library of babel in contrib/babel/library-of-babel.org)

Org-babel-notebook

  • State “TODO” from “” [2010-05-07 Fri 15:29]

How to make it possible to use org-babel interactively from within an interactive terminal (read: notebook, REPL, shell, or comint buffer – whatever name makes sense to you), rather than in it’s current context of an Org-mode file.

This would require a number of new features…

  1. a nice interface
    • embedded images, and videos, or arbitrary x-windows (e.g. a screen make by gnuplot), maybe a generic
    • allow input of multi-line code segments – maybe related to language selection…
    • allow for switching the current language
    • possibly allow function definitions
    • possibly support some form of export (maybe should export to an org-mode buffer which would then export arbitrarily)
  2. a way to maintain a persistent environment, s.t. variables and functions can maintain a persistent value over the course of a session
  3. sage for example provides a base library of operations, which can be called outside of the use of any particular package. In the same way we should provide for calling basic operations directly without having to open up a code block, examples could be a generic means of graphing objects, simple arithmetic and matrix operations, file I/O, etc…

    we should be able to use the existing lob and named code block functionality to define all of these primitives – probably using elisp as the preferred language for these code blocks.

  4. undoubtedly there is more…

Another popular notebook system is sage, maybe some inspiration can be drawn from there.

It could be nice to wrap up calc functions so that they can easily be called from this command line.

Hopefully there’s a rational way to extend the existing Org-babel functionality into this sort of an interface without either adding too many ad-hoc features or writing too much code.

interface – Org-mode comint buffer

just to stream-of-consciouses for a moment.

how will this comint buffer look, and should it even be a comint buffer?

Increasingly since Org-babel is fully capable of handing back org-mode formatted text from any execution, I’m thinking that the interactive notebook should have Org-mode as it’s major mode.

Then each execution (every time the user presses enter to send a line of code to the interpreter) could generate a new headline, and both the results and the related code could be placed under this headline.

In this way a series of executions at the “prompt” would result in a series of folded headlines above the prompt, with the latest headline always unfolded and immediately below the prompt. I think this could turn out to be a fairly natural interface.

normalize the expand header argument syntax

  • State “TODO” from “” [2010-04-27 Tue 11:35]

code block body expansion for table and preview

  • State “DONE” from “” [2010-04-23 Fri 09:44]

    In org-babel, code is “expanded” prior to evaluation. I.e. the code that is actually evaluated comprises the code block contents, augmented with the extra code which assigns the referenced data to variables. It is now possible to preview expanded contents, and also to expand code during during tangling. This expansion takes into account all header arguments, and variables.

preview
A new key-binding C-c M-b p bound to `org-babel-expand-src-block’ can be used from inside of a source code block to preview its expanded contents (which can be very useful for debugging).
tangling
The expanded body can now be tangled, this includes variable values which may be the results of other source-code blocks, or stored in headline properties or tables. One possible use for this is to allow those using org-babel for their emacs initialization to store values (e.g. usernames, passwords, etc…) in headline properties or in tables. The :no-expand header argument can be used to inhibit expansion of a code block body during tangling.

Here is an example of a code block and its resulting expanded body.

The data in the file.

usernamejohn-doe
passwordabc123

The actual code block

(setq my-special-username (first (first data)))
(setq my-special-password (first (second data)))

its expanded contents, as seen with C-c M-b p

(let ((data (quote (("john-doe") ("abc123")))))
(setq my-special-username (first (first data)))
(setq my-special-password (first (second data)))
)

Clarify situation re. org-babel-interpreters, org-babel-tangle-langs, src-lang-modes

  • State “DONE” from “STARTED” [2010-06-17 Thu 09:46]
  • State “STARTED” from “TODO” [2010-06-17 Thu 07:51]
  • State “TODO” from “” [2010-04-15 Thu 13:14]

implementation

removed all language lists (`org-babel-interpreters’ and `org-babel-tangle-langs’) from Org-babel. We no longer explicitly check if languages are on lists of supported languages, but rather simply check that the appropriate functions (e.g. `org-babel-execute:sh’) are defined at the times when they are needed.

changes

  • Instead of adding anything to external variables, we are going to rely on the existing header argument system to hold information on commentability and tangling of languages
  • From now on users will have to explicitly request comments in tangled code by setting the :comments header argument to yes.
  • We still want to be able to associate languages with file extensions, now done with `org-babel-tangle-lang-exts’. This is the only remaining list of languages used by Org-babel.

requirements

It seems that this has got overly complicated; time for some simplification and harmonisation.

See also bug Language mappings in org-src-lang-modes should be honoured by babel

-Lets just subvert org-src-lang-modes and use it for everything- (not actually making any changes to this variable). It would now hold

  • language major mode
  • language extension

Then when we need to check whether or not the language lang is executable we can just see if we have a org-babel-execute:lang function.

This can happen after the merge of our code into the main body of org-mode to make the shared variable a little bit more natural.

raw links in uses.org on Worg

  • State “TODO” from “” [2010-04-12 Mon 08:53]

∀ topics linked from the uses page we need to add a link to the raw version in some uniform manner – maybe right next to the html link in the headline.

org-babel-R show comint buffer patch from Julien Barnier

From: Julien Barnier <julien@no-log.org>
Subject: [Orgmode] [PATCH] [babel] Add option to display process buffer when editing R source code blocks
User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/23.1.91 (gnu/linux)
Date: Wed, 07 Apr 2010 16:41:55 +0200
To: emacs-orgmode@gnu.org

Hi,

This is a small patch to org-babel-R.el which allows to automatically
display the R process buffer when editing R source code blocks with
org-edit-src-code.

A custom variable allows to choose between no process buffer
(default), only the source code block and the process buffer, or the
org file, the source code block and the process buffer.

As I'm quite new to git, I hope my patch is usable, because I
generated it from a org-babel-R.el file which already had some
modifications from master.

Sincerely,

--
Julien


---
 contrib/babel/lisp/langs/org-babel-R.el |   27 ++++++++++++++++++++++++++-
 1 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/contrib/babel/lisp/langs/org-babel-R.el b/contrib/babel/lisp/langs/org-babel-R.el
index 8b333cc..3924089 100644
--- a/contrib/babel/lisp/langs/org-babel-R.el
+++ b/contrib/babel/lisp/langs/org-babel-R.el
@@ -218,7 +218,32 @@ Currently, insert hline if column names in output have been requested."
   (if column-names-p
       (cons (car result) (cons 'hline (cdr result)))
     result))
-
+
+
+(defcustom org-babel-R-edit-src-show-process nil
+  "Layout of windows while editing R source blocks in org files"
+  :group 'org-babel
+  :type '(choice (const :tag "No process buffer" nil)
+                (const :tag "Show source block and process buffer" "full")
+                (const :tag "Show org file, source block and process buffer" "split")))
+
+(defadvice org-edit-src-code (around org-edit-src-code-with-R-process activate)
+  "Display process buffer when eidting R source code blocks"
+  (if org-babel-R-edit-src-show-process
+    (let* ((info (org-babel-get-src-block-info))
+          (lang (first info))
+          (R-src-block (and info (string= (upcase lang) "R"))))
+      ad-do-it
+      (when R-src-block
+       (cond ((string= org-babel-R-edit-src-show-process "split")
+              (split-window-vertically)
+              (ess-switch-to-end-of-ESS)
+              (other-window -1))
+             ((string= org-babel-R-edit-src-show-process "full")
+              (delete-other-windows)
+              (ess-switch-to-end-of-ESS)
+              (other-window 1)))))
+    ad-do-it))

 (provide 'org-babel-R)
 ;;; org-babel-R.el ends here
--
1.7.0.4




_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Jump to line mapping to line i in tangled output

  • State “PROPOSED” from “” [2010-04-03 Sat 15:26] If a compiler/interpreter reports an error on line i of tangled output, jump to that line in the org buffer.

Turn off comments in tangling on per-buffer or per-block basis

  • State “PROPOSED” from “” [2010-04-03 Sat 11:55]

    To turn off comments in perl tangling the best I came up with was this. What is an easier method?

(setq org-babel-tangle-langs
      (cons '("perl" "pl" "#!/usr/bin/env perl" t)
            (remove-if (lambda (el) (equal (car el) "perl"))
                       org-babel-tangle-langs)))

Option to apply table formulae after C-c C-c

  • State “PROPOSED” from “” [2010-04-02 Fri 08:57]
'((1 2) (3 nil))

Can we make it so that after C-c C-c, the table looks like this?

12
33

Add append to results handling

  • State “DONE” from “PROPOSED” [2010-06-07 Mon 16:46]
  • State “PROPOSED” from “TODO” [2010-06-05 Sat 19:11]
  • State “TODO” from “PROPOSED” [2010-06-05 Sat 19:02]
  • State “PROPOSED” from “” [2010-04-01 Thu 10:34]

(implemented while fixing the results deletion and insertion bug)

I’m developing general purpose org-babel functions that run queries for frequently performed analyses. In some instances it isn’t possible to make the queries completely general, so I’m left with combining the results of two or more separate calls to the org-babel functions.

This is an example. Two runs are needed to make a table with coral abraders and basalt manuports, when there are also basalt abraders and coral manuports in the data table. If I set material=”(‘coral’, ‘basalt’)” and class=”(‘abrader’,’manuport’) then I get all four types of artifact in the results.

<<r-connect-to-data>>

q <- sprintf("SELECT c.context, a.class,
round(sum(a.weight),1) AS weight FROM artifacts a, baglist b,
context c WHERE a.bag = b.bag AND b.context = c.context AND
a.project = %i and b.project = %i and c.project = %i AND a.class IN %s
AND a.material IN %s AND c.context_type = 'cultural' AND c.duration IN %s  AND b.context >= %i AND b.context <= %i group by 1,
2 order by 1, 2", project, project, project, class, material, duration, start,
end)

art.wt <- dbGetQuery(con, q)
<<r-disconnect>>
art.wt

It would be useful in this case, and perhaps other cases, to have a :results append argument. An org-mode table would be created if it is absent, or the function results would be appended to an existing results table if present.

EMS thoughts

this seems like a good idea, and should be easy to implement. The next step would be to allow pre-pending or insertion at arbitrary points in the table, or to define combining functions (e.g. if the result is a scalar rather than a table to sum, or multiply, or concatenate the values), this then comes full-circle back to maybe the results block should just be another argument to the code block (which raises an issue of circular code block calls, and a related issue of controlling whether a block need be re-run if it’s results already exist in the buffer), and appending shouldn’t be explicitly built into org-babel.

Hmm, I see three good options here – there are certainly more;

  1. sort out all issues related to re-running code blocks which already have in-buffer results s.t. it is possible for a block to reference it’s own results
  2. add a simple append option
  3. add something like results incorporating functions which could provide a more generic functionality subsuming appending of functions

documentation for org-mode inclusion

  • State “DONE” from “TODO” [2010-06-23 Wed 13:26]
  • State “TODO” from “” [2010-04-01 Thu 08:27]

First of all, I would think we have a separate chapter about Org Babel. My feeling is that this chapter should be no longer than 10, maybe 15 pages to make it not dominate the manual. I am not quite sure what we can put into this envelope. It definitely should be enough to explain the basics, and one or two simple examples. This will not replace the extensive tutorials etc on the web, but my hope would be that all essential information from but maybe the information http://orgmode.org/worg/org-contrib/babel/ reference.php can be captured in there.

I have actually been thinking to switch from @example to @smallexample (and similar) for the entire manual - this would save quite some space, I believe. But I still need to make test prints to see if this is a viable option.

Then we will mention Org babel stuff where-ever necessary - obviously in the section about literal examples, and in the export chapter. I don’t see right now where else - yes, in the Introduction of course, with one sentence.

  • Carsten

I just created a babel-doc branch up in the babel forked repository on repo.or.cz which can hold this work. Hopefully it will largely be a straightforward process of converting reference.org from Worg into a texinfo and manual style document, and then similarly transferring over a couple of examples.

Allow return variable specification?

  • State “PROPOSED” from “” [2010-03-30 Tue 10:10]

    From Darlan Cavalcante

In fact, I don’t like to put a return statement in the end of the block just to make babel happy (python included), since I don’t want this when I tangle the file, but this is the most flexible way.

Maybe a good feature request for babel it that instead “exports: value” one has to write “exports: value variable” and this would have the effect of the “return variable” in the end of the block that we have right now for python.

[Dan] I think this might still suffer from the problem of not knowing where to insert the “return statement”. Because I believe a return statement will be necessary, whether inserted automatically or not.

Octave and Matlab tasks

  • State “TODO” from “” [2010-03-30 Tue 10:09]

From Darlan Cavalcante

If I understood correctly you always save the output to a file using the function org-babel-octave-wrapper-method, right? Maybe instead of using “%s save -ascii %s ans” to save the information to a file you can use “%s save -text %s ans” I tested this for numbers, strings and matrices and it seems to do the right thing. However, octaves adds some comment lines in the beginning of the file that you will probably need to remove, but you just need to remove lines that start with “#”.

Regarding which value is returned, maybe you can use the same approach employed for python, where a line “return variable” is required in the end of the block, instead of just returning the value of ans.

In fact, I don’t like to put a return statement in the end of the block just to make babel happy (python included), since I don’t want this when I tangle the file, but this is the most flexible way.

Maybe a good feature request for babel it that instead “exports: value” one has to write “exports: value variable” and this would have the effect of the “return variable” in the end of the block that we have right now for python.

Regarding the graphical output, It is probably better to use :results file as suggested in the TODO. There may be necessary a lot of details in the graphic output such as labels, setting axis, etc.. It is better to let the user handle this in the code and just put the link to the file.

Get matlab “Emacs Link” working

  • State “DONE” from “TODO” [2010-05-21 Fri 20:43]
  • State “TODO” from “” [2010-04-30 Fri 17:00]

Current solution is imperfect: it involves the matlab process signaling completion by removing a lock file rather than using the canonical org-babel comint completion functions.

How to detect if result is numeric or string?

  • State “TODO” from “” [2010-03-18 Thu 12:37]

    Strings were being returned as their ascii codes.

    We have adopted Darlan’s suggestion of using save -text %s ans (and delting the comment lines), which seems to provide a solution.

'octave string'
'string from matlab'

How to divert graphical output to file?

  • State “TODO” from “” [2010-03-17 Wed 23:3

    Currently we are like pyuthon and ruby in that we can use :results file and the block will output a link to that file, but unless your code takes care of it (e.g. write graphics to file), no content will be written to that file.

    However, would it be appropriate to move to the org-babel-R model, whereby if :results file is supplied, any graphical is automatically diverted to that file?

Related: getting graphical output from octave

In order to get graphical output without leaving open graph windows during evaluation, the following can be used:

figure( 1, "visible" , "off" );
sombrero;
print -dpng chart.png

The figure command opens an invisible graph window where the next plot commands should act.

Then, the print command exports the graph to a file.

Ensure that value is returned when expected

  • State “TODO” from “” [2010-03-18 Thu 13:57]

    In :results value mode, we return whatever value the variable ‘ans’ has at the end of the block. However, there are some problems with this as can be seen from the following session:

octave> 'hello'
ans = hello
octave> ans
ans = hello
octave> x = 19
x =  19
octave> x
x =  19
octave> ans
ans = hello

So although this works as expected:

z + 0

This doesn’t

z

As part of its internal operation, org-babel appends a special end-of-output string at the end of each block and it is this string’s value (integer ascii codes) that is being returned here.

Of course, this works

ans

Allow multiple sessions

  • State “TODO” from “” [2010-03-16 Tue 22:12]

Should we use these by default to save external process startup time?

  • State “TODO” from “” [2010-03-18 Thu 01:27]
  • -nodisplay ?
  • -nojvm ?

How to implement :results pp, :results code?

  • State “TODO” from “” [2010-03-18 Thu 01:10]

How to get rid of the matlab splash message

  • State “TODO” from “” [2010-03-18 Thu 01:09]

    -nosplash doesn’t seem to do the trick.

submit patch to matlab.el so that matlab session can be remote

  • State “TODO” from “” [2010-03-18 Thu 01:29]

At the moment I’m using a massive hack: insert

(cd “/ddavison@tombstone.princeton.edu:”) ;; DED

around line 4253 in matlab.el so that my matlab sessions run on a machine with matlab installed.

Customize interface

  • State “TODO” from “” [2010-03-26 Fri 00:31] Make sure we’re happy with the customize interface in the org-babel group.

literal values from tables

  • State “DONE” from “” [2010-03-25 Thu 13:23]

string values passed to :var header arguments are assumed to be source name references, so upon encountering

#+begin_src emacs-lisp :var data=something

babel will rush off in search of a table, result, or code block named “something” with which to initialize data, rather than passing the literal string value of something. This can lead to confusing behavior when dealing with tables, where for example

#+TBLNAME: system-host-ping :var host=system-hosts
| name   |             ip | ping   |
|--------+----------------+--------|
| host 1 | 192.168.10.200 | #ERROR |
| host 2 |  192.168.10.24 | #ERROR |
| host 3 |  192.168.42.24 | #ERROR |
#+TBLFM: $3='(sbe system-ping (ip $2))'

will result in all errors because each ip address is interpreted as a reference to be resolved rather than as a literal value. To fix this behavior the following $$ syntax has been added which can force table values to be interpreted as literal strings, resulting in the following

#+TBLNAME: system-host-ping :var host=system-hosts
| name   |             ip |           ping |
|--------+----------------+----------------|
| host 1 | 192.168.10.200 | 192.168.10.200 |
| host 2 |  192.168.10.24 |  192.168.10.24 |
| host 3 |  192.168.42.24 |  192.168.42.24 |
#+TBLFM: $3='(sbe system-ping (ip $$2))'

support stuff

#+source: system-ping
#+begin_src sh :var ip="192.168.42.24"
# Testing
echo $ip
#+end_src

:

#+results: system-ping
: 192.168.42.24

Add SQL functionality?

  • State “PROPOSED” from “” [2010-03-21 Sun 15:59]

I happened to notice this on #emacs

<spcshpopr8r> hallo emacsers
<spcshpopr8r> I have successfully, finally, got on board with org-babel and
             ess...earlier today, my head exploded  [15:39]
*** JayM (~jay@cpe-76-186-94-148.tx.res.rr.com) has quit: Ping timeout: 268
    seconds
*** CrustY (~klon@91.187.21.231) has quit: Read error: Connection reset by
    peer
<spcshpopr8r> now I want to execute an arbitary bunch of sql against an oracle
             database and then feed the results to an R block
*** CrustY (~klon@91.187.21.231) has joined channel #emacs  [15:40]
<jave> btw org vs planner?
*** gnuvince (~vince@72.0.219.6) has quit: Ping timeout: 256 seconds
*** kenshin (~kenshin@59.95.54.181) has quit: Quit: Ex-Chat
<spcshpopr8r> I've looked a little bit at org-babel-sql.el and it seems pretty
             spartan...like it just wants to talk to a mysql database at the
             moment  [15:41]
*** gnuvince (~vince@72.0.219.6) has joined channel #emacs
*** Colloguy (~flx@64.134.18.188) has joined channel #emacs
<spcshpopr8r> anybody here got a simple solution to get org-babel
             (org-babel-sql that is) to talk to my li'l ol oracle databases?

Ideally there’s a single sql-mode for emacs that we can interact with… it looks like sql-mode is the way to go. I’d say we add that as a requirement to org-babel-sql, and then start building up the integration with…

  1. start using sql-mode builtins for connection – which should extend us to other database engines
  2. add more header arguments (passwrd, database, username, etc…)
  3. support persistent sessions
  4. support returning query results as lists/tables
  5. support dropping lists/tables into insert statements

allow evaluation of emacs-lisp variable values

  • State “DONE” from “” [2010-03-22 Mon 08:40]

for example, the following simple example

#+begin_src emacs-lisp :var two=(+ 1 1)
  (sqrt two)
#+end_src

:

#+results:
: 1.4142135623730951

Or this more interesting usage, which pulls variable values from headline properties

*** example headline w/property
    :PROPERTIES:
    :special:  89
    :last-name: schulte
    :END:

:

#+begin_src emacs-lisp :var special=(string-to-number (org-entry-get nil "special" t))
  (+ special 1)
#+end_src

:

#+results:
: 90

:

#+begin_src emacs-lisp :var last-name=(org-entry-get nil "last-name" t))
  (message "hello %s" last-name)
#+end_src

:

#+results:
: hello schulte

Allow tramp syntax with :tangle

Email from Maurizio Vitale

From: Maurizio Vitale <mav@cuma.i-did-not-set--mail-host-address--so-tickle-me>
Subject: [Orgmode] [babel] using tramp when tangling
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (gnu/linux)
Date: Wed, 17 Mar 2010 14:39:43 -0400
To: emacs-orgmode@gnu.org


I'd find useful to use tramp syntax in the :tangle specification.
In my case it would be to specify sudo when tangling config files that
are supposed to go to areas not writable by the user running Emacs.
Something like:

...

other people might be interested in remote access to tangle targets.

Is there a way to achieve the above?

If I try to tangle the above, I get something along the lines that
"tramp cannot append to file". Would it be possible to have org-babel to
(optionally) tangle to a buffer and then save to file in one go?

Thanks a lot for the excellent addition to org-mode.

       Maurizio

Make org-babel-switch-to-buffer show code edit buffer

  • State “DONE” from “PROPOSED” [2010-08-29 Sun 18:53]
  • State “PROPOSED” from “” [2010-03-20 Sat 12:20]

    Done: see commit b05f8c91fed5d743adf5df787f2b28fb58274bf5

    See also org-babel-R show comint buffer patch from Julien Barnier

    This patch changes org-babel-switch-to-buffer so that the session and the code edit buffer are shown side-by-side, with point in the code edit buffer. With R, I find this to be the behaviour that I most often want, and I think it will be good for introducing ESS users to org-babel.

    I don’t know how whether we want to change org-babel-switch-to-buffer as below, or whether we make this functionality available via some other interface (a different function, prefix args, etc).

diff --git a/contrib/babel/lisp/org-babel.el b/contrib/babel/lisp/org-babel.el
index d01e6d6..5121fbc 100644
--- a/contrib/babel/lisp/org-babel.el
+++ b/contrib/babel/lisp/org-babel.el
@@ -281,9 +281,11 @@ of the source block to the kill ring."
     (with-temp-buffer (insert (org-babel-trim body)) (copy-region-as-kill (point-min) (point-max)))
     ;; if called with a prefix argument, then process header arguments
     (if arg (funcall (intern (concat "org-babel-prep-session:" lang)) session params))
-    ;; just to the session using pop-to-buffer
-    (pop-to-buffer (funcall (intern (format "org-babel-%s-initiate-session" lang)) session params))
-    (move-end-of-line 1)))
+    ;; switch to the session using pop-to-buffer
+    (save-excursion
+      (pop-to-buffer (funcall (intern (format "org-babel-%s-initiate-session" lang)) session params))
+      (move-end-of-line 1))
+    (org-edit-src-code)))

 (defalias 'org-babel-pop-to-session 'org-babel-switch-to-session)

Make some org-babel commands available from edit buffer?

  • State “DONE” from “PROPOSED” [2010-08-29 Sun 18:55]

    Done: see commit 441288ee72778931c3e1853b8221f7049289df91

For example, it might be nice to be able to do the following from the code edit buffer:

  1. switch back to parent org buffer, putting point at start of respective code block
  2. tangle

    I think Org-src-mode already saves the necessary information

(e.g. parent buffer, start of block).

Extend org executable link constructs

  • State “PROPOSED” from “” [2010-03-03 Wed 14:04] I just noticed the following link types in the org manual
shell:ls *.org                            A shell command
elisp:org-agenda                          Interactive Elisp command
elisp:(find-file-other-frame "Elisp.org") Elisp form to evaluate

The above can be placed within the standard construct:

[[link][description]]

I wonder whether we had that sufficiently in mind when we designed our in-line blocks? Some possibilities that come to mind are:

The above suggests that babel could extend org by supporting

[[ruby: 2+2]] and [[ruby: 2+2][description]]

etc.

And perhaps header args in {} rather than []

[[ruby{:session}]] and [[ruby{:session}: 2+2][description]]

If we needed to differentiate the babel handler from the org handler then we could use

[[src_ruby: 2+2]]

although I think I would like to keep the syntax identical, using variables to decide who handles what, so that it is invisible to the user.

Internals - non user-visible changes

  • State “TODO” from “” [2010-03-20 Sat 12:34]
[ ] org-babel-LANG-evaluate variable name
In some languages the argument list includes BUFFER but we then refer to SESSION in the function.
[ ] split org-babel-LANG-evaluate into session and non-session
Like clojure does.
[ ] tangle to buffers and write to file at end
Now that we don’t use append-to-file (see 73d8e5768570d62e79f19b117b599f668b6abc17), it would be more efficient to append the tangled output to buffers, and ultimately write the buffers to file and kill the buffers. I didn’t implement that, as it required a fair bit of kessing about with org-babel-tangle, but if someone has time it would be nice. The efficiency gain will presumably be considerable for remote tangle files.
[ ] Internals - Code sharing with org-babel-execute-buffer
Functions like org-babel-pop-to-session have to go through some initial variable binding which partially overlaps with the variable bindings in org-babel-execute-src-block. If possible, it would be nice to abstract the common set-up into shared code. For example, I recently added default-directory in the let-binding of o-b-execute-src-block, and subsequently realised that it also needed to be added to o-b-pop-to-session.
[ ] Make docstrings obey emacs rules
First descriptive sentence on own line less than 80 chars (or something like that).
[ ] Avoid multiple calls to org-babel-where-is-src-block-head
the posiion of head can be stored in the ‘info’ data structure. This would entail a bit of messing about as it would have to be one of the earlier elements in the list, thus pushing the others along by one.

org-icons for src blocks

  • State “STARTED” from “PROPOSED” [2010-08-29 Sun 19:01]
  • State “PROPOSED” from “” [2010-03-15 Mon 17:22]

    Started: see branch org-icons at git@github.com:dandavison/org-devel.git

I would like to investigate using org-icons to replace the code block boilerplate (#+begin_src LANG … #+end_src) with nice language-specific icons. Personally, even with code block folding, I find the remaining block boilerplate to be a bit intrusive.

As for default icons…ideas? I don’t know what it would look like using the icons associated with the respective software projects (ruby, python, R, etc) but it would be interesting to see.

In-buffer graphical results

  • State “DONE” from “STARTED” [2010-08-29 Sun 19:00]
  • State “STARTED” from “PROPOSED” [2010-03-15 Mon 17:21]
  • State “PROPOSED” from “” [2010-03-15 Mon 17:15]

    This functionality was implemented by Carsten. See `org-toggle-inline-images’.

    A proof-of-principle implementation of this is below. It uses org-babel-after-execute-hook to refresh the inline image displays in the whole buffer. This means that the code for plots and latex fragments can be edited and then the graphic updated with C-c C-c.

    However, after looking briefly at Nicolas Girard’s work on org-icons.el, I wonder whether it would be nicer to implement this using the font-lock machinery, similar to how the org icons are implemented.

(defun ded/iimage-mode-buffer (arg &optional refresh)
"Display/undisplay images.
With numeric ARG, display the images if and only if ARG is positive."
  (interactive)
  (let ((ing (if (numberp arg)
                 (> arg 0)
               iimage-mode))
        (modp (buffer-modified-p (current-buffer)))
        file img)
    (save-excursion
      (goto-char (point-min))
      (dolist (pair iimage-mode-image-regex-alist)
        (while (re-search-forward (car pair) nil t)
          (if (and (setq file (match-string (cdr pair)))
                   (setq file (iimage-locate-file file
                                   (cons default-directory
                                         iimage-mode-image-search-path))))
              (if ing
                  (let ((img (create-image file)))
                    (add-text-properties (match-beginning 0) (match-end 0) (list 'display img))
                    (if refresh (image-refresh img)))
                (remove-text-properties (match-beginning 0) (match-end 0) '(display)))))))
    (set-buffer-modified-p modp)))

(defun ded/org-iimage-refresh ()
  (interactive)
  (redisplay t)
  (set-face-underline-p 'org-link nil)
  (ded/iimage-mode-buffer 1 'refresh)
  (redisplay t))

(add-hook 'org-babel-after-execute-hook 'ded/org-iimage-refresh)

Language-specific session header args

  • State “PROPOSED” from “” [2010-03-15 Mon 14:41]

    One may want to specify the session for multiple blocks using properties. However, currently there is no way to specify a different session for different languages using properties. Two possibilities come to mind:

    1. Use org-babel-default-header-args:LANG
    2. Introduce new headers of the form session:LANG or session-LANG

    (1) is cleaner, in that it uses existing technology with no changes. However, it would be nice to be able to set these things in a property drawer. Perhaps(??) we could have a rule that says that session:LANG in a property drawer means to set the value of session in org-babel-default-header-args:LANG

How do we specify multiple :var references with property drawer?

  • State “PROPOSED” from “” [2010-02-23 Tue 19:58]

e.g. the following

*** example
,   :PROPERTIES:
,   :var: x=1
,   :var: y=2
,   :END:
,
#+begin_src emacs-lisp
,(list x y)
#+end_src

results in

Symbol's value as variable is void: y

Define function org-babel-version

  • State “REJECTED” from “PROPOSED” [2010-03-20 Sat 12:38]
  • Useful for compendium, where it documents the version of org-babel on which the compendium is known to work
  • org-version is appropriate for this

Check and document behaviour on Windows

  • State “TODO” from “” [2010-03-04 Thu 15:48]

Integration with GNU screen

  • State “PROPOSED” from “” [2010-02-28 Sun 15:30]

    It might be nice to support some relationship between babel sessions and screen sessions, so that babel sessions have persistency (particularly useful for remote sessions).

    This will be most straightforward for shell sessions. But the fact that ess-remote “converts” an emacs shell session running R into an inferior ESS comint R session suggests that it may be possible for R also.

Use org-insert-link machinery to create links

  • State “TODO” from “” [2010-02-28 Sun 15:25]

    One reason for doing this is to honour the variable org-link-file-path-type.

    Perhaps org-insert-link can be refactored along the lines of

(defun org-insert-link (...)
  ...
  (insert (org-create-link ...)))

in which case we could use org-create-link.

Are we happy with boolean header arg semantics?

  • State “PROPOSED” from “” [2010-02-24 Wed 15:51]

    For boolean header args like cache, noweb and colnames, the value is <on> if the header arg is yes; absent or any other value is <off>.

    Is it worth considering nil or absent = <off>, and any non-absent value = <on>, in order to be in line with lisp?

    I think this would also make sense from the point of view of least-surprise, because the default sense of those args is <off>; and anyone who is supplying them is likely to be wanting to turn them <on>, and so the only way for them to fail to do what they want would be if they supplied nil.

    In particular, users might be surprised that t, y, Y, on, Yes don’t work.

    I guess we’re going to have to really want to make this change to accept the backwards incompatibility cost, but I’m just noting it down. It could always be implemented by a variable.

Handling of table column names and hlines across languages

  • State “DONE” from “TODO” [2010-04-23 Fri 09:42]
  • State “TODO” from “PROPOSED” [2010-03-26 Fri 21:38]
  • State “PROPOSED” from “” [2010-02-23 Tue 20:04]
  • State “TODO” from “” [2010-02-23 Tue 12:22]

Org-babel now supports three new header arguments, and new default behavior for handling horizontal lines in tables (hlines), column names, and rownames across all languages. These are as follows

:hlines
Can take on the values of “yes” or “no”, with a default value of “no”. These values have the following effects.
“no”
results in all hlines being stripped from the input table. In most languages this is the desired effect, as a raw ‘hline symbol generally is interpreted as an unbound variable and leads to and error. The following table would previously have lead to an error but is now processed as shown.
#+tblname: many-cols
| a | b | c |
|---+---+---|
| d | e | f |
|---+---+---|
| g | h | i |
        

:

#+source: echo-table
#+begin_src python :var tab=many-cols
  return tab
#+end_src
        

:

#+results: echo-table
| a | b | c |
| d | e | f |
| g | h | i |
        
“yes”
leaves ‘hlines in the table. This is the default for emacs-lisp which may want to handle hline symbols explicitly.
:colnames
Can take on the values of “yes”, “no”, or nil for unassigned. The default value is nil. These values have the following effects
nil
If an input table looks like it has column names (meaning if it’s second row is an hline), then the column names will be removed from the table by org-babel before processing, then reapplied to the results, so for example the following code block has the effect shown.
#+tblname: less-cols
| a |
|---|
| b |
| c |
        

:

#+srcname: echo-table-again
#+begin_src python :var tab=less-cols
  return [[val + '*' for val in row] for row in tab]
#+end_src
        

:

#+results: echo-table-again
| a  |
|----|
| b* |
| c* |
        
“no”
No column name pre-processing will take place.
“yes”
Column names are removed and reapplied as with nil even if the table does not look like it has column names (i.e. the second row is not an hline)
:rownames
Can take on the values of “yes” or “no”, with a default value of “no”. These values have the following effects.
“no”
No row name pre-processing will take place.
“yes”
The first column of the table is removed from the table by org-babel before processing, and is then reapplied to the results. This has the effect shown below.
#+tblname: with-rownames
| one | 1 | 2 | 3 | 4 |  5 |
| two | 6 | 7 | 8 | 9 | 10 |
        

:

#+srcname: echo-table-once-again
#+begin_src python :var tab=with-rownames :rownames yes
  return [[val + 10 for val in row] for row in tab]
#+end_src
        

:

#+results: echo-table-once-again
| one | 11 | 12 | 13 | 14 | 15 |
| two | 16 | 17 | 18 | 19 | 20 |
        

Thanks to Julien Barnier for adding rownames support in R.

discussion

See also Support rownames and other org babel table features?

Julien Barnier has made a patch implementing rownames in R. This is in branch julien-barnier-R-rownames in the devel repo (commit f29c00432e6091bc1fbce8d1eb9052eff61da7b7)

There is a test file for column and rownames here.

From Tom Dye

IIUC, the difficulty is introduced by the difference between R, which keeps row names “under the hood,” and org-mode, which doesn’t have a concept of row names. So, the question becomes one of preserving R’s row names in cases where that is desirable. Because it is not possible, AFAIK, to distinguish between an org-mode table created through an R call to print a data frame and one made with org-tbl, the onus is on the user to preserve R row names

One way would be to establish an idiom for exporting and importing R data frames and put it on Worg. This one works for me.

So, on the way out: cbind(row=rownames(df),df)

And, on the way in: df <- data.frame(x, row.names=1)

If you want, I can add this, or something like it, to org-babel-doc-R.

Also, I’ve been using the reshape package and a melt, cast sequence, which I use frequently, keeps the row names in the first column, so I only have to be conscious of preserving row names on the way back into org-mode.

Examples

DateKg
2010-02-2195.0
2010-02-2293.0
2010-02-2392.0
2010-02-2491.5
2010-02-2591.0
2010-02-2692.0

As things stand

python

return d

Results in error because of ‘hline

We could remove the hline with the following, but need to think about whether to include the column names or not.

(defun org-babel-python-var-to-python (var)
  "Convert an elisp var into a string of python source code
specifying a var of the same value."
  (if (listp var)
      (concat "[" (mapconcat #'org-babel-python-var-to-python (remq 'hline var) ", ") "]")
      ;; (concat "[" (mapconcat #'org-babel-python-var-to-python var ", ") "]")
    (format "%S" var)))

That change would give this as result:

DateKg
2010-02-2195.0

NB Is it unfortunate that a named simple vector doesn’t get its names printed out with :colnames yes?

c(a=1,b=2)
x
1
2

This is because a 1d vector gets turned into a table with one column, and hence its names would be rownames, not column names. One has to transpose the vector in R to get the desired result.

t(c(a=1,b=2))
ab
12

shell

echo $d

Error:

org-babel-sh-var-to-sh wrongly converts ‘hline into “hline” resulting in error in orgtbl-to-generic. Could change last loine of org-babel-sh-var-to-sh

(if (stringp var) (format “%s” var) (format “%S” var))))

to

(cond ((eq var ‘hline) var) ((stringp var) (format “%s” var)) (t (format “%S” var)))

But need to think about whether the hline should even be there at this stage, or whether hlines and column names should have been removed (at least hlines as 2nd element of elisp table).

Store interpreter executable names and command-line arguments in variables

  • State “TODO” from “” [2010-02-20 Sat 12:22]

How do we check that block output is identical to previous output?

  • State “TODO” from “” [2010-02-14 Sun 18:24]

    A common situation for me is that I return to an org-babel file containing several blocks with output, and the first thing I want to do as a sanity check is execute all the blocks, verifying that the new output is the same as the old output.

    Maybe this would be possible if you rename the results, and then write a lob function which compares two results for equality? If you put that idea into a table then you’d basically have our test suite. I guess we could also develop support for doing this sort of thing automatically, although it’s not clear to me how you would indicate that you want to do a “checking” run by default. I do like the idea however, as it seems like it will have direct reproducible research applications. Maybe we just make this a header argument, and then provide some functions which call individual or multiple blocks with that header argument pre-set to true – or maybe header arguments aren’t the appropriate tool for this situation?

How do we open all source blocks in buffer?

  • State “DONE” from “STARTED” [2010-08-29 Sun 19:08]
  • State “STARTED” from “TODO” [2010-02-14 Sun 10:34]
  • State “TODO” from “” [2010-02-14 Sun 11:18]

This is an org-mode issue, but relevant to us. org visibility cycling just opens headings. An example of when you might want absolutely everything to be open is when using C-s to search through a buffer.

So, currently even closed source blocks will open temporarily for searching because of the way they are hidden. Also there is the `org-show-block-all’

org-show-block-all is a Lisp function in `org.el'.

(org-show-block-all)

Unfold all blocks in the current buffer.

function which will reveal all blocks in the current buffer, I guess the question then becomes – should this be bound to a key, or should it somehow hook into the org visibility cycling.

And should it be made interactive? If so let’s raise it on list. Personally I can deal with it not being part of visibility cycling, so I think this item may be almost DONE.

OK, you made it interactive (c4d385681f6c70), so I’m closing this item.

How do we mark a block as not to be executed?

  • State “DONE” from “STARTED” [2010-08-29 Sun 19:10]
  • State “STARTED” from “TODO” [2010-02-14 Sun 10:35]
  • State “TODO” from “” [2010-02-14 Sun 11:19]

    Done: (e52909d90274) The header arg is :eval no

E.g. by org-babel-execute-buffer

Hmm, should this be a new header argument, maybe :run which can take a yes or no argument?

Perhaps, if it’s the case that it won’t be used often, we should use :execute so that the terminology is kept consistent with the corresponding function (like “tangle”).

Make shell evaluation use user’s $PATH

  • State “PROPOSED” from “” [2010-01-17 Sun 20:10]

Shell evaluation is not picking up on the $PATH set in my ~/.bashrc. I think this is a general emacs/shell issue, but it would be nice to provide some guidance for users.

implement org-babel-load-session:* for all languages

  • State “STARTED” from “” [2010-01-11 Mon 10:00]

`org-babel-load-session:*’ is bound to M-up and can be called from inside of a code block to dump the body of the block to the related session. It is currently implemented for the following languages where…

NA
means the language doesn’t support sessions anyways
yes
means it is now implemented
??
means I need to look more closely at the language to determine if sessions are supported, or some prerequisite isn’t yet implemented
Languageorg-babel-load-session: implemented
AsymptoteNA
CNA
Clojureyes
cssNA
ditaaNA
GraphvizNA
Emacs LispNA
gnuplotyes
Haskellyes
LaTeXNA
Objective Caml??
Perl??
Pythonyes
Ryes
Rubyyes
SassNA
GNU Screen??
shellyes
SQL??

most of these all follow this simple basic form

(defun org-babel-load-session:R (session body params)
  "Load BODY into SESSION."
  (save-window-excursion
    (let ((buffer (org-babel-prep-session:R session params)))
      (with-current-buffer buffer
        (goto-char (process-mark (get-buffer-process (current-buffer))))
        (insert (org-babel-chomp body)))
      buffer)))

width and height set dimensions in latex output?

  • State “STARTED” from “PROPOSED” [2010-02-14 Sun 10:36]

See email thread started by Graham Smith 12 Dec.

Should we be setting ATTR_LaTeX in this situation?

So it looks like we somehow need to pass the width through to R as a variable. Are the current variable passing methods insufficient for this task? Or would it just be more convenient if the R execution code automatically checked for a value of ATTR_LaTeX and setup the R environment to correspond. I don’t really know R that well, so I’m not sure how this would work, but it does seem like it’d be handy if/once it was implemented. [Eric]

#+options toc:nil
#+title: R latex graphics

Text before figure.
#+CAPTION:    Trial boxplots with babel
#+LABEL:      fig:trial boxplots
#+ATTR_LaTeX: width=5cm
#+srcname:Boxplots Summary
#+begin_src R :session BirdData :file BoxplotSummary.pdf :exports both
  Wingcrd <- rnorm(100, 20) ; Tarsus <- rnorm(100, 5) ;
  Head <- rnorm(100, 3) ; Wt <- rnorm(100, 40)
  boxplot(Wingcrd,Tarsus,Head,Wt,names=c("Wingcrd","Tarsus","Head","Wt"))
#+end_src

#+results: Boxplots
[[file:BoxplotSummary.pdf]]

Text after figure.

Working directories and remote execution

  • State “STARTED” from “TODO” [2010-02-22 Mon 14:41]
  • State “TODO” from “DONE” [2010-02-21 Sun 01:16]
  • State “DONE” from “STARTED” [2010-02-21 Sun 00:50]
  • State “STARTED” from “PROPOSED” [2010-02-18 Thu 17:54]
  • State “PROPOSED” from “TODO” [2010-02-14 Sun 14:00]
  • State “TODO” from “TODO” [2010-02-14 Sun 10:41]

    Working directory is specified using :dir. If this is remote, then processes run remotely.

    There is a working implementation for R, ruby, python and shell (branch ded-babel-remote).

    One issue discussed below is that, as things stand in emacs, shell-command-on-region does not use tramp to handle the case of a remote default-directory (unlike shell-command).

    The underlying reason is that call-process-region does not use tramp. The current working solution is, instead of using call-process-region, to use an org-babel version of the tramp handler for call-process-region (this handler is present but unused in tramp).

    The current implementation does that only when default-directory is remote; otherwise we call the emacs version of call-process-region.

Improve the way call-process-region is handled

  • State “DONE” from “TODO” [2010-03-02 Tue 14:53]
  • State “TODO” from “” [2010-02-21 Sun 16:53]

    At line 217 of org-babel.el, we rebind call-process-region so that a handler from tramp is used:

(let (
      ...
      (call-process-region-original (symbol-function 'call-process-region))
      result)
  ;; (message "params=%S" params) ;; debugging
  (flet ((call-process-region (&rest args)
                              (apply 'org-babel-tramp-handle-call-process-region args)))
    ;; ...
    ))

Currently, there is a bug in that if we try to run the tests, a number of them fail. Furthermore, afterwards, it seems that

(symbol-function ‘call-process-region))

returns the rebound definition made in the flet, rather than the original definition, despite the fact that the flet has terminated.

This all needs to be fixed.

Improve temp file creation and remote reading

  • State “TODO” from “” [2010-02-21 Sun 17:02] See reply from Michael Albinus

    As an example from several such instances, consider this in ob-ruby.el:

     (case result-type
	(output (org-babel-eval org-babel-ruby-command body))
	(value (let ((tmp-file (make-temp-file "org-babel-ruby-results-")))
		 (org-babel-eval org-babel-ruby-command
				 (format (if (member "pp" result-params)
					     org-babel-ruby-pp-wrapper-method
					   org-babel-ruby-wrapper-method)
					 body tmp-file))
		 ((lambda (raw)
		    (if (or (member "code" result-params)
			    (member "pp" result-params))
			raw
		      (org-babel-ruby-table-or-string raw)))
		  (org-babel-eval-read-file tmp-file)))))

The prefix passed to make-temp-file is relative; if default-directory happens to be remote, then make-temp-file guarantees that the file name returned does not already exists on the local host, whereas what is wanted is a guarantee that it does not exist on thr remote host. It seems from Michael Albinu’s reply below that we should in that case be passing a prefix using remote syntax, expanded against the value of temporary-file-directory appropriate for the remote host.

Email from Dan

From: Dan Davison <davison@stats.ox.ac.uk> Subject: retrieving output from temp file User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux) Date: Sun, 21 Feb 2010 11:39:33 -0500 To: tramp-devel@gnu.org

I want to retrieve the contents of a file created by a shell process, which might be running remotely. My code (below) works, but I am trying to learn how to use tramp, and I think that this is not how it would be done by someone who knew what they were doing.

(defun retrieve-output ()
  (let ((default-directory "/user@host:dirpath")
        (output-file (make-temp-file "zzz-")))
    (shell-command (format "hostname > %s" output-file))
    (insert-file-contents
     (if (file-remote-p default-directory) (make-remote-file-name output-file) output-file))))

(defun make-remote-file-name (file)
  (let* ((vec (tramp-dissect-file-name default-directory))
         (user (tramp-file-name-user vec))
         (host (tramp-file-name-host vec)))
    (concat "/" user (when user "@") host ":" file)))

(retrieve-output)

If default-directory is not remote, then I want this to work for someone who does not have tramp installed (because aIui an XEmacs user might not have tramp?)

In my case I do need to store the output in a file. I.e. although in the example above the output is created by redirecting stdout to file, in general the output of the remote process will not be on stdout (the output file will be created in some other way by the shell process).

One thing that feels like a hack is the way that, when the process runs remotely, I manually convert the temp file path into a remote file path.

Another problem is that with my code there is no guarantee that the temp file name doesn’t already exist on the remote host.

Thanks a lot,

Dan

reply from Michael Albinus

From: Michael Albinus <michael.albinus@gmx.de> Subject: Re: retrieving output from temp file User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.92 (gnu/linux) Date: Sun, 21 Feb 2010 18:57:51 +0100 To: Dan Davison <davison@stats.ox.ac.uk> Cc: tramp-devel@gnu.org

Dan Davison <davison@stats.ox.ac.uk> writes:

> I want to retrieve the contents of a file created by a shell process, > which might be running remotely. My code (below) works, but I am trying > to learn how to use tramp, and I think that this is not how it would be > done by someone who knew what they were doing.

What about

(process-file “process” nil t)

> If default-directory is not remote, then I want this to work for someone > who does not have tramp installed (because aIui an XEmacs user might not > have tramp?)

It works also for a local `default-directory’. XEmacs comes with Tramp 2.0, but it doesn’t know `process-file’ (yet).

> In my case I do need to store the output in a file. I.e. although in > the example above the output is created by redirecting stdout to file, > in general the output of the remote process will not be on stdout (the > output file will be created in some other way by the shell process).

This case, I would do

(defun retrieve-output ()
  (let ((tmpfile
         (make-temp-file
          (concat (file-remote-p default-directory) "/tmp/zzz-"))))
    (unwind-protect
        (progn
          (process-file
           "process" nil nil nil
           (or (file-remote-p tmpfile 'localname) tmpfile))
          (insert-file-contents tmpfile))
      (delete-file tmpfile))))

(retrieve-output)

I have added the local file name part of tmpfile to the `process-file’ call; it depends on the “process” command, where it does expect the output file.

> One thing that feels like a hack is the way that, when the process runs > remotely, I manually convert the temp file path into a remote file path.

`make-temp-file’ works also wit a remote prefix, as you see.

> Another problem is that with my code there is no guarantee that the temp > file name doesn’t already exist on the remote host.

With this approach, `make-temp-file’ does it for you.

> Thanks a lot, > > Dan

Best regards, Michael.

Extend to other languages

  • State “TODO” from “” [2010-02-21 Sun 16:52]

    We need to extend some of the changes to some of the other languages, e.g. make sure that they construct remote file names when attemtping to read remote data, as in 010cd73feb4a1dcb2da6f9a7352a35cfb4dac00e.

Make gnuplot respond to default-directory

  • State “TODO” from “” [2010-02-28 Sun 15:34]

    gnuplot currently uses shell-command-to-string. I think this may be the reason why it does not respond to a change in default-directory. We may want to use shell-command-on-region (like many other languages) or shell-command instead.

Make sure file links are pointing into dir

  • State “STARTED” from “TODO” [2010-03-03 Wed 14:37]
  • State “TODO” from “” [2010-02-25 Thu 18:16]

    E.g. if I use :file pca.png, but the working directory of the R session is “/tmp”, then we need to ensure that the org file link points to the location of the file created by R.

relation of :dir and :exports

  • State “TODO” from “” [2010-03-03 Wed 14:48]

> on export the following generates a > broken link (link is relative but file is created in ~/images) > > #+begin_src ditaa :file images/ditaa.png :dir ~ :export results > > Shall we just leave that problem as is for now, or would you prefer it > to be solved before committing? Err, and do you happen to have a good > solution? :) >

Hmm,

I would be inclined to say we’ve given people alot of rope, and if they’re digging this deep into complex combinations of header arguments then we can’t stop themselves from hanging themselves. If that sounds good to you then I’d say we’re set and go ahead and commit.

Since remote directories are working, the user could always put the path to the directory on their webserver in the :dir option, in which case the links may actually resolve.

should we allow :results file without explicitly giving path?

I.e. should we create a file in /tmp or in the current directory?

Improve support for :session t :results output

  • State “DONE” from “TODO” [2010-02-21 Sun 00:48]
  • State “TODO” from “” [2010-02-16 Tue 13:59]

:dir has no effect on existing session

  • State “DEFERRED” from “TODO” [2010-02-21 Sun 00:48]
  • State “TODO” from “” [2010-02-16 Tue 15:53]

    Should we warn user in this case? (In general o-b tends to silently ignore some nonsensical header args at the moment I think)

:session t :results value

  • State “DONE” from “TODO” [2010-02-21 Sun 00:49]
  • State “TODO” from “PROPOSED” [2010-02-11 Thu 14:40]

    A remote session can be started using tramp (the way I do it at the moment is to visit a remote file and then issue e.g. M-x R. It will run on the remote machine) or M-x ssh. This will work with :results output. However :results value currently (in the target language) writes the data to a file on the remote machine and then (in elisp) attempts to retrieve it from the local machine. Thus the necessary network data transmission is not currently attempted. Perhaps tramp can be used to achieve this?

    Hmm, If there is some programmatic way to detect that the current buffer is visiting a remote machine, then it should be easy to change the code which is writing and reading to temporary files to explicitly do all such operations on the local host. [Eric]

    I’m not quite clear on this. With “:results value” in (say) R, the R process has to write the return data to file. If the R process is running remotely then someone (R?, emacs?, shell?) has to transfer the data between machines. If this is to be robust across languages, then we probably want to allow the langauge processes to continue to write locally as they do currently (because the ability of languages to deal with the network transfer will vary), and therefore we would require a new data transfer stage (implemented perhaps in emacs or by standard shell utilities) in order for the data to be read into org-babel [Dan]

    I currently don’t ever really run anything on a remote machine, but if someone else got this going I’d be happy to help iron out the details. [Eric]

    For the record, my situation is the opposite – all the data is stored remotely (and the remote machines have more appropriate computing power than my netbook), but I don’t really want to run emacs over ssh, for a variety of reasons (performance, code libraries) – so having org-babel execute remotely is very desirable for me. Having said that, ‘:session :results output’ is working quite adequately; I’ve only occasionally felt the need for ‘:results value’. [Dan]

external process evaluation on remote machines

  • State “DONE” from “TODO” [2010-02-21 Sun 00:50]
  • State “TODO” from “” [2010-02-11 Thu 14:49]

    We need:

    • A way to specify the machine on which a remote process should be

run (and to retrieve stdout/stderr)

  • A way of retrieving file contents in the case of :results value.

The second of the above should be handled by the remote-results-value case above. As for telling org-babel to start the process on a remote machine, it’s not immediately clear to me how that would work.

The first thing that occurs to me is that maybe if we accept a path so that a buffer on the remote machine can be opened using tramp then we can switch to that buffer and remote execution will happen naturally in a language-agnostic way. This is a slight automation of the process described above. So something like

#+begin_src clojure :host my-other-machine:~/some-file
  (println (System/getProperties))
#+end_src

would result in Org-babel first opening a buffer with the value of :host and then running through the same execution pattern as above.

byte-compile elisp on tangle

  • State “TODO” from “PROPOSED” [2009-12-21 Mon 12:50]

from mailing list

1) When the org configuration file aren't changed, a way to automatically load
their tangled version (bytecompiled ?) in order to improve the initialization
time of Emacs.
  • are there times when this wouldn’t be desirable?
  • maybe this should be controlled by another header argument

also we need to address some :tangle header argument related bugs in `org-babel-load-file’

2) If I include a source code block without a :tangle argument, then the .el file is truncated in odd
ways.  For instance, if the last source code block doesn't have a tangle argument but all the source code
blocks above it have either :tangle yes or :tangle no, then the .el file only contains the tangled source
code block without the tangle argument.

I've had other permutations of 2)
 where only source code blocks after a :tangle no source code block get tangled.

If all of the source code blocks have :tangle arguments, then all seems to be well.

strip ansi-color characters (sh)

This should definitely be done in org-babel-sh, I don’t think it is common enough to move the functionality to org-babel-comint, but I could be wrong…

Export issues

reference source blocks that are themselves excluded from export

restrictions on locations of org-exp-blocks interblocks

inline source code blocks [5/8]

Like the \R{ code } blocks

not sure what the format should be, maybe just something simple like src_lang[]{} where lang is the name of the source code language to be evaluated, [] is optional and contains any header arguments and {} contains the code.

evaluation with \C-c\C-c

Putting aside the header argument issue for now we can just run these with the following default header arguments

:results
silent
:exports
results

inline exportation

Need to add an interblock hook (or some such) through org-exp-blocks

header arguments

We should make it possible to use header arguments.

Bring export of inline code back to life

Uses session even when not requested

fontification

we should color these blocks differently

refine html exportation

should use a span class, and should show original source in tool-tip

Allow export of results of #+lob lines

Inline fragment not evaluated unless another code block in buffer

External shell issues

  • State “TODO” from “” [2010-03-20 Sat 12:32]

org-babel-sh explicitly specify the shell

this should be possible as source-code execution shouldn’t depend on the values of user-specific information (i.e. the user’s shell)

Improve external shell issues under Windows (with R at least)

  • State “DEFERRED” from “PROPOSED” [2010-02-03 Wed 13:21]
  • State “PROPOSED” from “” [2010-01-13 Wed 12:18]

I’m wondering whether we should implement some windows-specific hackery to make it less likely that users will have problems with R under windows.

If someone wants to submit a Windows specific patch I’d be happy to apply it, I however have no access to a Windows machine [Eric].

Two users have now mailed the list reporting that they can’t get org-babel to work with R under Windows. The problem stems from the fact that we use the external shell execution as default, but under Windows it is quite often the case that users have not set their system up so that “R” is in the shell path. One possibility would be to make session (ESS) the R Windows default when available. Another approach is suggested by this email by d.tchin:

d.tchin R Windows shell email

Bernd Weiss email

http://www.mail-archive.com/emacs-orgmode@gnu.org/msg19104.html

Work on tangling

  • State “STARTED” from “TODO” [2010-06-16 Wed 20:36]

allow tangle to be called on a single source block

this should have a reasonable binding

make tangled files read-only?

  • State “PROPOSED” from “REJECTED” [2010-09-07 Tue 14:00]
  • State “REJECTED” from “PROPOSED” [2010-06-16 Wed 20:36] With a file-local variable setting, yea that makes sense. Maybe the header should reference the related org-mode file.

I fear the only time this sort of feature would even be noticed is when it is not wanted.

Ah, not sure I agree with that. I’ve done it plenty of times. In fact, I have a project which has drifted out of Org because I realised too late that I was editing tangled files and haven’t got round to putting the bits back into the relevant Org blocks. Plus, I just came across the same complaint on-line:

http://splash-of-open-sauce.blogspot.com/2010/01/moving-to-emacs-init-files.html

make tangled files executable?

  • State “DONE” from “PROPOSED” [2010-06-16 Wed 20:35]

At least if using shebang line

optionally do not output comment and links

  • State “DONE” from “PROPOSED” [2010-06-16 Wed 20:35]

inserting empty results lines

check if org-babel inserts empty results lines, if so don’t do this

function for executing source blocks

Do you think a user-visible function taking the name of a file and a list of source-code block names in that file would be sufficient. That shouldn’t be hard to implement given our current setup.

customizable noweb syntax

given that some languages do allow <<foo>> syntax it would be preferable if we could make our noweb syntax customizable.

macro expansion

allow some generic system of macro expansion, maybe borrowing from the noweb expansion system…

export source code block names on export

  • State “DONE” from “TODO” [2010-08-29 Sun 19:15]

    We do this now. There is currently a pending patch by david O’Toole which concerns this. See Review David O’Toole export patch.

  • perhaps using an org-mode macro
  • noweb has a syntax for doing this on latex export
  • maybe even include a list of where it is used…
>>> That sounds like a good idea.  It would be nice to make the
>>> appearance
>>> of the exported names customizable as I can see many different
>>> possibilities and would prefer to leave the actual choice up to the
>>> user.  Would macros allow for over-definition/customization by the
>>> user?
>>
>> I am afraid you have to ask the Org mode developers this question.
>>
>
> understood :) I'll look into this
  

: :

I don't know whether you can call Emacs Lisp functions in an Org
macro, but perhaps you can talk the Org developers into allowing for
that. Emacs Lisp functions can be over written.. Even better, you
could then define a hook, and have a clean extendable solution.
  

asynchronous

This may duplicate some of the contents of the rework-running-proc-interaction task.

Basically it would be great to be able to run processes asynchronously in the background. Emacs has support for and even encourages this, it would just be a question of make the changes both for sessions and non-session evaluation.

documentation

our documentation has fallen behind our development. We should:

  1. troll through the babel-related git logs and adjust the documentation from all major commits
  2. stub out a page for each language
  3. adjust the instructions WRT :results value and :results output
  4. expand the discussion of noweb references
  5. ensure that all new header arguments (e.g. results_switches) are represented in the documentation
  6. add the org-babel-template.el new language template

documentation odds and ends

this is just a place to hold random notes for items that should make it into the documentation at some point.

  • org-src-preserve-indentation

new layout

what should this look like? I’m thinking something like.

This is really off the cuff, so please make changes

file#subheadingdescription
org-babelintroduction, setup instructions, intro examples
org-babel-docactual documentation, header arguments, etc…
languages/directory to hold language specific documentation
org-babel-testsmain general in elisp and shell only
org-babel-tests#rubyruby specific tests
org-babel-tests#pythonpython specific tests
etc…

further work on dependencies of header args?

For example, pp and code should probably imply value. It would be possible in principle to have a general mechanism for specifying and resolving dependencies, which would be used by o-b-merge-params.

:hide header argument for automatically folding source blocks

from the mailing list

My suggestion is that if a source block has the :hide header argument it should be closed by default as if the user had pressed tab. The user could then press tab at the “#BEGIN_SRC …” line to show the content of the block and, maybe, the block could be closed again if the cursor leaves the block. This can be useful for other blocks as well.

allow hiding of code blocks with <tab> on srcname line?

  • State “REJECTED” from “PROPOSED” [2010-08-29 Sun 19:18] I think this is out of keeping with other Org behaviour

Allow hiding of results blocks?

  • State “DONE” from “PROPOSED” [2010-08-29 Sun 19:17]

Support passing of data to source block on stdin?

add data serialization language result types (XML, YAML, JSON, etc…)

these could be cached in source-code blocks of the appropriate serialization language, and could be very useful, especially for languages (like ruby) which support dumping object to/from these serialization languages.

allow references to bound emacs lisp variables?

I don’t think we can do this currently. Something like this?

(defun org-babel-ref-literal (ref)
  "Determine if the right side of a header argument variable
assignment is a literal value or is a reference to some external
resource.  If REF is literal then return it's value, otherwise
return nil."
  (if (boundp (intern ref)) (eval (intern ref))
    (let ((out (org-babel-read ref)))
      (if (equal out ref)
	  (if (string-match "^\"\\(.+\\)\"$" ref)
	      (read ref))
	out))))

Need to be careful that an attempt is not made to interpret quoted strings as elisp variables. It would allows stuff like this

(setq
 evecfile (concat dir "/" "evecs")
 numpcs   10)
x <- matrix(scan(evecf), ncol=numpcs)
plot(x[,1:2], pch="+")

jumping between results and source blocks (evaluation from results)

see discussion on the Org-mode list http://thread.gmane.org/gmane.emacs.orgmode/18407/focus=18419

it is possible to open results from a source block, but yea I think jumping would be nice. The hardest part of this implementation will be selecting a key sequence.

:results org should be org block

  • State “DONE” from “PROPOSED” [2010-08-29 Sun 19:21]
  • State “PROPOSED” from “REJECTED” [2010-02-28 Sun 15:10]
  • State “REJECTED” from “PROPOSED” [2009-12-22 Tue 14:40]

    This is done now, albeit motivated differently. See commits 2c33b2eb66 and 3d2dbf8604.

The reason that :results org was introduced was to support seamless insertion of results into an org-mode buffer. This proposal would defeat that initial purpose. [Eric]

No, we already have :results raw which is currently synonymous with org. [Dan]

Also, the idea of org source blocks in an org-mode document seems needlessly complex and contorted. Of course I could be missing something here. [Eric]

Org already supports org source blocks (e.g. line 50 org-src.el). And there’s at least one clear use case for them, i.e. demonstrating, in HTML, what an org buffer looks like:

Test export

You’re reading HTML at the moment, but this is what org looks like in your emacs buffer.

#+SEQ_TODO: TODO | DONE
* TODO todo item
,  SCHEDULED: <2010-02-28 Sun>

I admit that I don’t know how often one will want to generate org blocks using babel, but I think that to use org as a synonym of raw is inconsistent with :results latex, :results html, :results code. [Dan]

support for working with org-src-mode edit buffers [7/8]

Patch against org source.

I’ve worked on several related changes to source code edit buffer behaviour in the org core. My current patch (below) does the following. Detailed explanation / working notes are below.

  • C-x s offers to save edit buffers
  • C-x C-c offers to save edit buffers
  • C-x k warns that you’re killing an edit buffer
  • If you do kill an edit buffer, the overlay in the parent buffer is removed
  • Edit buffers are named Org Src <orgbuf>[<lang>], where <orgbuf> is the name of the org-mode buffer containing this source code block, and lang is the language major mode. The latter might be unnecessary?

These changes were added to the main org repository in commit 4b6988bf36cb458c9d113ee4332e016990c1eb04

Detailed working notes to go with that patch

Recap of current org-src-mode

If you use C-c ’ to work on code in a begin_source block, the code buffer is put in minor mode org-src-mode, which features the following two useful key-bindings:

C-x sorg-edit-src-savesave the code in the source code block in the parent org file
C-c ’org-edit-src-exitreturn to the parent org file with new code

Furthermore, while the edit buffer is alive, the originating code block is subject to a special overlay which links to the edit buffer when you click on it.

This is all excellent, and I use it daily, but I think there’s still a couple of improvements that we should make.

Proposed bug I

C-x k kills the buffer without questions; the overlay remains, but now links to a deleted buffer.

Proposed bug II

C-x C-c kills a modified edit buffer silently, without offering to save your work. I have lost work like that a number of times recently.

Proposed bug III

C-x s does not offer to save a modified edit buffer

Notes on solution
write-contents-functions

A good start seems to be to use org-src-mode-hook to add org-edit-src-save to the write-contents-functions list. This means that when it comes to saving, org-edit-src-save will be called and no subsequent attempt will be made to save the buffer in the normal way. (This should obviate the remapping of C-x C-s to org-edit-src-save in org-src.el)

buffer-offer-save

We also want to set this to t.

Where does this get us?
  • C-x s still does not offer to save the edit buffer. That’s

because buffer-file-name is nil.

  • C-x C-c does ask us whether we want to save the

edit buffer. However, since buffer-file-name is nil it asks us for a file name. The check in org-edit-src-exit throws an error unless the buffer is named ‘* Org Edit ‘…

  • C-x k kills the buffer silently, leaving a broken overlay

link. If buffer-file-name were set, it would have warned that the buffer was modified.

buffer-file-name

So, that all suggests that we need to set buffer-file-name, even though we don’t really want to associate this buffer with a file in the normal way. As for the file name, my current suggestion is parent-org-filename[edit-buffer-name].

[I had to move the (org-src-mode) call to the end of org-edit-src-code to make sure that the required variables were defined when the hook was called.]

And so where are we now?
  • C-x s does offer to save the edit buffer, but in saving

produces a warning that the edit buffer is modified.

  • C-x k now gives a warning that the edit buffer is modified

(even if it’s not).

  • C-x C-c is working as desired, except that again we get

warnings that the edit buffer is modified, once when we save, and again just before exiting emacs.

  • And C-c ’ now issues a warning that the edit buffer is

modified when we leave it, which we don’t want.

So, we need to get rid of the buffer modification warnings.

I’ve made buffer-file-name nil inside the let binding in org-edit-src-exit.

And?
  • C-x s behaves as desired, except that as was already the case,

the edit buffer is always considered modified, and so repeated invocations keep saving it.

  • As was already the case, C-x k always gives a warning that the

edit buffer has been modified.

  • C-x C-c is as desired (offers to save the edit buffer) except

that it warns of the modified buffer just before exiting.

  • C-c ’ is as it should be (silent)
Conclusion

We’ve got the desired behaviour, at the cost of being forced to assign a buffer-file-name to the edit buffer. The consequence is that the edit buffer is considered to always be modified, since a file of that name is never actually written to (doesn’t even exist). I couldn’t see a way to trick emacs into believing that the buffer was unmodified since last save. But in any case, I think there’s an argument that these modifications warnings are a good thing, because one should not leave active edit buffers around: you should always have exited with C-c ’ first.

Doesn’t currently work with ess-load-file

ess-load-file contains these two lines

 (let ((source-buffer (get-file-buffer filename)))
   (if (ess-check-source filename)
	(error "Buffer %s has not been saved" (buffer-name source-buffer)))

which have the effect of, in the course of saving, deleting the buffer `source-buffer’, and then attempting to use it subsequently. The only solution I have thought of so far is submitting a patch to ess which would, e.g. reverse the order of those two lines (perform the error check outside the let binding).

In fact, even after doing that there are further problems generated by the fact that the edit buffer has an associated filename for which the file doesn’t exist. I think this worked OK in the past when the edit buffer had no associated filename. So this is a problem which needs addressing. Maybe defadvice could be used on ess functions where necessary to make org/org-babel play nicely with ess?

C-x s steals focus

With two modified edit buffers open, make one of them the current buffer and issue C-x s. It will offer to save both of them, but the second one to be saved will become the current buffer at the end.

name edit buffer according to #+srcname (and language?)

See above patch agains org.

optionally evaluate header references when we switch to *Org Edit Src* buffer

That seems to imply that the header references need to be evaluated and transformed into the target language object when we hit C-c ’ to enter the Org Edit Src buffer [DED]

Good point, I heartily agree that this should be supported [Eric]

(or at least before the first time we attempt to evaluate code in that buffer – I suppose there might be an argument for lazy evaluation, in case someone hits C-c ’ but is “just looking” and not actually evaluating anything.) Of course if evaluating the reference is computationally intensive then the user might have to wait before they get the Org Edit Src buffer. [DED]

I fear that it may be hard to anticipate when the references will be needed, some major-modes do on-the-fly evaluation while the buffer is being edited. I think that we should either do this before the buffer is opened or not at all, specifically I think we should resolve references if the user calls C-c ’ with a prefix argument. Does that sound reasonable? [Eric]

Yes [Dan]

[Dan] So now that we have org-src-mode and org-src-mode-hook, I guess org-babel should do this by using the hook to make sure that, when C-c C-’ is issued on a source block, any references are resolved and assignments are made in the appropriate session.

12
34
table.size.times.do |n|
  puts n
end

set buffer-local-process variables appropriately [DED]

  • State “DONE” from “TODO” [2010-08-29 Sun 19:23]

When switching to a edit buffer for R, if :session is active then we should set ess-current-process-name (or is it ess-local-process-name?) to the session name. Implementation? Presumably this uses org-src-mode-hook somehow.

old discussion

I think something like this would be great. You’ve probably already thought of this, but just to note it down: it would be really nice if org-babel’s notion of a buffer’s ‘session/process’ played nicely with ESS’s notion of the buffer’s session/process. ESS keeps the current process name for a buffer in a buffer-local variable ess-current-process-name (or is it ess-local-process-name?). So one thing we will probably want to do is make sure that the Org Edit Src Example buffer sets that variable appropriately. [DED]

I had not thought of that, but I agree whole heartedly. [Eric]

Once this is done every variable should be able to dump regions into their inferior-process buffer using major-mode functions.

send code to inferior process

Another thought on this topic: I think we will want users to send chunks of code to the interpreter from within the Org Edit Src buffer, and I think that’s what you have in mind already. In ESS that is done using the ess-eval-* functions. [DED]

I think we can leave this up to the major-mode in the source code buffer, as almost every source-code major mode will have functions for doing things like sending regions to the inferior process. If anything we might need to set the value of the buffer local inferior process variable. [Eric]

some possible requests/proposed changes for Carsten [4/4]

While I remember, some possible requests/proposed changes for Carsten come to mind in that regard:

Remap C-x C-s to save the source to the org buffer?

I’ve done this personally and I find it essential. I’m using

(defun org-edit-src-save ()
  "Update the parent org buffer with the edited source code, save
the parent org-buffer, and return to the source code edit
buffer."
  (interactive)
  (let ((p (point)))
    (org-edit-src-exit)
    (save-buffer)
    (org-edit-src-code)
    (goto-char p)))

(define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)

which seems to work.

I think this is great, but I think it should be implemented in the org-mode core

Rename buffer and minor mode?

Something shorter than Org Edit Src Example for the buffer name. org-babel is bringing org’s source code interaction to a level of maturity where the ‘example’ is no longer appropriate. And if further keybindings are going to be added to the minor mode then maybe org-edit-src-mode is a better name than org-exit-edit-mode.

Maybe we should name the buffer with a combination of the source code and the session. I think that makes sense.

[ES] Are you also suggesting a new org-edit-src minor mode? [DED] org-exit-edit-mode is a minor mode that already exists:

Minor mode installing a single key binding, “C-c ‘” to exit special edit.

org-edit-src-save now has a binding in that mode, so I guess all I’m saying at this stage is that it’s a bit of a misnomer. But perhaps we will also have more functionality to add to that minor mode, making it even more of a misnomer. Perhaps something like org-src-mode would be better.

Changed minor mode name and added hooks

a hook called when the src edit buffer is created

This should be implemented in the org-mode core

resolve references to other org buffers/files

This would allow source blocks to call upon tables, source-blocks, and results in other org buffers/files.

See…

resolve references to other non-org files

  • tabular data in .csv, .tsv etc format
  • files of interpreted code: anything stopping us giving such files similar status to a source code block?
  • Would be nice to allow org and non-org files to be remote

command line execution

Allow source code blocks to be called form the command line. This will be easy using the sbe function in org-babel-table.el.

This will rely upon resolve references to other buffers.

LoB: start to provide some useful functionality

Now that things are more settled down than a few months ago it may be a good time to start working out what functionality LoB should provide. Let’s collect ideas here for now.

table calculations

Two table requests have recently appeared on the list that are better met with org-babel than with pure org-mode:

plotting

  • plotting and analysis functions in R (incl. re-implement those from org-R?)
  • plotting with gnuplot (incl. re-implement org-plot??)

google API

See library of babel for google API

Improve error checking

  • State “STARTED” from “TODO” [2010-03-01 Mon 10:00]
  • State “TODO” from “DEFERRED” [2010-03-01 Mon 05:49]

    Current solution for :results value mode:

    • store shell exit code and stderr
    • if non-zero exit code:
      • write stderr to Org-Babel Error Output
      • display Org-Babel Error Output

    :results output mixes stderr with stdout and does not otherwise notify on error.

    Questions:

    • What should we do when stderr is non-empty but exit code is zero?
    • What should we do in the case of shell non-session :results value?
    • What should we do in the case of :session?

Return error structure?

Could use the following at outset of org-babel-insert-result.

(if (and (consp result) (eq (first result 'org-babel-error-flag)))
    (progn
      (message "Shell command exited with error %d" (second result))
      (unless (= (length (third result)) 0)
        (let (error-buffer (get-buffer-create "*Org-Babel Error Output"))
          (with-current-buffer error-buffer (insert (third result)))
          (display-buffer error-buffer))))
    )

older notes

E.g. when trying to execute sass block, I did not have sass installed, and so shell-command returned code 127, but org-babel did not warn me that anything had gone wrong. I expect it will be hard to do this properly, but ultimately it would be nice to be able to specify somewhere to receive STDERR, and to be warned if it is non-empty.

Probably simpler in non-session evaluation than session? At least the mechanism will be different I guess.

R has a try function, with error handling, along the lines of python. I bet ruby does too. Maybe more of an issue for functional style; in my proposed scripting style the error just gets dumped to the org buffer and the user is thus alerted.

For now I think the current behavior of returning any error messages generated by the source language is sufficient.

Error checking in R sessions

A simple thing to do is to wrap the R code in try(…), as in the patch below. That results in some improved behaviour:

  • You get the error message from R
  • Execution halts at first error E.g.
f <- function() {
    cat("hello\n")
    x <- log("a")
    cat("bye\n")
}
f()
+ hello
Error in log("a") : Non-numeric argument to mathematical function

patch

diff –git a/contrib/babel/lisp/langs/org-babel-R.el b/contrib/babel/lisp/langs/org-babel-R.el index 1ef21db..45f8409 100644 — a/contrib/babel/lisp/langs/org-babel-R.el +++ b/contrib/babel/lisp/langs/org-babel-R.el @@ -103,8 +103,8 @@ last statement in BODY, as elisp.” (out-tmp-file (make-temp-file “R-out-functional-results”))) (case result-type (output

  • (with-temp-file in-tmp-file (insert body))
  • (shell-command-to-string (format “R –slave –no-save < ‘%s’ > ‘%s’”
  • (with-temp-file in-tmp-file (insert (concat “try({” body “})”)))
  • (shell-command-to-string (format “R –slave –no-save < ‘%s’ > ‘%s’ 2>&1” in-tmp-file out-tmp-file)) (with-temp-buffer (insert-file-contents out-tmp-file) (buffer-string))) (value

@@ -124,7 +124,7 @@ last statement in BODY, as elisp.” (format “write.table(.Last.value, file="%s", sep="\t", na="nil",row.names=FALSE, col.names=%s, quote=FALSE)” tmp-file (if column-names-p “TRUE” “FALSE”)) org-babel-R-eoe-indicator) “\n”)) (output

  • (mapconcat #’org-babel-chomp (list body org-babel-R-eoe-indicator) “\n”))))
  • (mapconcat #’org-babel-chomp (list (concat “try({” body “})”) org-babel-R-eoe-indicator) “\n”)))) (raw (org-babel-comint-with-output buffer org-babel-R-eoe-output nil (insert full-body) (inferior-ess-send-input))) broke results)

diff –git a/contrib/babel/lisp/org-babel-ref.el b/contrib/babel/lisp/org-babel-ref.el index 0e8695f..060f880 100644 — a/contrib/babel/lisp/org-babel-ref.el +++ b/contrib/babel/lisp/org-babel-ref.el @@ -139,7 +139,7 @@ return nil.” (‘results-line (org-babel-read-result)) (‘table (org-babel-read-table)) (‘source-block

  • (setq result (org-babel-execute-src-block t nil args))
  • (setq result (org-babel-execute-src-block t (org-babel-get-src-block-info) args)) (if (symbolp result) (format “%S” result) result)) (‘lob (setq result (org-babel-execute-src-block t lob-info args)))))))

Finalise argument-passing syntax

x
7

In general we need to have a full set of rules for how a string of supplied arguments (some possibly named) interact with the arguments in the definition (some possibly with defaults) to give values to the variables in the function body.

share org-babel [3/7]

how should we share org-babel?

post to org-mode

post to ess mailing list

I’d like to not rush in to this, get some feedback from the org list first and let my R usage of org-babel settle down. [DD]

create a org-babel page on worg

Getting hold of it instructions

  • What about non-git users?
  • Are we moving/copying to contrib/?

Fixed width HTML output created by ... is ugly!

create a short screencast demonstrating org-babel in action

a peer-reviewed publication?

(see paper.org)

examples

we need to think up some good examples

interactive tutorials

This could be a place to use org-babel assertions.

for example the first step of a tutorial could assert that the version of the software-package (or whatever) is equal to some value, then source-code blocks could be used with confidence (and executed directly from) the rest of the tutorial.

answering a text-book question w/code example

org-babel is an ideal environment enabling both the development and demonstrationg of the code snippets required as answers to many text-book questions.

something using tables

maybe something along the lines of calculations from collected grades

file sizes

Maybe something like the following which outputs sizes of directories under the home directory, and then instead of the trivial emacs-lisp block we could use an R block to create a nice pie chart of the results.

du -sc ~/*
(mapcar #'car sizes)

Answer to question on list

From: Hector Villafuerte <hectorvd@gmail.com> Subject: [Orgmode] Merge tables Date: Wed, 19 Aug 2009 10:08:40 -0600 To: emacs-orgmode@gnu.org

Hi, I’ve just discovered Org and are truly impressed with it; using it for more and more tasks.

Here’s what I want to do: I have 2 tables with the same number of rows (one row per subject). I would like to make just one big table by copying the second table to the right of the first one. This is a no-brainer in a spreadsheet but my attempts in Org have failed. Any ideas?

By the way, thanks for this great piece of software! – hector

Suppose the tables are as follows

abc
123
789
def
456
101112

Here is an answer using R in org-babel

cbind(a, b)
“a”“b”“c”“d”“e”“f”
123456
789101112

Alternatively

Use org-table-export, do it in external spreadsheet software, then org-table-import

allow for stripping of header rows from table data

  • State “DONE” from “PROPOSED” [2010-08-29 Sun 19:25]

    See documentation of :colnames and :rownames header args

maybe controlled by an argument

Control precision of numerical output

Does org have an option controlling precision of numbers in tables?

allow `anonymous’ function block with function call args?

  • State “REJECTED” from “PROPOSED” [2010-08-29 Sun 19:26]

My question here is simply whether we’re going to allow

# whatever

I think we’re probably not going to do this seeing as the functionality is available with :var, and it’s dubious syntax anyway.

but with preference given to #+srcname blockname(arg=ref)

allow :result as synonym for :results?

Creating presentations

  • State “DONE” from “PROPOSED” [2010-08-29 Sun 19:29]

    This should now be possible as beamer support has been added to Org mode. It might be nice to have other presentation formats. I’m marking this as done as the babel part should already be working via existing export mechanisms

The recent thread containing posts by Nick Dokos and Sebastian Vaubán on exporting to beamer looked very interesting, but I haven’t had time to try it out yet. I would really like it if, eventually, we can generate a presentation (with graphics generated by code blocks) from the same org file that contains all the notes and code etc. I just wanted that to be on record in this document; I don’t have anything more profound to say about it at the moment, and I’m not sure to what extent it is an org-babel issue.

conversion between org-babel and noweb (e.g. .Rnw) format

I haven’t thought about this properly. Just noting it down. What Sweave uses is called “R noweb” (.Rnw).

I found a good description of noweb in the following article (see the pdf).

I think there are two parts to noweb, the construction of documentation and the extraction of source-code (with notangle).

documentation: org-mode handles all of our documentation needs in a manner that I believe is superior to noweb.

source extraction At this point I don’t see anyone writing large applications with 100% of the source code contained in org-babel files, rather I see org-babel files containing things like

  • notes with active code chunks
  • interactive tutorials
  • requirements documents with code running test suites
  • and of course experimental reports with the code to run the experiment, and perform analysis

Basically I think the scope of the programs written in org-babel (at least initially) will be small enough that it wont require the addition of a tangle type program to extract all of the source code into a running application.

On the other hand, since we already have named blocks of source code which reference other blocks on which they rely, this shouldn’t be too hard to implement either on our own, or possibly relying on something like noweb/notangle.

Modify results value implementation

  • State “PROPOSED” from “” [2010-03-03 Wed 14:27]

Are results transferred to elisp via file?

languagesessionshell process
Ryesyes
rubynoyes
pythonnoyes
shellnono
haskellnoNA

The main point is that in ruby and python, the printed values from the interpreter can be used as results, whereas that’s not really attractive for R. However, one minor point is: ought we to consider (in the interests of consistency) not using files for ruby shell-process and python shell-process?

optional timestamp for output

DEFERRED: I’m deferring this in deference to the better caching system proposed by Carsten. (see sha1-caching)

Add option to place an (inactive) timestamp at the #+resname, to record when that output was generated.

source code block timestamps (optional addition)

[Eric] If we did this would we then want to place a timestamp on the source-code block, so that we would know if the results are current or out of date? This would have the effect of caching the results of calculations and then only re-running if the source-code has changed. For the caching to work we would need to check not only the timestamp on a source-code block, but also the timestamps of any tables or source-code blocks referenced by the original source-code block.

[Dan] I do remember getting frustrated by Sweave always having to re-do everything, so this could be desirable, as long as it’s easy to over-ride of course. I’m not sure it should be the default behaviour unless we are very confident that it works well.

maintaining source-code block timestamps

It may make sense to add a hook to `org-edit-special’ which could update the source-code blocks timestamp. If the user edits the contents of a source-code block directly I can think of no efficient way of maintaining the timestamp.

source-name visible in LaTeX and html exports

Maybe this should be done in backend specific manners.

The listings package may provide for naming a source-code block…

Actually there is no obvious simple and attractive way to implement this. Closing this issue for now.

Support rownames and other org babel table features?

(see Handling of table column names and hlines across languages)

The full org table features are detailed in the manual here.

rownames

Perhaps add a :rownames header arg. This would be an integer (usually 1) which would have the effect of post-processing all the variables created in the R session in the following way: if the integer is j, set the row names to the contents of column j and delete column j. Perhaps it is artificial to allow this integer to take any value other than 1. The default would be nil which would mean no such behaviour.

Actually I don’t know about that. If multiple variables are passed in, it’s not appropriate to alter them all in the same way. The rownames specification would normally refer to just one of the variables. For now maybe just say this has to be done in R. E.g.

collectionsizeexcludeincludeexclude2include2
58C2936829282562680
MS585277150817715081
NBS29296428654022527
POBI27171271612716
58C+MS+NBS+POBI1359013004
rownames(size) <- size[,1]
size <- size[,-1]

Old notes

[I don’t think it’s as problematic as this makes out] This is non-trivial, but may be worth doing, in particular to develop a nice framework for sending data to/from R.

Notes

In R, indexing vector elements, and rows and columns, using strings rather than integers is an important part of the language.

  • elements of a vector may have names
  • matrices and data.frames may have “column names” and “row names” which can be used for indexing
  • In a data frame, row names must be unique

Examples

> # a named vector
> vec <- c(a=1, b=2)
> vec["b"]
b
2
> mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
> mat
   c1 c2
r1  1  3
r2  2  4
> # The names are separate from the data: they do not interfere with operations on the data
> mat * 3
   c1 c2
r1  3  9
r2  6 12
> mat["r1","c2"]
[1] 3
> df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
> df$var2
 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10  9  8  7  6  5  4  3  2  1
> df["g",]
  var1 var2
g    7   20

So it’s tempting to try to provide support for this in org-babel. For example

  • allow R to refer to columns of a :var reference by their names
  • When appropriate, results from R appear in the org buffer with “named columns (and rows)”

    However none (?) of the other languages we are currently supporting really have a native matrix type, let alone “column names” or “row names”. Names are used in e.g. python and perl to refer to entries in dicts / hashes.

    It currently seems to me that support for this in org-babel would require setting rules about when org tables are considered to have named columns/fields, and ensuring that (a) languages with a notion of named columns/fields use them appropriately and (b) languages with no such notion do not treat then as data.

  • Org allows something that looks like column names to be separated by a hline
  • Org also allows a row to function as column names when special markers are placed in the first column. An hline is unnecessary (indeed hlines are purely cosmetic in org [correct?]
  • Org does not have a notion of “row names” [correct?]

    The full org table functionality exeplified here has features that we would not support in e.g. R (like names for the row below).

Initial statement: allow tables with hline to be passed as args into R

This doesn’t seem to work at the moment (example below). It would also be nice to have a natural way for the column names of the org table to become the column names of the R data frame, and to have the option to specify that the first column is to be used as row names in R (these must be unique). But this might require a bit of thinking about.

col1col2col3
123
4schulte6
123
4schulte6
tabel
“col1”“col2”“col3”
123
4“schulte”6

Another example is in the grades example.

use textConnection to pass tsv to R?

When passing args from the org buffer to R, the following route is used: arg in buffer -> elisp -> tsv on file -> data frame in R. I think it would be possible to avoid having to write to file by constructing an R expression in org-babel-R-assign-elisp, something like this

(org-babel-R-input-command
 (format  "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
	  name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))

I haven’t tried to implement this yet as it’s basically just fiddling with something that works. The only reason for it I can think of would be efficiency and I haven’t tested that.

This Didn’t work after an initial test. I still think this is a good idea (I also think we should try to do something similar when writing out results frmo R to elisp) however as it wouldn’t result in any functional changes I’m bumping it down to deferred for now. [Eric]

for quick tests

123
mean(mean(vec))
2
2

Rework Interaction with Running Processes [2/5]

robust to errors interrupting execution

sleep(10)
:patton_is_an_grumpy

use C-g keyboard-quit to push processing into the background

This may be possible using the `run-with-timer’ command.

I have no idea how this could work…

sleep(10)
:patton_is_an_grumpy

ability to select which of multiple sessions is being used

Increasingly it is looking like we’re going to want to run all source code blocks in comint buffer (sessions). Which will have the benefits of

  1. allowing background execution
  2. maintaining state between source-blocks
    • allowing inline blocks w/o header arguments

R sessions

(like ess-switch-process in .R buffers)

Maybe this could be packaged into a header argument, something like :R_session which could accept either the name of the session to use, or the string prompt, in which case we could use the ess-switch-process command to select a new process.

evaluation of shell code as background process?

After C-c C-c on an R code block, the process may appear to block, but C-g can be used to reclaim control of the .org buffer, without interrupting the R evalution. However I believe this is not true of bash/sh evaluation. [Haven’t tried other languages] Perhaps a solution is just to background the individual shell commands.

The other languages (aside from emacs lisp) are run through the shell, so if we find a shell solution it should work for them as well.

Adding an ampersand seems to be a supported way to run commands in the background (see external-commands). Although a more extensible solution may involve the use of the call-process-region function.

Going to try this out in a new file org-babel-proc.el. This should contain functions for asynchronously running generic shell commands in the background, and then returning their input.

partial update of org-mode buffer

The sleekest solution to this may be using a comint buffer, and then defining a filter function which would incrementally interpret the results as they are returned, including insertion into the org-mode buffer. This may actually cause more problems than it is worth, what with the complexities of identifying the types of incrementally returned results, and the need for maintenance of a process marker in the org buffer.

‘working’ spinner

It may be nice and not too difficult to place a spinner on/near the evaluating source code block

conversion of output from interactive shell, R (and python) sessions to org-babel buffers

[DED] This would be a nice feature I think. Although an org-babel purist would say that it’s working the wrong way round… After some interactive work in a R buffer, you save the buffer, maybe edit out some lines, and then convert it to org-babel format for posterity. Same for a shell session either in a shell buffer, or pasted from another terminal emulator. And python of course.

improve the source-block snippet

any real improvement seems somewhat beyond the ability of yasnippet for now.

file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title

,#name : Chapter title
,# --
${1:Chapter}
${1:$(make-string (string-width text) ?\=)}

$0

sb – snippet

waiting for guidance from those more familiar with yasnippets

allow reference to file type results

  • State “DONE” from “TODO” [2010-02-09 Tue 20:29]

So files can be referenced by source code blocks

  • Here’s a simple example counting the size of the user’s .profile file.

    The raw org-mode text

    #+results: my-profile
    [[file:~/.profile][my-profile]]
    
    #+begin_src sh :var profile=my-profile
      wc $profile
    #+end_src
    
    #+results:
    : 22 109 675 /home/eschulte/.profile
        

    Now exported to html

wc $profile
  • Or perhaps a slightly fancier example using the host command to grab IP information for the org-mode homepage.

    The raw org-mode text

    #+results: org-mode-page
    [[http://orgmode.org/][org-mode-page]]
    
    #+begin_src sh :var page=org-mode-page
      host $page |head -1
    #+end_src
    
    #+results:
    : http://orgmode.org/ has address 63.123.155.104
        

    or exported to html

host $page |head -1

evaluation of latex source code blocks

  • State “DONE” from “” [2010-02-06 Sat 10:04]

It is now possible to generate .png and .pdf files when an appropriate :file header arguments is supplied to a latex source code block.

for example the following block will generate a small png of the latex logo using the mechanisms used for in-buffer preview of latex fragments.

#+begin_src latex :file latex-logo.png
  \LaTeX
#+end_src

data/latex-logo.png

To specify that the buffer-colors should not be used for example when exporting to an external file the :buffer no header argument can be passed to latex src blocks generating .png files.

The following more complex example uses the excellent pgf/tikz package to generate a complex figure (converted to png for web display).

#+begin_src latex :file fsa.pdf :packages '(("" "tikz")) :pdfwidth 3in :pdfheight 3in
  % Define block styles
  \usetikzlibrary{shapes,arrows}
  \tikzstyle{astate} = [circle, draw, text centered, font=\footnotesize, fill=blue!25]
  \tikzstyle{rstate} = [circle, draw, text centered, font=\footnotesize, fill=red!25]

:

  \begin{tikzpicture}[->,>=stealth', shorten >=1pt, auto, node distance=2.8cm, semithick]
    \node [astate] (1) at (0,0) {1};
    \node [astate] (2) at (1,0) {2};
    \node [rstate] (3) at (2,0) {3};
    \path (1) edge [bend left] node {b} (2)
          (2) edge node {b} (3)
          (2) edge [bend left] node {a} (1)
          (3) edge [loop above] node {(a, b)} (3);
  \end{tikzpicture}
#+end_src

data/fsa.png

The :pdfwidth and :pdfheight header arguments can be used to control the size of the generated pdf image, and the :packages header argument can be used to temporarily add new packages to your org-export-latex-packages-alist.

allow customization of shebang lines and commenting through header argument

  • State “DONE” from “TODO” [2010-02-05 Fri 19:05]
  • State “TODO” from “” [2010-02-03 Wed 13:07]

this is from suggestions coming through the mailing list

I think the best names for these new header-arguments would be :shebang (which could be multiple lines long if stored in an elisp variable) and (to me at least) :comments is more intuitive than :headers to control the insertion of org-babel comments with links back to the original tangling document.

This is now implemented using the :shebang and :comments header arguments, so for example the following block will tangle with no comments

#+begin_src emacs-lisp :comments no
  (message "I should have no comments")
#+end_src

and the following will tangle using a custom shebang line

#+begin_src ruby :shebang #!/usr/bin/ruby
  puts :nonstandard_shebang
#+end_src

>>>>>>> origin/master:development.org

indent noweb references to the level of the initial ref

  • State “DONE” from “DONE” [2009-12-17 Thu 11:22]

augment `org-babel-expand-noweb-references’ so that it rigidly indents the body of the block to the depth of the original noweb reference.

In fact, a better specification is: copy whatever (any character) is in front
of `<<<code here>>>' in front of every line of the referenced block (same
amount of characters, be it spaces or something else).

That way, a commented SQL source block like the following would be correctly
outputted:

    -- -- set flag
    -- UPDATE dossier
    -- SET DossierSentToSector = @now
    -- WHERE ID
    --     IN (SELECT actID_fk
    --         FROM actions
    --         WHERE (actID = 338 AND actEtat = 3))
    --     AND F1SignDate < @firstDayOfThisMonth
    --     AND DossierSentToSector IS NULL
    --     AND
    --     <<ConditionForSector>>

Currently, the first line of the "condition" block is commented, the others
not, resulting in incorrect code being tangled.

With the above "spec", I guess all cases are covered. What do you think?

Suitable export of #+srcname and #+resname lines

  • State “DONE” from “STARTED” [2010-02-11 Thu 09:46]
  • State “STARTED” from “TODO” [2010-01-04 Mon 15:05]
  • State “TODO” from “STARTED” [2010-01-04 Mon 09:31]

update – exporting arguments with source names

We are now exporting arguments with source-block names and wrapping source-code blocks and names in div which is given the org-src-container class. This allows for styling of the name and block, and for explicit association of the name and block using css like the following which we will apply here with an HTML block

#+begin_html
  <style type="text/css">
    .org-src-container {
    border-left: 4px solid gray; /* gray bar offsetting code and name */
    padding: 0.5em 0.5em 0.5em 1em; }
    .org-src-container pre {
      margin-left: 1em; }        /* indentation of code blocks w/names */
  </style>
#+end_html
.org-src-container { border-left: 4px solid gray; /* gray bar offsetting code and name */ padding: 0.5em 0.5em 0.5em 1em; } .org-src-container pre { margin-left: 1em; } /* indentation of code blocks w/names */

So, for example the following raw org-mode text

#+source: fibonacci
#+begin_src emacs-lisp :var input=0
  (defun fib (n)
    (if (> n 1)
        (+ (fib (- n 1)) (fib (- n 2)))
        1))
  (fib input)
#+end_src

:

#+results: fibonacci
: 1

:

now applying our Fibonacci function

:

#+call: fibonacci(input=5)

:

#+results: fibonacci(input=5)
: 8

exports to the following html

(defun fib (n)
  (if (> n 1)
      (+ (fib (- n 1)) (fib (- n 2)))
      1))
(fib input)

now applying our Fibonacci function

older

The current fix here takes the simplest possible approach. The name of a source-code block is placed in the org-caption text property and `org-export-format-source-code-or-example’ then reads this property and adds the name to the export.

So the following source-code block

#+source: square
#+begin_src emacs-lisp :var input=1
  (* input input)
#+end_src

exports to the following html

(* input input)

which is the following raw html

<label class="org-src-name">square</label>
<pre class="src src-emacs-lisp">
<span style="color: #7f7f7f;">(</span>* input input<span style="color: #7f7f7f;">)</span>
</pre>

and the following raw latex

\lstset{language=Lisp}\begin{lstlisting}[title={square}]
(* input input)
\end{lstlisting}

There is much room here for stylistic improvement and hopefully this initial implementation will spur discussion/suggestions for how to improve the appearance and content of these exported source-names.

sha1 hash based caching

  • State “DONE” from “STARTED” [2009-12-22 Tue 14:42]

This has been implemented. Results can now be cached using the :cache header argument. See the following example.

#+begin_src emacs-lisp :cache yes
  (+ 1 2)
#+end_src

:

#+results[e1b5...]:
: 3

historical

So we now have two caching solutions, one which is entirely in-buffer using sha1 hashes stuffed into the resname lines, and another which saves the results either in RAM or optionally on disk. It is not immediately clear which combination of the two approaches would be best.

[EMS]
I find the saving of cached results in external files to

be very upsetting. It pollutes the user’s disk, and it breaks what is to me a very fundamental part of org-mode, namely the fact that all data is saved in plain text in org-mode files.

Currently I’m leaning towards some combination of file-local variable (RAM) caching and in-buffer caching. I have more comments in-line below.

[DED]
I agree about caching to external files. And now that

the hash is hidden in the resname, I think we definitely want the in-buffer mechanism. Apart from anything else it improves the mechanism by which we decide whether or not to over-write existing results.

The only slight drawback I can see is export: someone who doesn’t want the results in their org file is forced to regenerate them on every export.

[EMS]
alright, that sounds good to me. I can’t think of a

good solution to the export problem right now. The approach taken in org-exp-blocks – suggested by Carsten – is to add the hash to the file name where the results were stored, so I guess that could be an option, but it would be fairly intrusive, and it would share the problem of saving state outside of the actual org-mode buffer. So if you don’t object, I’ll merge in the in-buffer caching, and we can keep the export caching as an open issue, then possibly add the RAM/local-variable caching on top (which seems like it only require a couple of lines of code).

How well do org buffers function with large folded tables?

I have no idea. Emacs seems to be pretty capable of handling huge files, but once we get to millions of lines there would probably be some noticeable delays, and I doubt hiding the results behind an overlay would help.

Maybe this would be a good place for some LOB functions. One for serializing data and one for reading in serialized data. I’m familiar with YAML which at least has ruby, python (and I believe elisp) bindings, so that’s my first thought, but there are probably more efficient solutions.

Given one function for writing to a file, taking a piece of data and a file-name and another function for reading from a file given a file name that should be sufficient for most large storage needs. And there’s also of course SQL support in org-babel.

cache in buffer

Plus
  • Fits cleanly into existing org-babel paradigm.
  • simple to implement – minimal code changes
  • doesn’t rely on anything external to the org-mode file
    • Persistent across emacs sessions
Minus
  • I don’t think it will work for export will it?
  • This does work for exporting results. If a result line is already in the buffer then it will be used instead of re-evaluation of the result on export. note I did however notice two bugs when checking this out which I just pushed up a fix to in the ems-babel branch.
  • So still a slight drawback, as the results must be in the org buffer.
    • The result is editable; no promise that repeat evaluation will

give the original result. true, but that is also true of results stored in local variables (RAM) or on disk – although admittedly it would be harder in those cases. It would be cool if we could automatically remove the hash when a result is edited by hand…

  • [Therefore difficult to confirm that cache is working] nope,

I’ve tested it and it works :)

  • Not good for large tables, yes storing large tables in org-mode

buffers can be a pain, perhaps some sort of result folding would be generally useful – beyond cached results

  • sha1 hashes are ugly and not-for-humans: hide them? yes, they

are hidden in the most recent version in branch ems-babel. If you need to know the hash value for some reason pressing C-c C-c on the small visible portion of the hash will copy it to your kill ring

cache in RAM

Plus
  • Result is not editable – without editing local variables
  • Good for very large tables – as long as we don’t mind

persisting large tables in memory

  • Fastest of the three
Minus
  • Not part of babel paradigm (but should be very unobtrusive)
  • Not persistent across emacs sessions
  • Not sharable (impossible to send a file to someone else and include cached results)
    • user can’t read cached data

cache on disk

Plus
  • Good for large tables
  • Result is not easily editable
  • Persistent across emacs sessions
Minus
  • Not currently part of babel paradigm
  • (but we will probably want to implement external table access) meaning tables in foreign org-mode files? because I think that is already implemented. If some other sort of foreign table then I’m not sure what you mean.
  • Not sharable (impossible to send a file to someone else and include cached results)
    • pollutes user’s directories with new files
    • saves state outside of the org-mode buffer
    • no longer “everything in plain text”
    • currently saving data in /tmp directories where it won’t

survive reboot

  • using (format "%S" object) to serialize data will not work

for large lists/tables

  • elisp may not be the ideal serialization language
  • the cached data is not visible or readable by the user

How do we distinguish a nil result from a lack of a cached result?

I wonder if we should consider some cashing of images, also for export. I think we could have an alist with sha1 hashes as keys and image files as values. The sha1 hash could be made from the entire code and the command that is used to create the image..

– Carsten

(sha1 stuff) seems to work.

org-feed.el has a (require ‘sha1) and org-publish.el uses it too.

– Bernt

fold-able results

  • State “DONE” from “DONE” [2009-11-19 Thu 15:47]

As mentioned in the caching discussion it will often be desirable to hide results in org-mode buffers. This should be fairly easily implemented using the same mechanisms used to fold source-code blocks in tandem with the `org-babel-result-end’ function.

this is currently implemented in the head of the ems-babel branch

Should tangle stop guessing file extensions?

  • State “DONE” from “PROPOSED” [2010-03-20 Sat 12:28]
  • State “PROPOSED” from “” [2010-02-28 Sun 18:33]

    See org commits

    • 29a2ab0047b3a6f0669f1ab84877e166e00aefff
    • aa0e4b86ce2753909ef71081f8c437ad67af0fb1

Email from Martin G. Skjæveland

Rename org-babel-pop-to-session org-babel-switch-to-session

  • State “DONE” from “STARTED” [2010-03-16 Tue 12:59]
  • State “STARTED” from “PROPOSED” [2010-03-16 Tue 12:44]
  • State “PROPOSED” from “” [2010-03-16 Tue 12:33]

    This would bring us into line with ruby-switch-to-inf, python-switch-to-python and ess-switch-to-ESS.

    I’m finding o-b-pop-to-session a very useful function and suggest that we add it to org-babel-keys: z is the compelling choice as C-c C-z is the binding fior ruby, python and ESS.

    I’ve gone ahead and made those changes using a defalias for now.

,—-

org-babel-pop-to-session is an alias for `org-babel-switch-to-session’
in `org-babel.el’.
(org-babel-pop-to-session &optional arg info)
Switch to the session of the current source-code block.
If called with a prefix argument then evaluate the header arguments
for the source block before entering the session. Copy the body
of the source block to the kill ring.

`—-

,—-

org-babel-switch-to-session is an interactive Lisp function.
It is bound to C-c M-b z.
(org-babel-switch-to-session &optional arg info)
Switch to the session of the current source-code block.
If called with a prefix argument then evaluate the header arguments
for the source block before entering the session. Copy the body
of the source block to the kill ring.

`—-

,—-

ruby-switch-to-inf is an interactive compiled Lisp function in
`inf-ruby.el’.
It is bound to C-c C-z.
(ruby-switch-to-inf eob-p)
Switch to the ruby process buffer.
With argument, positions cursor at end of buffer.

`—-

,—-

python-switch-to-python is an interactive compiled Lisp function in
`python.el’.
It is bound to C-c C-z, <menu-bar> <Python> <Switch to interpreter>.
(python-switch-to-python eob-p)
Switch to the Python process buffer, maybe starting new process.
With prefix arg, position cursor at end of buffer.

`—-

,—-

ess-switch-to-end-of-ESS is an interactive compiled Lisp function in
`ess-inf.el’.
It is bound to C-c C-z, <menu-bar> <ESS> <Motion…> <Goto end of ESS
buffer>, C-M-r.
(ess-switch-to-end-of-ESS)
Switch to the end of the inferior ESS process buffer.

`—-

Allow tramp syntax with :tangle

Email from Maurizio Vitale

From: Maurizio Vitale <mav@cuma.i-did-not-set--mail-host-address--so-tickle-me>
Subject: [Orgmode] [babel] using tramp when tangling
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (gnu/linux)
Date: Wed, 17 Mar 2010 14:39:43 -0400
To: emacs-orgmode@gnu.org


I'd find useful to use tramp syntax in the :tangle specification.
In my case it would be to specify sudo when tangling config files that
are supposed to go to areas not writable by the user running Emacs.
Something like:

...

other people might be interested in remote access to tangle targets.

Is there a way to achieve the above?

If I try to tangle the above, I get something along the lines that
"tramp cannot append to file". Would it be possible to have org-babel to
(optionally) tangle to a buffer and then save to file in one go?

Thanks a lot for the excellent addition to org-mode.

       Maurizio

allow ‘output mode to return stdout as value?

We do allow this now. It turned out to be necessary for lob calls using output mode.

Maybe we should allow this. In fact, if block x is called with :results output, and it references blocks y and z, then shouldn’t the output of x contain a concatenation of the outputs of y and z, together with x’s own output? That would raise the question of what happens if y is defined with :results output and z with :results value. I guess z’s (possibly vector/tabular) output would be inside a literal example block containing the whole lot.

Reworking output: verbatim, pretty-print, vector/scalar etc

See emails on subject between ES, BA, DED.

Draft of conclusions of email discussion

  • We add two new :results options (which only take effect with :results value):
    • [X] parseable (code) output
    • [X] pretty-printed output
  • We get rid of :results scalar: [EMS] I don’t see why results scalar is on the chopping block? Did we want another name for this value, or is it overcome by the verbatim/code options? For now I propose we leave the scalar option unchanged.
  • [X] We rename :results vector -> table (even though this will be

the default it is nice to have a name for it, and we need a symbolic name for implementing – and allowing users to change – the default behavior)

  • [X] We force :results output to be empty for elisp

Parseable output

  • output as code block
  • possible :results option names
    • parseable
    • verbatim
    • code

Pretty-printed output

  • output as literal text block (not as code block)
  • possible :results option names
    • pretty
    • pretty-print
    • pp

Language-specific implementation

languageparseableprettyNotes
emacs-lisppppp
ruby?ppppis pp output parseable?
pythonpprint.pprintpprint.pprint
perl??
Rdputdefault interpreter output?
shellNANAno such thing as :results value

pretty print source results

[ see above ]

add a result type for the display of source-code objects. results of this type should be wrapped in source-code blocks.

inspired by Benny’s pp patch for emacs-lisp (below)

  diff --git a/contrib/babel/lisp/langs/org-babel-emacs-lisp.el b/contrib/babel/lisp/langs/org-babel-emacs-lisp.el
index 39f5cc7..60671ac 100644
--- a/contrib/babel/lisp/langs/org-babel-emacs-lisp.el
+++ b/contrib/babel/lisp/langs/org-babel-emacs-lisp.el
@@ -39,10 +39,14 @@
   "Execute a block of emacs-lisp code with org-babel.  This
 function is called by `org-babel-execute-src-block' via multiple-value-bind."
   (message "executing emacs-lisp code block...")
-  (save-window-excursion
-    (let ((print-level nil) (print-length nil))
-      (eval `(let ,(mapcar (lambda (var) `(,(car var) ',(cdr var))) vars)
-	       ,(read (concat "(progn " body ")")))))))
+  (let ((results (cdr (assoc :results params))))
+    (save-window-excursion
+      (let ((print-level nil) (print-length nil))
+        (eval `(let ,(mapcar (lambda (var) `(,(car var) ',(cdr var))) vars)
+                 ,(read (concat "(progn " (if (string-match "pp$" results)
+                                              (concat "(pp " body ")")
+                                              body)
+                                ")"))))))))

 (provide 'org-babel-emacs-lisp)
 ;;; org-babel-emacs-lisp.el ends here

something similar for ruby exists here http://www.ruby-doc.org/stdlib/libdoc/pp/rdoc/index.html

Finalise behaviour regarding vector/scalar output

[ see above ]

Stop spaces causing vector output

This simple example of multilingual chaining produces vector output if there are spaces in the message and scalar otherwise.

[Not any more]

paste(msg, "und R", sep=" ")
org-babel speaks elisp y python und R
msg + " y python"
(concat msg " elisp")

support for passing paths to files between source blocks

Maybe this should be it’s own result type (in addition to scalars and vectors). The reason being that some source-code blocks (for example ditaa or anything that results in the creation of a file) may want to pass a file path back to org-mode which could then be inserted into the org-mode buffer as a link to the file…

This would allow for display of images upon export providing functionality similar to org-exp-blocks only in a more general manner.

Incorporate Benny Andresen’s GNU screen interpreter

Ensure that #+lob calls honour header args

Currently, the header args defined in a source block are not honoured when that source block is referenced by a #+lob call.

Solutions

Maybe we will have to either stop using an emacs-lisp block in the lob implementation, or provide some mechanism for the emacs-lisp blocks to pass unused header arguments through to their :var blocks. At first glance the former seems much cleaner. [Eric]

Allow header args on lob line

commit da1f07620862a2f8701597fbd6d8ceca93183840

Example

Here’s a source block that defines the action we want to do.

And here’s some more text.

Another heading

Finally

Here is where we actually want the image to appear

blue.png

dynamic clock tables (as input to source blocks)

something like…

#+BEGIN: clocktable :maxlevel 2 :block today :scope tree1 :link t :name todays-clock
#+END: clocktable

These actually work given the current setup, the tables simply need to be named using a #+TBLNAME: line.

#+TBLNAME: todays-times
#+BEGIN: clocktable :maxlevel 2 :block today :scope tree1 :link t :name todays-clock
#+END: clocktable

figure out how to handle graphic output

This is listed under graphical output in out objectives.

This should take advantage of the :results file option, and languages which almost always produce graphical output should set :results file to true by default (this is currently done for the gnuplot and ditaa languages). That would handle placing these results in the buffer. Then if there is a combination of silent and file :results headers we could drop the results to a temp buffer and pop open that buffer…

Display of file results is addressed in the #+begin_src R :session *R* :results org cat("***** This is a table\n") cat("| 1 | 2 | 3 |\n") cat("[[http://google.com][Google it here]]\n" #+end_src

:

#+resname:
***** This is a table
| 1 | 2 | 3 |

: Google it here

We actually might want to remove the #+resname line if the results type is org-mode, not sure… Either way I don’t think there is a good way to capture/remove org type results.

LaTeX

"this should be inside of a LaTeX block"

Html

"this should be inside of a HTML block



and more



is long"
this should be inside of a HTML block and more is long

raw

Added a raw results header argument, which will insert the results of a source-code block into an org buffer un-escaped. Also, if the results look like a table, then the table will be aligned.

puts "| root | square |"
puts "|---"
10.times do |n|
  puts "| #{n} | #{n*n} |"
end
rootsquare
00
11
24
39
416
525
636
749
864
981

Not sure how/if this would work, but it may be desirable.

org-bable-tangle - no default extension if one already exists

take default values for header args from properties

Use file-wide and subtree wide properties to set default values for header args.

[DED] One thing I’m finding when working with R is that an org file may contain many source blocks, but that I just want to evaluate a subset of them. Typically this is in order to take up where I left off: I need to recreate a bunch of variables in the session environment. I’m thinking maybe we want to use a tag-based mechanism similar to :export: and :noexport: to control evaluation on a per-subtree basis.

test-header with properties

Ahh… as is so often the case, just had to wrap `org-babel-params-from-properties’ in a `save-match-data’ form.

1234
(+ (length my-lis) def)

new reference syntax inside source code blocks

This is from an email discussion on the org-mode mailing list with Sébastien. The goal here is to mimic the source-block reference style of Noweb. Upon export and/or tangle these references could be replaced with the actual body of the referenced source-code block.

See the following for an example.

puts "---------------------------header---------------------------"
puts "---------------------------footer---------------------------"
# <<ems-ruby-print-header>>
puts "                            Ruby                            "
# <<ems-ruby-print-footer>>

Upon export the previous source-code block would result in a file being generated at ruby-noweb.rb with the following contents

puts "---------------------------header---------------------------"
puts "                            Ruby                            "
puts "---------------------------footer---------------------------"

the body of a source-code block with all <<src-name>> references expanded can now be returned by `org-babel-expand-noweb-references’. This function is now called by default on all source-code blocks on export.

re-work tangling system

Sometimes when tangling a file (e.g. when extracting elisp from a org-mode file) we want to get nearly every source-code block.

Sometimes we want to only extract those source-code blocks which reference a indicate that they should be extracted (e.g. traditional literate programming along the Noweb model)

I’m not sure how we can devise a single simple tangling system that naturally fits both of these use cases.

new setup

the tangle header argument will default to no meaning source-code blocks will not be exported by default. In order for a source-code block to be tangled it needs to have an output file specified. This can happen in two ways…

  1. a file-wide default output file can be passed to `org-babel-tangle’ which will then be used for all blocks
  2. if the value of the tangle header argument is anything other than no or yes then it is used as the file name
(org-babel-load-file "test-tangle.org")
(if (string= test-tangle-advert "use org-babel-tangle for all your emacs initialization files!!")
    "succeed"
  "fail")
succeed

\C-c \C-o to open results of source block

by adding a defadvice to org-open-at-point we can use the common \C-c \C-o keybinding to open the results of a source-code block. This would be especially useful for source-code blocks which generate graphical results and insert a file link as the results in the org-mode buffer. (see TODO figure out how to handle graphic output). This could also act reasonably with other results types…

file
use org-open-at-point to open the file
scalar
open results unquoted in a new buffer
tabular
export the table to a new buffer and open that buffer

when called with a prefix argument the block is re-run

+---------+
| cBLU    |
|         |
|    +----+
|    |cPNK|
|    |    |
+----+----+

blue.png

'((1 2) (3 4))
12
34
8.times do |n|
  puts "row #{n}"
end
row 0
row 1
row 2
row 3
row 4
row 5
row 6
row 7

Stop spaces causing vector output

This simple example of multilingual chaining produces vector output if there are spaces in the message and scalar otherwise.

[Not any more]

paste(msg, "und R", sep=" ")
org-babel speaks elisp y python und R
msg + " y python"
(concat msg " elisp")

add :tangle family of header arguments

values are

no
don’t include source-code block when tangling
yes
do include source-code block when tangling

this is tested in test-tangle.org

extensible library of callable source blocks

Current design

This is covered by the Library of Babel, which will contain ready-made source blocks designed to carry out useful common tasks.

Initial statement [Eric]

Much of the power of org-R seems to be in it’s helper functions for the quick graphing of tables. Should we try to re-implement these functions on top of org-babel?

I’m thinking this may be useful both to add features to org-babel-R and also to potentially suggest extensions of the framework. For example one that comes to mind is the ability to treat a source-code block like a function which accepts arguments and returns results. Actually this can be it’s own TODO (see source blocks as functions).

Objectives [Dan]

  • We want to provide convenient off-the-shelf actions (e.g. plotting data) that make use of our new code evaluation environment but do not require any actual coding.

Initial Design proposal [Dan]

  • Input data will be specified using the same mechanism as :var references, thus the input data may come from a table, or another source block, and it is initially available as an elisp data structure.
  • We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that line will apply an action to the referenced data.
  • Actions correspond to source blocks: our library of available actions will be a library of org-babel source blocks. Thus the code for executing an action, and the code for dealing with the output of the action will be the same code as for executing source blocks in general
  • Optionally, the user can have the relevant source block inserted into the org buffer after the (say) #+BABELDO line. This will allow the user to fine tune the action by modifying the code (especially useful for plots).
  • So maybe a #+BABELDO line will have header args
    • :data (a reference to a table or source code block)
    • :action (or should that be :srcname?) which will be something

like :action pie-chart, referring to a source block which will be executed with the :data referent passed in using a :var arg.

  • :showcode or something controlling whether to show the code

Modification to design

I’m implementing this, at least initially, as a new interpreter named ‘babel’, which has an empty body. ‘babel’ blocks take a :srcname header arg, and look for the source-code block with that name. They then execute the referenced block, after first appending their own header args on to the target block’s header args.

If the target block is in the library of babel (a.o.t. e.g. the current buffer), then the code in the block will refer to the input data with a name dictated by convention (e.g. __data__ (something which is syntactically legal in all languages…). Thus the babel block will use a :var __data__ = whatever header arg to reference the data to be plotted.

Column names in R input/output

This has been implemented: Automatic on input to R; optional in output. Note that this equates column names with the header row in an org table; whereas org actually has a mechanism whereby a row with a ‘!’ in the first field defines column names. I have not attempted to support these org table mechanisms yet. See this DEFERRED todo item.

use example block for large amounts of stdout output?

We’re currently `examplizing’ with : at the beginning of the line, but should larger amounts of output be in a \#+begin_example…\#+end_example block? What’s the cutoff? > 1 line? This would be nice as it would allow folding of lengthy output. Sometimes one will want to see stdout just to check everything looks OK, and then fold it away.

I’m addressing this in branch ‘examplizing-output’. Yea, that makes sense. (either that or allow folding of large blocks escaped with :).

Proposed cutoff of 10 lines, we can save this value in a user customizable variable.

add ability to remove such results

exclusive exports params

:this_is_a_test
:this_is_a_test

LoB: allow output in buffer

allow default header arguments by language

org-babel-default-header-args:lang-name

An example of when this is useful is for languages which always return files as their results (e.g. ditaa, and gnuplot).

singe-function tangling and loading elisp from literate org-mode file [3/3]

This function should tangle the org-mode file for elisp, and then call `load-file’ on the resulting tangled file.

(setq test-tangle-advert nil)
(setq test-tangle-loading nil)
(setq results (list :before test-tangle-loading test-tangle-advert))
(org-babel-load-file "test-tangle.org")
(setq results (list (list :after test-tangle-loading test-tangle-advert) results))
(delete-file "test-tangle.el")
(reverse results)
:beforenilnil
:after“org-babel tangles”“use org-babel-tangle for all your emacs initialization files!!”

add optional language limiter to org-babel-tangle

This should check to see if there is any need to re-export

ensure that org-babel-tangle returns the path to the tangled file(s)

(mapcar #'file-name-nondirectory (org-babel-tangle-file "test-tangle.org" "emacs-lisp"))
“test-tangle.el”

only tangle the file if it’s actually necessary

add a function to jump to a source-block by name

I’ve had an initial stab at that in org-babel-find-named-block (library-of-babel branch).

At the same time I introduced org-babel-named-src-block-regexp, to match src-blocks with srcname.

This is now working with the command `org-babel-goto-named-source-block’, all we need is a good key binding.

add :none session argument (for purely functional execution) [4/4]

This would allow source blocks to be run in their own new process

  • These blocks could then also be run in the background (since we can detach and just wait for the process to signal that it has terminated)
  • We wouldn’t be drowning in session buffers after running the tests
  • we can re-use much of the session code to run in a more functional mode

While session provide a lot of cool features, like persistent environments, pop-to-session, and hints at exportation for org-babel-tangle, they also have some down sides and I’m thinking that session-based execution maybe shouldn’t be the default behavior.

Down-sides to sessions

  • much more complicated than functional evaluation
    • maintaining the state of the session has weird issues
    • waiting for evaluation to finish
    • prompt issues like shell-prompt-escapes-bug
  • can’t run in background
  • litter emacs with session buffers

ruby

puts :eric
puts :schulte
[1, 2, 3]
“eric”
“schulte”

python

print 'something'
print 'output'
[1, 2, 3]
123

sh

echo "first"
echo "second"
“first”
“second”

R

a <- 8
b <- 9
a + b
b - a
“[1]”17
“[1]”1

fully purge org-babel-R of direct comint interaction

try to remove all code under the ;; functions for evaluation of R code line

Create objects in top level (global) environment [5/5]

sessions

initial requirement statement [DED]

At the moment, objects created by computations performed in the code block are evaluated in the scope of the code-block-function-body and therefore disappear when the code block is evaluated {unless you employ some extra trickery like assign(‘name’, object, env=globalenv()) }. I think it will be desirable to also allow for a style wherein objects that are created in one code block persist in the R global environment and can be re-used in a separate block.

This is what Sweave does, and while I’m not saying we have to be the same as Sweave, it wouldn’t be hard for us to provide the same behaviour in this case; if we don’t, we risk undeservedly being written off as an oddity by some.

IOW one aspect of org-babel is that of a sort of functional meta-programming language. This is crazy, in a very good way. Nevertheless, wrt R I think there’s going to be a lot of value in providing for a working style in which the objects are stored in the R session, rather than elisp/org buffer. This will be a very familiar working style to lots of people.

There are no doubt a number of different ways of accomplishing this, the simplest being a hack like adding

for(objname in ls())
    assign(objname, get(objname), envir=globalenv())

to the source code block function body. (Maybe wrap it in an on.exit() call).

However this may deserve to be thought about more carefully, perhaps with a view to having a uniform approach across languages. E.g. shell code blocks have the same semantics at the moment (no persistence of variables across code blocks), because the body is evaluated in a new bash shell process rather than a running shell. And I guess the same is true for python. However, in both these cases, you could imagine implementing the alternative in which the body is evaluated in a persistent interactive session. It’s just that it’s particularly natural for R, seeing as both ESS and org-babel evaluate commands in a single persistent R session.

sessions [Eric]

Thanks for bringing this up. I think you are absolutely correct that we should provide support for a persistent environment (maybe called a session) in which to evaluate code blocks. I think the current setup demonstrates my personal bias for a functional style of programming which is certainly not ideal in all contexts.

While the R function you mention does look like an elegant solution, I think we should choose an implementation that would be the same across all source code types. Specifically I think we should allow the user to specify an optional session as a header variable (when not present we assume a default session for each language). The session name could be used to name a comint buffer (like the R buffer) in which all evaluation would take place (within which variables would retain their values –at least once I remove some of the functional method wrappings currently in place– ).

This would allow multiple environments to be used in the same buffer, and once this setup was implemented we should be able to fairly easily implement commands for jumping between source code blocks and the related session buffers, as well as for dumping the last N commands from a session into a new or existing source code block.

Please let me know if you foresee any problems with this proposed setup, or if you think any parts might be confusing for people coming from Sweave. I’ll hopefully find some time to work on this later in the week.

can functional and interpreted/interactive models coexist?

Even though both of these use the same *R* buffer the value of a is not preserved because it is assigned inside of a functional wrapper.

a <- 9
b <- 21
a + b
a

This functional wrapper was implemented in order to efficiently return the results of the execution of the entire source code block. However it inhibits the evaluation of source code blocks in the top level, which would allow for persistence of variable assignment across evaluations. How can we allow both evaluation in the top level, and efficient capture of the return value of an entire source code block in a language independent manner?

Possible solutions…

  1. we can’t so we will have to implement two types of evaluation depending on which is appropriate (functional or imperative)
  2. we remove the functional wrapper and parse the source code block into it’s top level statements (most often but not always on line breaks) so that we can isolate the final segment which is our return value.
  3. we add some sort of “#+return” line to the code block
  4. we take advantage of each languages support for meta-programming through eval type functions, and use said to evaluate the entire blocks in such a way that their environment can be combined with the global environment, and their results are still captured.
  5. I believe that most modern languages which support interactive sessions have support for a last_result type function, which returns the result of the last input without re-calculation. If widely enough present this would be the ideal solution to a combination of functional and imperative styles.

None of these solutions seem very desirable, but for now I don’t see what else would be possible.

Of these options I was leaning towards (1) and (4) but now believe that if it is possible option (5) will be ideal.

(1) both functional and imperative evaluation

Pros

  • can take advantage of built in functions for sending regions to the inferior process
  • retains the proven tested and working functional wrappers

Cons

  • introduces the complication of keeping track of which type of evaluation is best suited to a particular context
  • the current functional wrappers may require some changes in order to include the existing global context

(4) exploit language meta-programming constructs to explicitly evaluate code

Pros

  • only one type of evaluation

Cons

  • some languages may not have sufficient meta-programming constructs

(5) exploit some last_value functionality if present

Need to ensure that most languages have such a function, those without will simply have to implement their own similar solution…

languagelast_value function
R.Last.value
ruby_
python_
shellsee last command for shells
emacs-lispsee special-case
82 + 18
last command for shells

Do this using the tee shell command, and continually pipe the output to a file.

Got this idea from the following email-thread.

suggested from mailing list

while read line
do
  bash -c "$line" | tee /tmp/last.out1
  mv /tmp/last.out1 /tmp/last.out
done

another proposed solution from the above thread

#!/bin/bash
# so - Save Output. Saves output of command in OUT shell variable.
OUT=`$*`
echo $OUT

and another

emacs-lisp will be a special case

While it is possible for emacs-lisp to be run in a console type environment (see the elim function) it is not possible to run emacs-lisp in a different session. Meaning any variable set top level of the console environment will be set everywhere inside emacs. For this reason I think that it doesn’t make any sense to worry about session support for emacs-lisp.

Further thoughts on ‘scripting’ vs. functional approaches

These are just thoughts, I don’t know how sure I am about this. And again, perhaps I’m not saying anything very radical, just that it would be nice to have some options supporting things like receiving text output in the org buffer.

I can see that you’ve already gone some way down the road towards the ‘last value’ approach, so sorry if my comments come rather late. I am concerned that we are not giving sufficient attention to stdout / the text that is returned by the interpreters. In contrast, many of our potential users will be accustomed to a ‘scripting’ approach, where they are outputting text at various points in the code block, not just at the end. I am leaning towards thinking that we should have 2 modes of evaluation: ‘script’ mode, and ‘functional’ mode.

In script mode, evaluation of a code block would result in all text output from that code block appearing as output in the org buffer, presumably as an #+begin_example…#+end_example. There could be an :echo option controlling whether the input commands also appear in the output. [This is like Sweave].

In functional mode, the result of the code block is available as an elisp object, and may appear in the org buffer as an org table/string, via the mechanisms you have developed already.

One thing I’m wondering about is whether, in script mode, there simply should not be a return value. Perhaps this is not so different from what exists: script mode would be new, and what exists currently would be functional mode.

I think it’s likely that, while code evaluation will be exciting to people, a large majority of our users in a large majority of their usage will not attempt to actually use the return value from a source code block in any meaningful way. In that case, it seems rather restrictive to only allow them to see output from the end of the code block.

Instead I think the most accessible way to introduce org-babel to people, at least while they are learning it, is as an immensely powerful environment in which to embed their ‘scripts’, which now also allows them to ‘run’ their ‘scripts’. Especially as such people are likely to be the least capable of the user-base, a possible design-rule would be to make the scripting style of usage easy (default?), perhaps requiring a special option to enable a functional style. Those who will use the functional style won’t have a problem understanding what’s going on, whereas the ‘skript kiddies’ might not even know the syntax for defining a function in their language of choice. And of course we can allow the user to set a variable in their .emacs controlling the preference, so that functional users are not inconveniennced by having to provide header args the whole time.

Please don’t get the impression that I am down-valuing the functional style of org-babel. I am constantly horrified at the messy ‘scripts’ that my colleagues produce in perl or R or whatever! Nevertheless that seems to be how a lot of people work.

I think you were leaning towards the last-value approach because it offered the possibility of unified code supporting both the single evaluation environment and the functional style. If you agree with any of the above then perhaps it will impact upon this and mean that the code in the two branches has to differ a bit. In that case, functional mode could perhaps after all evaluate each code block in its own environment, thus (re)approaching ‘true’ functional programming (side-effects are hard to achieve).

ls > files
echo "There are `wc -l files` files in this directory"

even more thoughts on evaluation, results, models and options

Thanks Dan, These comments are invaluable.

What do you think about this as a new list of priorities/requirements for the execution of source-code blocks.

  • Sessions
    1. we want the evaluation of the source code block to take place in a session which can persist state (variables, current directory, etc…).
    2. source code blocks can specify their session with a header argument
    3. each session should correspond to an Emacs comint buffer so that the user can drop into the session and experiment with live code evaluation.
  • Results
    1. each source-code block generates some form of results which (as we have already implemented) is transfered into emacs-lisp after which it can be inserted into the org-mode buffer, or used by other source-code blocks
    2. when the results are translated into emacs-lisp, forced to be interpreted as a scalar (dumping their raw values into the org-mode buffer), as a vector (which is often desirable with R code blocks), or interpreted on the fly (the default option). Note that this is very nearly currently implemented through the results-type-header.
    3. there should be two means of collecting results from the execution of a source code block. Either the value of the last statement of the source code block, or the collection of all that has been passed to STDOUT during the evaluation.

header argument or return line (header argument)

Rather than using a header argument to specify how the return value should be passed back, I’m leaning towards the use of a #+RETURN line inside the block. If such a line is not present then we default to using STDOUT to collect results, but if such a line is present then we use it’s value as the results of the block. I think this will allow for the most elegant specification between functional and script execution. This also cleans up some issues of implementation and finding which statement is the last statement.

Having given this more thought, I think a header argument is preferable. The #+return: line adds new complicating syntax for something that does little more than we would accomplish through the addition of a header argument. The only benefit being that we know where the final statement starts, which is not an issue in those languages which contain ‘last value’ operators.

new header :results arguments

script
explicitly states that we want to use STDOUT to initialize our results
return_last
stdout is ignored instead the value of the final statement in the block is returned
echo
means echo the contents of the source-code block along with the results (this implies the script :results argument as well)

rework evaluation lang-by-lang [4/4]

This should include…

  • functional results working with the comint buffer
  • results headers
    script
    return the output of STDOUT
    • write a macro which runs the first redirection, executes the body, then runs the second redirection
    last
    return the value of the last statement -
  • sessions in comint buffers

Ruby [4/4]

  • [X] functional results working with comint
  • [X] script results
  • [X] ensure scalar/vector results args are taken into consideration
  • [X] ensure callable by other source block
a = 2
b = 4
c = a + b
[a, b, c, 78]
24678
last.flatten.size + 1
5
ruby sessions
schulte = 27
schulte + 3
schulte

R [4/4]

  • [X] functional results working with comint
  • [X] script results
  • [X] ensure scalar/vector results args are taken into consideration
  • [X] ensure callable by other source block

To redirect output to a file, you can use the sink() command.

a <- 9
b <- 10
b - a
a + b
83
twoentyseven + 9
28

Python [4/4]

  • [X] functional results working with comint
  • [X] script results
  • [X] ensure scalar/vector results args are taken into consideration
  • [X] ensure callable by other source block
8
9
10
tasking + 2
12

Shells [4/4]

  • [X] functional results working with comint
  • [X] script results
  • [X] ensure scalar/vector results args are taken into consideration
  • [X] ensure callable by other source block
echo 'eric'
date
echo $other ' is the old date'
$ Fri Jun 12 13:08:37 PDT 2009  is the old date

implement a session header argument [4/4]

:session header argument to override the default session buffer

ruby

schulte = :in_schulte
:in_schulte
schulte
:in_schulte
:in_schulte
:in_schulte

python

what = 98
what
98

shell

WHAT='patton'
echo $WHAT
patton

R

a <- 9
b <- 8
a + b
17
a + b

function to bring up inferior-process buffer [4/4]

This should be callable from inside of a source-code block in an org-mode buffer. It should evaluate the header arguments, then bring up the inf-proc buffer using pop-to-buffer.

For lack of a better place, lets add this to the `org-metadown-hook’ hook.

To give this a try, place the cursor on a source block with variables, (optionally git a prefix argument) then hold meta and press down.

ruby

num.times{|n| puts another}

python

another * num

R

a * b

shell

echo $NAME

function to dump last N lines from inf-proc buffer into the current source block

Callable with a prefix argument to specify how many lines should be dumped into the source-code buffer.

comint notes

Implementing comint integration in org-babel-comint.el.

Need to have…

  • handling of outputs
    • split raw output from process by prompts
    • a ring of the outputs, buffer-local, `org-babel-comint-output-ring’
    • a switch for dumping all outputs to a buffer
  • inputting commands

Lets drop all this language specific stuff, and just use org-babel-comint to split up our outputs, and return either the last value of an execution or the combination of values from the executions.

comint filter functions

;;  comint-input-filter-functions	hook	process-in-a-buffer
;;  comint-output-filter-functions	hook	function modes.
;;  comint-preoutput-filter-functions   hook
;;  comint-input-filter			function ...
1
2
3
4
5

Remove protective commas from # comments before evaluating

org inserts protective commas in front of ## comments in language modes that use them. We need to remove them prior to sending code to the interpreter.

,# this one might break it??
:comma_protection

pass multiple reference arguments into R

Can we do this? I wasn’t sure how to supply multiple ‘var’ header args. Just delete this if I’m being dense.

This should be working, see the following example…

n + m
10

ensure that table ranges work

when a table range is passed to org-babel as an argument, it should be interpreted as a vector.

12simple
23Fixnum:1
34Array:123456
45
56
67
"simple"
"#{n.class}:#{n}"
n
Array:123
ar.size
3

global variable indicating default to vector output

how about an alist… org-babel-default-header-args this may already exist… just execute the following and all source blocks will default to vector output

(setq org-babel-default-header-args '((:results . "vector")))

name named results if source block is named

currently this isn’t happening although it should be

:namer
:namer

(simple caching) check for named results before source blocks

see the TODO comment in org-babel-ref.el#org-babel-ref-resolve-reference

set :results silent when eval with prefix argument

'silentp

results-type header (vector/file) [3/3]

In response to a point in Dan’s email. We should allow the user to force scalar or vector results. This could be done with a header argument, and the default behavior could be controlled through a configuration variable.

:scalar

since it doesn’t make sense to turn a vector into a scalar, lets just add a two values…

vector
forces the results to be a vector (potentially 1 dimensional)
file
this throws an error if the result isn’t a string, and tries to treat it as a path to a file.

I’m just going to cram all of these into the :results header argument. Then if we allow multiple header arguments it should work out, for example one possible header argument string could be :results replace vector file, which would replace any existing results forcing the results into an org-mode table, and interpreting any strings as file paths.

“:scalar”

multiple :results headers

:schulte

file result types

When inserting into an org-mode buffer create a link with the path being the value, and optionally the display being the file-name-nondirectory if it exists.

"something"

something

This will be useful because blocks like ditaa and dot can return the string path of their files, and can add file to their results header.

vector result types

8
8

results name

In order to do this we will need to start naming our results. Since the source blocks are named with #+srcname: lines we can name results with #+resname: lines (if the source block has no name then no name is given to the #+resname: line on creation, otherwise the name of the source block is used).

This will have the additional benefit of allowing results and source blocks to be located in different places in a buffer (and eventually in different buffers entirely).

'schulte

Once source blocks are able to find their own #+resname: lines we then need to…

(sbe "developing-resnames")
schulte

change the results insertion functions to use these lines

teach references to resolve #+resname lines.

org-babel tests org-babel [1/1]

since we are accumulating this nice collection of source-code blocks in the sandbox section we should make use of them as unit tests. What’s more, we should be able to actually use org-babel to run these tests.

We would just need to cycle over every source code block under the sandbox, run it, and assert that the return value is equal to what we expect.

I have the feeling that this should be possible using only org-babel functions with minimal or no additional elisp. It would be very cool for org-babel to be able to test itself.

This is now done, see * Tests.

org-babel assertions (may not be necessary)

These could be used to make assertions about the results of a source-code block. If the assertion fails then the point could be moved to the block, and error messages and highlighting etc… could ensue

make C-c C-c work anywhere within source code block?

This seems like it would be nice to me, but perhaps it would be inefficient or ugly in implementation? I suppose you could search forward, and if you find #+end_src before you find #+begin_src, then you’re inside one. [DED]

Agreed, I think inside of the #+srcname: line would be useful as well.

'schulte

integration with org tables

We should make it easy to call org-babel source blocks from org-mode table formulas. This is practical now that it is possible to pass arguments to org-babel source blocks.

See the related sandbox header for tests/examples.

digging in org-table.el

In the past org-table.el has proven difficult to work with.

Should be a hook in org-table-eval-formula.

Looks like I need to change this if statement (line 2239) into a cond expression.

source blocks as functions

Allow source code blocks to be called like functions, with arguments specified. We are already able to call a source-code block and assign it’s return result to a variable. This would just add the ability to specify the values of the arguments to the source code block assuming any exist. For an example see

When a variable appears in a header argument, how do we differentiate between it’s value being a reference or a literal value? I guess this could work just like a programming language. If it’s escaped or in quotes, then we count it as a literal, otherwise we try to look it up and evaluate it.

folding of code blocks? [2/2]

[DED] In similar way to using outline-minor-mode for folding function bodies, can we fold code blocks? #+begin whatever statements are pretty ugly, and in any case when you’re thinking about the overall game plan you don’t necessarily want to see the code for each Step.

folding of source code block

Sounds good, and wasn’t too hard to implement. Code blocks should now be fold-able in the same manner as headlines (by pressing TAB on the first line).

folding of results

So, lets do a three-stage tab cycle… First fold the src block, then fold the results, then unfold.

There’s no way to tell if the results are a table or not w/o actually executing the block which would be too expensive of an operation.

selective export of text, code, figures

[DED] The org-babel buffer contains everything (code, headings and notes/prose describing what you’re up to, textual/numeric/graphical code output, etc). However on export to html / LaTeX one might want to include only a subset of that content. For example you might want to create a presentation of what you’ve done which omits the code.

[EMS] So I think this should be implemented as a property which can be set globally or on the outline header level (I need to review the mechanics of org-mode properties). And then as a source block header argument which will apply only to a specific source code block. A header argument of :export with values of

code
just show the code in the source code block
none
don’t show the code or the results of the evaluation
results
just show the results of the code evaluation (don’t show the actual code)
both
show both the source code, and the results

this will be done in (sandbox) selective export.

a header argument specifying silent evaluation (no output)

This would be useful across all types of source block. Currently there is a :replace t option to control output, this could be generalized to an :output option which could take the following options (maybe more)

t
this would be the default, and would simply insert the results after the source block
replace
to replace any results which may already be there
silent
this would inhibit any insertion of the results

This is now implemented see the example in the sandbox

assign variables from tables in R

This is now working (see (sandbox-table)-R). Although it’s not that impressive until we are able to print table results from R.

insert 2-D R results as tables

everything is working but R and shell

shells

R

This has already been tackled by Dan in org-R:check-dimensions. The functions there should be useful in combination with R-export-to-csv as a means of converting multidimensional R objects to emacs lisp.

It may be as simple as first checking if the data is multidimensional, and then, if so using write to write the data out to a temporary file from which emacs can read the data in using org-table-import.

Looking into this further, is seems that there is no such thing as a scalar in R R-scalar-vs-vector In that light I am not sure how to deal with trivial vectors (scalars) in R. I’m tempted to just treat them as vectors, but then that would lead to a proliferation of trivial 1-cell tables…

allow variable initialization from source blocks

Currently it is possible to initialize a variable from an org-mode table with a block argument like table=sandbox (note that the variable doesn’t have to named table) as in the following example

123
4schulte6
(message (format "table = %S" table))
"table = ((1 2 3) (4 \"schulte\" 6))"

It would be good to allow initialization of variables from the results of other source blocks in the same manner. This would probably require the addition of #+SRCNAME: example lines for the naming of source blocks, also the table=sandbox syntax may have to be expanded to specify whether the target is a source code block or a table (alternately we could just match the first one with the given name whether it’s a table or a source code block).

At least initially I’ll try to implement this so that there is no need to specify whether the reference is to a table or a source-code block. That seems to be simpler both in terms of use and implementation.

This is now working for emacs-lisp, ruby and python (and mixtures of the three) source blocks. See the examples in the sandbox.

This is currently working only with emacs lisp as in the following example in the emacs lisp source reference.

add a :requires header argument

Given the use of noweb references we no longer need to have a need for this sort of functionality.

this header argument would specify a named source-code block which should be appended to the body of the current source-code block before evaluation.

import types
types.FunctionType

The initial implementation of this should be fairly easy and straightforward. It may become more complicated when it comes to

  • ensuring that blocks aren’t repeated on tangling
  • ensuring that blocks aren’t repeated during session based evaluation

additional comment (Juan Reyero)

Sounds like a good solution. Another possibility would be to add an option that makes chunks dependent on other chunks that appear earlier in the buffer. It is less general, but possibly simpler to implement (you don’t have to worry about circular dependencies) and less verbose. If you could assume a functional style without side effects you could even track which chunks are up-to-date, and only re-compute from the first one not up-to-date in the buffer onwards to the chunk you are being asked to process. This could be yet another option.

Add languages [17/21]

I’m sure there are many more that aren’t listed here. Please add them, and bubble any that you particularly care about up to the top.

Any new language should be implemented in a org-babel-lang.el file. Follow the pattern set by org-babel-shell.el and org-babel-R.el.

(defun org-babel-list-supported-languages ()
  (interactive)
  (sort
   (set-difference
   (mapcar
    (lambda (s) (intern (progn (string-match "^ob-\\(.+\\)\.el$" s)
                          (match-string 1 s))))
    (directory-files
     (save-window-excursion
       (file-name-directory
        (buffer-file-name (find-library "ob"))))
     nil "^ob-.+\.el$"))
   '(comint eval exp keys lob ref table tangle))
   (lambda (x y) (string< (downcase (symbol-name x))
                     (downcase (symbol-name y))))))

(mapcar 'list (org-babel-list-supported-languages))
(length langs)

matlab and octave

clojure

  • State “DONE” from “” [2010-03-01 Mon 10:26]

add session support for clojure

  • State “DONE” from “” [2010-01-06 Wed 17:41]

Thanks to Joel Boehland org-babel-clojure now supports multiple sessions. This can be exercised through the following

raw org-mode
#+srcname: set-clojure-session-var-s1
#+begin_src clojure :session s1 :results value
(def *var* [1 2 3])
#+end_src

:

#+results: set-clojure-session-var-s1
: #'user/*var*

: :

#+srcname: set-clojure-session-var-s2
#+begin_src clojure :session s2 :results value
(def *var* [3 4 5 6 7 8 9])
#+end_src

:

#+results: set-clojure-session-var-s2
: #'user/*var*

: :

#+srcname: get-clojure-session-var-s1
#+begin_src clojure :session s1 :results value
(count *var*)
#+end_src

:

#+results: get-clojure-session-var-s1
: 3

: :

#+srcname: get-clojure-session-var-s2
#+begin_src clojure :session s2 :results value
(count *var*)
#+end_src

:

#+results: get-clojure-session-var-s2
: 7
exported to html
(def *var* [1 2 3])
(def *var* [3 4 5 6 7 8 9])
(count *var*)
(count *var*)

oz

  • State “DONE” from “” [2010-03-01 Mon 10:27]

added support for Oz

  • State “DONE” from “DONE” [2010-02-23 Tue 07:33]
  • State “DONE” from “” [2010-02-17 Wed 08:24]

Thanks to Torsten Anders for sharing Org-babel support for the Oz programming language!

For complete information see the org-babel-doc-oz page on Worg.

screen

  • State “DONE” from “” [2010-03-01 Mon 10:28]

Haskell

"hello Haskell"
hello Haskell
let fac n = if n == 0 then 1 else n * fac (n - 1)
fac 4
24
[1, 2, 3, 4, 5]
12345

allow non-interactive evaluation

ocaml [2/3]

  • [X] Working for the simple case (no arguments, simple output)
  • [X] correct handling of vector/list output
  • [ ] ability to import arguments
let rec fib x =
  match x with
    | 0 -> 1
    | 1 -> 1
    | n -> fib(n - 1) + fib(n - 2) in
  fib 12
233
"string"
"string"
[1; 2; 3; 4]
1234
[|"ocaml"; "array"|]
“ocaml”“array”

perl

  • State “DONE” from “TODO” [2010-03-01 Mon 05:22]

This could probably be added to org-babel-script.el

It has been implemented on its own (for external process only). I think org-babel-script.el may be obsolete now.

$a = "1 2 3";
split " ", $a;

java

SQL

Things left to do

  • support for sessions
  • add more useful header arguments (user, passwd, database, etc…)
  • support for more engines (currently only supports mysql)
  • what’s a reasonable way to drop table data into SQL?
show databases
“Database”
“information_schema”
“test”

SASS

Sass is a very nice extension of CSS, which is much nicer to read and write (see sass-lang).

#me
  position: absolute
  top: 1em
  left: 1em
  .head
    text-align: center

stylesheet.css

CSS

trivial org-babel-css.el

ditaa

(see file result types)

+---------+
| cBLU    |
|         |
|    +----+
|    |cPNK|
|    |    |
+----+----+

blue.png

latex

  • State “DONE” from “” [2010-03-01 Mon 10:27]

gnuplot [7/7]

(see file result types)

independent varfirst dependent varsecond dependent var
0.10.4250.375
0.20.31250.3375
0.30.249999930.28333338
0.40.2750.28125
0.50.260.27
0.60.258333380.24999993
0.70.246428450.23928553
0.80.231250.2375
0.90.233333230.2333332
10.22250.22
1.10.209090750.22272708
1.20.199999980.21458333
1.30.196153680.21730748
set title "Implementing Gnuplot"
plot data using 1:2 with lines

add variables

gnuplot 4.2 and up support user defined variables. This is how we will handle variables with org-babel (meaning we will need to require gnuplot 4.2 and up for variable support, which can be install using macports on Mac OSX).

  • scalar variables should be replaced in the body of the gnuplot code
  • vector variables should be exported to tab-separated files, and the variable names should be replaced with the path to the files

direct plotting w/o session

gnuplot support for column/row names

This should be implemented along the lines of the R-colname-support.

We can do something similar to the :labels param in org-plot, we just have to be careful to ensure that each label is aligned with the related data file.

This may be walking too close to an entirely prebuilt plotting tool rather than straight gnuplot code evaluation. For now I think this can wait.

a file header argument

to specify a file holding the results

plot data using 1:2, data using 1:3 with lines

plot.png

helpers from org-plot.el

There are a variety of helpers in org-plot which can be fit nicely into custom gnuplot header arguments.

These should all be in place by now.

header argument specifying 3D data

00000000000000000
00000000000000000
00000000000000000
00110010000011100
01001010000100010
01001010000100010
01001010000101110
01001010000100000
01001010000100000
01001010000100010
01001011010100010
00110010110011100
00000000000000000
00000000000000000

gnuplot sessions

Working on this, we won’t support multiple sessions as `gnuplot-mode’ isn’t setup for such things.

Also we can’t display results with the default :none session, so for gnuplot we really want the default behavior to be :default, and to only run a :none session when explicitly specified.

set title "Implementing Gnuplot Sessions"
plot data using 1:2 with lines

session.png

dot

(see file result types)

digraph data_relationships {
  "data_requirement" [shape=Mrecord, label="{DataRequirement|description\lformat\l}"]
  "data_product" [shape=Mrecord, label="{DataProduct|name\lversion\lpoc\lformat\l}"]
  "data_requirement" -> "data_product"
}

test-dot.png

asymptote

(see file result types)

for information on asymptote see http://asymptote.sourceforge.net

import graph;

size(0,4cm);

real f(real t) {return 1+cos(t);}

path g=polargraph(f,0,2pi,operator ..)--cycle;
filldraw(g,pink);

xaxis("$x$",above=true);
yaxis("$y$",above=true);

dot("$(a,0)$",(1,0),N);
dot("$(2a,0)$",(2,0),N+E);

asymptote-test.png

ruby

python

R

emacs-lisp

sh

Bugs [91/128]

lob-one-liners export bug?

  • State “DONE” from “PROPOSED” [2011-02-07 Mon 22:30]
  • State “PROPOSED” from “” [2011-02-01 Tue 15:05]

Debugger entered–Lisp error: (wrong-type-argument stringp nil) replace-match(nil t t) (while (and (< (point) end) (re-search-forward org-babel-lob-one-liner-regexp nil t)) (setq replacement (let ((lob-info (org-babel-lob-get-info))) (save-match-data (org-babel-exp-do-export (list “emacs-lisp” “results” (org-babel-merge-params org-babel-default-header-args (org-babel-params-from-buffer) (org-babel-params-from-properties) (org-babel-parse-header-arguments …)) (car (last lob-info))) (quote lob))))) (setq end (+ end (- (length replacement) (length (match-string 0))))) (replace-match replacement t t)) (save-excursion (goto-char start) (while (and (< (point) end) (re-search-forward org-babel-lob-one-liner-regexp nil t)) (setq replacement (let ((lob-info (org-babel-lob-get-info))) (save-match-data (org-babel-exp-do-export (list “emacs-lisp” “results” (org-babel-merge-params org-babel-default-header-args … … …) (car …)) (quote lob))))) (setq end (+ end (- (length replacement) (length (match-string 0))))) (replace-match replacement t t))) (let (replacement) (save-excursion (goto-char start) (while (and (< (point) end) (re-search-forward org-babel-lob-one-liner-regexp nil t)) (setq replacement (let ((lob-info (org-babel-lob-get-info))) (save-match-data (org-babel-exp-do-export (list “emacs-lisp” “results” … …) (quote lob))))) (setq end (+ end (- (length replacement) (length (match-string 0))))) (replace-match replacement t t)))) org-babel-exp-lob-one-liners(1 574) funcall(org-babel-exp-lob-one-liners 1 574) (lambda (pair) (funcall (second pair) start end))((lob org-babel-exp-lob-one-liners)) mapcar((lambda (pair) (funcall (second pair) start end)) ((lob org-babel-exp-lob-one-liners) (src org-babel-exp-inline-src-blocks))) (catch (quote –cl-block-interblock–) (mapcar (lambda (pair) (funcall (second pair) start end)) org-export-interblocks)) (cl-block-wrapper (catch (quote –cl-block-interblock–) (mapcar (lambda (pair) (funcall (second pair) start end)) org-export-interblocks))) (block interblock (mapcar (lambda (pair) (funcall (second pair) start end)) org-export-interblocks)) interblock(1 574) (progn (interblock start (match-beginning 0))) (unwind-protect (progn (interblock start (match-beginning 0))) (set-match-data save-match-data-internal (quote evaporate))) (let ((save-match-data-internal (match-data))) (unwind-protect (progn (interblock start (match-beginning 0))) (set-match-data save-match-data-internal (quote evaporate)))) (save-match-data (interblock start (match-beginning 0))) (while (re-search-forward “^\([ ]*\)#\+begin_\(\S-+\)[ ]*\(.*\)?[ \n]\([^

sbe failing when results are in buffer

  • State “TODO” from “” [2011-01-31 Mon 16:10]

sbe: Invalid function: (“Variant” “TP” “FP” “TN” “FN” “MOD” “TOT” “MODW” “TOTW”)

looks like some a level of quoting is missing.

#+call / sbe with :rownames / :colnames

  • State “TODO” from “” [2011-01-31 Mon 17:50]

Need to find a minimal exampe for this. Try using #+call to call a block which uses :rownames yes :colnames yes and also pulls in a table as input using :var tab=block-that-returns-a-table.

'((1 2 3) (4 5 6))
tab

(sbe “main-block”)

export not honoring org-src-lang-modes mappings

  • State “TODO” from “” [2011-01-21 Fri 17:16]

python prep-session failure

  • State “TODO” from “” [2011-01-14 Fri 22:10]

C-u C-c C-v z on a python block using a session results in

first time
ansi-color-filter-apply: Marker does not point anywhere
subsequent times
check-face: Not a face: nil

The latter error is thrown in org-babel-comint-wait-for-output, in which (face-at-point) somehow starts returning nil.

Naming session changes noweb expansion on export

  • State “TODO” from “” [2010-12-08 Wed 06:43]

Reported by Andreas Leha.

I think this might qualify as a bug. Using the example code below, if I don’t run R in a session, or if I run R in a session but don’t name the session, everything works as expected. The behavior changes for me when the session gets a name.

On Dec 7, 2010, at 9:46 AM, Andreas Leha wrote:

Hi Tom,

thanks for the answer and thanks for spotting the typo. But even with the typo corrected during export the noweb links are still exported.

print("mod1")
print("mod2")
<<mod1>>
<<mod2>>

Result: … \lstset{language=R} \begin{lstlisting} print(“mod1”) print(“mod2”) \end{lstlisting} …

python: unicode inserted into buffer incorrectly

  • State “TODO” from “” [2010-12-04 Sat 20:52]
[u'this is unicode'] * 2

Multiple #+babel lines don’t work

  • State “TODO” from “” [2010-11-17 Wed 21:30]

:exports both in #+babel does not work

  • State “TODO” from “” [2010-12-04 Sat 22:34]

exports results in header line doesn’t work

  • State “TODO” from “” [2011-01-20 Thu 13:15]

R tests are broken

  • State “TODO” from “” [2010-10-31 Sun 15:04]

Not sure when this was broken.

R: col names in Rtable-R-colnames-30expected “-3” but was “0”
R: col names in orgtable-R-colnames-org169#ERRORexpected “169” but was “#ERROR”

elisp can’t deal with single cons cell

  • State “TODO” from “” [2010-10-29 Fri 14:53]
'(a . b)

Debugger entered–Lisp error: (wrong-type-argument listp b)

I’m starting to doubt a fix I made recently:

The general issue we’ve struck in a couple of places is that something like

(length ‘(a . b))

is an error.

In `org-babel-sha1-hash’ I added this (when (and v (not (and (sequencep v) (not (consp v)) (= (length v) 0))))

but I may have been under the impression that consp was nil for a regular list, but that is of course not true:

(consp ‘(a b)) => t

In-buffer results not being used when sha1 present

  • State “DONE” from “TODO” [2010-10-21 Thu 07:53]
  • State “TODO” from “” [2010-10-20 Wed 23:04]

When executing “caller” below I expect it to returned the cached result; instead it re-evaluates “random” each time.

runif(1)

#+results[a2a72cd647ad44515fab62e144796432793d68e1]: random

0.4659510825295
x

#+results[bec9c8724e397d5df3b696502df3ed7892fc4f5f]: caller

0.254227238707244

Results appending rather than replacing

  • State “DONE” from “TODO” [2010-10-21 Thu 07:54]
  • State “TODO” from “” [2010-10-20 Wed 23:03]

Originally a :cache yes header was present, but I have now removed it. The results are being appended to the results block rather than replacing.

runif(1)

#+results[cbe5590c35d34b4cb69ad08c58f081e9cd30802f]: random2

0.190289322985336

What does the arg argument do in `org-babel-expand-src-block’?

  • State “PROPOSED” from “” [2010-10-20 Wed 18:36]

lob issues

  • State “PROPOSED” from “” [2010-10-20 Wed 10:27]

This lob line didn’t complete, which resulted in the active buffer switching to a buffer visiting psi.org, and the following message in the R buffer.

> Warning message: In file.create(transfer.file) : cannot create file ‘/tmp/babel-4519BQr/R-4519bsb’, reason ‘No such file or directory’

Should ob-execute-{buffer,subtree} execute lob lines?

  • State “PROPOSED” from “” [2010-10-18 Mon 14:07]

R variables import

  • State “STARTED” from “TODO” [2010-10-16 Sat 15:37]
  • State “TODO” from “” [2010-10-09 Sat 11:59]
table(x)

With the R-vectors patch applied, the variable seems to be imported OK using C-u C-c C-v z, but fails with straight C-c C-c. This is related to the fact that there is (I believe) confusing variable resolution code duplication between `org-babel-expand-body:R’ and `org-babel-prep-session:R’.

tab
row1ab
row212
row1ab
row212

Org block should remove protective commas

  • State “DONE” from “TODO” [2010-10-09 Sat 11:53]
  • State “TODO” from “” [2010-10-08 Fri 18:15]
,|a|

can’t evaluate org block on export

  • State “DONE” from “TODO” [2010-09-19 Sun 10:52]
  • State “TODO” from “” [2010-09-17 Fri 10:12]

This involves nested calls to the org exporter which breaks because the temporary buffer name org-mode-tmp is hard-coded in `org-export-preprocess-string’.

For now the solution here is to not set the :results header argument of the org code block, which will result in the code block safely evaluating without requiring an Org-mode export, then the content of the code block will export successfully with the rest of the org-mode file.

Non-elisp is getting into elisp tangle with org-babel-load-file

  • State “DONE” from “TODO” [2010-09-18 Sat 13:25]
  • State “TODO” from “” [2010-09-05 Sun 20:26]

org-babel-load-file on a file containing the following:

elisp

(setq dan 'here)

R

"This shouldn't get into the elisp tangle output"

different properties when exporting than when manually evaluating

  • State “DONE” from “TODO” [2010-09-18 Sat 20:01]
  • State “TODO” from “” [2010-08-25 Wed 12:04]

Thanks to Tom Short for pointing this out. Email from Tom Short: {Orgmode} Re: {babel} exports,

So for example see looks like this is a problem with info collection in the scraps.org buffer. It output different values on export and on manual execution.

Two different issues here,

  1. First, the order of the arguments to header arguments may vary, however `org-babel-sha1-hash’ now controls for this as of this commit.
    commit fbb828a1077d4d8879b00beab2923876d93416e7
    Author: Eric Schulte <schulte.eric@gmail.com>
    Date:   Sat Sep 18 14:02:33 2010 -0400
        

    :

        ob: hash construction invariant to order of header arguments
        
  2. Second, when exporting from a narrowed region, any header arguments specified outside of the region are not collected, because only the narrowed region is coppied to the temporary export buffer. This requires some complicated jumping back to the original file to collect header arguments during export, implemented as of this commit
    commit efdf78172d9f7c0070c781d136a9b49a2a56fcc4
    Author: Eric Schulte <schulte.eric@gmail.com>
    Date:   Sat Sep 18 19:01:49 2010 -0600
        

    :

        ob-exp: resolving code block parameters in the original file on export
        

    :

        * lisp/ob-exp.el (org-babel-exp-src-blocks): now switching back to the
          original file before resolving code block parameters to ensure
          headline and buffer wide parameters are taken into consideration
          when only a narrowed portion of the file is exported
        

shell sessions hang in many cases

  • State “TODO” from “” [2010-08-25 Wed 11:24]

for example

cd ~/src/org/
make

maybe rather than sending an eoe character in shell, we should just check for a prompt…

How do I get colnames with e.g. shell?

  • State “PROPOSED” from “” [2010-08-15 Sun 14:26]
return [[1,2],None, [3,4]]
12
34
echo 1, 2
echo 3, 4

eval and noeval

  • State “DONE” from “STARTED” [2010-08-26 Thu 17:05]
  • State “STARTED” from “PROPOSED” [2010-08-25 Wed 11:32]
  • State “PROPOSED” from “” [2010-08-15 Sun 14:06]

In general if `org-confirm-babel-evaluate’ is set to nil, then code blocks can be evaluated without the need for explicit confirmation and if it is set to a truthy value then explicit confirmation is required by the user for every code block evaluation.

The :eval header argument can never be used to decrease the level of confirmation required from the user, because that could result in a security risk, however it can create additional barriers to evaluation. For example the following should make you nervous if the :eval no header argument were not present.

#+begin_src sh :eval never
  sudo rm -rf /
#+end_src

The :eval header argument can take the following values.

no or never
the source block can not be evaluated, no matter what
query
the user will be queried before every evaluation regardless of the value of `org-confirm-babel-evaluate’

For backwards compatibility the :noeval header argument is an alias for :eval no.

The following demonstrates this header argument. On export the user is queried once, for the this-is-ls which is called by the final emacs-lisp code block.

*** eval and noeval
date, should export both, but won't output results because of presence
of the =:noeval= header argument.
#+begin_src sh :noeval :exports both
  date
#+end_src

:

should export code, so no need to do anything
#+begin_src sh
  date
#+end_src

:

should export nothing, and should not query
#+source: this-is-ls
#+begin_src sh :eval query :exports none
  date
#+end_src

:

should export results, and should trigger query above
#+begin_src emacs-lisp :var ls=this-is-ls :exports results
  ls
#+end_src

It exports to the following…

eval and noeval

date, should export both, but won’t output results because of presence of the :noeval header argument.

date

should export code, so no need to do anything

date

should export nothing, and should not query

should export results, and should trigger query above

shell remote dir bug

  • State “TODO” from “” [2010-08-15 Sun 12:58]

This seems to be a bug in tramp. If the target directory is specified with a relative path (as in ‘:bennch’ below) then while it is fine on the first evaluation, subsequent evaluations seem to “remember” the location of the default directory. So they are already in it when they try to cd into it again. Thus there is an error because ~/bench exists while ~/bench/bench does not.

ls

sbe doesn’t work during export

  • State “TODO” from “” [2010-08-11 Wed 16:59]

Alternatively, I believe we should be able to use sbe to put the elisp code in blocks:

plot sin(x)
(if (and (boundp 'latexp) latexp) "postscript" "png")
(if (and (boundp 'latexp) latexp) "sin.eps" "sin.png")

sbe functionality is not documented in manual

  • State “TODO” from “” [2010-08-08 Sun 18:34]

Appended newline character when using indexing

  • State “TODO” from “” [2010-08-08 Sun 16:25]
  my $total = 0;
  foreach my $row  (@$details) {
      $total += @$row[2];
  }

return $total;

(sbe “totals(details=expenses[1:-1])”)

thius sbe call returns “155…”

The ellipsis comes from `org-babel-table-truncate-at-newline’, which encounters a newline char that shouldn’t be there.

sbe adds extra ()s

  • State “TODO” from “” [2010-08-08 Sun 16:22]

(sbe “totals()”)

I think this patch should be applied so that () is not added in the above sbe call

Modified lisp/ob-table.el diff –git a/lisp/ob-table.el b/lisp/ob-table.el index 4a0454c..5e2770d 100644 — a/lisp/ob-table.el +++ b/lisp/ob-table.el @@ -104,7 +104,7 @@ example above.” (eval `(org-babel-parse-header-arguments (concat “:var results=” ,source-block

  • ”(”
  • (if ,variables (concat “(” (mapconcat (lambda (var-spec) (if (> (length (cdr var-spec)) 1)

@@ -113,7 +113,7 @@ example above.” (format “%S=%s” (car var-spec) (cadr var-spec)))) ‘,variables “, “)

  • ”)”)))))
  • ”)”))))))) (org-babel-execute-src-block nil (list “emacs-lisp” “results” (org-babel-merge-params ‘((:results . “silent”)) params))))

org-babel-read bug

  • State “TODO” from “” [2010-08-07 Sat 18:36]

The second evaluation results in “Symbol’s function definition is void: a”, during `org-babel-read’

'((a . b))

#+results[bb636ec8540a08d49cd3b04273d14984ba4ec31d]:

(a . b)

org-babel-prep-session:R sometimes fails to evaluate code

  • State “TODO” from “” [2010-08-07 Sat 18:01]

Sometimes the R-prep-session function fails to evaluate the code, because there is some text lying around at the prompt in the session buffer. Is there a way of guarenteeing that the session is ready to accept input?

src block regexp doesn’t match with empty body

  • State “TODO” from “” [2010-08-05 Thu 00:25]

Sometimes I like to create an empty body just to do C-c C-v z and start my session.

This seems to fix it but I haven’t tested and not confident about the +? *? stuff.

Modified lisp/ob.el diff –git a/lisp/ob.el b/lisp/ob.el index 6e74c15..eb06488 100644 — a/lisp/ob.el +++ b/lisp/ob.el @@ -113,7 +113,7 @@ remove code block execution from the C-c C-c keybinding.” ;; (4) header arguments “\([^\n]*\)\n” ;; (5) body

  • “\([^\000]+?\n\)[ \t]*#\+end_src”)
  • “\([^\000]*?\n\)[ \t]*#\+end_src”) “Regexp used to identify code blocks.”)

    (defvar org-babel-inline-src-block-regexp

remove octave prompt from session output

  • State “TODO” from “” [2010-08-01 Sun 11:53]

>> #+tblname: test >> | 1 | 2 | 3 | >> >> #+source: outtest >> #+begin_src octave :session out :var vec=test :results output >> vecb=vec; >> vecb >> #+end_src >> >> –8<—————cut here—————end—————>8— >> >> You will get following output >> >> #+results: outtest >> : vec = >> : >> : 1 2 3 >> : octave.exe> vecb = >> : >> : 1 2 3 >> >> As you see I get two outputs : vec variable and vecb (with octave prompt). >> It is what I expected, excepted vec output.

sha1 hash visibility with org-mode-restart

  • State “TODO” from “” [2010-07-29 Thu 21:28]

In the example below, first execute the block to get a hidden sha1 hash. Then if you use C-c C-c on the #+ line, the hash becomes fully visible.

1

#+results[6a71fb0c2a1a1d0e79f72d9391908cf396bef3a0]:

1

#+whatever

Basically what happens on C-c C-c is that the outline overlays are recorded, org-mode-restart gets called, then outline visibility is restored. The problem is that the hash overlay is not dealt with by this. A simple workaround is

  • (overlay-put ov2 ‘invisible ‘org-babel-hide-hash)
  • (overlay-put ov2 ‘invisible ‘outline)

But this is not nice from a design point of view. One solution might be to alter `org-save-outline-visibility’ such that it saves a broader class of overlays?

There’s a (I think a separate?) problem that with the hash fully displayed, folding the headline leaves the hash displayed in a nasty way.

conditionally remove existing results of source-code block on export

  • State “DONE” from “TODO” [2010-06-07 Mon 13:17]
  • State “TODO” from “” [2010-06-07 Mon 13:07]

so for example the following

** stripping existing results
#+results: trickily-located-somehwere-else
,: I shouldn't be exported

,Neither of the result strings for the following two code blocks should
,be included in the export.  And only one of the bodies should be
,included...

#+begin_src emacs-lisp :exports code
,  (+ 1 1 1 1)
#+end_src

#+results:
,: don't include me in the export!!!!!!!

#+srcname: trickily-located-somehwere-else
#+begin_src emacs-lisp :exports none
,  (message "I shouldn't be exported")
#+end_src

Should export only the bodies of the code blocks, but should remove the existing results.

This is now the case as the above now exports to the following (showing only the germane HTML)

<p>
Neither of the result strings for the following two code blocks should
be included in the export.  And only one of the bodies should be
included&hellip;
</p>



<pre class="src src-emacs-lisp"><span class="org-esk-paren">(</span>+ 1 1 1 1<span class="org-esk-paren">)</span>
</pre>

org-babel-tangle mis-handles empty code blocks

  • State “DONE” from “STARTED” [2010-05-26 Wed 07:06]
  • State “STARTED” from “TODO” [2010-05-25 Tue 20:25]
  • State “TODO” from “” [2010-04-25 Sun 22:13]
This source:
*** Cook Observations on Hawaiian Homes
#+srcname: homestead-cook
#+begin_src latex :tangle yes

:

#+end_src

now yields the following LaTeX

\begin{verbatim}
nil
\end{verbatim}

which works

however this source (notice latex vs. LaTeX)

*** Cook Observations on Hawaiian Homes
#+srcname: homestead-cook
#+begin_src LaTeX :tangle yes

:

#+end_src

is noticed by org-babel, and is eliminated by the org-babel pre-processing because we are exporting to LaTeX and the body is empty.

This may be a good argument for adding language aliases to org-babel, so that both of the above blocks are recognized and treated equally.

org-babel-tangle should check if in org-mode

  • State “DEFERRED” from “TODO” [2010-05-26 Wed 06:56]
  • State “TODO” from “” [2010-04-17 Sat 19:12]

Why? I just successfully tangled from within my scratch buffer without any problems. If no org-mode functions are explicitly required, then why not let people tangle from wherever they like?

Should strip protective commas from org blocks on tangling

  • State “DONE” from “TODO” [2010-06-07 Mon 11:43]
  • State “TODO” from “” [2010-04-15 Thu 13:14]

This has been fixed with a simple call to `replace-regexp-in-string’, one may thing that there would be a complex function for removing the commas, but org-mode uses a simple regexp as well – so apparently this approach is safe.

unwind-protect needed in functions that use narrowing

  • State “DONE” from “TODO” [2010-06-07 Mon 11:52]
  • State “TODO” from “” [2010-04-12 Mon 21:58]

e.g. when calling `org-babel-execute-subtree’ If there’s an error you end up with a narrowed buffer.

We’re eating up org code block flags

  • State “DEFERRED” from “TODO” [2010-04-17 Sat 13:42]
  • State “TODO” from “” [2010-04-11 Sun 09:59]

    We document that org switches should come before the header args, so this may not be a bug, assuming that we’re happy that tangle can create files with spaces in it.

    This creates a file called “src/msg.pl +n”

\#+begin_src perl :tangle src/msg.pl +n

Language mappings in org-src-lang-modes should be honoured by babel

  • State “DEFERRED” from “DONE” [2010-06-07 Mon 11:53]
  • State “TODO” from “” [2010-04-09 Fri 09:39]

this has been deferred because it should be addressed in another task

Email from me to Andrea Crotti:

first use of new session in ruby errors out

  • State “DONE” from “TODO” [2010-06-07 Mon 12:08]
  • State “TODO” from “” [2010-06-07 Mon 11:58]

a problem with the following error

ansi-color-process-output: Marker does not point anywhere

prep session may fail if comint prompt is not empty

  • State “DONE” from “TODO” [2010-06-07 Mon 12:20]
  • State “TODO” from “” [2010-04-08 Thu 15:22]

    I think the current code assumes that we can just go to the comint buffer and insert stuff and do comint-send-input. However, the user might have left some characters lying around after the comint prompt.

noweb R error?

  • State “DONE” from “PROPOSED” [2010-04-08 Thu 22:47]
  • State “PROPOSED” from “” [2010-04-03 Sat 20:50]

    This is an oddity of ggplot2 in R. It behaves differently at the top level versus when wrapped in a function body.

Execute in dir with dir “temp” present

12
library(ggplot2)
g <- ggplot(x, aes(x = weight))
g + geom_histogram(aes(y=..density..)) + geom_density(weight=2) + scale_x_log10()
<<plotting-code>>

Error: unexpected input in: “x <- read.table(“/tmp/org-babel-R-import10613QvG”, header=TRUE, sep=”\t”, as.is=TRUE) <<” Execution halted

Tangling sometimes needs newline before comments

  • State “DONE” from “PROPOSED” [2010-06-07 Mon 12:26]
  • State “PROPOSED” from “” [2010-04-03 Sat 11:53]

    If code block starts with comment it can get appended to end of previous block without intervening newline. (This happened with uncommentable t).

    this should already be fixed by recent changes to org-babel-tangle.el

Do we really want inheritance of params?

  • State “DONE” from “PROPOSED” [2010-03-27 Sat 18:51]
  • State “PROPOSED” from “” [2010-03-27 Sat 16:56]

    Got rid if it.

    a23ca4529f5b5467a4f7d520db24c9818c5eb148

The following example suggest that inheritance of params by the referenced block may be undesirable. What do we actually need it for?

4
print x

Missing values in R tables

  • State “TODO” from “” [2010-03-25 Thu 15:58]

This is undesirable:

t
x
x

point movement during #+call

  • State “REJECTED” from “DEFERRED” [2010-06-07 Mon 12:28]
  • State “TODO” from “” [2010-03-25 Thu 14:47]

[Eric] I was unable to reproduce this bug. When I evaluated the #+call line with C-c C-c the point moved up to the source-code block temporarily during execution, but then moved safely back to the call line.

Please let me know if I’ve mis-understood the problem.

This caused movement of point/buffer scrolling during execution of the #+call

x <- array("", dim=c(8,12),dimnames=list(LETTERS[1:8], 1:12))
cbind(rownames(x), x)
4
123456789101112
A
B
C
D
E
F
G
H

source block regexp matches delimiters inside source block

  • State “DONE” from “DONE” [2010-03-23 Tue 11:30]

Thanks to Maurizio for pointing out this error.

#+begin_src emacs-lisp
;;; evaluating the following C-x C-e with the point at the end is ok
(message "%d" (org-babel-where-is-src-block-head))
(defun foo () (insert "#+begin_src emacs-lisp"))
;;; evaluating the following C-x C-e with the point at the end is not ok
(message "%d" (org-babel-where-is-src-block-head))
#+end_src

:

#+results:
: 182800

:

#+begin_src emacs-lisp
;;; evaluating the following C-x C-e with the point at the end is ok
(org-babel-where-is-src-block-head)
(defun foo () (insert "#+end_src emacs-lisp"))
;;; evaluating the following C-x C-e with the point at the end is not ok
(org-babel-where-is-src-block-head)
#+end_src

:

#+results:
: 183148

:

#+begin_src ruby
  "#+end_src"
#+end_src

:

#+results:
: #+end_src

Why are referenced source blocks evaluated twice?

  • State “DONE” from “PROPOSED” [2010-03-27 Sat 17:00]
  • State “PROPOSED” from “” [2010-03-20 Sat 12:48]

    Fixed at 47571965d4b4977d72781f8a80ac29c6a609e731

Example

return 'from python'
x

With this debugging patch

diff --git a/contrib/babel/lisp/org-babel.el b/contrib/babel/lisp/org-babel.el
index d01e6d6..40331a0 100644
--- a/contrib/babel/lisp/org-babel.el
+++ b/contrib/babel/lisp/org-babel.el
@@ -198,6 +198,7 @@ block."
   (interactive)
   ;; (message "supplied params=%S" params) ;; debugging
   (let* ((info (or info (org-babel-get-src-block-info)))
+        (remove-me (message (concat "\n" (pp info) "\n")))
          (lang (first info))
         (params (setf (third info)
                        (sort (org-babel-merge-params (third info) params)

We get this output

("emacs-lisp" "x\n"
 ((:comments . "")
  (:shebang . "")
  (:cache . "no")
  (:noweb . "no")
  (:tangle . "")
  (:exports . "code")
  (:results . "replace")
  (:var . "x=python-block")
  (:session . "none"))
 "")


("emacs-lisp" "x\n"
 ((:comments . "")
  (:shebang . "")
  (:cache . "no")
  (:noweb . "no")
  (:tangle . "")
  (:exports . "code")
  (:results . "replace")
  (:var . "x=python-block")
  (:session . "none"))
 "")
 [2 times]
executing emacs-lisp code block...
("python" "return 'from python'\n"
 ((:comments . "")
  (:shebang . "")
  (:cache . "no")
  (:noweb . "no")
  (:tangle . "")
  (:exports . "code")
  (:results . "replace")
  (:session . "none"))
 "" "python-block")


("python" "return 'from python'\n"
 ((:comments . "")
  (:shebang . "")
  (:cache . "no")
  (:noweb . "no")
  (:tangle . "")
  (:exports . "code")
  (:results . "replace")
  (:session . "none"))
 "" "python-block")
 [2 times]
executing Python source code block
"from python"
("python" "return 'from python'\n"
 ((:comments . "")
  (:shebang . "")
  (:cache . "no")
  (:noweb . "no")
  (:tangle . "")
  (:exports . "code")
  (:results . "replace")
  (:session . "none"))
 "" "python-block")


("python" "return 'from python'\n"
 ((:comments . "")
  (:shebang . "")
  (:cache . "no")
  (:noweb . "no")
  (:tangle . "")
  (:exports . "code")
  (:results . "replace")
  (:session . "none"))
 "" "python-block")
 [2 times]
executing Python source code block
"from python"
finished

Is this o-b-comint code mac-specific?

  • State “DONE” from “PROPOSED” [2010-06-07 Mon 12:57]
  • State “PROPOSED” from “” [2010-03-17 Wed 23:59]
(replace-regexp-in-string "\n" "\r\n" (regexp-quote ,full-body)) string-buffer))

if it is either Mac or langauge specific, then it was probably required at some point. I’ll replace “\n\r” with “[\n\r]+” so that it won’t break any instance where the comint process returns “\n” instead of “\n\r”.

Newlines etc in table cells mess up table formatting

  • State “STARTED” from “TODO” [2010-06-18 Fri 12:35]
  • State “TODO” from “” [2010-03-16 Tue 23:36]

Put together an org-table.el patch and submitted to the list, so we’ll see. For some reason the patch still doesn’t fix the aberrant R behavior.

Perhaps we should remove or protect characters like {\n,\r,\f,\v} when they occur in table cells.

Is this something that should be implemented in `orgtbl-to-orgtbl’? Maybe this is worth a post to the mailing list…

(orgtbl-to-orgtbl '(("first" "line") ("second" "line
with
newlines")) '())
  '((1 2 3 4) ("eric
michael
schulte"))
  [["one", "two", "three
four"]]
data.frame(numbers=1:10, strings=c(letters[1:9], "an entry with a
newline character"))
numbers strings
1 a
2 b
3 c
4 d
5 e
6 f
7 g
8 h
9 i
10 an entry with a
newline character

src block regexp fails if block contains block delimiters

  • State “DONE” from “TODO” [2010-06-07 Mon 16:23]
  • State “TODO” from “” [2010-03-11 Thu 11:32]

this appears to be fixed now…

(concat "#+begin_src\n"
        "hello\n"
        "#+end_src\n")

used to be that C-c C-c → End of file during parsing

report by Maurizio Vitale

From: Maurizio Vitale <mav@cuma.i-did-not-set--mail-host-address--so-tickle-me>
Subject: [Orgmode] Re: [babel] problems with org-babel-src-block-regexp (full text)
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1.50 (gnu/linux)
Date: Thu, 11 Mar 2010 12:19:09 -0500
To: emacs-orgmode@gnu.org

: :

[sorry for the double posting, but the control char in the regexp
prevented the full message from appearing. This is what I meant to send]

:

There're problems with org-babel-src-block-regexp when the src block
contains the block delimiters (for instance in strings or comments).

:

Granted, this is an uncommon occurrence (which I discovered by accident)
editing an org-file containing elisp code that needed to insert
#+begin_src...#+end_src pairs, but if it is not too difficult to fix it
would be nice to have a more useful behaviour.

:

This is my setting (replaced 0x0 with ^@ to avoid posting problems):
org-babel-src-block-regexp is a variable defined in `org-babel.el'.
Its value is
"^[ 	]*#\\+begin_src[ 	]+\\(python\\|sh\\|emacs-lisp\\)[ 	]*\\([^\":\n]*\"[^\"\n*]*\"[^\":\n]*\\|[^\":\n]*\\)\\([^\n]*\\)\n\\([^^@]+?\\)#\\+end_src"

:

And these are two test cases. I'm particularly puzzled by the first,
because the regexp seems to be anchored to the begin of line and hence
it shouldn't match the begin_src inside the string.
For end_src, it should probably be anchored to the begin of line.

:

#+begin_src emacs-lisp
;;; evaluating the following C-x C-e with the point at the end is ok
(org-babel-where-is-src-block-head)
(defun foo () (insert "#+begin_src emacs-lisp"))
;;; evaluating the following C-x C-e with the point at the end is not ok
(org-babel-where-is-src-block-head)
#+end_src

:

#+begin_src emacs-lisp
;;; evaluating the following C-x C-e with the point at the end is ok
(org-babel-where-is-src-block-head)
(defun foo () (insert "#+end_src emacs-lisp"))
;;; evaluating the following C-x C-e with the point at the end is not ok
(org-babel-where-is-src-block-head)
#+end_src

:

Best regards,

:

     Maurizio

results deletion and insertion

  • State “DONE” from “PROPOSED” [2010-06-07 Mon 16:46]
  • State “PROPOSED” from “” [2010-03-09 Tue 09:33]

Graham’s problem was that on C-c C-c, this

#+ATTR_LaTeX: width=10cm
#+results: BoxplotFlowering
,[[file:FloweringBoxplot.pdf]]

turns into this:

#+results: BoxplotFlowering
,[[file:FloweringBoxplot.pdf]]
#+ATTR_LaTeX: width=10cm

Two questions:

  1. Is it desirable that we remove the corresponding results block anywhere in the buffer, but then re-insert it beneath the code block? Or should we promise to re-insert in situ, in the case that we find a pre-existing results block?
  2. How should a user use a #+ATTR_LaTeX line?

Solved

Yes, we should certainly be updating the result “in situ”.

alright, this has now been implemented, in addition I’ve added append and prepend results arguments to for adding new results either before or after existing results. The following demonstrates this new behavior

#+results: in-situ
,: update me in place please -- Mon Jun  7 16:44:44 2010
,: update me in place please -- Mon Jun  7 16:44:43 2010
,: update me in place please -- Mon Jun  7 16:44:42 2010
,: update me in place please -- Mon Jun  7 16:44:37 2010
,: update me in place please -- Mon Jun  7 16:42:14 2010
,: update me in place please (at the bottom) -- Mon Jun  7 16:44:59 2010
,: update me in place please (at the bottom) -- Mon Jun  7 16:45:00 2010
,: update me in place please (at the bottom) -- Mon Jun  7 16:45:02 2010

,the results should be *above* the block

#+srcname: in-situ
#+begin_src emacs-lisp :results prepend
,  (format "update me in place please -- %s"
,          (current-time-string))
#+end_src

#+srcname: in-situ
#+begin_src emacs-lisp :results append
,  (format "update me in place please (at the bottom) -- %s"
,          (current-time-string))
#+end_src

Graham Smith email

From: Dan Davison <davison@stats.ox.ac.uk>
Subject: Re: [Orgmode] Re: [babel] captions and figure size on export
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux)
Date: Tue, 09 Mar 2010 09:32:19 -0500
To: Graham Smith <myotisone@gmail.com>
Cc: emacs-orgmode@gnu.org

Graham Smith <myotisone@gmail.com> writes:

> This is driving me mad here:  I have removed all the fig and size code
> and replaced it with onky one line that should be resizing the first
> graph(FloweringBoxplot.pdf).  However, its resizing the second graph
> (NonFloweringBoxplot/pdf) and not affecting the first graph. This
> works on other files, so I am really at a loss here.

Hi Graham,

We can see what's going on here by executing the block manually with C-c
C-c, which transforms your results block from this:

#+ATTR_LaTeX: width=10cm
#+results: BoxplotFlowering
[[file:FloweringBoxplot.pdf]]

into this:

#+results: BoxplotFlowering
[[file:FloweringBoxplot.pdf]]
#+ATTR_LaTeX: width=10cm

(The reason is that the results block corresponding to the code block is
first deleted and then re-inserted immediately below the code block).

You are using ':exports both' which means that the above transformation
actually occurs during export, but in a special pre-export org buffer
which the user never sees. The fact that the ATTR_LaTeX line ends up
last explains why it modifies the *second* block.

Thanks for pointing this out -- we will revisit this behaviour as it
does seem surprising and on the face of it undesirable.

As a workaround, is it necessary for you to execute the block on export?
Or could you instead use :exports code? I.e. before export, execute the
block manually and ensure that the #+ATTR_LaTeX and #+results lines are
arranged in the correct way.

Dan

Revisit field delimiter behaviour

  • State “PROPOSED” from “” [2010-02-21 Sun 01:04]

    The table below might be considered not to be the desired table, in that it should perhaps have had just one column.

c(date(), system("hostname", intern=TRUE), getwd())

shell-command-on-region calls

  • State “DONE” from “TODO” [2010-03-07 Sun 17:22]
  • State “TODO” from “PROPOSED” [2010-02-25 Thu 16:14]
    • State “PROPOSED” from “” [2010-02-20 Sat 22:24]

      Org-babel now uses its own function `org-babel-shell-command-on-region’ in which the various shell-command-on-region bugs can be fixed.

original notes

Can we double check this at some point:

(shell-command-on-region (point-min) (point-max) “R –slave –no-save” ‘replace)

Note that replace is 5th argument, not 4th:

(shell-command-on-region start end command &optional output-buffer replace error-buffer display-error-buffer)

Yes, people agree that this is a bug in emacs. A bug report has been filed. So those calls should be something like

(shell-command-on-region (point-min) (point-max) “R –slave –no-save” ‘current-buffer ‘replace)

session bugs

  • State “TODO” from “” [2010-02-28 Sun 14:54]

shell different results for :session and external process

  • State “PROPOSED” from “TODO” [2010-02-25 Thu 16:58]
  • State “TODO” from “” [2010-02-19 Fri 13:27]
echo 1
echo 2

But without :session this gives

echo 1
echo 2

python ‘session output’ evaluation is not outputting last line

  • State “TODO” from “PROPOSED” [2010-02-25 Thu 18:13]

This is a bit weird. The first evaluation in a session is OK, but second and subsequent evaluations fail to output the last line.

print 'hello'
print 'bye'

But evaluate it again and the second line will not be output.

(R) session evaluation fails if no new line terminating output

cat("There's no newline ending this")

python :session :results value not working

  • State “DONE” from “TODO” [2010-02-25 Thu 16:59]
  • State “TODO” from “” [2010-02-18 Thu 17:49]

    22fc3b2d333e0d440cdc972a6478a834180b188d and f19692578a958d42fb03048e574949231a83c2f7

import os, time
[time.ctime(), os.uname(),os.getcwd()]

fibonacci test failing

quotes getting chomped?

  • State “DONE” from “PROPOSED” [2010-02-23 Tue 22:25]
  • State “PROPOSED” from “” [2010-02-16 Tue 16:07]

commit

commit 22fc3b2d333e0d440cdc972a6478a834180b188d Author: Dan Davison <davison@stats.ox.ac.uk> Date: Fri Feb 19 11:34:16 2010 -0500

babel: Fix unquoting of strings

Before this change we have

["1", "3"]

After, we have

["1", "3"]

bug

Look at second line of output below. Looks like initial and terminal single quotes have been chomped.

python

code
import os, time
print time.ctime()
print os.uname()
print os.getcwd()
print '\n' ## We have a bug involving losing final lines
local
<<dir-test-python>>

so for example

#+results: haskell-hiding

:

#+begin_src haskell :var num=haskell-hiding
  num + 1
#+end_src

results in a

reference 'haskell-hiding' not found in this buffer

error

table slicing doesn’t work with function style references

  • State “DONE” from “TODO” [2010-02-14 Sun 11:49]
  • State “TODO” from “” [2010-02-12 Fri 13:45]
echo $rs

reference ‘karafet-hgR[1’ not found in this buffer

this is now fixed in the following commit, in which ensure that `org-babel-ref-split-args’ will maintain balanced square brackets.

commit 692c569215453f4283d3ecfa24d3d6e1f2715c02
Author: Eric Schulte
Date:   Sun Feb 14 11:48:28 2010 -0700

    babel: smarter `org-babel-ref-split-args' -- fixes bug parsing indexed function-style args

	Modified contrib/babel/lisp/org-babel-ref.el
diff --git a/contrib/babel/lisp/org-babel-ref.el b/contrib/babel/lisp/org-babel-ref.el
index dc51020..c0e6a55 100644
--- a/contrib/babel/lisp/org-babel-ref.el
+++ b/contrib/babel/lisp/org-babel-ref.el
@@ -193,8 +193,8 @@ which case the entire range is returned."
         (when (= depth 0)
           (setq return (reverse (cons (substring buffer 0 -1) return)))
           (setq buffer "")))
-       ((string= holder "(") (setq depth (+ depth 1)))
-       ((string= holder ")") (setq depth (- depth 1)))))
+       ((or (string= holder "(") (string= holder "[")) (setq depth (+ depth 1)))
+       ((or (string= holder ")") (string= holder "]")) (setq depth (- depth 1)))))
     (mapcar #'org-babel-trim (reverse (cons buffer return)))))

 (defun org-babel-ref-at-ref-p ()

on export babel overwrites unrecognized source blocks

  • State “DONE” from “TODO” [2010-01-18 Mon 13:37]
  • State “TODO” from “” [2010-01-13 Wed 09:14]

Thanks to Sébastien for this bug report.

You see that when the language is unknown to Org-babel (Delphi in this example), it gets ignored during export, and gets replaced by the previous code block!!

So, the current workaround is to take whatever other language know to Babel (here: perl) for the export to (at least) output the right code in the generated document (even if not correctly highlighted).

demonstrated in the attached babel-overwrite.org

Probably has something to do with the conflicting regular expressions between org-babel-exp.el and org-exp-blocks.el…

This has been fixed in the following commit

which introduces a local variable to org-ex-blocks which is used by org-babel-exp to ensure that previous blocks do not overwrite the current block.

R language shadows ruby

  • State “DONE” from “TODO” [2010-03-26 Fri 20:54]
  • State “TODO” from “TODO” [2010-01-18 Mon 13:39]
  • State “TODO” from “” [2009-12-31 Thu 12:57]

simple fix, we just sort the language names by length, that way the longer names will be tried first and the shorter names won’t have a chance to overshadow them.

(setq org-babel-interpreters
      (sort (cons interpreter org-babel-interpreters)
            (lambda (left right)
              (> (length left) (length right)))))

DED: I think this requires tightening up of org-babel-src-block-regexp, so that the language identifiers have appropriate delimiters around them.

Thanks to Daniel Doherty for submitting the following report

I’m not on this mailing list, but thought this would be the right place to pass along a bug in org-babek that had me baffled for a long time.

I was initializing org-babel with the following from my init.el file

(require 'org-babel-init)
(require 'org-babel-ruby)
(require 'org-babel-R)
(require 'org-babel-python)
(org-babel-load-library-of-babel)

When executing a ruby src block, I would get an error saying that the “r” language was not a babel-interpreter.

When I change the order of the language initializations, the error goes away:

(require 'org-babel-init)
(require 'org-babel-R)
(require 'org-babel-ruby)
(require 'org-babel-python)
(org-babel-load-library-of-babel)

Now, ruby is recognized.  Looks like the function for parsing the src block is seeing the ‘r’ in ruby and stopping because ‘R” is a valid language, but then choking when it tries to run ‘r’.

subtree execution can’t find blocks in other sub-trees

  • State “DONE” from “TODO” [2010-06-07 Mon 17:39]
  • State “TODO” from “” [2009-12-30 Wed 17:05]

This is most likely due to the fact that `org-babel-execute-subtree’ calls `org-narrow-to-subtree’, which narrows the buffer making functions defined in other sub-trees invisible. This is demonstrated by calling `org-babel-execute-subtree’ (bound to C-c M-b s) inside of “tree 2” below.

Thanks to Tom Dye for pointing this out.

example

tree 1

a <- 1

tree 2

<<foo>>
a + 1
a + 2

header argument values are not being collected from properties

  • State “DONE” from “TODO” [2009-12-28 Mon 18:11]
  • State “TODO” from “” [2009-12-28 Mon 16:07]

Many types of header arguments are not having their values properly read from outline-level properties.

There were two problems here

  1. we weren’t searching for all needed header arguments in properties
  2. we weren’t inheriting header arguments from enclosing properties

both of these issues were resolved in changes to org-babel-params-from-properties (see below).

  1. we are now tracking the following properties
    '("cache" "cmdline" "exports" "file" "noweb" "results"
                   "session" "tangle" "var")
        
  2. we now search the properties of enclosing headlines by adding t as the last argument to
    (org-entry-get (point) header-arg t)
        

code for `org-babel-params-from-properties’

(defun org-babel-params-from-properties ()
  "Return an association list of any source block params which
may be specified in the properties of the current outline entry."
  (save-match-data
    (delq nil
          (mapcar
           (lambda (header-arg)
             (let ((val (or (condition-case nil
                                (org-entry-get (point) header-arg t)
                              (error nil))
                            (cdr (assoc header-arg org-file-properties)))))
               (when val
                 ;; (message "prop %s=%s" header-arg val) ;; debugging
                 (cons (intern (concat ":" header-arg)) val))))
           '("cache" "cmdline" "exports" "file" "noweb" "results"
             "session" "tangle" "var")))))

Error in R if block is merely defining a function

First idea which occurs to me is we could check whether the object inherits from a blacklist of classes before attempting to write.table.

Or wrap the write.table in a try statement, defaulting to something that will work for all objects, e.g. str().

f <- function(x) x

Error in as.data.frame.default(xi, optional = TRUE) : cannot coerce class “function” into a data.frame Calls: write.table … data.frame -> as.data.frame -> as.data.frame.default Execution halted

Insert results fails if eob or ensuing lines non-empty

This (end of buffer after block, with or without newline after “end_src”) fails with “End of buffer”

echo output

This

echo output

text here

results in

t#+results:

output

ext here

My working suggestion is this (branch results-insert-bug)

commit 0ab4b84375060e750764347b3335f110b2bfee20
Author: Dan Davison <davison@stats.ox.ac.uk>
Date:   Fri Nov 27 14:03:25 2009 -0500

    org-babel: Fix insertion of new results

    Insertion of new results was failing if the block was followed by
    end-of-buffer. Also, if the block was followed by non-empty lines, the
    \#+resname was being inserted away from column 0.

diff --git a/contrib/babel/lisp/org-babel.el b/contrib/babel/lisp/org-babel.el
index 0b68be3..9e7e038 100644
--- a/contrib/babel/lisp/org-babel.el
+++ b/contrib/babel/lisp/org-babel.el
@@ -639,7 +639,8 @@ following the source block."
                                  (looking-at (concat org-babel-result-regexp "\n"))))
                          ;; or (with optional insert) back up and make one ourselves
                           (when insert
-                            (goto-char end) (forward-char 1)
+                            (goto-char end)
+                           (insert "\n")
                             (insert (concat "#+results" (if hash (concat "["hash"]"))
                                             ":"(if name (concat " " name)) "\n"))
                             (move-beginning-of-line 0)

This certainly fixes the issue above but has the effect of inserting additional newlines with each repeat evaluation eliminating idempotency of source block evaluation. For example with the above patch applied evaluate the following 10 times

(+ 1 1)

I just committed a slight variation on the above patch which should fix that problem.

:results code not dealt with correctly?

E.g. here in org-babel-python-evaluate. Should that be (or (member “pp” result-params) (member “code” result-params)) ?

(format
 (if (member "pp" result-params)
     org-babel-python-pp-wrapper-method
   org-babel-python-wrapper-method)
 (let ((lines (split-string
               ;; ...
               )))))

before jumping to the solution, what’s the problem here? Are we not printing out results correctly when the code option has been selected?

lob does not inherit :session from file properties

  • State “DEFERRED” from “TODO” [2009-12-28 Mon 16:07]

This is now being tracked in the more general bug regarding header arguments from properties.

org-babel-insert-result uninvestigated worry

What happens if org-babel-where-is-src-block-result doesn’t return non-nil? these lines in org-babel.el – this link doesn’t resolve for me…

tangle needs to check that buffer has file name

  • State “DONE” from “TODO” [2010-06-07 Mon 17:42]

Tangle extension names

  • State “DONE” from “PROPOSED” [2010-02-03 Wed 13:17]
  • State “PROPOSED” from “” [2010-01-13 Wed 12:50]

I found this patch hanging around in my repo and seem to remember thinking this was a good change but can’t remember now the details.

   diff --git a/contrib/babel/lisp/org-babel-tangle.el b/contrib/babel/lisp/org-babel-tangle.el
index c805373..1d7fe3b 100644
--- a/contrib/babel/lisp/org-babel-tangle.el
+++ b/contrib/babel/lisp/org-babel-tangle.el
@@ -102,9 +102,10 @@ exported source code blocks by language."
                                      ((> (length tangle) 0) tangle))
                                     target-file))
                      (file-name (when base-name
-                                  (if (string= base-name
-                                               (file-name-sans-extension base-name))
-                                      (concat base-name "." ext) base-name))))
+                                  (if (and ext
+                                          (string= base-name
+                                                   (file-name-sans-extension base-name)))
+                                     (concat base-name "." ext) base-name))))
                 ;; ;; debugging
                 ;; (message "tangle=%S base-name=%S file-name=%S"
                 ;;          tangle base-name file-name)

This has been applied – it was a good idea – as it’s now possible to export a Makefile without an extension.

python :results value fails with result ending in newline

  • State “DONE” from “TODO” [2010-06-07 Mon 17:48]

which is superficially surprising, but anyway, I think the regexp in org-babel-number-p needs a tweak. Not quite sure exactly how though.

return "x\n"

ah, improper assumption that results would be single line, nice catch.

should be fixed now…

(if (org-babel-number-p "x\n")
  "yes" "no")

(org-babel-merge-params nil) doesn’t return nil

  • State “REJECTED” from “PROPOSED” [2010-06-07 Mon 18:05]
(org-babel-merge-params nil)

Is that desirable?

Yes, I think this is fine, because all of the above are only triggered when “yes” (cdr (assoc :keyword params)) instances are found.

Allow #+srcname to be indented

  • State “DONE” from “PROPOSED” [2010-06-07 Mon 18:11]

done, e.g.

   #+srcname: i-am-indented
   #+begin_src emacs-lisp
,     (message "i am indented")
   #+end_src

#+results: i-am-indented
,: i am indented

#+begin_src emacs-lisp :var output=i-am-indented
,  (length output)
#+end_src

#+results:
,: 13

Juan Reyero bug report

  • State “REJECTED” from “TODO” [2010-02-25 Thu 18:19]

> … However I can’t replicate this > behaviour under linux. I get > > #+resname: > : 2 > > for all three examples. > > I’m using org-version 6.31trans in emacs-version 23.0.91.1 under ubuntu > jaunty with python 2.6.2. Is this definitely replicable under OSX?

Yes, definitely. I am using emacs version 22.3.1, and python 2.6.1. I have stripped bare my .emacs, and still:

2
0

jm

should delete temp files after evaluation

  • State “STARTED” from “TODO” [2010-08-29 Sun 18:49]

tangle is outputting multiple shebang lines with python code

o-b-execute-subtree places output outside of folded subtree

  • State “DONE” from “TODO” [2010-06-07 Mon 18:39]

This appears to have been fixed by some of the recent changes

Example

Try M-x org-babel-execute-subtree with the subtree folded and point at the beginning of the heading line. The 7 output doesn’t appear and the 6 output ends up outside the folded heading.

echo 7

point should go to end of comint buffer after execution

  • State “STARTED” from “TODO” [2010-03-01 Mon 10:07]

    NB the variable comint-scroll-to-bottom-on-output

    At the moment it seems that point goes to the end when the comint window is visible, but stays where it is when it is not.

shell

echo hello

python

Same as for R.

'goodbye'

example on list This is because python statements are broken up by newlines in org-babel-python-evaluate.

I think this is a somewhat uncomfortable issue. Both perl and python demand a return statement.

Edit: Not true for perl

That currently means that we have to separate the code into “statements” (not sure that’s the right term), in order to stick a “return” in front of the last one. But in general, I think you would have to parse the language in order to know how to break the language up in this way. It’s easy to come up with examples of valid python code, involving mutliline statements and/or semicolons that don’t work in current org-babel. E.g.

def f(x,y):
    return x + y
a = f(4,
      5)
return a ## Added return statement to demonstrate solution

Yea, I don’t know how to do this either. I searched extensively on how to isolate the last output of a series of shell commands (see <a href=”* last command for shells”>last command for shells). The results of the search were basically that it was not possible (or at least not accomplish-able with a reasonable amount of effort).

That fact combined with the tendancy to always use standard out in shell scripts led me to treat these two options (output and value) as identical in shell evaluation. Not ideal but maybe good enough for the moment.

In the `results’ branch I’ve changed this so that they’re not quite identical: output results in raw stdout contents, whereas value converts it to elisp, perhaps to a table if it looks tabular. This is the same for the other languages. [Dan]

Maintain correct indentation in tangled (e.g.) python code

This is solved by setting new variable org-src-preserve-indentation to t, either globally or buffer locally.

Code

Class X

class X(object):
Init method
def __init__(self):
    self.data = 11
Do stuff method
def do_stuff(self):
    print('Doing stuff')

Tangle output

This has invalid indentation #!/usr/bin/env python

class X(object):

#!/usr/bin/env python

def __init__(self): self.data = 11

#!/usr/bin/env python

def do_stuff(self): print(‘Doing stuff’)

prompt characters appearing in output with R

See commit 6947d0ecd2fbcc9e7d1330143ae0a51f2fe347bb which makes a start on solving this.

commit note

org-babel: prevent comint prompt characters from appearing in R output.

This is a hack, but is does fix the problem (on my system at least), and it does demonstrate where the problem lies. It seems that the ‘raw’ output often contains lines consisting of multiple concatenated elements, each element being of the form ‘x ’ where x is either ‘>’ or ‘+’. ‘+’ is inferior-ess-secondary-prompt, and ‘>’ is (related to) inferior-ess-primary-prompt. The problem can be triggered by leaving blank lines in the middle of R code, and using :results output.

example

3

7

x <- 8

cat("hello\n")


1
[1] 3
[1] 7
hello
[1] 1

changed srcname doesn’t change resname

echo a
a

a sub-header

sub-stuff

another sub-header

and finally the results

aa

blank srcname takes srcname from next line

Commit 1c7009e28e5c16992b2c219808acff35b8d38c2b

5
5

stripping indentation from source-code blocks

This is a problem in org-babel-exp.el.

failing to write srcname to resname when evaluating a named block

"I'm the result"
I'm the result

Python session evaluation bug

The following block evaluates correctly with :session none (set :results to output), but fails with session-based evaluation (with :results value, as below, you see the error message)

I edebug’ed it and it seems fine until we go to comint.

import os
from subprocess import *

chunk = 10000
format = '.gen.gz'

cc = [('58C','NBS'),
      ('58C','POBI'),
      ('NBS','POBI')]

for outdir in ['none', 'noscots', 'lax', 'strict']:
    outdir = os.path.join('exclusion-study', outdir)
    for case, control in cc:
        outfile = os.path.join(outdir, '%s-vs-%s-direct' % (case, control))
        cmd = 'snptest %s -frequentist 1 -hwe ' % ('-gen_gz' if format == '.gen.gz' else '')
        cmd += '-cases %s %s ' % (case + format, case + '.sample')

        cmd += '-controls %s %s ' % (control + format, control + '.sample')
        cmd += '-exclude_samples %s ' % os.path.join(outdir, 'exclusions')
        cmd += '-o %s ' % outfile
        cmd += '-chunk %d ' % chunk
        cmd += '> %s' % outfile + '.log'
        # os.system(cmd)
        print(cmd)
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/none/exclusions -o exclusion-study/none/58C-vs-NBS-direct -chunk 10000 > exclusion-study/none/58C-vs-NBS-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/none/exclusions -o exclusion-study/none/58C-vs-POBI-direct -chunk 10000 > exclusion-study/none/58C-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/none/exclusions -o exclusion-study/none/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/none/NBS-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/noscots/exclusions -o exclusion-study/noscots/58C-vs-NBS-direct -chunk 10000 > exclusion-study/noscots/58C-vs-NBS-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/noscots/exclusions -o exclusion-study/noscots/58C-vs-POBI-direct -chunk 10000 > exclusion-study/noscots/58C-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/noscots/exclusions -o exclusion-study/noscots/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/noscots/NBS-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/lax/exclusions -o exclusion-study/lax/58C-vs-NBS-direct -chunk 10000 > exclusion-study/lax/58C-vs-NBS-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/lax/exclusions -o exclusion-study/lax/58C-vs-POBI-direct -chunk 10000 > exclusion-study/lax/58C-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/lax/exclusions -o exclusion-study/lax/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/lax/NBS-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/strict/exclusions -o exclusion-study/strict/58C-vs-NBS-direct -chunk 10000 > exclusion-study/strict/58C-vs-NBS-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/strict/exclusions -o exclusion-study/strict/58C-vs-POBI-direct -chunk 10000 > exclusion-study/strict/58C-vs-POBI-direct.log
snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/strict/exclusions -o exclusion-study/strict/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/strict/NBS-vs-POBI-direct.log

require users to explicitly turn on each language

As we continue to add more languages to org-babel, many of which will require new major-modes we need to re-think how languages are added to org-babel.

Currently we are requiring all available languages in the org-babel-init.el file. I think we need to change this to a user setting so that only the language which have their requirements met (in terms of system executables and emacs major modes) are loaded. It is one more step for install, but it seems to me to be the only solution. Thoughts?

proposed

we add something like the following to the instillation instructions

;; Uncomment each of the following require lines if you want org-babel
;; to support that language.  Each language has a comment explaining
;; it's dependencies.  See the related files in lisp/langs for more
;; detailed explanations of requirements.
;;
;; (require 'org-babel-R)         ;; ess-mode
;; (require 'org-babel-asymptote) ;; asymptote be installed on your system
;; (require 'org-babel-css)       ;; none
;; (require 'org-babel-ditaa)     ;; ditaa be installed on your system
;; (require 'org-babel-dot)       ;; dot be installed on your system
;; (require 'org-babel-gnuplot)   ;; gnuplot-mode
;; (require 'org-babel-python)    ;; python-mode
;; (require 'org-babel-ruby)      ;; inf-ruby mode, ruby and irb must be installed on your system
;; (require 'org-babel-sql)       ;; none

note that org-babel-sh, org-babel-emacs-lisp are not included in the list as they can safely be assumed to work on any system.

impetus

we should come up with a way to gracefully degrade when support for a specific language is missing

> To demonstrate creation of documents, open the “test-export.org” file in > the base of the org-babel directory, and export it as you would any > other org-mode file. The “exports” header argument controls how > source-code blocks are exported, with the following options > > - none :: no part of the source-code block is exported in the document > - results :: only the output of the evaluated block is exported > - code :: the code itself is exported > - both :: both the code and results are exported

I have this error showing up:

executing Ruby source code block apply: Searching for program: no such file or directory, irb

problem with newlines in output when :results value

'\n'.join(map(str, range(4)))
0
1
2
3

Whereas I was hoping for

0
1
2
3

Note: to generate the above you can try using the new raw results header.

'|'+'|\n|'.join(map(str, range(4)))+'|'
0
1
2
3

This is now working, it doesn’t return as a table because the value returned is technically a string. To return the table mentioned above try something like the following.

[[0], [1], [2], [3]]
0
1
2
3

This is some sort of non-printing char / quoting issue I think. Note that

'\\n'.join(map(str, range(4)))
0\n1\n2\n3

Also, note that

print('\n'.join(map(str, range(4))))
0
1
2
3

collapsing consecutive newlines in string output

This is an example of the same bug

"the first line ends here


     and this is the second one

even a third"

This doesn’t produce anything at all now. I believe that’s because I’ve changed things so that :results output really does not get the value of the block, only the STDOUT. So if we add a print statement this works OK.

print "the first line ends here


     and this is the second one

even a third"
the first line ends here

: :

     and this is the second one

:

even a third

However, the behaviour with :results value is wrong

"the first line ends here


     and this is the second one

even a third"
0

o-b-execute-subtree overwrites heading when subtree is folded

I’m not seeing this any more, so am closing this bug.

Example

Try M-x org-babel-execute-subtree with the subtree folded and point at the beginning of the heading line.

echo 7
echo 6
6

non-orgtbl formatted lists

for example

'((:results . "replace"))
(:results . “replace”)
'(eric schulte)
ericschulte

adding blank line when source-block produces no output

find . \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o  -type f \( -iname \*.el \) -exec grep -i -nH -e org-babel-trim {} \;

Allow source blocks to be recognised when #+ are not first characters on the line

I think Carsten has recently altered the core so that #+ can have preceding whitespace, at least for literal/code examples. org-babel should support this.

(message "i'm indented")
(message "I'm not indented")
(message "I said %s" speech)

are the org-babel-trim s necessary?

at the end of e.g. org-babel-R-evaluate, org-babel-python-evaluate, but not org-babel-ruby-evaluate

I think it depends on the language, if we find that extra blank lines are being inserted in a particular language that is a good indication that the trim or chomp functions may be appropriate.

org-babel-trim and the related org-babel-chomp are use throughout org-babel…

find lisp/ \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o  -type f \( -iname \*.el \) -exec grep -i -nH org-babel-trim {} \;
lisp//langs/org-babel-python.el:49:		     vars "\n") "\n" (org-babel-trim body) "\n")) ;; then the source block body
lisp//langs/org-babel-python.el:143:			       (org-remove-indentation (org-babel-trim body)) "[\r\n]")))
lisp//langs/org-babel-python.el:158:	       #'org-babel-trim
lisp//langs/org-babel-python.el:166:                                           (reverse (mapcar #'org-babel-trim raw)))))))
lisp//langs/org-babel-python.el:169:	  (output (org-babel-trim (mapconcat #'identity (reverse (cdr results)) "\n")))
lisp//langs/org-babel-python.el:170:	  (value (org-babel-python-table-or-string (org-babel-trim (car results)))))))))
lisp//langs/org-babel-ruby.el:149:                                                  (mapcar #'org-babel-trim raw)))))))
lisp//langs/org-babel-sh.el:148:                                                  (mapcar #'org-babel-trim raw)))))))
lisp//langs/org-babel-sh.el:151:            (output (org-babel-trim (mapconcat #'org-babel-trim (reverse results) "\n")))
lisp//org-babel-ref.el:161:    (mapcar #'org-babel-trim (reverse (cons buffer return)))))
lisp//org-babel.el:198:    (with-temp-buffer (insert (org-babel-trim body)) (copy-region-as-kill (point-min) (point-max)))
lisp//org-babel.el:465:           (org-babel-trim
lisp//org-babel.el:706:(defun org-babel-trim (string &optional regexp)
find lisp/ \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o  -type f \( -iname \*.el \) -exec grep -i -nH org-babel-chomp {} \;
lisp//langs/org-babel-R.el:122:             (full-body (mapconcat #'org-babel-chomp
lisp//langs/org-babel-R.el:143:	      (delete nil (mapcar #'extractor (mapcar #'org-babel-chomp raw))) "\n"))))))))
lisp//langs/org-babel-ruby.el:143:	     #'org-babel-chomp
lisp//langs/org-babel-sh.el:142:	   (full-body (mapconcat #'org-babel-chomp
lisp//org-babel-tangle.el:163:      (insert (format "\n%s\n" (org-babel-chomp body)))
lisp//org-babel.el:362:			 (org-babel-chomp (match-string 2 arg)))
lisp//org-babel.el:698:(defun org-babel-chomp (string &optional regexp)
lisp//org-babel.el:707:  "Like `org-babel-chomp' only it runs on both the front and back of the string"
lisp//org-babel.el:708:  (org-babel-chomp (org-babel-reverse-string
lisp//org-babel.el:709:                    (org-babel-chomp (org-babel-reverse-string string) regexp)) regexp))

LoB is not populated on startup

org-babel-library-of-babel is nil for me on startup. I have to evaluate the org-babel-lob-ingest line manually.

12
24
39
416
525

I’ve added a section to org-babel-init.el which will load the library of babel on startup.

Note that this needs to be done in org-babel-init.el rather than in org-babel-lob.el, not entirely sure why, something about it being required directly?

Also, I’m now having the file closed if it wasn’t being visited by a buffer before being loaded.

use new merge function here?

And at other occurrences of org-combine-plists?

creeping blank lines

There’s still inappropriate addition of blank lines in some circumstances.

Hmm, it’s a bit confusing. It’s to do with o-b-remove-result. LoB removes the entire (#+resname and result) and starts from scratch, whereas #+begin_src only removes the result. I haven’t worked out what the correct fix is yet. Maybe the right thing to do is to make sure that those functions (o-b-remove-result et al.) are neutral with respect to newlines. Sounds easy, but…

E.g.

b=5

Compare the results of

22

23
23

#+srcname arg parsing bug

arg
2
arg
40
arg
40

Fix nested evaluation and default args

The current parser / evaluator fails with greater levels of nested function block calls (example below).

Initial statement [ded]

If we want to overcome this I think we’d have to redesign some of the evaluation mechanism. Seeing as we are also facing issues like dealing with default argument values, and seeing as we now know how we want the library of babel to behave in addition to the source blocks, now might be a good time to think about this. It would be nice to do the full thing at some point, but otoh we may not consider it a massive priority.

AIui, there are two stages: (i) construct a parse tree, and (ii) evaluate it and return the value at the root. In the parse tree each node represents an unevaluated value (either a literal value or a reference). Node v may have descendent nodes, which represent values upon which node v’s evaluation depends. Once that tree is constructed, then we evaluate the nodes from the tips towards the root (a post-order traversal).

[This would also provide a solution for concatenating the STDOUTs of called blocks, which is a task below; we concatenate them in whatever order the traversal is done in.]

In addition to the variable references (i.e. daughter nodes), each node would contain the information needed to evaluate that node (e.g. lang body). Then we would pass a function postorder over the tree which would call o-b-execute-src-block at each node, finally returning the value at the root.

Fwiw I made a very tentative small start at stubbing this out in org-babel-call.el in the ‘evaluation’ branch. And I’ve made a start at sketching a parsing algorithm below.

Parse tree algorithm

Seeing as we’re just trying to parse a string like f(a=1,b=g(c=2,d=3)) it shouldn’t be too hard. But of course there are ‘proper’ parsers written in elisp out there, e.g. Semantic. Perhaps we can find what we need – our syntax is pretty much the same as python and R isn’t it?

Or, a complete hack, but maybe it would be we easy to transform it to XML and then parse that with some existing tool?

But if we’re doing it ourselves, something very vaguely like this? (I’m sure there’re lots of problems with this)

## we are currently reading a reference name: the name of the root function
whereami = "refname"
node = root = Node()
for c in call_string:
    if c == '(':
        varnum = 0
        whereami = "varname" # now we're reading a variable name
    if c == '=':
        new = Node()
        node.daughters = [node.daughters, new]
        new.parent = node
        node = new
        whereami = "refname"
    if c == ',':
        whereami = "varname"
        varnum += 1
    elif c == ')':
        node = node.parent
    elif c == ' ':
        pass
    else:
        if whereami = "varname":
            node.varnames[varnum] += c
        elif whereami = "refname":
            node.name += c

discussion / investigation

I believe that this issue should be addressed as a bug rather than as a point for new development. The code in org-babel-ref.el already resolves variable references in a recursive manner which should work in the same manner regardless of the depth of the number of nested function calls. This recursive evaluation has the effect of implicitly constructing the parse tree that your are thinking of constructing explicitly.

Through using some of the commented out debugging statements in org-babel-ref.el I have looked at what may be going wrong in the current evaluation setup, and it seems that nested variables are being set using the :var header argument, and these variables are being overridden by the default variables which are being entered through the new functional syntax (see the demonstration header below).

I believe that once this bug is fixed we should be back to fully resolution of nested arguments. We should capture this functionality in a test to ensure that we continue to test it as we move forward. I can take a look at implementing this once I get a chance.

Looks like the problem may be in org-babel-merge-params, which seems to be trampling the provided :vars values.

Nope, now it seems that we are actually looking up the results line, rather than the actual source-code block, which would make sense given that the results-line will return the same value regardless of the arguments supplied. See the output of this debug-statement.

We need to be sure that we don’t read from a #+resname: line when we have a non-nil set of arguments.

demonstration

After uncommenting the debugging statements located here and more importantly here, we can see that the current reference code does evaluate the references correctly, and it uses the :var header argument to set a=8, however the default variables specified using the functional syntax in adder(a=3, b=2) is overriding this specification.

doesn’t work with functional syntax
a + b
5
arg
5
still does work with :var syntax

so it looks like regardless of the syntax used we’re not overriding the default argument values.

a + b
5
arg
5

Set of test cases

Both defaults provided in definition

a+b
30
Rely on defaults
30

## should be 30 ## OK, but

empty parens () not recognised as lob call E.g. remove spaces between parens above

updated org-babel-lob-one-liner-regexp

One supplied, one default
20

## should be 20

## should be 10

10
Both supplied
3

One arg lacks default in definition

a+b
Rely on defaults (one of which is missing)

[no output]

## should be error: b has no default

Maybe we should let the programming language handle this case. For example python spits out an error in the #+lob line above. Maybe rather than catching these errors our-selves we should raise an error when the source-block returns an error. I’ll propose a task for this idea, I’m not sure how/if it would work…

Default over-ridden

See the above deferred and the new proposed task, I think it may be more flexible to allow the source block language to handle the error.

[no output ] ## should be error: b has no default

Missing default supplied
11

## should be 11 ## OK

One over-ridden, one supplied
3

## should be 3

Example that fails

return a+b
return 1
1

nesting

return arg
4
return arg
12

deeply nested arguments still fails

return arg
4
4

Used to result in this error

supplied params=nil
new-refere="adder", new-referent="a=adder(a=one(),b=one()),b=adder(a=one(),b=one())"
args=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
type=source-block
supplied params=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
new-refere="adder", new-referent="a=one("
args=((:var . "a=one("))
type=source-block
supplied params=((:var . "a=one("))
reference 'one(' not found in this buffer

Need to change the regexp in org-babel-ref-resolve-reference so that it only matches when the parenthesis are balanced. Maybe look at this.

Still some problems with deeply nested arguments and defaults

sandbox

Parsing / defaults bug

Try inserting a space between ‘a=0,’ and ‘b=0’ and comparing results

arg
99
arg
10

Nesting problem II

This generates parsing errors

Fixed: c2bef96b7f644c05be5a38cad6ad1d28723533aa

arg
106
arg
8

Why does this give 8?

It was picking up the wrong definition of adder

arg
101

Problem with empty argument list

This gives empty list with () and ‘no output’ with ( )

I think this is OK now.

arg
99
arg
10

Nesting problem II

This generates parsing errors

Fixed: c2bef96b7f644c05be5a38cad6ad1d28723533aa

arg
106
arg
8

Why does this give 8?

It was picking up the wrong definition of adder

arg
101

Problem with empty argument list

This gives empty list with () and ‘no output’ with ( )

I think this is OK now.

arg
99
arg

Arg lacking default

This would be good thing to address soon. I’m imagining that e.g. here, the ‘caller’ block would return the answer 30. I believe there’s a few issues here: i.e. the naked ‘a’ without a reference is not understood; the default arg b=6 is not understood.

a+b
var

allow srcname to omit function call parentheses

Someone needs to revisit those regexps. Is there an argument for moving some of the regexps used to match function calls into defvars? (i.e. in o-b.el and o-b-ref.el)

This seems to work now. It still might be a good idea to separate out some of the key regexps into defvars at some point.

3.times {puts 'x'}
x
x
x

avoid stripping whitespace from output when :results output

This may be partly solved by using o-b-chomp rather than o-b-trim in the o-b-LANG-evaluate functions.

function calls in #+srcname: refs

My srcname references don’t seem to be working for function calls. This needs fixing.

59

srcname function call doesn’t work for calling a source block

var1
59

They do work for a simple reference

var1
59

and they do work for :var header arg

var1
58

LoB: with output to buffer, not working in buffers other than library-of-babel.org

Initial report

I haven’t fixed this yet. org-babel-ref-resolve-reference moves point around, inside a save-excursion. Somehow when it comes to inserting the results (after possible further recursive calls to org-babel-ref-resolve-reference), point hasn’t gone back to the lob line.

11
2.5
3.333
11

Now

I think this got fixed in the bugfixes before merging results into master.

cursor movement when evaluating source blocks

E.g. the pie chart example. Despite the save-window-excursion in org-babel-execute:R. (I never learned how to do this properly: org-R jumps all over the place…)

I don’t see this now [ded]

LoB: calls fail if reference has single character name

commit 21d058869df1ff23f4f8cc26f63045ac9c0190e2

This doesn’t work

11
2.5
3.3333
4.25
5.2
6.1666

But this is OK

11
2.5
3.3333
4.25
5.2
6.1666

make :results replace the default?

I’m tending to think that appending results to pre-existing results creates mess, and that the cleaner `replace’ option should be the default. E.g. when a source block creates an image, we would want that to be updated, rather than have a new one be added.

I agree.

ruby evaluation not working under ubuntu emacs 23

With emacs 23.0.91.1 on ubuntu, for C-h f run-ruby I have the following, which seems to conflict with this line in org-babel-ruby.el.

run-ruby is an interactive compiled Lisp function.

(run-ruby cmd)

Run an inferior Ruby process, input and output via buffer *ruby*.
If there is a process already running in `*ruby*', switch to that buffer.
With argument, allows you to edit the command line (default is value
of `ruby-program-name').  Runs the hooks `inferior-ruby-mode-hook'
(after the `comint-mode-hook' is run).
(Type C-h m in the process buffer for a list of commands.)

So, I may have a non-standard inf-ruby.el. Here’s my version of run-ruby.

run-ruby is an interactive Lisp function in `inf-ruby.el'.

(run-ruby &optional COMMAND NAME)

Run an inferior Ruby process, input and output via buffer *ruby*.
If there is a process already running in `*ruby*', switch to that buffer.
With argument, allows you to edit the command line (default is value
of `ruby-program-name').  Runs the hooks `inferior-ruby-mode-hook'
(after the `comint-mode-hook' is run).
(Type C-h m in the process buffer for a list of commands.)

It seems we could either bundle my version of inf-ruby.el (as it’s the newest). Or we could change the use of `run-ruby’ so that it is robust across multiple distributions. I think I’d prefer the former, unless the older version of inf-ruby is actually bundled with emacs, in which case maybe we should go out of our way to support it. Thoughts?

I think for now I’ll just include the latest inf-ruby.el in the newly created utility directory. I doubt anyone would have a problem using the latest version of this file.

test failing forcing vector results with test-forced-vector-results ruby code block

Note that this only seems to happen the second time the test table is evaluated

8
triv.class.name

mysteriously this seems to be fixed…

defunct R sessions

Sometimes an old R session will turn defunct, and newly inserted code will not be evaluated (leading to a hang).

This seems to be fixed by using `inferior-ess-send-input’ rather than `comint-send-input’.

ruby fails on first call to non-default session

:patton

when reading results from #+resname line

Errors when trying to read from resname lines.

8
buggy

R-code broke on “org-babel” rename

8 * 2

error on trivial R results

So I know it’s generally not a good idea to squash error without handling them, but in this case the error almost always means that there was no file contents to be read by org-table-import, so I think it’s ok.

pie(c(1, 2, 3), labels = c(1, 2, 3))
8
8
c(1,2,3)
1
2
3

ruby new variable creation (multi-line ruby blocks)

Actually it looks like we were dropping all but the last line.

total = 0
table.each{|n| total += n}
total/table.size
2

R code execution seems to choke on certain inputs

Currently the R code seems to work on vertical (but not landscape) tables

"schulte"
num
schulte
(setq debug-on-error t)
'(1 2 3)
mean(mean(table))
2
1
2
3
mean(table)

org bug/request: prevent certain org behaviour within code blocks

E.g. [[]] gets recognised as a link (when there’s text inside the brackets). This is bad for R code at least, and more generally could be argued to be inappropriate. Is it difficult to get org to ignore text in code blocks? [DED]

I believe Carsten addressed this recently on the mailing list with the comment that it was indeed a difficult issue. I believe this may be one area where we could wait for an upstream (org-mode) fix.

[Dan] Carsten has fixed this now in the core.

with :results replace, non-table output doesn’t replace table output

And vice versa. E.g. Try this first with table and then with len(table) [DED]

table
123
4“schulte”6
2

Yes, this is certainly a problem. I fear that if we begin replacing anything immediately following a source block (regardless of whether it matches the type of our current results) we may accidentally delete hand written portions of the user’s org-mode buffer.

I think that the best solution here would be to actually start labeling results with a line that looks something like…

This would have a couple of benefits…

  1. we wouldn’t have to worry about possibly deleting non-results (which is currently an issue)
  2. we could reliably replace results even if there are different types
  3. we could reference the results of a source-code block in variable definitions, which would be useful if for example we don’t wish to re-run a source-block every time because it is long-running.

Thoughts? If no-one objects, I believe I will implement the labeling of results.

extra quotes for nested string

Well R appears to be reading the tables without issue…

these should be quoted

ls
“COPYING”
“README.markdown”
“block”
“examples.org”
“existing_tools”
“intro.org”
“org-babel”
“rorg.org”
“test-export.html”
“test-export.org”
tab[1][0]
README.markdown
as.matrix(tab[2,])
README.markdown

simple ruby arrays not working

As an example eval the following. Adding a line to test

345
ar.first.first

space trailing language name

fix regexp so it works when there’s a space trailing the language name

:schulte

Args out of range error

The following block resulted in the error below [DED]. It ran without error directly in the shell.

cd ~/work/genopca
for platf in ill aff ; do
    for pop in CEU YRI ASI ; do
	rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
	cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
	cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
    done
done

executing source block with sh… finished executing source block string-equal: Args out of range: “”, -1, 0

the error string-equal: Args out of range: "", -1, 0 looks like what used to be output when the block returned an empty results string. This should be fixed in the current version, you should now see the following message no result returned by source block.

ruby arrays not recognized as such

Something is wrong in lisp/org-babel-script.el related to the recognition of ruby arrays as such.

[1, 2, 3, 4]
1234
[1, 2, 3, 4]
1234

weird escaped characters in shell prompt break shell evaluation

E.g. this doesn’t work. Should the shell sessions set a sane prompt when they start up? Or is it a question of altering comint-prompt-regexp? Or altering org-babel regexps?

black=30 ; red=31 ; green=32 ; yellow=33 ; blue=34 ; magenta=35 ; cyan=36 ; white=37
prompt_col=$red
prompt_char='>'
export PS1="\[\033[${prompt_col}m\]\w${prompt_char} \[\033[0m\]"

I just pushed a good amount of changes, could you see if your shell problems still exist?

The problem’s still there. Specifically, aIui, at this line of org-babel-sh.el, raw gets the value

(“” “�[0m Sun Jun 14 19:26:24 EDT 2009\n” “�[0m org_babel_sh_eoe\n” “�[0m “)

and therefore (member org-babel-sh-eoe-output …) fails

I think that `comint-prompt-regexp’ needs to be altered to match the shell prompt. This shouldn’t be too difficult to do by hand, using the `regexp-builder’ command and should probably be part of the user’s regular emacs init. I can’t think of a way for us to set this automatically, and we are SOL without a regexp to match the prompt.

elisp reference fails for literal number

That’s a bug in Dan’s elisp, not in org-babel.

(message a)
30

Tests

General Tests

Evaluate all the cells in this table for a comprehensive test of the org-babel functionality.

Note: if you have customized org-babel-default-header-args then some of these tests may fail.

functionalityblockargexpectedresultspass
basic evaluationpass
emacs lispbasic-elisp55pass
shellbasic-shell66pass
rubybasic-rubyorg-babelorg-babelpass
pythonbasic-pythonhello worldhello worldpass
Rbasic-R1313pass
tablespass
emacs lisptable-elisp33pass
rubytable-ruby1-2-31-2-3pass
table-from-ruby44pass
pythontable-python55pass
table-from-python44pass
Rtable-R3.53.5pass
R: col names in Rtable-R-colnames-30expected “-3” but was “0”
R: col names in orgtable-R-colnames-org169#ERRORexpected “169” but was “#ERROR”
source block referencespass
all languageschained-ref-lastArrayArraypass
source block functionspass
emacs lispdefun-fibbfibbdfibbdpass
run overtest-fibonacci011pass
atest-fibonacci111pass
varietytest-fibonacci222pass
oftest-fibonacci333pass
differenttest-fibonacci455pass
argumentstest-fibonacci588pass
bugs and taskspass
simple ruby arraysruby-array-test33pass
R number evaluationbug-R-number-evaluation22pass
multi-line ruby blocksmulti-line-ruby-test22pass
forcing vector resultstest-forced-vector-resultsArrayArraypass
deeply nested argumentsdeeply-nested-args-bug44pass
sessionspass
set ruby sessionset-ruby-session-var:set:setpass
get from ruby sessionget-ruby-session-var33pass
set python sessionset-python-session-varsetsetpass
get from python sessionget-python-session-var44pass
set R sessionset-R-session-varsetsetpass
get from R sessionget-R-session-var55pass

The second TBLFM line can be used to blank out the table results, in the absence of a better method.

basic tests

(+ 1 4)
expr 1 + 5
date
"org-babel"
return 'hello world'
b <- 9
b + 4

read tables

123
456
var1var2var3
12213
415567
(length (car table))
table.first.join("-")
[[1, 2], [3, 4]]
tab[1][1]
return table[1][1]
return [[1, 2], [3, 4]]
return tab[1][1]
mean(mean(table))
sum(table$var2 - table$var3)
x^2

This should return 169. The fact that R is able to use the column name to index the data frame (x$var3) proves that a table with column names (a header row) has been recognised as input for the R-square function block, and that the R-square block has output an elisp table with column names, and that the colnames have again been recognised when creating the R variables in this block.

x$var3[1]

references

Lets pass a references through all of our languages…

Lets start by reversing the table from the previous examples

table.reverse()
table
456
123

Take the first part of the list

table[1]
4
1

Turn the numbers into string

(mapcar (lambda (el) (format "%S" el)) table)
“(4)”“(1)”

and Check that it is still a list

table.class.name

source blocks as functions

(defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
(fibbd n)

sbe tests (these don’t seem to be working…)

Testing the insertion of results into org-mode tables.

"the first line ends here


     and this is the second one

even a third"
the first line ends here\n\n\n     and this is the second one\n\neven a third
raise "oh nooooooooooo"
oh nooooooooooo
the first line ends here…-:5: warning: parenthesize argument(s) for future version…

forcing results types tests

8
triv.class.name

sessions

var = [1, 2, 3]
:set
var.size
var=4
'set'
var
a <- 5
'set'
a

Clojure Tests

Evaluate all the cells in this table for a comprehensive test of the org-babel functionality.

Note: if you have customized org-babel-default-header-args then some of these tests may fail.

functionalityblockargexpectedresultspass
basic evaluationpass
clojurebasic-clojure55pass
clojurehello-clojurehello clojurehello clojurepass
rubybasic-rubyorg-babelorg-babelpass
tablespass
clojuretable-clojure33pass
clojuretable-clojure-join1-2-31-2-3pass
rubytable-ruby1-2-31-2-3pass
source block referencespass
all languagesclojure-chained-ref-lastclass clojure.lang.PersistentListclass clojure.lang.PersistentListpass
source block functionspass
clojuredefn-fib#’user/fib#’user/fibpass
clojure arg1clojure-fibonacci000pass
clojure arg2clojure-fibonacci111pass
clojure arg3clojure-fibonacci322pass
sessionspass
set ruby sessionset-python-session-varsetsetpass
get from ruby sessionget-python-session-var44pass
set clojure sessionset-clojure-session-var:set:setpass
get from clojure sessionget-clojure-session-var33pass

basic tests

(+ 1 4)
[ 1 2 3 4]
1234
"hello clojure"
hello clojure
"org-babel"
(println "hello output-no-session clojure")
hello output-no-session clojure
(println '(6 7 8 9 ))
(6 7 8 9)

read tables

123
456
table.first.join("-")
(count (first table))
(apply str (interpose "-" (first table)))

write tables

(list '(A B C) (reverse (first table)))
ABC
321

references

Lets pass a references through all of our languages…

Lets start by reversing the table from the previous examples

(reverse table)
456
123

Take the first part of the list

(first table)
4
1

Turn the numbers into string

(map #(format "%S" %) table)
“(4)”“(1)”

and Check that it is still a list

(type table)

source blocks as functions

(defn fib [n]
  (if (<= n 1)
    n
    (+ (fib (dec n)) (fib (- n 2)))))
(fib n)

result replace

0

(format "I'm going to format the number: %d" n)
I'm going to format the number: 7

sessions

var=4
'set'
var
(def *var* [1 2 3])
:set
(count *var*)
(println "hello clojure")
hello clojure