ox-ipynb does not always preserve org subtrees using Markdown headings

[xelibrion]

Hi John, I just discovered that there are cases when level of nesting is not being preserved with Markdown headings - often a block that was a subtree in org file, becomes a H1 heading in Markdown.

The simplest example I could come up with would be content below, if you were to export it, are not preserved will become H1 heading instead of H2.

* Markdown levels
Some comment
#+BEGIN_SRC ipython
import numpy as np
#+END_SRC

** are not preserved
#+BEGIN_SRC ipython
import pandas as pd
#+END_SRC

The end.

[xelibrion]

It also might make sense to put headings for each subtree to a separate cell in notebook, as Collapsible headings (https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/collapsible_headings) extension does not work great with the way headings get exported at the moment.

[jkitchin]

I cannot reproduce the first comment, for me the headings are preserved.

The collapsible headings is a nice idea. I have implemented that I think. That raises the notebook up to 1.1% of the goodness of org-mode ;)

[singletS]

I cannot reproduce the heading promotion problem described in the first comment either.

However, with f9c485e I'm getting two cases where specious markdown cells are inserted. I think I've narrowed them to these conditions, but someone should try to reproduce:

• The org file starts with a heading (no text or #+OPTIONS or anything)
• A src block followed by (possibly whitespace and) a heading. Eg:

* First heading
#+BEGIN_SRC ipython :session
import string
#+END_SRC

** Second heading
 More text.

Also, +1 adding this. I haven't used nbextensions yet, but they look intriguing...

[jkitchin]

Those are both issues. I think I fixed them both. thanks for testing them out!

[jkitchin]

btw, the collapsible headings makes big notebooks much more usable!

[xelibrion]

Export for one of the large-ish org files I was working with recently stopped working after the change. This is the exception I get

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("[ 	\n
]+\\'" nil)
  (if (string-match "[ 	\n
]+\\'" s) (replace-match "" t t s) s)
  (progn (if (string-match "[ 	\n
]+\\'" s) (replace-match "" t t s) s))
  (unwind-protect (progn (if (string-match "[ 	\n
]+\\'" s) (replace-match "" t t s) s)) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal (match-data))) (unwind-protect (progn (if (string-match "[ 	\n
]+\\'" s) (replace-match "" t t s) s)) (set-match-data save-match-data-internal (quote evaporate))))
  s-trim-right(nil)
  (s-trim-left (s-trim-right s))
  s-trim(nil)
  (progn (fset (quote org-export-get-relative-level) vnew) (s-trim (org-export-string-as s (quote md) t (quote (:with-toc nil :with-tags nil)))))
  (unwind-protect (progn (fset (quote org-export-get-relative-level) vnew) (s-trim (org-export-string-as s (quote md) t (quote (:with-toc nil :with-tags nil))))) (fset (quote org-export-get-relative-level) old))
  (let* ((vnew (function (lambda (headline info) (org-element-property :level headline)))) (old (symbol-function (quote org-export-get-relative-level)))) (unwind-protect (progn (fset (quote org-export-get-relative-level) vnew) (s-trim (org-export-string-as s (quote md) t (quote (:with-toc nil :with-tags nil))))) (fset (quote org-export-get-relative-level) old)))
  (let* ((org-export-filter-latex-fragment-functions (quote (ox-ipynb-filter-latex-fragment))) (org-export-filter-link-functions (quote (ox-ipynb-filter-link))) (org-export-filter-keyword-functions (quote (ox-ipynb-keyword-link))) (md (let* ((vnew (function (lambda ... ...))) (old (symbol-function (quote org-export-get-relative-level)))) (unwind-protect (progn (fset (quote org-export-get-relative-level) vnew) (s-trim (org-export-string-as s ... t ...))) (fset (quote org-export-get-relative-level) old))))) (if (not (string= "" md)) (list (quote (cell_type . "markdown")) (cons (quote metadata) (make-hash-table)) (cons (quote source) (vconcat (list md)))) nil))
  export-ipynb-markdown-cell("")
  (and t (export-ipynb-markdown-cell (s-trim s)))
  (let* ((md (and t (export-ipynb-markdown-cell (s-trim s))))) (if md (setq cells (cons md cells))))
  (if (string= "" s) (progn) (let* ((md (and t (export-ipynb-markdown-cell (s-trim s))))) (if md (setq cells (cons md cells)))))
  (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ((md (and t (export-ipynb-markdown-cell (s-trim s))))) (if md (setq cells (cons md cells))))) (setq --cl-var-- (cdr --cl-var--)))
  (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ((md (and t (export-ipynb-markdown-cell ...)))) (if md (setq cells (cons md cells))))) (setq --cl-var-- (cdr --cl-var--))) nil)
  (let ((text (buffer-substring-no-properties end (org-element-property :begin (car current-source))))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ((md (and t ...))) (if md (setq cells (cons md cells))))) (setq --cl-var-- (cdr --cl-var--))) nil))
  (progn (let ((text (buffer-substring-no-properties end (org-element-property :begin (car current-source))))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ((md ...)) (if md (setq cells ...)))) (setq --cl-var-- (cdr --cl-var--))) nil)))
  (if (not (string= "" (s-trim (buffer-substring end (org-element-property :begin (car current-source)))))) (progn (let ((text (buffer-substring-no-properties end (org-element-property :begin (car current-source))))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* (...) (if md ...))) (setq --cl-var-- (cdr --cl-var--))) nil))))
  (if current-source (if (not (string= "" (s-trim (buffer-substring end (org-element-property :begin (car current-source)))))) (progn (let ((text (buffer-substring-no-properties end (org-element-property :begin ...)))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ... ...)) (setq --cl-var-- (cdr --cl-var--))) nil)))) (let ((text (buffer-substring-no-properties end (point-max)))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ((md ...)) (if md (setq cells ...)))) (setq --cl-var-- (cdr --cl-var--))) nil)))
  (while current-source (setq cells (cons (export-ipynb-code-cell current-source) cells)) (setq result-end (cdr current-source) result (car result-end) result-end (cdr result-end)) (setq end (max (or result-end 0) (org-element-property :end (car current-source)))) (setq current-source (car-safe (prog1 src-results (setq src-results (cdr src-results))))) (if current-source (if (not (string= "" (s-trim (buffer-substring end (org-element-property :begin ...))))) (progn (let ((text (buffer-substring-no-properties end ...))) (let* ((--cl-var-- ...) (s nil)) (while (consp --cl-var--) (setq s ...) (if ... ... ...) (setq --cl-var-- ...)) nil)))) (let ((text (buffer-substring-no-properties end (point-max)))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* (...) (if md ...))) (setq --cl-var-- (cdr --cl-var--))) nil))))
  (let* ((cells (if (export-ipynb-keyword-cell) (list (export-ipynb-keyword-cell)) (quote nil))) (ox-ipynb-language (ox-ipynb-get-language)) (metadata (list (quote metadata) (cons (quote org) (let* ((all-keywords ...) (ipynb-keywords ...) (include-keywords ...) (keywords ...)) keywords)) (cdr (assoc ox-ipynb-language ox-ipynb-kernelspecs)) (cdr (assoc ox-ipynb-language ox-ipynb-language-infos)))) (ipynb (or (and (boundp (quote export-file-name)) export-file-name) (concat (file-name-base (buffer-file-name)) ".ipynb"))) src-blocks src-results current-src result result-end end data) (setq src-blocks (org-element-map (org-element-parse-buffer) (quote src-block) (function (lambda (src) (if (string= (symbol-name ox-ipynb-language) (org-element-property :language src)) (progn src)))))) (setq src-results (let* ((--cl-var-- src-blocks) (src nil) (result=nil nil) (--cl-var-- nil)) (while (consp --cl-var--) (setq src (car --cl-var--)) (setq result (save-excursion (goto-char (org-element-property :begin src)) (let (... start end result-content) (if location ...)))) (setq --cl-var-- (cons (cons src result) --cl-var--)) (setq --cl-var-- (cdr --cl-var--))) (nreverse --cl-var--))) (setq current-source (car-safe (prog1 src-results (setq src-results (cdr src-results))))) (if (car current-source) (if (string= "" (s-trim (buffer-substring-no-properties (point-min) (org-element-property :begin (car current-source))))) nil (let ((text (buffer-substring-no-properties (point-min) (org-element-property :begin ...)))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" ...) (progn) (let* ... ...)) (setq --cl-var-- (cdr --cl-var--))) nil))) (let ((text (buffer-substring-no-properties (point-min) (point-max)))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" (s-trim s)) (progn) (let* (...) (if md ...))) (setq --cl-var-- (cdr --cl-var--))) nil))) (while current-source (setq cells (cons (export-ipynb-code-cell current-source) cells)) (setq result-end (cdr current-source) result (car result-end) result-end (cdr result-end)) (setq end (max (or result-end 0) (org-element-property :end (car current-source)))) (setq current-source (car-safe (prog1 src-results (setq src-results (cdr src-results))))) (if current-source (if (not (string= "" (s-trim (buffer-substring end ...)))) (progn (let ((text ...)) (let* (... ...) (while ... ... ... ...) nil)))) (let ((text (buffer-substring-no-properties end (point-max)))) (let* ((--cl-var-- (s-slice-at "^\\*" text)) (s nil)) (while (consp --cl-var--) (setq s (car --cl-var--)) (if (string= "" s) (progn) (let* ... ...)) (setq --cl-var-- (cdr --cl-var--))) nil)))) (setq data (append (list (cons (quote cells) (reverse cells))) (list metadata) (quote ((nbformat . 4) (nbformat_minor . 0))))) (save-current-buffer (set-buffer (get-buffer-create "*ox-ipynb*")) (erase-buffer) (insert (json-encode data))) (switch-to-buffer "*ox-ipynb*") (set (make-local-variable (quote export-file-name)) ipynb) (get-buffer "*ox-ipynb*"))
  ox-ipynb-export-to-buffer()
  (setq buf (ox-ipynb-export-to-buffer))
  (save-current-buffer (set-buffer "*Org ORG Export*") (set (make-local-variable (quote export-file-name)) ipynb) (setq buf (ox-ipynb-export-to-buffer)) (save-current-buffer (set-buffer buf) (set (make-local-variable (quote export-file-name)) ipynb)) (prog1 buf (kill-buffer "*Org ORG Export*")))
  (let ((ipynb (or (if (boundp (quote export-file-name)) (progn export-file-name)) (concat (file-name-base (buffer-file-name)) ".ipynb"))) buf) (org-org-export-as-org async subtreep visible-only ext-plist) (save-current-buffer (set-buffer "*Org ORG Export*") (set (make-local-variable (quote export-file-name)) ipynb) (setq buf (ox-ipynb-export-to-buffer)) (save-current-buffer (set-buffer buf) (set (make-local-variable (quote export-file-name)) ipynb)) (prog1 buf (kill-buffer "*Org ORG Export*"))))
  ox-ipynb-export-to-ipynb-buffer(nil nil nil nil)
  (set-buffer (ox-ipynb-export-to-ipynb-buffer async subtreep visible-only ext-plist))
  (save-current-buffer (set-buffer (ox-ipynb-export-to-ipynb-buffer async subtreep visible-only ext-plist)) (let* ((efn export-file-name) (buf (find-file-noselect efn))) (write-file efn) (save-current-buffer (set-buffer buf) (set (make-local-variable (quote export-file-name)) efn)) (kill-buffer buf) efn))
  ox-ipynb-export-to-ipynb-file(nil nil nil nil)
  org-export-dispatch(nil)
  funcall-interactively(org-export-dispatch nil)
  call-interactively(org-export-dispatch nil nil)
  command-execute(org-export-dispatch)

[jkitchin]

I guess that is happening from s-trim(nil) somewhere. If you can get a small working example it is probably easy to fix. I can't reproduce it on the ones I have tried so far though.

[jkitchin]

Closing this until a small example that reproduces the issue can be made.