OrderedList and BulletList break when empty in lua scripts

[CodeSandwich]

Explain the problem.

OrderedLists and BulletLists break internally when empty, they start collapsing their content into one entry.

Example

The example consists of 5 examples, list1 to list5. list3 and list4 present the bug. list5 would be very convenient if it worked. It behaves the same for both BulletList and OrderedList.

Tested using a dummy command:

echo "" | pandoc -f markdown -t markdown -L bug.lua

bug.lua has the following content:

function Pandoc()
    -- LIST1
    -- Initialize a list with a single item - works fine
    local list1 = pandoc.BulletList(pandoc.Plain "a")
    -- local list1 = pandoc.OrderedList(pandoc.Plain "a")
    list1.content:insert(pandoc.Plain "b")
    print("list 1", pandoc.json.encode(list1))

    -- LIST2
    -- Initialize a list with a list of items - works fine
    local list2 = pandoc.BulletList({ pandoc.Plain "a" })
    -- local list2 = pandoc.OrderedList({ pandoc.Plain "a" })
    list2.content:insert(pandoc.Plain "b")
    print("list 2", pandoc.json.encode(list2))

    -- LIST3
    -- Initialize a list with an empty list - fails
    local list3 = pandoc.BulletList({ })
    -- local list3 = pandoc.OrderedList({ })
    list3.content:insert(pandoc.Plain "a")
    list3.content:insert(pandoc.Plain "b")
    print("list 3", pandoc.json.encode(list3))

    -- LIST4
    -- Initialize a list with a single item, then remove it - fails
    local list4 = pandoc.BulletList(pandoc.Plain "x")
    -- local list4 = pandoc.OrderedList(pandoc.Plain "x")
    list4.content:remove()
    list4.content:insert(pandoc.Plain "a")
    list4.content:insert(pandoc.Plain "b")
    print("list 4", pandoc.json.encode(list4))

    -- LIST5
    -- Initialize a list with a nil - throws an exception
    -- local list5 = pandoc.BulletList()
    -- local list5 = pandoc.OrderedList()

    return pandoc.Pandoc({
        list1,
        pandoc.HorizontalRule(),
        list2,
        pandoc.HorizontalRule(),
        list3,
        pandoc.HorizontalRule(),
        list4,
    })
end

The output is:

list 1  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"}],[{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
list 2  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"}],[{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
list 3  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"},{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
list 4  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"},{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}

-   a
-   b

------------------------------------------------------------------------

-   a
-   b

------------------------------------------------------------------------

-   a
    b

------------------------------------------------------------------------

-   a
    b

Pandoc version?

pandoc 3.1.3
Features: +server +lua
Scripting engine: Lua 5.4
OS: Linux

[StephanMeijer]

pandoc 3.1.4
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /Users/steve/.local/share/pandoc
Copyright (C) 2006-2023 John MacFarlane. Web: https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

Pandoc 3.1.4 output:

list 1  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"}],[{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
list 2  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"}],[{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
list 3  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"},{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
list 4  {"c":[[{"c":[{"c":"a","t":"Str"}],"t":"Plain"},{"c":[{"c":"b","t":"Str"}],"t":"Plain"}]],"t":"BulletList"}
-   a
-   b

------------------------------------------------------------------------

-   a
-   b

------------------------------------------------------------------------

-   a
    b

------------------------------------------------------------------------

-   a
    b

[tarleb]

This is indeed very counterintuitive behavior, but I don't consider it a bug.

When pandoc retrieves the list from Lua, it expects a list of items as the contents property, where each item is a list of blocks. Pandoc tries to be as forgiving as possible, and converts a single item to a singleton list. Here it sees a list of blocks and assumes that it's the list of blocks making up the first item.

Using the correct types will ensure that pandoc won't interpret the input differently to what's intended, e.g.

local item = {pandoc.Plain "x"}
local list = pandoc.BulletList{item}
list.content:insert{pandoc.Plain "a"}

Note the use of curly braces instead of parentheses.

[CodeSandwich]

Thank you for the great explanation! It makes sense and I agree that it's not a bug at all, the problem clearly is in my code that abuses Pandoc's generic fault tolerance.

[tarleb]

On second thought, maybe pandoc should do better, and maybe we can adjust the behavior to match your expectations. Let me look into that.

[tarleb]

More arguments to change this:

local content = {pandoc.Plain "one", pandoc.Plain "two"}
local bl1 = pandoc.BulletList(content)
local bl2 = pandoc.BulletList{}
bl2.content = content
print(bl1)
print(bl2)

The lists are different, but should probably be equal. I'll make the change.