trouble building pandoc-2.9 on armv7 without -O0

[juhp]

It looks like both Debian and Fedora experienced problems recently building pandoc-2.9 on 32bit armv7:

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965121 (for 2.9.1.1)

| [112 of 158] Compiling Text.Pandoc.Writers.Markdown
| ghc: out of memory (requested 2097152 bytes)

• https://koji.fedoraproject.org/koji/taskinfo?taskID=47414930 (2.9.2.1 for Fedora Rawhide)

[101 of 166] Compiling Text.Pandoc.Writers.LaTeX
ghc: out of memory (requested 1048576 bytes)

Anyway it seems possible to build it with -O0 fortunately, but I thought I would report it here
just in case anyone thinks of anything else that could be done to mollify this, thanks.

cc @jonassmedegaard

[mb21]

Thanks! from the core team, nobody builds to ARM... but if somebody figures it out, a pull for #5450 is certainly welcome!

btw. seems your build system is just running out of memory? how much RAM does it have?

[juhp]

seems your build system is just running out of memory?

That's my understanding yes.

how much RAM does it have?

I checked https://kojipkgs.fedoraproject.org//work/tasks/4930/47414930/hw_info.log
and it seems to have 24GB, but I am not intimate with armv7 process memory limitations, etc.

[ickc]

Probably not cross-compiling and hit the 4GiB limit on 32 bit system? Compiling pandoc crosses the 4GiB limit since version ~2.8.

[jgm]

Seems that T.P.Readers.LaTeX is the biggest module and the usual problem with oom errors in compiling. We can try to break it up into smaller modules.

[jgm]

I wrote a small perl script that collects timing info dumped by ghc and produces a table. (in tools, parseTimings.pl). Output is:

 Time (ms) Alloc (Mb) LOC
      2084       3055    256 Text.Pandoc.App.OutputSettings
       459        717    125 Text.Pandoc.Writers.Roff
       304        390     74 Text.Pandoc.Class.PandocIO
       128        200     97 Text.Pandoc.Readers.Odt
       922       1546    225 Text.Pandoc.Readers.HTML.Parsing
        42         68     53 Text.Pandoc.Readers.CslJson
      2886       3512    270 Text.Pandoc.SelfContained
        76        107     68 Text.Pandoc.Lua.Module.Types
     22683      27717    773 Text.Pandoc.Writers.RST
      1015       1593    263 Text.Pandoc.Readers.HTML.Table
      5772       7042   1263 Text.Pandoc.Readers.DocBook
      1623       2563    296 Text.Pandoc.Readers.Org.Meta
      1385       2111    193 Text.Pandoc.XML.Light.Types
       229        418    221 Text.Pandoc.Writers
      7021       7928    357 Text.Pandoc.Options
       184        225     43 Text.Pandoc.Lua.Marshaling.List
       808       1147    250 Text.Pandoc.Lua.Filter
        13         15     24 Text.Pandoc.Lua.Marshaling.AnyValue
     67194      74010   2452 Text.Pandoc.Readers.LaTeX
      4380       5976    294 Text.Pandoc.Writers.LaTeX.Table
      9965      12615    988 Text.Pandoc.Readers.Muse
        72        105     59 Text.Pandoc.Lua.Marshaling.SimpleTable
       128        212     53 Text.Pandoc.Writers.Math
      4570       7014    642 Text.Pandoc.Citeproc
      4745       7139    515 Text.Pandoc.Readers.DokuWiki
      2105       2881    721 Text.Pandoc.Class.PandocMonad
       528        917     79 Text.Pandoc.Writers.OPML
      8488      11641   1574 Text.Pandoc.Parsing
      1658       2412    314 Text.Pandoc.Writers.Jira
       797        661     31 Text.Pandoc.Citeproc.Data
      3157       4644    459 Text.Pandoc.Readers.FB2
      4445       5359    615 Text.Pandoc.Extensions
     10224      13493    337 Text.Pandoc.Writers.Man
       129        225     48 Text.Pandoc.Readers.CSV
       116        164     24 Text.Pandoc.Emoji
      1363       1582    380 Text.Pandoc.Lua.Marshaling.AST
      2118       3156    295 Text.Pandoc.Readers.Creole
       706       1102    248 Text.Pandoc.Writers.JATS.Table
      1208       1916    239 Text.Pandoc.Writers.Ipynb
     25969      31210   2211 Text.Pandoc.Readers.Markdown
       364        628    292 Text.Pandoc.Readers.Org.ParserState
     10842      15157    635 Text.Pandoc.Writers.ICML
      3603       4789    232 Text.Pandoc.Class.IO
      1171       1638    506 Text.Pandoc.Writers.Shared
       210        272    144 Text.Pandoc.UTF8
       114        182    103 Text.Pandoc.Lua.Init
       304        429     37 Text.Pandoc.Readers.LaTeX.Types
       334        528     81 Text.Pandoc.CSV
     14672      18166   1528 Text.Pandoc.Writers.HTML
      3831       5760   1005 Text.Pandoc.Writers.Powerpoint.Presentation
        55         88     50 Text.Pandoc.Slides
       190        264    213 Text.Pandoc.Readers.Odt.Arrows.Utils
       366        421     93 Text.Pandoc.Readers.Odt.Namespaces
        20         30     35 Text.Pandoc.Filter.Path
      6832       7948   1023 Text.Pandoc.Shared
       314        464    117 Text.Pandoc.Templates
      7003       6850   1070 Text.Pandoc.App.CommandLineOptions
      6560       9362    556 Text.Pandoc.Writers.JATS
       180        294     99 Text.Pandoc.Writers.OOXML
       966       1515    322 Text.Pandoc.Readers.Docx.Parse.Styles
      1882       2806    408 Text.Pandoc.Readers.Org.DocumentTree
       546        803    784 Text.Pandoc.Readers.Odt.Generic.XMLConverter
       587        786    218 Text.Pandoc.Class.PandocPure
      7432       9656    272 Text.Pandoc.Writers.Haddock
       104        159    112 Text.Pandoc.Process
       425        571     37 Text.Pandoc.Citeproc.CslJson
       912        911    158 Text.Pandoc.Lua.Walk
       644        964     34 Text.Pandoc.Writers.LaTeX.Notes
     13119      16946    496 Text.Pandoc.Writers.Textile
       132        214     79 Text.Pandoc.Lua.Marshaling.ReaderOptions
        31         47     37 Text.Pandoc.Image
      1068       1692    302 Text.Pandoc.Readers.EPUB
        98        145     61 Text.Pandoc.Lua.ErrorConversion
      5329       7762    711 Text.Pandoc.Readers.Textile
        34         52     83 Text.Pandoc.Class.CommonState
       134        189     56 Text.Pandoc.CSS
      3374       5111    353 Text.Pandoc.Readers.LaTeX.SIunitx
     23562      29610    598 Text.Pandoc.Writers.Ms
       310        526    218 Text.Pandoc.Readers.Org.Parsing
      6275       8874    918 Text.Pandoc.Readers.Org.Inlines
        60         99     47 Text.Pandoc.Filter.Lua
       345        506     81 Text.Pandoc.MediaBag
       982       1520    177 Text.Pandoc.Readers.Haddock
       329        536    208 Text.Pandoc.Readers.Docx.Lists
        98        180     97 Text.Pandoc.Readers.HTML.Types
      3520       4774    323 Text.Pandoc.Writers.AnnotatedTable
     16027      19610    470 Text.Pandoc.Writers.Org
      9604      13372    908 Text.Pandoc.Writers.OpenDocument
      8425      11371   1556 Text.Pandoc.Writers.Docx
       226        405     70 Text.Pandoc.Citeproc.Util
      8080      11373   1910 Text.Pandoc.Writers.Powerpoint.Output
        57         91     61 Text.Pandoc.Writers.BibTeX
       647       1034    190 Text.Pandoc.Readers.Org.ExportSettings
      2559       3510    937 Text.Pandoc.Readers.Odt.ContentReader
     42606      50867   1410 Text.Pandoc.Writers.Markdown
      7884       8857    453 Text.Pandoc.Writers.Docbook
      4901       6985    570 Text.Pandoc.Readers.Txt2Tags
     39793      46866   1565 Text.Pandoc.Writers.LaTeX
        23         26     30 Text.Pandoc.Readers.Odt.Generic.SetMap
       638        987    131 Text.Pandoc.Readers.Org.BlockStarts
       356        375     78 Text.Pandoc.Readers.HTML.TagCategories
       700       1068    175 Text.Pandoc.Lua.Module.Utils
      4778       6593    267 Text.Pandoc.Writers.XWiki
       299        480     80 Text.Pandoc.Readers.Org.Shared
      3748       5326   1057 Text.Pandoc.Readers.Docx.Parse
        86        167     63 Text.Pandoc.Readers.Docx.Util
         3          4     22 Text.Pandoc.Readers.Odt.Base
         4          5     23 Text.Pandoc.Lua
      4684       5905    200 Text.Pandoc.Error
      7565       9832    916 Text.Pandoc.Readers.LaTeX.Parsing
      3967       5084    524 Text.Pandoc.Readers.JATS
       245        477    405 Text.Pandoc.Asciify
       703       1087    100 Text.Pandoc.Filter
         3          4     17 Text.Pandoc.Writers.CommonMark
        83        144    140 Text.Pandoc.XML.Light.Proc
      1083       1584    234 Text.Pandoc.XML.Light.Output
     15561      21454    598 Text.Pandoc.Writers.AsciiDoc
     20603      26156    574 Text.Pandoc.Writers.ConTeXt
        97        132    104 Text.Pandoc.Readers.Odt.Generic.Fallible
     12139      15613    391 Text.Pandoc.Writers.ZimWiki
       367        643    128 Text.Pandoc.Lua.Module.Pandoc
         3          5     29 Text.Pandoc.Class
      8453      11978   1091 Text.Pandoc.Readers.HTML
       547        914    173 Text.Pandoc.Readers.Metadata
      1620       2497    284 Text.Pandoc.Writers.ODT
      1662       2461    373 Text.Pandoc.App
       638        912    421 Text.Pandoc.RoffChar
       367        444    154 Text.Pandoc.Lua.Marshaling.Version
        26         27     45 Text.Pandoc.Readers.Odt.Generic.Namespaces
       132        200    116 Text.Pandoc.Readers.Odt.Generic.Utils
      8829      12660    700 Text.Pandoc.Readers.MediaWiki
      2631       3878     76 Text.Pandoc.Readers.CommonMark
        42         63     71 Text.Pandoc.Readers.BibTeX
      1623       2591    537 Text.Pandoc.MIME
      3829       5696    737 Text.Pandoc.Readers.Docx
       240        358     80 Text.Pandoc.Filter.JSON
       175        225    145 Text.Pandoc.Lua.Util
       172        246     89 Text.Pandoc.Lua.Global
       147        243     89 Text.Pandoc.XML.Light
      5220       7632    399 Text.Pandoc.Writers.RTF
     11672      16239    520 Text.Pandoc.Writers.DokuWiki
        35         39     28 Text.Pandoc.Lua.Marshaling.Context
     13371      15719   1474 Text.Pandoc.Writers.EPUB
        68         84     33 Text.Pandoc.Lua.Module.System
      1277       1953    163 Text.Pandoc.Writers.JATS.References
        83        117     73 Text.Pandoc.Lua.Marshaling.MediaBag
       123        227     49 Text.Pandoc.Readers.Org
      6229       7896     93 Text.Pandoc.Writers.Native
        38         60     47 Text.Pandoc.Writers.Docx.StyleMap
       583        785     74 Text.Pandoc.Readers.Docx.Fields
      7220       9812    921 Text.Pandoc.Readers.Org.Blocks
       256        443    181 Text.Pandoc.Readers
      1133       1598    255 Text.Pandoc.Citeproc.MetaValue
       316        489    188 Text.Pandoc.Readers.Docx.Combine
     10191      12585    393 Text.Pandoc.Logging
       278        392    141 Text.Pandoc.Lua.PandocLua
       120        201     60 Text.Pandoc.Lua.Packages
      2114       3060    638 Text.Pandoc.Readers.Odt.StyleReader
      3374       5056    304 Text.Pandoc.Writers.TEI
       795       1087    216 Text.Pandoc.Highlighting
      1929       2923    228 Text.Pandoc.Readers.Ipynb
        26         37     43 Text.Pandoc.Writers.Powerpoint
      1296       1906    492 Text.Pandoc.XML
      5559       8138    730 Text.Pandoc.Readers.Roff
       743       1136    155 Text.Pandoc.Readers.LaTeX.Lang
     10917      14227   1105 Text.Pandoc.Writers.MediaWiki
     11160      14365    482 Text.Pandoc.Writers.Texinfo
     21925      27161   1677 Text.Pandoc.Readers.RST
     10465      15027    659 Text.Pandoc.App.Opt
        48         94     82 Text.Pandoc.Writers.LaTeX.Types
       171        251    103 Text.Pandoc.Lua.Module.MediaBag
       106        196     61 Text.Pandoc.UUID
        45         62     65 Text.Pandoc.Lua.Marshaling.PandocError
     25635      30444   1466 Text.Pandoc.Citeproc.BibTeX
         5         10     71 Text.Pandoc
      2918       4338    575 Text.Pandoc.Writers.FB2
        68        103     48 Text.Pandoc.Writers.LaTeX.Caption
      1069       1642    230 Text.Pandoc.Writers.Custom
      5006       7593    513 Text.Pandoc.Readers.Man
      7166       9615    666 Text.Pandoc.Readers.Vimwiki
      5682       7440    656 Text.Pandoc.Readers.TikiWiki
       259        391    102 Text.Pandoc.Lua.Marshaling.CommonState
      1401       2291    279 Text.Pandoc.Citeproc.Locator
       206        276     84 Text.Pandoc.App.FormatHeuristics
         3          4     19 Text.Pandoc.Lua.Marshaling
       195        258    139 Text.Pandoc.Readers.Odt.Arrows.State
       869       1158    121 Text.Pandoc.Translations
       527        761     90 Text.Pandoc.Writers.CslJson
      1419       2204    379 Text.Pandoc.ImageSize
      1915       2741    525 Text.Pandoc.PDF
     15091      18719    717 Text.Pandoc.Writers.Muse
       955       1467    199 Text.Pandoc.Readers.Jira
      4050       6007    512 Text.Pandoc.Readers.TWiki
       396        676    107 Text.Pandoc.Readers.OPML
       881       1302    112 Text.Pandoc.BCP47
       666       1168     32 Text.Pandoc.Data
        33         48     54 Text.Pandoc.Readers.Native
        32         51     48 Text.Pandoc.Writers.JATS.Types

[jgm]

The biggest compilation bottlenecks are:

     39793      46866   1565 Text.Pandoc.Writers.LaTeX
     42606      50867   1410 Text.Pandoc.Writers.Markdown
     67194      74010   2452 Text.Pandoc.Readers.LaTeX

If we could focus on these three, we might make some progress.

[jgm]

Note that the "megabytes" field here is just the sum of the megabytes used in various compilation phases for the module; it does not mean that this much memory was required at one time, but it is a good relative "heaviness" measure.

[jgm]

In latest master (2faa57e) I have improved things by further splitting of the LaTeX reader. Now we get

     37754      46851   1565 Text.Pandoc.Writers.LaTeX
     39387      50842   1410 Text.Pandoc.Writers.Markdown
     49976      59906   1872 Text.Pandoc.Readers.LaTeX

Since the original issue noted above affects the LaTeX and Markdown writers as well, we still need further splitting (also of these writers), but this is a step in the right direction.

[jgm]

Latest (6a6291d):

     38073      46702   1563 Text.Pandoc.Writers.LaTeX
     39875      50870   1410 Text.Pandoc.Writers.Markdown
     47762      55771   1681 Text.Pandoc.Readers.LaTeX

It's notable that the LaTeX reader is now around the same size as many other modules that seem much less expensive to compile (e.g. the RST reader, 1677 lines of code, 21528/27159). I wish I understood what features of the expensive modules are making them expensive. @mpickering do you have any insight?

[jgm]

As of 7f1b933:

     38568      46700   1563 Text.Pandoc.Writers.LaTeX
     42981      50852   1410 Text.Pandoc.Writers.Markdown
     46226      49473   1542 Text.Pandoc.Readers.LaTeX

So now the LaTeX reader is right in the same ballpark as the LaTeX and Markdown writers.
I'll be curious to hear whether this is enough to enable compiling on armv7 without -O0.
My guess is that it isn't, since the original post shows oom on the writers.

[jgm]

After further trimming down these big modules (da5e9e5), I can compile pandoc (with optimizations) on a build machine with 2 GB of memory (though I had to restart the build several times as it got killed with oom errors). There is room for further improvement, but I'm hoping these changes will make it easier to build pandoc on low-memory machines.

The most recent numbers from my tool (top three modules):

     23514      30495   1466 Text.Pandoc.Citeproc.BibTeX
     24160      31787   2237 Text.Pandoc.Readers.Markdown
     37421      44369   1344 Text.Pandoc.Readers.LaTeX

[sjakobi]

jgm/doclayout#1 seems relevant to this issue.

[tarleb]

Can this be considered fixed, or should we keep the issue open?

[jgm]

In my experience it still takes a great deal of memory to compile on ARM specifically.
I think the issue may be fixed when ghc comes out with their native code generator for ARM.

[juhp]

Well I think the NCG will only be for arm64 (aarch64) fwiw.

[tarleb]

Most large modules have been partitioned into smaller one. I believe it should be ok to close this now.