Repository navigation
Building Filee's own document engine: the story so far (with pictures) #891
Replies: 9 comments
Update: what landed since this postSix more pieces are in the PDF compression: fonts subset again, object streams (#895)Compress PDF now shrinks the fonts a PDF embeds down to the glyphs its pages draw. That helps most with fonts embedded whole, and with pages split off a long document. It also packs the file's objects into object streams. Against Ghostscript's
On PDFs that are already compact, Ghostscript stays 2–12 % smaller, because it rewrites their page content more tightly. So Also try Ghostscript stays as an option. The new engine is a setting now (#899, #900)Settings → Engines → Use the Filee renderer (preview), off by default.
macOS: a false alarm (#896)A font test that skipped on macOS turned out to be a test problem, not a rendering one. macOS reports "no outline" for the space character, which has none. The test now runs on macOS too. RTF without Pandoc (#904)RTF → DOCX, HWPX, PDF, HTML, Markdown is built in now.
Checked with RTF that 한글 itself saved, and with Pandoc's. ODT without Pandoc or LibreOffice (#907)ODT is read and written by Filee itself:
Coming next: an EMF / WMF renderer that works on every platform (clip art in HWP files, Word's EMF pictures), and Filee's own HWP 5.0 reader. |
Update: HWP files and EMF / WMF picturesTwo more pieces are in the Filee reads HWP files itself (#909)
What changes:
To check it, every HWP was compared with the HWPX that 한글 saves from the same file, field by field:
Most of the remaining differences are the document title. When a file has no summary information, 한글's HWPX names it after its first line of text; Filee leaves it empty. The rest is one dropped field result, one distribution document (refused on purpose) and two test artefacts. 21 HWPX files written this way were opened in 한글, and all of them open with the same page count as the original HWP. EMF / WMF pictures on every platform (#908)EMF and WMF are the picture formats of HWP clip art and older Word documents. Until now only Windows could draw them, through GDI+. Filee now has its own EMF / WMF renderer, so they work the same on Windows, Linux and macOS:
Left GDI+, right Filee. The metafiles were recorded on Windows with GDI and GDI+: The one visible difference is the thin dotted line at the top right, which comes out as a grey line. That's on the list. Measured with SSIM (1.0 = identical):
Text scores lowest because GDI snaps letters to the pixel grid and Filee doesn't. Not drawn yet: EMF+ only files (made by GDI+ without an EMF fallback). On Windows those still go to GDI+. Coming next: making the Filee renderer the default, and then removing rhwp. |
Update: Markdown → PowerPoint without PandocFilee now writes PPTX itself (#932, in the The new writer takes anything Filee can read as a document: Markdown, HTML, DOCX, ODT, HWP / HWPX, EPUB, and the Pandoc formats when Pandoc is installed. How the slides are splitThe rules are Pandoc's, so a deck you already make with Pandoc splits the same way:
On a slide: bullet and numbered lists keep their levels, tables get a header row and banding, pictures are scaled to fit (never enlarged), and code goes in a monospace box. When a slide is fullText isn't shrunk to fit. It goes on to a new slide with the same title, like PowerPoint's Continue on a new slide. Lists keep their numbering, and long tables repeat their header row. Checks
Not yet
Coming next: Filee's own slide renderer, so PPTX → PDF / PNG no longer needs LibreOffice. |
Update: old e-books and Kindle output without calibreTwo more pieces are in the Reading CHM, PDB, PML, TCR, RB, SNB and OEB (#931)These formats needed calibre. Filee now reads them itself, so they convert to EPUB, PDF, DOCX, ODT, HWPX, PPTX, TXT and Markdown without calibre.
Three real help files on Windows converted cleanly to EPUB and Markdown: 7-Zip's, PuTTY's, and Python 3.10's (9 MB, 974 pages, 954 pictures). A Korean CHM built in code, converted to PDF with neither calibre nor rhwp installed: The missing space under the table is a bug in the Filee renderer, not in the CHM reader. It's on the list for the renderer work. Writing AZW3 and MOBI (#937)Filee now writes Kindle books itself, from the same XHTML it uses for EPUB. A book gets:
Every book Filee writes is read back with Filee's own MOBI reader and compared with the EPUB of the same book.
These AZW3 and MOBI files haven't been opened in Kindle Previewer or on a Kindle yet. If you try one, a report in this thread helps. What calibre is still for
If you rank calibre first in the engine order, it still writes AZW3 and MOBI. Coming next: the built-in slide renderer (PPTX → PDF / PNG without LibreOffice). |
Update: Unhwp, H2Orestart and Java are goneThree more pieces are in the HWP / HWPX → TXT, Markdown, HTML with Filee's own code (#929)These conversions used Unhwp, a native library. Filee now reads the file itself and writes the text with the same writers it uses for every other document. Because it's all managed code, it works the same on Windows ARM64 and Linux arm64, where Unhwp needed its own builds. What's better than before:
Measured on 30 documents built in code, before Unhwp was removed:
The last 3 differ on purpose: Filee's TXT leaves out headers, footers, page numbers and tracked deletions, and puts notes at the end. LibreOffice without H2Orestart and Java (#934)LibreOffice used to read HWP through H2Orestart, which needs a Java runtime. Filee now reads HWP itself, so LibreOffice no longer opens HWP files. The one target that still needs another engine is RTF: Filee writes a DOCX first and Pandoc or LibreOffice turns it into RTF.
EMF / WMF inside HWPX on every platform (#938)When Filee writes an HWPX, EMF and WMF pictures are now turned into PNG with Filee's own metafile renderer (#908), on Linux and macOS too. Coming next: Filee's own HWP writer, so HWP output no longer needs rhwp. |
Update: Filee writes HWP files itselfFilee now saves The writer is the reverse of the HWP reader from #909. It turns the same HWPX package 한글 saves into HWP records, using Hancom's published format document. Everything the package holds is carried over, including what Filee's document model doesn't keep:
Where the format document leaves a value open, it was matched against HWP files that 한글 saved from the same HWPX, record by record. A file Filee wrote, opened in 한글 2024 (left) and drawn by Filee's renderer (right). The picture, the text flowing around it and the text boxes sit where 한글 puts them: Checks
Eight more test documents (tables, tabs, line spacing, several scripts) still need opening in 한글. The PC was too busy for 한글's automation to start. That check comes next. rhwp stays as a fallback for now. Coming next: with HWP reading (#909) and writing both built in, what's left before rhwp can go is making the Filee renderer the default for HWP → PDF. |
Update: less Pandoc, less 7-ZipTwo more pieces are in the Seven markup languages without Pandoc (#936, #939, #943, #947)Filee now reads and writes these itself:
Each converts to and from Markdown, HTML and TXT, and to DOCX, ODT, HWPX, PPTX and PDF. With these in, Pandoc is only needed for LaTeX, AsciiDoc, Typst and man pages. A notebook built in code (left) and the PDF Filee made from it, with neither Pandoc nor LaTeX installed: The spacing around code and under the table in that picture has since been fixed in the renderer (#946, #949). Every reader was fuzzed with random and damaged input, and none failed. The fuzzer also caught a regular expression in Textile that took minutes on a line of 73 dashes; that is fixed. Unpacking archives without 7-Zip (#950)Filee now unpacks the common archive formats inside the app instead of starting 7-Zip:
The same safety checks apply to every format: password-protected entries, paths that leave the folder, links, archive bombs and too many entries are refused. If an archive uses something the built-in reader doesn't handle, 7-Zip tries it next. A refused archive is never handed on. Unpacking 100 MB in 100 files, compared with the bundled 7-Zip (lower is better): ZIP, TAR, TAR.GZ and TAR.XZ are as fast as 7-Zip or close to it. Solid 7z and bzip2 are much slower. For solid 7z archives with many files, Filee still hands the archive to 7-Zip when it's installed. 7-Zip is still needed to write 7z, XZ and BZ2, and to read CAB, UDF disc images, DMG, LHA and ARJ. |
Update: the Filee renderer now beats rhwp on HWP → PDFThe Filee renderer, Filee's own layout engine (Use the Filee renderer (preview)), now draws more of what 한글 prints. Fourteen fixes are in the What it draws now
Measured against 한글's own PDFsEach document was compared with the PDF 한글 prints. The columns are pages that match, line breaks that match, and median SSIM (a picture similarity score, 1.0 = identical):
No document got worse. SSIM can't get close to 1.0 against these PDFs. When 한글 exports to PDF, it draws all 함초롬바탕 text in the bold face, even where the document doesn't ask for bold. That happens in 44 of the 89 PDFs. What's nextFilee is now ahead of rhwp on every measure, so the next step is making the Filee renderer the default for HWP and HWPX → PDF. rhwp can be removed after that. Still missing: 덧말 (ruby text), line numbers, overlapping characters (글자 겹치기), text art, and some equations. |
Update: slides, spreadsheets and Word 97–2003 without LibreOfficeThree more pieces are in the Slides to PDF and pictures (#942)With Use the Filee renderer (preview) on, PPTX, PPTM, POTX and PPSX now go to PDF, PNG and JPG with Filee's own slide renderer. The renderer draws:
A deck with something it doesn't draw yet (shadows and other effects, 3D, WordArt warps, charts without a stored picture) still goes to LibreOffice. Filee on the left, LibreOffice on the right: A 10-slide deck takes 4–6 seconds. LibreOffice takes about a minute when it starts with a fresh profile. Spreadsheets printed the way Excel prints them (#951)XLSX, XLS and ODS now go to PDF with Filee's own print engine, which follows Excel's page logic:
On 12 test workbooks, page counts and page breaks matched values worked out by hand in Excel's units. Compared with LibreOffice, most break the same way. Where they differ, LibreOffice measures its default column width and its row heights in a different way from Excel. Spreadsheets → PDF through LibreOffice stays in the settings for anyone who prefers it. Charts, shapes and comments aren't printed yet. Word 97–2003 documents (#953)
Password-protected files are refused with a message. Word 6.0 / 95 and Works files still go to LibreOffice. Filee's renderer (left) and LibreOffice (right), from the same DOC: Comments, tracked changes and grouped shapes aren't read yet. Coming next: PPT, ODP / ODG and Visio readers on top of the slide renderer. |














Uh oh!
There was an error while loading. Please reload this page.
Building Filee's own document engine: where we are
Filee turns HWP / HWPX (and later DOCX and others) into PDF with rhwp today, and sends harder documents to LibreOffice. Both are separate programs. For the past days we have been building an engine of Filee's own that lays out and draws documents in-process, on every platform Filee runs on.
This post tells the story so far, step by step, with pictures. Your comments are very welcome: which documents break for you today, which features matter most, and anything that looks wrong below.
Note
The engine is not in a release yet. It lives on the
builtin-enginesbranch and is tracked on the Filee: built-in engines board. Today's converters are unchanged until the engine is wired in.How to read the pictures
Each picture shows the same document three times:
Most documents come from the test matrix, which is written in code: there are no sample files in the repository. Hangul's PDFs were made by driving Hangul itself through automation. The rest come from files attached to our issues.
1. The groundwork
PDF output and font subsetting (#683)
The first question was how to write PDF. We tried two ways on one mixed page (Korean, Latin, Arabic, Devanagari):
Both gave about the same size, so we kept A. Filee's own subsetter keeps only the glyphs a document uses:
Fonts, shaping and Unicode (#695)
A way to measure (#693)
Before writing layout code, we built the yardstick:
It also showed where rhwp falls short today:
2. Paragraphs and lines (#834, #839)
Line breaking
Following Hangul's own lines
An HWPX that Hangul saved stores every line of every paragraph, and Hangul draws from those lines when it opens the file. Filee now uses them too, when the fonts are installed.
3. Pages, tables and columns (#840, #842, #845)
Page composer
Tables
The two forms from #49 were the hardest test. Against Hangul's own PDFs:
In the instructor card, rhwp draws the consent sentence and the line under it on top of each other. Filee keeps them apart, as Hangul does. (Personal details in the file were already replaced by its author; the signature is blurred here.)
Columns, footnotes and endnotes
4. Floating objects (#847)
Pictures, shapes and text boxes anchored to the paper, the page, the column or the paragraph go where Hangul puts them. On a probe file with one anchor per page, every anchor lands within 0.7 pt of Hangul's.
rhwp loses the wrap and draws two paragraphs on one line. Filee matches Hangul's export of the same file.
5. Equations (#848)
Hangul's equation script is read the way Hangul reads it, then laid out and drawn:
over,sqrt,root … of,LEFT … RIGHT, matrices,cases, sums and integrals;Two exam papers from our issue files, with 488 and 249 equations, were the test bed.
6. Chinese, Japanese and Korean typography (#849)
#280: a table of contents twice as long
In #280's Word document, every page number of the table of contents wrapped onto a line of its own. Word measures tab stops from the margin, Hangul from the paragraph's indent, and that difference pushed the right tab past the margin. Now the numbers stay on their lines at the right margin.
#237: quotes drawn over the text
#237's document showed Chinese quotes drawn over the next character. Filee draws them in place.
7. Right-to-left text and complex scripts (#861)
Bidirectional text
The full Unicode Bidirectional Algorithm (UAX #9) now runs: embeddings, isolates, numbers and brackets. It passes all 861,948 cases of Unicode's two test files.
Shaping
Arabic joining, Indic conjuncts and vowel signs, and Thai, Lao, Khmer and Myanmar marks all come out shaped (#185, #202, #216).
Lines without spaces
Thai, Lao, Khmer and Myanmar put no spaces between words, so lines need a dictionary to know where a word ends. Filee carries ICU's word lists, about 500 KB for the four scripts. Where a document says how to break, Hangul's setting wins:
What's next
Roughly in this order:
.hwpfiles take the same path as HWPX.Not done yet: kashida justification for Arabic, vertical text, CJK punctuation compression, and Word's document grid.
We'd love your input
All reactions