In LuaLaTeX RTL mode, empty lines appear before every paragraph

[zauguin]

Brief outline of the bug

When LuaLaTeX is used and \textdir is not set to the default value, an empty paragraph is added in front of every normal paragraph, leading to empty lines being added before every paragraph. This is caused by LuaTeX inserting direction nodes into paragraphs to set the text direction, therefore the paragraph isn't empty when \par is called during \everypar.

Minimal example showing the bug

\RequirePackage{latexbug}       % <--should be always the first line (see CONTRIBUTING)!
\documentclass{article}

\showoutput

\begin{document}

\textdir TRT
test

test

\end{document}

Log file (required) and possibly PDF file

Logfile

This is LuaHBTeX, Version 1.14.0 (TeX Live 2022/dev)  (format=lualatex-dev 2021.6.21)  21 JUN 2021 01:59
 restricted system commands enabled.
**testrtl.tex
(./testrtl.tex
LaTeX2e <2021-11-15> pre-release-1 (develop 2021-6-21 branch)
Lua module: luaotfload 2021-05-21 3.19-dev Lua based OpenType font support
Lua module: lualibs 2021-05-20 2.74 ConTeXt Lua standard libraries.
Lua module: lualibs-extended 2021-05-20 2.74 ConTeXt Lua libraries -- extended c
ollection.
luaotfload | conf : Root cache directory is "/home/marcel/.texlive2021/texmf-var
/luatex-cache/generic-dev/names".
luaotfload | init : Loading fontloader "fontloader-2021-05-20.lua" from kpse-res
olved path "/home/marcel/texmf/tex/latex-dev/luaotfload/fontloader-2021-05-20.lu
a".
Lua-only attribute luaotfload@noligature = 1
Lua-only attribute luaotfload@syllabe = 2
luaotfload | init : Context OpenType loader version 3.116
Inserting `luaotfload.node_processor' at position 1 in `pre_linebreak_filter'.
Inserting `luaotfload.node_processor' at position 1 in `hpack_filter'.
Inserting `luaotfload.glyph_stream' at position 1 in `glyph_stream_provider'.
Inserting `luaotfload.define_font' at position 1 in `define_font'.
Lua-only attribute luaotfload_color_attribute = 3
luaotfload | conf : Root cache directory is "/home/marcel/.texlive2021/texmf-var
/luatex-cache/generic-dev/names".
Inserting `luaotfload.harf.strip_prefix' at position 1 in `find_opentype_file'.
Inserting `luaotfload.harf.strip_prefix' at position 1 in `find_truetype_file'.
Inserting `luaotfload.harf.finalize_vlist' at position 1 in `post_linebreak_filt
er'.
Inserting `luaotfload.harf.finalize_hlist' at position 2 in `hpack_filter'.
Inserting `luaotfload.cleanup_files' at position 1 in `wrapup_run'.
Inserting `luaotfload.harf.finalize_unicode' at position 1 in `finish_pdffile'.
Inserting `luaotfload.glyphinfo' at position 1 in `glyph_info'.
Lua-only attribute luaotfload.letterspace_done = 4
Inserting `luaotfload.aux.set_sscale_dimens' at position 1 in `luaotfload.patch_
font'.
Inserting `luaotfload.aux.set_font_index' at position 2 in `luaotfload.patch_fon
t'.
Inserting `luaotfload.aux.patch_cambria_domh' at position 3 in `luaotfload.patch
_font'.
Inserting `luaotfload.aux.fixup_fontdata' at position 1 in `luaotfload.patch_fon
t_unsafe'.
Inserting `luaotfload.aux.set_capheight' at position 4 in `luaotfload.patch_font
'.
Inserting `luaotfload.aux.set_xheight' at position 5 in `luaotfload.patch_font'.
Inserting `luaotfload.rewrite_fontname' at position 6 in `luaotfload.patch_font'
. L3 programming layer <2021-06-18>
Inserting `tracingstacklevels' at position 1 in `input_level_string'.
(/usr/local/texlive/2021/texmf-dist/tex/latex/latexbug/latexbug.sty
Package: latexbug 2021/03/05 v1.0j Bug-classification
)
(/home/marcel/texmf/tex/latex-dev/base/article.cls
Document Class: article 2021/02/12 v1.4n Standard LaTeX document class
(/home/marcel/texmf/tex/latex-dev/base/size10.clo
File: size10.clo 2021/02/12 v1.4n Standard LaTeX file (size option)
luaotfload | db : Font names database loaded from /home/marcel/.texlive2021/texm
f-var/luatex-cache/generic-dev/names/luaotfload-names.luc)
\c@part=\count179
\c@section=\count180
\c@subsection=\count181
\c@subsubsection=\count182
\c@paragraph=\count183
\c@subparagraph=\count184
\c@figure=\count185
\c@table=\count186
\abovecaptionskip=\skip47
\belowcaptionskip=\skip48
\bibindent=\dimen138
)
(/usr/local/texlive/2021/texmf-dist/tex/latex/l3backend/l3backend-luatex.def
File: l3backend-luatex.def 2021-05-07 L3 backend support: PDF output (LuaTeX)
\l__color_backend_stack_int=\count187
\l__pdf_internal_box=\box50
)
(./testrtl.aux)
\openout1 = testrtl.aux

LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 6.
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 6.
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 6.
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 6.
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for TS1/cmr/m/n on input line 6.
LaTeX Font Info:    Trying to load font information for TS1+cmr on input line 6
.
 (/home/marcel/texmf/tex/latex-dev/base/ts1cmr.fd
File: ts1cmr.fd 2019/12/16 v2.5j Standard LaTeX font definitions
)
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for TU/lmr/m/n on input line 6.
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 6.
LaTeX Font Info:    ... okay on input line 6.
LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 6.
LaTeX Font Info:    ... okay on input line 6.


Completed box being shipped out [1]
\vbox(633.0+0.0)x407.0, direction TLT
.\glue 16.0
.\vbox(617.0+0.0)x345.0, shifted 62.0, direction TLT
..\vbox(12.0+0.0)x345.0, glue set 12.0fil, direction TLT
...\glue 0.0 plus 1.0fil
...\hbox(0.0+0.0)x345.0, direction TLT
....\hbox(0.0+0.0)x345.0, direction TLT
..\glue 25.0
..\glue(\lineskip) 0.0
..\vbox(550.0+0.0)x345.0, glue set 503.94617fil, direction TLT
...\write-{}
...\glue(\topskip) 10.0
...\hbox(0.0+0.0)x345.0, glue set 345.0fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\begindir TRT
....\penalty 10000
....\enddir TRT
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0
...\glue(\baselineskip) 5.85
...\hbox(6.15+0.11)x345.0, glue set 313.84fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\begindir TRT
....\hbox(0.0+0.0)x15.0, direction TLT
....\TU/lmr/m/n/10 t
....\TU/lmr/m/n/10 e
....\TU/lmr/m/n/10 s
....\TU/lmr/m/n/10 t
....\penalty 10000
....\enddir TRT
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0 plus 1.0
...\glue(\baselineskip) 11.89
...\hbox(0.0+0.0)x345.0, glue set 345.0fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\begindir TRT
....\penalty 10000
....\enddir TRT
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue(\parskip) 0.0
...\glue(\baselineskip) 5.85
...\hbox(6.15+0.11)x345.0, glue set 313.84fil, direction TLT
....\localpar
.....\localinterlinepenalty=0
.....\localbrokenpenalty=0
.....\localleftbox=null
.....\localrightbox=null
....\begindir TRT
....\hbox(0.0+0.0)x15.0, direction TLT
....\TU/lmr/m/n/10 t
....\TU/lmr/m/n/10 e
....\TU/lmr/m/n/10 s
....\TU/lmr/m/n/10 t
....\penalty 10000
....\enddir TRT
....\glue(\parfillskip) 0.0 plus 1.0fil
....\glue(\rightskip) 0.0
...\glue -0.11
...\glue 0.0 plus 1.0fil
...\glue 0.0
...\glue 0.0 plus 0.0001fil
..\glue(\baselineskip) 23.34
..\hbox(6.66+0.0)x345.0, direction TLT
...\hbox(6.66+0.0)x345.0, glue set 170.0fil, direction TLT
....\glue 0.0 plus 1.0fil
....\TU/lmr/m/n/10 1
....\glue 0.0 plus 1.0fil


{/usr/local/texlive/2021/texmf-var/fonts/map/pdftex/updmap/pdftex.map}
(./testrtl.aux))

Here is how much of LuaTeX's memory you used:
 961 strings out of 478053
 100000,383344 words of node,token memory allocated
 406 words of node memory still in use:
   3 hlist, 1 vlist, 1 rule, 2 glue, 3 kern, 1 glyph, 4 attribute, 48 glue_spec,
 4 attribute_list, 1 write nodes
   avail lists: 1:1,2:27,3:10,4:1,5:30,6:8,7:68,9:26
 21468 multiletter control sequences out of 65536+600000
 23 fonts using 2647927 bytes
 33i,5n,40p,127b,500s stack positions out of 5000i,500n,10000p,200000b,80000s
</usr/local/texlive/2021/texmf-dist/fonts/opentype/public/lm/lmroman10-regular.o
tf>
Output written on testrtl.pdf (1 page, 2880 bytes).

PDF statistics: 15 PDF objects out of 1000 (max. 8388607)
 8 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 131072)
 1 words of extra memory for PDF output out of 10000 (max. 100000000)

Picture of output

[FrankMittelbach]

In my opinion this is a bug in luatex. The TeX engine is pretty clear about what should happen in case of \noindent\par, i.e., that paragraph should be discarded (and the \localpar stuff is handled this way but the \textdir node is not).

Not sure how to best deal with this. Is there anything in luatex to query the state from TeX (or alter it?) or do we need some lua code to drop the node and reinsert it later? My guess that would be best approach unless luatex changes here.
Doing it with simple TeX macros leaves you with some extra box in the stream which is guaranteed to cause trouble later on.
E.g.:

\ExplSyntaxOn \makeatletter
\tl_gset:Nn \g__para_standard_everypar_tl {
  \box_gset_to_last:N \g_para_indent_box
  \group_begin:
% added:  
    \textdir TRT
%-----
    \tex_par:D
  \group_end:
  \tex_everypar:D { \msg_error:nnnn { hooks }{ para-mode }{before}{vertical} }
  \@kernel@before@para@before
  \hook_use:n {para/before}
  \group_begin:
    \tex_everypar:D {}
% changed:    
%    \skip_zero:N \tex_parskip:D
    \tex_parskip:D\dimexpr-\baselineskip\relax
%--------
    \tex_noindent:D
  \group_end:
  \tex_everypar:D{\g__para_standard_everypar_tl}
  \@kernel@before@para@begin
  \hook_use:n {para/begin}
  \if_mode_horizontal: \else:
    \msg_error:nnnn { hooks }{ para-mode }{begin}{vertical} \fi:
  \__para_handle_indent:
}
\ExplSyntaxOff\makeatother

[zauguin]

In my opinion this is a bug in luatex.

I agree, but I think we still have to work around it for now.

Not sure how to best deal with this. Is there anything in luatex to query the state from TeX (or alter it?) or do we need some lua code to drop the node and reinsert it later?

I think we need some Lua code here (see the linked PR), but there is no need to reinsert anything since the dir nodes are automatically reinserted in the new paragraph. We just have to discard them in the empty one.

[FrankMittelbach]

so far so good, but the question is how that get's distributed. I would consider that as first-aid for luatex and put it into first-aid on main and first-aid on dev but not roll out a new patch level for this. Or do you think it is enough to just put it into -dev as you did so far?

[zauguin]

I would consider that as first-aid for luatex and put it into first-aid on main and first-aid on dev but not roll out a new patch level for this.

I agree. I prepared a first-aid version in #594 for main which we can then cherry-pick into develop.