Fix a bug that file:read(standard_io, ..) unexpectedly returns eof

[sile]

This PR fixes a bug that file:read(standard_io, ..) call in escript could unexpectedly return eof if io:setopts(standard_io, [binary]) is set.
This bug seems introduced by 72a5fc1 that replaced user module by group module.

Bug reproduce code

Creates an escript file that calls file:read/2 twice.

#!/usr/bin/env escript
%% filename: foo.erl

main([]) ->
    ok = io:setopts(standard_io, [binary]),
    io:format("1st: ~p", [file:read(standard_io, 3)]),
    io:format("2nd: ~p", [file:read(standard_io, 3)]).

Starts Erlang shell and invokes the above escript via open_port/2.
After erlang:port_command(P, ["foo"]). is called, the second file:read/2 call in the escript file will return falsey eof.

$ chmod +x foo.erl
$ erl
Erlang/OTP 26 [RELEASE CANDIDATE 1] [erts-13.1.4] [source-ea903ec2bb] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Eshell V13.1.4 (press Ctrl+G to abort, type help(). for help)
1> P = open_port({spawn, "./foo.erl"}, []).                                 
#Port<0.3>
2> erlang:port_command(P, ["foo"]).        
true
3> flush().
Shell got {#Port<0.3>,{data,"1st: {ok,<<\"foo\">>}"}}
Shell got {#Port<0.3>,{data,"2nd: eof"}}   % Unexpected falsey eof
ok
4> erlang:port_command(P, ["bar"]).
** exception error: bad argument
     in function  port_command/2
        called as port_command(#Port<0.3>,["bar"])

By applying the change in this PR, the escript behaves as follows:

$ erl
Erlang/OTP 26 [RELEASE CANDIDATE 1] [erts-13.1.4] [source-ea903ec2bb] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Eshell V13.1.4 (press Ctrl+G to abort, type help(). for help)
1> P = open_port({spawn, "./foo.erl"}, []).
#Port<0.3>
2> erlang:port_command(P, ["foo"]).        
true
3> flush().
Shell got {#Port<0.3>,{data,"1st: {ok,<<\"foo\">>}"}}
ok
4> erlang:port_command(P, ["bar"]).
true
5> flush().
Shell got {#Port<0.3>,{data,"2nd: {ok,<<\"bar\">>}"}}
ok

[github-actions]

CT Test Results

       3 files     142 suites   1h 39m 36s ⏱️
3 190 tests 2 919 ✔️ 269 💤 2 ❌
3 691 runs  3 373 ✔️ 316 💤 2 ❌

For more details on these failures, see this check.

Results for commit 3139ec6.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[sile]

Additional context

I ran into this bug when trying to use erlang_ls with OTP 26-rc1.

[sile]

Although Test Erlang/OTP (kernel) and Test Erlang/OTP (stdlib) checks still fail, let me mark this PR as review ready as those failures seems not relevant to this PR.

I tried running the failed test cases on my laptop (m1 macbook pro) as follows, but it succeeded:

$ gh repo clone otp
$ cd otp
$ gh pr checkout 6881
$ export ERL_TOP=`pwd`
$ ./otp_build setup -a
$ make stdlib_test ARGS="-suite peer_SUITE"
$ make kernel_test ARGS="-suite global_SUITE"

Besides, I created a branch (master-for-stdlib-kernek-ci) from otp/master and added only extra newlines to lib/stdlib/src/io_lib.erl and lib/kernel/src/group.erl. There were no actual meaningful diff with the otp/master but the CI failed by the same reasons with this PR (CI result).

[jhogberg]

Thanks for the PR, and don't sweat the failed testcases, they're a bit unstable on GitHub actions. I've added it to our nightly runs :)

[jhogberg]

Merged, thanks again for the PR!

[sile]

🎉

[garazdawi]

@sile I had to revert your commit as it cause problems in other situations. I've done a solution that fixes the problem in the example you posted in #7384 . Can you check if it also solves your original problem?

[sile]

@garazdawi Thank you for fixing my mistake. But unfortunately, it seems #7384 doesn't solve my original problem.

The issue at hand is that the erlang_ls process crashes immediately after attempting to open an Erlang file using Emacs.
Let me share commands to reproduce that:

// Build Erlang/OTP using #7384
$ gh repo clone erlang/otp
$ cd otp
$ gh pr checkout 7384
$ ./configure
$ make
$ sudo make install
$ cd ../

// Build erlang_ls
$ gh repo clone erlang-ls/erlang_ls
$ cd erlang_ls
$ git checkout 0.47.1
$ make
$ sudo make install

// Open an Erlang file (NOTE: you need to have been set up .emacs file to enable erlang_ls when opening Erlang files)
$ emacs /path/to/some-erlang-file.erl
// The erlang_ls process crashes. The following error messages are shown in the *erlang-ls::stderr* Emacs buffer.
[2023-06-09T21:54:35.068498+09:00] [error] Supervisor: {<0.64.0>,user_sup}. Context: child_terminated. Reason: {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}. Offender: pid=<0.69.0>,mod=user_sup. [supervisor_bridge:report_error/3 L152] <0.64.0>
[2023-06-09T21:54:35.068481+09:00] [error] Error in process <0.69.0> with exit value:, {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]} [ L] <0.69.0>
[2023-06-09T21:54:35.072422+09:00] [error] Generic server <0.64.0> terminating. Reason: {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}. Last message: {'EXIT',<0.69.0>,{function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}}. State: {state,user_sup,undefined,<0.69.0>,{<0.64.0>,user_sup}}. [gen_server:error_info/8 L1373] <0.64.0>
[2023-06-09T21:54:35.072692+09:00] [error] crasher: initial call: supervisor_bridge:user_sup/1, pid: <0.64.0>, registered_name: [], exit: {{function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,1208}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [kernel_sup,<0.47.0>], message_queue_len: 0, messages: [], links: [<0.49.0>], dictionary: [], trap_exit: true, status: running, heap_size: 6772, stack_size: 28, reductions: 39347; neighbours: [proc_lib:crash_report/4 L584] <0.64.0>
[2023-06-09T21:54:35.072935+09:00] [error] Supervisor: {local,kernel_sup}. Context: child_terminated. Reason: {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}. Offender: id=user,pid=<0.64.0>. [supervisor:do_restart/3 L759] <0.49.0>

When I built Erlang/OTP using OTP-26.0 branch, the problem didn't appear.

[garazdawi]

What Emacs and OS versions are you using? Could you show the contents of the .emacs file?

I followed your steps on Ubuntu using Emacs 27.1 and the config example provided by Erlang-LS and could not reproduce the crash.

[sile]

Hmm...

My OS and Emacs versions are as follows:

• OS: macOS Ventura 13.4
• Emacs: GNU Emacs 28.2

The following is the .emacs file to be needed to reproduce the crash in my environment:

(require 'package)

(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)
(require 'use-package)

(add-hook 'erlang-mode-hook #'lsp)

I'll re-investigate this issue in more detail tomorrow 🙏

[sile]

@garazdawi Let me share another reproduce procedure that directly run erlang_ls command (without editors).

1. Save the first JSON RPC request for LSP with the namelsp.jsonrpc

Content:

Content-Length: 3002

{"jsonrpc":"2.0","method":"initialize","params":{"processId":2934,"rootPath":"/path/to/app","clientInfo":{"name":"emacs","version":"GNU Emacs 28.2 (build 1, aarch64-apple-darwin22.1.0)\n of 2022-10-22"},"rootUri":"file:///path/to/app","capabilities":{"general":{"positionEncodings":["utf-32","utf-16"]},"workspace":{"workspaceEdit":{"documentChanges":true,"resourceOperations":["create","rename","delete"]},"applyEdit":true,"symbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"executeCommand":{"dynamicRegistration":false},"didChangeWatchedFiles":{"dynamicRegistration":true},"workspaceFolders":true,"configuration":true,"codeLens":{"refreshSupport":true},"fileOperations":{"didCreate":false,"willCreate":false,"didRename":true,"willRename":true,"didDelete":false,"willDelete":false}},"textDocument":{"declaration":{"dynamicRegistration":true,"linkSupport":true},"definition":{"dynamicRegistration":true,"linkSupport":true},"references":{"dynamicRegistration":true},"implementation":{"dynamicRegistration":true,"linkSupport":true},"typeDefinition":{"dynamicRegistration":true,"linkSupport":true},"synchronization":{"willSave":true,"didSave":true,"willSaveWaitUntil":true},"documentSymbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]},"hierarchicalDocumentSymbolSupport":true},"formatting":{"dynamicRegistration":true},"rangeFormatting":{"dynamicRegistration":true},"onTypeFormatting":{"dynamicRegistration":true},"rename":{"dynamicRegistration":true,"prepareSupport":true},"codeAction":{"dynamicRegistration":true,"isPreferredSupport":true,"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["","quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"resolveSupport":{"properties":["edit","command"]},"dataSupport":true},"completion":{"completionItem":{"snippetSupport":false,"documentationFormat":["markdown","plaintext"],"resolveAdditionalTextEditsSupport":true,"insertReplaceSupport":true,"deprecatedSupport":true,"resolveSupport":{"properties":["documentation","detail","additionalTextEdits","command"]},"insertTextModeSupport":{"valueSet":[1,2]}},"contextSupport":true,"dynamicRegistration":true},"signatureHelp":{"signatureInformation":{"parameterInformation":{"labelOffsetSupport":true}},"dynamicRegistration":true},"documentLink":{"dynamicRegistration":true,"tooltipSupport":true},"hover":{"contentFormat":["markdown","plaintext"],"dynamicRegistration":true},"foldingRange":{"dynamicRegistration":true},"selectionRange":{"dynamicRegistration":true},"callHierarchy":{"dynamicRegistration":false},"typeHierarchy":{"dynamicRegistration":true},"publishDiagnostics":{"relatedInformation":true,"tagSupport":{"valueSet":[1,2]},"versionSupport":true},"linkedEditingRange":{"dynamicRegistration":true}},"window":{"workDoneProgress":true,"showDocument":{"support":true}}},"initializationOptions":null,"workDoneToken":"1"},"id":1}

2. Run erlang_ls command and give the above file to the standard input

// Using OTP-26.0
$ cat lsp.jsonrpc | erlang_ls
Content-Length: 178

{"params":{"message":"[2023-06-13T11:25:26.024900+09:00] [info] Started erlang_ls server [erlang_ls:main/1 L32] <0.9.0>\n","type":4},"jsonrpc":"2.0","method":"window/logMessage"}Content-Length: 977

{"id":1,"result":{"capabilities":{"callHierarchyProvider":true,"codeActionProvider":true,"codeLensProvider":{"resolveProvider":false},"completionProvider":{"resolveProvider":true,"triggerCharacters":[":","#","?",".","-","\"","{"," "]},"definitionProvider":true,"documentFormattingProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"executeCommandProvider":{"commands":["84757:server-info","84757:ct-run-test","84757:show-behaviour-usages","84757:suggest-spec","84757:function-references","84757:add-behaviour-callbacks"]},"foldingRangeProvider":true,"hoverProvider":true,"implementationProvider":true,"referencesProvider":true,"renameProvider":{"prepareProvider":true},"semanticTokensProvider":{"full":false,"range":false,"legend":{"tokenModifiers":[],"tokenTypes":[]}},"textDocumentSync":{"save":{"includeText":false},"change":2,"openClose":true},"workspaceSymbolProvider":true},"serverInfo":{"name":"Erlang LS","version":"0.47.1"}},"jsonrpc":"2.0"}Content-Length: 236

{"params":{"message":"The current project is missing an erlang_ls.config file. Need help configuring Erlang LS for your project? Visit: https://erlang-ls.github.io/configuration/","type":2},"jsonrpc":"2.0","method":"window/showMessage"}Content-Length: 202

{"params":{"message":"[2023-06-13T11:25:26.050106+09:00] [info] OTP Path: \"/usr/local/lib/erlang\" [els_config:do_initialize/4 L135] <0.152.0>\n","type":4},"jsonrpc":"2.0","method":"window/logMessage"}Content-Length: 235

{"params":{"message":"[2023-06-13T11:25:26.085050+09:00] [info] Excluded OTP Applications: [\"megaco\",\"diameter\",\"snmp\",\"wx\"] [els_config:do_initialize/4 L151] <0.152.0>\n","type":4},"jsonrpc":"2.0","method":"window/logMessage"}Content-Length: 287

{"params":{"message":"[2023-06-13T11:25:26.085165+09:00] [info] Config=#{\"apps_dirs\" => [\"_build/default/lib/*\"],\"include_dirs\" => [\"_build/default/lib/*/include\",\"include\"]} [els_config:do_initialize/4 L221] <0.152.0>\n","type":4},"jsonrpc":"2.0","method":"window/logMessage"}Content-Length: 205

{"params":{"message":"[2023-06-13T11:25:26.157159+09:00] [info] Language server stopping... [els_general_provider:handle_request/1 L100] <0.152.0>\n","type":4},"jsonrpc":"2.0","method":"window/logMessage"}

// Using PR 7384 branch
$ cat lsp.jsonrpc | erlang_ls
[2023-06-13T11:24:14.540626+09:00] [error] Error in process <0.69.0> with exit value:, {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]} [ L] <0.69.0>
[2023-06-13T11:24:14.540644+09:00] [error] Supervisor: {<0.64.0>,user_sup}. Context: child_terminated. Reason: {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}. Offender: pid=<0.69.0>,mod=user_sup. [supervisor_bridge:report_error/3 L152] <0.64.0>
[2023-06-13T11:24:14.544212+09:00] [error] Generic server <0.64.0> terminating. Reason: {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}. Last message: {'EXIT',<0.69.0>,{function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}}. State: {state,user_sup,undefined,<0.69.0>,{<0.64.0>,user_sup}}. [gen_server:error_info/8 L1373] <0.64.0>
[2023-06-13T11:24:14.544418+09:00] [error] crasher: initial call: supervisor_bridge:user_sup/1, pid: <0.64.0>, registered_name: [], exit: {{function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,1208}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}, ancestors: [kernel_sup,<0.47.0>], message_queue_len: 0, messages: [], links: [<0.49.0>], dictionary: [], trap_exit: true, status: running, heap_size: 6772, stack_size: 28, reductions: 39337; neighbours: [proc_lib:crash_report/4 L584] <0.64.0>
[2023-06-13T11:24:14.544637+09:00] [error] Supervisor: {local,kernel_sup}. Context: child_terminated. Reason: {function_clause,[{group,edit_line,[<<>>,[]],[{file,"group.erl"},{line,888}]},{group,get_line_echo_off,4,[{file,"group.erl"},{line,827}]},{group,get_chars_loop,10,[{file,"group.erl"},{line,502}]},{group,io_request,6,[{file,"group.erl"},{line,190}]},{group,server_loop,3,[{file,"group.erl"},{line,116}]}]}. Offender: id=user,pid=<0.64.0>. [supervisor:do_restart/3 L759] <0.49.0>

Note: The crash was able to be reproduced in Ubuntu 22.04 in addition to macOS.

[garazdawi]

Managed to reproduce the error with this example. I did a fix in my PR, can you check if it works for you?

[sile]

The latest commit fixed my problem. Thank you!

[garazdawi]

Great!