jemalloc issue in ripgrep on aarch64-linux

[andreamah]

From #179121 (comment)
@mycraftmw

2023-04-11 11:01:28.153 [debug] RipgrepSearchEH#search /home/tongyu/.vscode-server-insiders/bin/7f03ec6f7a48791d8027761b22773b454fc52478/node_modules/@vscode/ripgrep/bin/rg --hidden --ignore-case -g '!/.git' -g '!/.svn' -g '!/.hg' -g '!/CVS' -g '!/.DS_Store' -g '!/Thumbs.db' -g '!/node_modules' -g '!/bower_components' -g '!**/*.code-search' --no-ignore-parent --follow --crlf --fixed-strings --no-config --no-ignore-global --json -- 'hello' '.'

• cwd: /home/tongyu/bitswiss
  2023-04-11 11:01:28.161 [debug] RipgrepSearchEH#search : Unsupported system page size
  : Unsupported system page size
  memory allocation of 5 bytes failed

2023-04-11 11:01:28.220 [debug] RipgrepSearchEH#search No data from stdout
2023-04-11 11:01:28.220 [debug] RipgrepSearchEH#search No result from parser
2023-04-11 11:01:28.220 [debug] RipgrepSearchEH#search

Probably a regression from making aarch64-linux ripgrep build statically compiled to fix microsoft/vscode-ripgrep#34. We might need to dynamically compile ripgrep instead of using musl, and find a container to compile it in.

related BurntSushi/ripgrep#2180

// cc @roblourens

[teohhanhui]

Looks like jemalloc just needs to be configured with --with-lg-page to support 16K page size.

This can be achieved by setting the environment variable:

JEMALLOC_SYS_WITH_LG_PAGE=14 (16K page size)

Or to support 64K page size as well:

JEMALLOC_SYS_WITH_LG_PAGE=16 (64K page size)

In either case, it'd still work for 4K/16K(/64K) page sizes.

[mycraftmw]

@andreamah Do we have any good news on this? I found the similar thing happened before on other application like sharkdp/fd#1085, probably we can use the same approach to solve this one?

[erifan]

Any update on this issue ?

[gene203]

Temporary Workaround:

To address the issue until it is fixed, follow these steps:

Install ripgrep and obtain its path. For Rocky Linux or any RHEL-based Linux distribution, run the following command with root privileges:
sudo dnf install -y ripgrep

For Ubuntu, use the following command:
sudo apt install ripgrep

Alternatively, you can install ripgrep using Cargo:
cargo install ripgrep

Verify the installation and retrieve the path by running:
which rg

Replace the broken rg with a symlink by executing the following script. Please note that you should remove the echo commands to commit the changes:
(RIPGREP=/usr/bin/rg; find ~/.vscode-server/bin -name rg | while read file _ ; do echo rm -f $file ; echo ln -sf $RIPGREP $file; done)

This workaround should help resolve the issue temporarily.

[andreamah]

This should be fixed in tomorrow's VS Code Insiders. Please try it out and let me know if it works for you!

[gene203]

I tried

Version: 1.79.0-insder (user setup)
Commit: 6d3febd
Date: 2023-05-19T20:39:14 (about 3 hours ago)
OS: Windows_NT x65 10.0.22621`

but it seems not fixed or not included the fix. execute from the remote server(aarch64)
$ find .vscode-server-insiders -name rg -exec {} \;
and got

: Unsupported system page size
: Unsupported system page size
memory allocation of 5 bytes failed
find: ‘.vscode-server-insiders/bin/6d3fedb7c2cf965915c785dc07701eb978947506/node_modules/@vscode/ripgrep/bin/rg’ terminated by signal 6

(after resetting the remote environment by deleting .vscode-server-insiders folder on the remote, and vscode reloaded. - 'Developer: Reload Window')

[andreamah]

Are you running 16K page size or 64K?

[gene203]

I got

$ getconf PAGE_SIZE
65536

on the aarch64 remote (Rocky Linux 8, aws r7g instance/Graviton3 processors).

Meanwhile, executing the same command above on x86-64 remote, I got the normal rg help print out as like

$ find .vscode-server-insiders/bin -name rg -exec {} ;`
error: The following required arguments were not provided:

USAGE:

rg [OPTIONS] PATTERN [PATH ...]
rg [OPTIONS] -e PATTERN ... [PATH ...]
rg [OPTIONS] -f PATTERNFILE ... [PATH ...]
rg [OPTIONS] --files [PATH ...]
rg [OPTIONS] --type-list
command | rg [OPTIONS] PATTERN
rg [OPTIONS] --help
rg [OPTIONS] --version

For more information try --help

and correcting the hand typed typo above of vscode insider version info

Version: 1.79.0-insider (user setup)
Commit: 6d3fedb
Date: 2023-05-19T20:39:14.598Z
Electron: 22.5.2
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: 10.8.168.25-electron.0
OS: Windows_NT x64 10.0.22621

[andreamah]

Hmm ok. I tried setting the env variable to increase max page size for jemalloc, but it seems to not be working. I'll try another approach.

[RuinanSun]

Is this fixed in the latest release version of VScode?
I just check on my arm64 container but it seems not working.

VScode Version: 1.79.0 (Universal)

[bashimao]

Same with @RuinanSun

Switched to newest insider build. 1.80.0. Problem not solved.

Went to server, killed .vscode server and deleted $HOME/.vscode-server-insiders. Tried again to bootstrap. Problem is still NOT resolved. Search is still limited to open files.

[bashimao]

[bashimao]

Here is some useful link detailing what I think is the problem. Not sure if it is resolved by now. But if info from the discussion is current, I think you will have to either compile on the target system, or ship different precompiled versions of binary and select the correct one depending on the system specs (see jemalloc/jemalloc#467 ).

[andreamah]

We do cross-compile for different systems for ripgrep https://github.com/microsoft/ripgrep-prebuilt. I'll have to take another look.

[bashimao]

So, my interpretation is, that the page-size is hard-coded into jemalloc. When the system page size at runtime differs from the one it is compiled for, it will fail.

In other words, the one shipped right now fails out of the box:

mlanger@localhost:~/.vscode-server-insiders/bin/db6bcdf309cc1565c502a54b73c33a63bdabfbd8/node_modules/@vscode/ripgrep/bin$ ./rg
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted (core dumped)

From your linked repo, I understand that you now focus on 16k page size. That would also not help me (if my suspicion is right), as my system's page size is: 64k

$ getconf PAGESIZE
65536

[bashimao]

@andreamah @RuinanSun : Temporary fix.

1. Bootstrap vscode-server on target ARM system.
   You should have a directory called ~/.vscode-server-insiders now. That is where the vscode server lives.

2. Download and compile ripgrep:

   sudo apt-get install cargo
   git clone https://github.com/BurntSushi/ripgrep.git
   cd ripgrep
   cargo build --release
   

3. Replace binary deployed by vscode-server with custom compiled version.
   VSCODE_RG=`find ~/.vscode-server-insiders -name rg`; rm $VSCODE_RG; ln -s $PWD/target/release/rg $VSCODE_RG

Now search should work as expected.

[deepak1556]

@bashimao can you check if the following ripgrep build runs fine on your aarch64 setup ? The build is generated from microsoft/ripgrep-prebuilt#30 to use 64k for jemalloc.

[RuinanSun]

Thanks @bashimao. ripgrep built from source works for me!

[andreamah]

@RuinanSun @gene203 @bashimao can any of you verify this:

can you check if the following ripgrep build runs fine on your aarch64 setup ? The build is generated from microsoft/ripgrep-prebuilt#30 to use 64k for jemalloc.

[gene203]

I untar-ed the file and the binary 'rg' is working good with remote search. (I sym-linked it with vscode 1.79)

$ tar zxvf ripgrep--aarch64-unknown-linux-musl.tar.gz
rg
$ ls -l rg
-rwxr-xr-x 1 sysadmin sysadmin 4111744 Jun 12 21:34 rg
$ find ~/.vscode-server/bin -name rg -exec ln -sfn ~/rg {} ;

[VSCodeTriageBot]

This bug has been fixed in the latest release of VS Code Insiders!

@andreamah, you can help us out by commenting /verified if things are now working as expected.

If things still don't seem right, please ensure you're on version 45b31e9 of Insiders (today's or later - you can use Help: About in the command palette to check), and leave a comment letting us know what isn't working as expected.

Happy Coding!

[gene203]

I can confirm it is fixed with the vscode insider downloaded a minute ago of

Version: 1.80.0-insider (user setup)
Commit: da7f05a
Date: 2023-06-27T15:13:35.848Z
Electron: 22.3.14
ElectronBuildId: 21893604
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: 10.8.168.25-electron.0
OS: Windows_NT x64 10.0.22621

(assume the commit hash changed since)

❯ find ~/.vscode-server-insiders -name rg -exec {} -V \;
ripgrep 13.0.0

[andreamah]

thanks so much for checking! @gene203