"bus error" when running via Rosetta on Apple silicon mac

[niyue]

Problem

When running a x86_64 version of vector via Rosetta on Apple silicon mac, I ran into "bus error" immediately when launching vector (actually my use case is more complex since I built it from source with some modification).

After some binary searching, I found this is caused by this commit 5fe1767. This commit introduced jemalloc for memory allocation, however, jemalloc has some issue when running via Rosetta on Apple silicon mac:

• SIGNAL, Code 10 Bus error: 10 Will running on apple m1 through rosetta 2.  tikv/jemallocator#17
• Illegal instruction with x86-64 jemalloc on Rosetta jemalloc/jemalloc#2014

Since the jemalloc issue (jemalloc/jemalloc#2014) has been there for nearly 2 years, I don't think it will be fixed any time soon. I wonder if there is any possibility to:

1. provide an optional feature for other memory allocators
2. or even better, switch the default memory allocator from jemalloc to other memory allocators

There are some other memory allocator like tcmalloc and mimalloc, which may provide comparable performance.

I've tried switch to mimalloc locally and doesn't run into such issue any more:

• https://github.com/microsoft/mimalloc
• https://docs.rs/mimalloc/latest/mimalloc/ (I used this library locally)

I don't run any performance benchmark so I have no idea how this impact vector's performance currently.

If such an option is acceptable, I am glad to give it a try. Thanks.

Configuration

No response

Version

0.24

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @niyue !

Thanks for this report.

We have previously evaluated the other allocators you mentioned there, but the performance was not as good for Vector's allocation patterns as jemalloc was. It's been a while though, so we could certainly see if that is still the case. If you open a PR swapping the allocator, there is a Github Actions workflow that runs our performance tests and will post the difference.

I'm surprised there hasn't been much movement on jemalloc/jemalloc#2014.

Alternatively, we also have an issue to publish native M1 builds. Would that be sufficient for you? Vector does build under M1, so you could build it locally, we just haven't set up the scaffolding to publish the builds yet.

[spencergilbert]

FWIW, also on M1 and using tagged versions for x86 I've not run into this error - it's possibly related to a specific configuration or change you've made locally?

[niyue]

it's possibly related to a specific configuration or change you've made locally

It is possible since I am not 100% sure how this happens. The binary got loaded and crashed immediately without enter any code path yet (static variable like the memory allocator got initialized I think), and I don't have too much information except the bus error printed on the console (there is not even any crash log I can find)

I found it may be caused by jemalloc because I bisect to find the git commit, and then discovered that jemallocator and jemalloc's issue tracker indeed has related issue.

It's been a while though, so we could certainly see if that is still the case

Is there any place I can get more info about the result we evaluated last time, such as when/what version of allocator was evaluated?

If you open a PR swapping the allocator, there is a Github Actions workflow that runs our performance tests and will post the difference

Thanks for the pointer, I only did some minimal change locally to make it work, and I am not sure how it impacts perf yet, I will give it a try.

we also have #10193. Would that be sufficient for you?

It helps a lot in most cases, but my scenario is like this:
Some of our developers uses x86_64 machines for development, and some of our developers uses apple silicon machines for development. Since x86_64 is our primary target CPU architecture, instead of dispatching all x86_64 issues to developers with x86_64 machines, I would like to make everyone has the capability to fix all issues (for all target CPU architectures), so ideally I would like to make such use case possible (running x86_64 build on apple silicon)

[niyue]

If you open a PR swapping the allocator, there is a Github Actions workflow that runs our performance tests and will post the difference

I've submitted a PR to replace jemalloc with mimalloc, and see what the performance impact will be.

[jszwedko]

I'm leaning towards just using the system allocator on OSX to resolve this given we don't have any performance tests on OSX to motivate using a different allocator.