Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache::FastMmap segfaults with files larger than 4GB or so #11

Closed
toddr opened this issue Apr 5, 2018 · 3 comments
Closed

Cache::FastMmap segfaults with files larger than 4GB or so #11

toddr opened this issue Apr 5, 2018 · 3 comments

Comments

@toddr
Copy link
Contributor

toddr commented Apr 5, 2018

https://rt.cpan.org/Public/Bug/Display.html?id=92518

@autarch opened on Mon Jan 27 16:11:25 2014

I'm guessing this is because all the internals are based on 32-bit integers.?
@robmueller
Copy link
Owner

Yes. I'd like to update it so everything is 64bit, but just haven't had time to work on it recently.

@toddr
Copy link
Contributor Author

toddr commented Apr 6, 2018

@robmueller thanks for the response. Just didn't want to get this lost in RT.

In the mean time, I submitted #12 so people will know where to report issues going forward.

@robmueller
Copy link
Owner

So I was just looking at the code again today, and realised that I didn't need to make everything 64bit, just the file offsets and cache size could be 64bit, while offsets within each page could be 32bit just fine (limiting page sizes to 4G seems fine for now).

So I've just uploaded a new version 1.48 to cpan which does this.

oschwald added a commit to oschwald/cache-fastmmap that referenced this issue May 22, 2019
@oschwald
Fix race condition when creating file
ba1be41 
This fixes a race condition when creating a new file with multiple
processes that could result in a SIGBUS, e.g.:

```
    Program terminated with signal SIGBUS, Bus error.
    #0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:141
    robmueller#1  0x00007f9b476c1bf9 in _mmc_init_page (cache=0x562e26507730, p_cur=1807) at mmap_cache.c:1077
    robmueller#2  0x00007f9b476c1f2d in mmc_init (cache=0x562e26507730) at mmap_cache.c:180
    robmueller#3  0x00007f9b476c0633 in XS_Cache__FastMmap_fc_init (cv=<optimized out>) at FastMmap.c:252
    robmueller#4  0x0000562e1d78e718 in Perl_pp_entersub () at pp_hot.c:5232
    robmueller#5  0x0000562e1d784e63 in Perl_runops_standard () at run.c:41
    robmueller#6  0x0000562e1d705924 in Perl_call_sv (sv=sv@entry=0x562e1f862788, flags=flags@entry=13) at perl.c:3021
    robmueller#7  0x0000562e1d70800e in Perl_call_list (oldscope=oldscope@entry=2, paramList=0x562e1f8628d8) at perl.c:5061
    robmueller#8  0x0000562e1d6e8fb1 in S_process_special_blocks (floor=floor@entry=45, fullname=<optimized out>, fullname@entry=0x562e1f86d930 "BEGIN",
        cv=cv@entry=0x562e1f862788, gv=<optimized out>) at op.c:10350
    robmueller#9  0x0000562e1d6ff856 in Perl_newATTRSUB_x (floor=floor@entry=45, o=<optimized out>, proto=<optimized out>, proto@entry=0x0, attrs=<optimized out>,
        attrs@entry=0x0, block=0x562e1f86d840, block@entry=0x562e1f86d8c0, o_is_gv=o_is_gv@entry=false) at op.c:10275
    robmueller#10 0x0000562e1d7026d6 in Perl_utilize (aver=<optimized out>, floor=45, version=<optimized out>, idop=0x562e1f86c748, arg=<optimized out>) at op.c:7484
    robmueller#11 0x0000562e1d735e1c in Perl_yyparse (gramtype=gramtype@entry=258) at perly.y:329
    robmueller#12 0x0000562e1d70b83a in S_parse_body (xsinit=0x562e1d6e7230 <xs_init>, env=0x0) at perl.c:2509
    robmueller#13 perl_parse (my_perl=<optimized out>, xsinit=0x562e1d6e7230 <xs_init>, argc=<optimized out>, argv=<optimized out>, env=0x0) at perl.c:1803
    robmueller#14 0x0000562e1d6e7066 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:121
```

This happens as one process could create the file and be filling it with
0s while another process would memory map and try to initialize the
file.

This also addresses races during the file size check.

This does not address the race during initialization of the pages.
Although those should not result in segfaults, we would ideally lock the
file until it was fully initialized. I avoided doing this primarily
because I didn't want to deal with the win32 code.
oschwald added a commit to oschwald/cache-fastmmap that referenced this issue May 22, 2019
@oschwald
Fix race condition when creating file
7a83754 
This fixes a race condition when creating a new file with multiple
processes that could result in a SIGBUS, e.g.:

```
    Program terminated with signal SIGBUS, Bus error.
    #0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:141
    robmueller#1  0x00007f9b476c1bf9 in _mmc_init_page (cache=0x562e26507730, p_cur=1807) at mmap_cache.c:1077
    robmueller#2  0x00007f9b476c1f2d in mmc_init (cache=0x562e26507730) at mmap_cache.c:180
    robmueller#3  0x00007f9b476c0633 in XS_Cache__FastMmap_fc_init (cv=<optimized out>) at FastMmap.c:252
    robmueller#4  0x0000562e1d78e718 in Perl_pp_entersub () at pp_hot.c:5232
    robmueller#5  0x0000562e1d784e63 in Perl_runops_standard () at run.c:41
    robmueller#6  0x0000562e1d705924 in Perl_call_sv (sv=sv@entry=0x562e1f862788, flags=flags@entry=13) at perl.c:3021
    robmueller#7  0x0000562e1d70800e in Perl_call_list (oldscope=oldscope@entry=2, paramList=0x562e1f8628d8) at perl.c:5061
    robmueller#8  0x0000562e1d6e8fb1 in S_process_special_blocks (floor=floor@entry=45, fullname=<optimized out>, fullname@entry=0x562e1f86d930 "BEGIN",
        cv=cv@entry=0x562e1f862788, gv=<optimized out>) at op.c:10350
    robmueller#9  0x0000562e1d6ff856 in Perl_newATTRSUB_x (floor=floor@entry=45, o=<optimized out>, proto=<optimized out>, proto@entry=0x0, attrs=<optimized out>,
        attrs@entry=0x0, block=0x562e1f86d840, block@entry=0x562e1f86d8c0, o_is_gv=o_is_gv@entry=false) at op.c:10275
    robmueller#10 0x0000562e1d7026d6 in Perl_utilize (aver=<optimized out>, floor=45, version=<optimized out>, idop=0x562e1f86c748, arg=<optimized out>) at op.c:7484
    robmueller#11 0x0000562e1d735e1c in Perl_yyparse (gramtype=gramtype@entry=258) at perly.y:329
    robmueller#12 0x0000562e1d70b83a in S_parse_body (xsinit=0x562e1d6e7230 <xs_init>, env=0x0) at perl.c:2509
    robmueller#13 perl_parse (my_perl=<optimized out>, xsinit=0x562e1d6e7230 <xs_init>, argc=<optimized out>, argv=<optimized out>, env=0x0) at perl.c:1803
    robmueller#14 0x0000562e1d6e7066 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:121
```

This happens as one process could create the file and be filling it with
0s while another process would memory map and try to initialize the
file.

This also addresses races during the file size check.

This does not address the race during initialization of the pages.
Although those should not result in segfaults, we would ideally lock the
file until it was fully initialized. I avoided doing this primarily
because I didn't want to deal with the win32 code.
oschwald added a commit to oschwald/cache-fastmmap that referenced this issue May 23, 2019
@oschwald
Fix race condition when creating file
76d3f70 
This fixes a race condition when creating a new file with multiple
processes that could result in a SIGBUS, e.g.:

```
    Program terminated with signal SIGBUS, Bus error.
    #0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:141
    robmueller#1  0x00007f9b476c1bf9 in _mmc_init_page (cache=0x562e26507730, p_cur=1807) at mmap_cache.c:1077
    robmueller#2  0x00007f9b476c1f2d in mmc_init (cache=0x562e26507730) at mmap_cache.c:180
    robmueller#3  0x00007f9b476c0633 in XS_Cache__FastMmap_fc_init (cv=<optimized out>) at FastMmap.c:252
    robmueller#4  0x0000562e1d78e718 in Perl_pp_entersub () at pp_hot.c:5232
    robmueller#5  0x0000562e1d784e63 in Perl_runops_standard () at run.c:41
    robmueller#6  0x0000562e1d705924 in Perl_call_sv (sv=sv@entry=0x562e1f862788, flags=flags@entry=13) at perl.c:3021
    robmueller#7  0x0000562e1d70800e in Perl_call_list (oldscope=oldscope@entry=2, paramList=0x562e1f8628d8) at perl.c:5061
    robmueller#8  0x0000562e1d6e8fb1 in S_process_special_blocks (floor=floor@entry=45, fullname=<optimized out>, fullname@entry=0x562e1f86d930 "BEGIN",
        cv=cv@entry=0x562e1f862788, gv=<optimized out>) at op.c:10350
    robmueller#9  0x0000562e1d6ff856 in Perl_newATTRSUB_x (floor=floor@entry=45, o=<optimized out>, proto=<optimized out>, proto@entry=0x0, attrs=<optimized out>,
        attrs@entry=0x0, block=0x562e1f86d840, block@entry=0x562e1f86d8c0, o_is_gv=o_is_gv@entry=false) at op.c:10275
    robmueller#10 0x0000562e1d7026d6 in Perl_utilize (aver=<optimized out>, floor=45, version=<optimized out>, idop=0x562e1f86c748, arg=<optimized out>) at op.c:7484
    robmueller#11 0x0000562e1d735e1c in Perl_yyparse (gramtype=gramtype@entry=258) at perly.y:329
    robmueller#12 0x0000562e1d70b83a in S_parse_body (xsinit=0x562e1d6e7230 <xs_init>, env=0x0) at perl.c:2509
    robmueller#13 perl_parse (my_perl=<optimized out>, xsinit=0x562e1d6e7230 <xs_init>, argc=<optimized out>, argv=<optimized out>, env=0x0) at perl.c:1803
    robmueller#14 0x0000562e1d6e7066 in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:121
```

This happens as one process could create the file and be filling it with
0s while another process would memory map and try to initialize the
file.

This also addresses races during the file size check.

This does not address the race during initialization of the pages.
Although those should not result in segfaults, we would ideally lock the
file until it was fully initialized. I avoided doing this primarily
because I didn't want to deal with the win32 code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@toddr @robmueller