Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Weird error #89

Closed
scriby opened this Issue Sep 4, 2012 · 19 comments

Comments

Projects
None yet
3 participants

scriby commented Sep 4, 2012

I was verifying some behavior in fibers today, when I ran across a weird error in this sample program. The strange thing is that the error occurs after 120 levels of recursion, but this program only creates a single fiber. I verified this using the Fiber.poolSize and Fiber.createdFibers counters. The other weird thing is even if I change the pool size it still happens at 120 levels of recursion.

Another thing I noticed is if I use flow.add and flow.wait (which creates a new fiber for each call to t2) instead of .sync() , the seg fault doesn't occur, and it recurses until it runs out of memory (which is pretty fast -- still need to investigate this). But the weird thing is that it doesn't seem to be preventing new fibers from being created past the max pool size. But if the pool is for concurrent running fibers, then that would make sense as all but one of the fibers would be yielded.

var asyncblock = require('asyncblock');

if(asyncblock.enableTransform(module, __filename)){
    return;
}

var t2 = function(i, callback){
    console.log(i);

    if(i > 100000){
        return callback();
    }

    asyncblock(function(){
        t2(i + 1).sync();
    }, callback);
};

t2(0, function(){
    console.log('done');
});

This error occurs:

ubuntu@ubuntu:~/sourcecode/folkLOR$ node test.js
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
done
*** glibc detected *** node: double free or corruption (out): 0x00007fb2d00279d0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x78a8f)[0x7fb2d5ecda8f]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x73)[0x7fb2d5ed18e3]
node(_ZN2v88internal22HandleScopeImplementer19FreeThreadResourcesEv+0xd)[0x6c1f0d]
node(_ZN2v88internal13ThreadManager19FreeThreadResourcesEv+0x14)[0x8687e4]
node(_ZN2v86LockerD1Ev+0x5c)[0x868cec]
node(_ZN4node5StartEiPPc+0x1a8)[0x582608]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xff)[0x7fb2d5e73eff]
node[0x579a5d]
======= Memory map: ========
00400000-00baf000 r-xp 00000000 07:00 808414                             /usr/local/bin/node
00dae000-00daf000 r--p 007ae000 07:00 808414                             /usr/local/bin/node
00daf000-00dc0000 rw-p 007af000 07:00 808414                             /usr/local/bin/node
00dc0000-00dca000 rw-p 00000000 00:00 0 
00f1a000-00f88000 rw-p 00000000 00:00 0                                  [heap]
36ab7e00000-36ab7e85000 rw-p 00000000 00:00 0 
66c6cb00000-66c6cc00000 rw-p 00000000 00:00 0 
97f93828000-97f93840000 ---p 00000000 00:00 0 
97f93840000-97f93860000 rw-p 00000000 00:00 0 
97f93860000-97f93888000 ---p 00000000 00:00 0 
11b35bbdf000-11b35bc00000 ---p 00000000 00:00 0 
11b35bc00000-11b35bc05000 rw-p 00000000 00:00 0 
11b35bc05000-11b35bc06000 ---p 00000000 00:00 0 
11b35bc06000-11b35bcff000 rwxp 00000000 00:00 0 
11b35bcff000-11b35bd00000 ---p 00000000 00:00 0 
11b35bd00000-11b35bd05000 rw-p 00000000 00:00 0 
11b35bd05000-11b35bd06000 ---p 00000000 00:00 0 
11b35bd06000-11b35bdff000 rwxp 00000000 00:00 0 
11b35bdff000-11b35be00000 ---p 00000000 00:00 0 
11b35be00000-11b35be05000 rw-p 00000000 00:00 0 
11b35be05000-11b35be06000 ---p 00000000 00:00 0 
11b35be06000-11b35beff000 rwxp 00000000 00:00 0 
11b35beff000-11b35bf00000 ---p 00000000 00:00 0 
11b35bf00000-11b35bf05000 rw-p 00000000 00:00 0 
11b35bf05000-11b35bf06000 ---p 00000000 00:00 0 
11b35bf06000-11b35bfff000 rwxp 00000000 00:00 0 
11b35bfff000-11b37bbdf000 ---p 00000000 00:00 0 
145401302000-145401303000 r-xp 00000000 00:00 0 
155e4e42d000-155e4e42e000 r-xp 00000000 00:00 0 
15a8a4a6d000-15a8a4a6e000 r-xp 00000000 00:00 0 
175194400000-17519441d000 rw-p 00000000 00:00 0 
1a06b8000000-1a06b8e00000 ---p 00000000 00:00 0 
1a06b8e00000-1a06b9000000 rw-p 00000000 00:00 0 
1a06b9000000-1a06b9e00000 ---p 00000000 00:00 0 
1a06b9e00000-1a06ba000000 rw-p 00000000 00:00 0 
1b212d600000-1b212d625000 rw-p 00000000 00:00 0 
2816707b2000-2816707b3000 r-xp 00000000 00:00 0 
2fb29002b000-2fb29022b000 rw-p 00000000 00:00 0 
37af7d600000-37af7d635000 rw-p 00000000 00:00 0 
3ab1cab00000-3ab1cac00000 rw-p 00000000 00:00 0 
3b8366300000-3b8366400000 rw-p 00000000 00:00 0 
3c631c849000-3c631c84a000 r-xp 00000000 00:00 0 
7fb2d0000000-7fb2d019f000 rw-p 00000000 00:00 0 
7fb2d019f000-7fb2d4000000 ---p 00000000 00:00 0 
7fb2d5c4c000-7fb2d5c54000 r-xp 00000000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7fb2d5c54000-7fb2d5e53000 ---p 00008000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7fb2d5e53000-7fb2d5e54000 r--p 00007000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7fb2d5e54000-7fb2d5e55000 rw-p 00008000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7fb2d5e55000-7fb2d5fdf000 r-xp 00000000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb2d5fdf000-7fb2d61de000 ---p 0018a000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb2d61de000-7fb2d61e2000 r--p 00189000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb2d61e2000-7fb2d61e3000 rw-p 0018d000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb2d61e3000-7fb2d61e9000 rw-p 00000000 00:00 0 
7fb2d61e9000-7fb2d6201000 r-xp 00000000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb2d6201000-7fb2d6401000 ---p 00018000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb2d6401000-7fb2d6402000 r--p 00018000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb2d6402000-7fb2d6403000 rw-p 00019000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb2d6403000-7fb2d6407000 rw-p 00000000 00:00 0 
7fb2d6407000-7fb2d641c000 r-xp 00000000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb2d641c000-7fb2d661b000 ---p 00015000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb2d661b000-7fb2d661c000 r--p 00014000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb2d661c000-7fb2d661d000 rw-p 00015000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb2d661d000-7fb2d66a1000 r-xp 00000000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb2d66a1000-7fb2d68a0000 ---p 00084000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb2d68a0000-7fb2d68a1000 r--p 00083000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb2d68a1000-7fb2d68a2000 rw-p 00084000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb2d68a2000-7fb2d698a000 r-xp 00000000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7fb2d698a000-7fb2d6b89000 ---p 000e8000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7fb2d6b89000-7fb2d6b91000 r--p 000e7000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7fb2d6b91000-7fb2d6b93000 rw-p 000ef000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7fb2d6b93000-7fb2d6ba8000 rw-p 00000000 00:00 0 
7fb2d6ba8000-7fb2d6baa000 r-xp 00000000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb2d6baa000-7fb2d6daa000 ---p 00002000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb2d6daa000-7fb2d6dab000 r--p 00002000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb2d6dab000-7fb2d6dac000 rw-p 00003000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb2d6dac000-7fb2d6db3000 r-xp 00000000 07:00 1050558                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb2d6db3000-7fb2d6fb2000 ---p 00007000 07:00 1050558                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb2d6fb2000-7fb2d6fb3000 r--p 00006000 07:00 1050558                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb2d6fb3000-7fb2d6fb4000 rw-p 00007000 07:00 1050558                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb2d6fb4000-7fb2d6fd5000 r-xp 00000000 07:00 1050476                    /lib/x86_64-linux-gnu/ld-2.13.so
7fb2d71ad000-7fb2d71b3000 rw-p 00000000 00:00 0 
7fb2d71c1000-7fb2d71c2000 ---p 00000000 00:00 0 
7fb2d71c2000-7fb2d71d4000 rw-p 00000000 00:00 0 
Aborted

scriby commented Sep 4, 2012

There's a small problem in the program I included in the original issue with the callback. With this updated version, a different seg fault occurs after 122 levels of recursion.

var asyncblock = require('asyncblock');

if(asyncblock.enableTransform(module, __filename)){
    return;
}

var t2 = function(i, callback){
    console.log(i);

    if(i > 100000){
        return callback();
    }

    asyncblock(function(){
        t2(i + 1).sync();
    });
};

t2(0, function(){
    console.log('done');
});
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122

*** glibc detected *** node: free(): invalid next size (normal): 0x0000000002afd3a0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x78a8f)[0x7f19cc308a8f]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x73)[0x7f19cc30c8e3]
node(_ZN2v88internal9AssemblerD1Ev+0x25)[0x872e45]
node(_ZN2v88internal17FullCodeGenerator8MakeCodeEPNS0_15CompilationInfoE+0x243)[0x712583]
node[0x6d85d8]
node(_ZN2v88internal8Compiler11CompileLazyEPNS0_15CompilationInfoE+0x20a)[0x6d9f8a]
node[0x7aac08]
node(_ZN2v88internal10JSFunction11CompileLazyENS0_6HandleIS1_EENS0_18ClearExceptionFlagE+0x54)[0x7b3df4]
node(_ZN2v88internal11CallIC_MissENS0_9ArgumentsEPNS0_7IsolateE+0x2a4)[0x8fc8a4]
[0xe4050b0618e]
======= Memory map: ========
00400000-00baf000 r-xp 00000000 07:00 808414                             /usr/local/bin/node
00dae000-00daf000 r--p 007ae000 07:00 808414                             /usr/local/bin/node
00daf000-00dc0000 rw-p 007af000 07:00 808414                             /usr/local/bin/node
00dc0000-00dca000 rw-p 00000000 00:00 0 
02a9d000-02c23000 rw-p 00000000 00:00 0                                  [heap]
52822688000-52822689000 r-xp 00000000 00:00 0 
60e6fc04000-60e6fc05000 r-xp 00000000 00:00 0 
a61acd00000-a61acd25000 rw-p 00000000 00:00 0 
e4050a99000-e4050b00000 ---p 00000000 00:00 0 
e4050b00000-e4050b05000 rw-p 00000000 00:00 0 
e4050b05000-e4050b06000 ---p 00000000 00:00 0 
e4050b06000-e4050bff000 rwxp 00000000 00:00 0 
e4050bff000-e4050c00000 ---p 00000000 00:00 0 
e4050c00000-e4050c05000 rw-p 00000000 00:00 0 
e4050c05000-e4050c06000 ---p 00000000 00:00 0 
e4050c06000-e4050cff000 rwxp 00000000 00:00 0 
e4050cff000-e4050d00000 ---p 00000000 00:00 0 
e4050d00000-e4050d05000 rw-p 00000000 00:00 0 
e4050d05000-e4050d06000 ---p 00000000 00:00 0 
e4050d06000-e4050dff000 rwxp 00000000 00:00 0 
e4050dff000-e4070a99000 ---p 00000000 00:00 0 
f62dfd00000-f62dfe00000 rw-p 00000000 00:00 0 
13af89400000-13af8941d000 rw-p 00000000 00:00 0 
1968cc7b0000-1968cc7b1000 r-xp 00000000 00:00 0 
19705fb29000-19705fd29000 rw-p 00000000 00:00 0 
262cc1022000-262cc1023000 r-xp 00000000 00:00 0 
299f8fe00000-299f8fe85000 rw-p 00000000 00:00 0 
2c904dd00000-2c904dd35000 rw-p 00000000 00:00 0 
2fb064f34000-2fb064f35000 r-xp 00000000 00:00 0 
340403e00000-340403f00000 rw-p 00000000 00:00 0 
3735db300000-3735db400000 rw-p 00000000 00:00 0 
3c13de000000-3c13dee00000 ---p 00000000 00:00 0 
3c13dee00000-3c13df000000 rw-p 00000000 00:00 0 
3c13df000000-3c13dfe00000 ---p 00000000 00:00 0 
3c13dfe00000-3c13e0000000 rw-p 00000000 00:00 0 
3f2b24094000-3f2b240c0000 ---p 00000000 00:00 0 
3f2b240c0000-3f2b240e0000 rw-p 00000000 00:00 0 
3f2b240e0000-3f2b240f4000 ---p 00000000 00:00 0 
7f19c8000000-7f19c8021000 rw-p 00000000 00:00 0 
7f19c8021000-7f19cc000000 ---p 00000000 00:00 0 
7f19cc087000-7f19cc08f000 r-xp 00000000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7f19cc08f000-7f19cc28e000 ---p 00008000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7f19cc28e000-7f19cc28f000 r--p 00007000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7f19cc28f000-7f19cc290000 rw-p 00008000 07:00 1324905                    /home/ubuntu/sourcecode/folkLOR/node_modules/asyncblock/node_modules/fibers/bin/linux-x64-v8-3.11/fibers.node
7f19cc290000-7f19cc41a000 r-xp 00000000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7f19cc41a000-7f19cc619000 ---p 0018a000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7f19cc619000-7f19cc61d000 r--p 00189000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7f19cc61d000-7f19cc61e000 rw-p 0018d000 07:00 1050489                    /lib/x86_64-linux-gnu/libc-2.13.so
7f19cc61e000-7f19cc624000 rw-p 00000000 00:00 0 
7f19cc624000-7f19cc63c000 r-xp 00000000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f19cc63c000-7f19cc83c000 ---p 00018000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f19cc83c000-7f19cc83d000 r--p 00018000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f19cc83d000-7f19cc83e000 rw-p 00019000 07:00 1050554                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7f19cc83e000-7f19cc842000 rw-p 00000000 00:00 0 
7f19cc842000-7f19cc857000 r-xp 00000000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f19cc857000-7f19cca56000 ---p 00015000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f19cca56000-7f19cca57000 r--p 00014000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f19cca57000-7f19cca58000 rw-p 00015000 07:00 1050517                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f19cca58000-7f19ccadc000 r-xp 00000000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7f19ccadc000-7f19cccdb000 ---p 00084000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7f19cccdb000-7f19cccdc000 r--p 00083000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7f19cccdc000-7f19cccdd000 rw-p 00084000 07:00 1050526                    /lib/x86_64-linux-gnu/libm-2.13.so
7f19cccdd000-7f19ccdc5000 r-xp 00000000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7f19ccdc5000-7f19ccfc4000 ---p 000e8000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7f19ccfc4000-7f19ccfcc000 r--p 000e7000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7f19ccfcc000-7f19ccfce000 rw-p 000ef000 07:00 533248                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.14
7f19ccfce000-7f19ccfe3000 rw-p 00000000 00:00 0 
7f19ccfe3000-7f19ccfe5000 r-xp 00000000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f19ccfe5000-7f19cd1e5000 ---p 00002000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f19cd1e5000-7f19cd1e6000 r--p 00002000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f19cd1e6000-7f19cd1e7000 rw-p 00003000 07:00 1050499                    /lib/x86_64-linux-gnu/libdl-2.13.so
7f19cd1e7000-7f19cd1ee000 r-xp 00000000 07:00 1050558                    /lib/x86_64-linux-gnu/librt-2.13.so
7f19cd1ee000-7f19cd3ed000 ---p 00007000 07:00 1050558                    /lib/x86_64-linux-gnu/librt-2.13.soAborted
Owner

laverdet commented Sep 4, 2012

Hey there's a lot of logic going on in asyncblock and I'm trying to wrap my head around it.. is it possible to distil this down to something that doesn't have external dependencies? If not that's fine it's just going to take longer to debug.

scriby commented Sep 4, 2012

I attempted to write a test using plain Fibers and couldn't get the same behavior to occur. I did notice something else strange, though. If I write a recursive "loop" in plain js, I can get about 15,000 recursive calls before stack overflow.

If I create a fiber, then do the recursive "loop" from within the fiber, stack overflow occurs at about 960 recursive calls. But I think this is totally unrelated to the current issue...

For the program in question, not too much code in asyncblock is going to run (probably about 30 lines). I think if you step through it in the debugger you should be able to follow. The .sync() gets transformed into a flow.addAndReuseFiber(key) and a flow.wait(key) -- which tells asyncblock to not create a new fiber for asyncblocks made from within the current fiber (as we know the other fiber is yielding anyway). Flow.wait(key) is like Fiber.yield, but waits for a particular piece of data to be ready before continuing.

Owner

laverdet commented Sep 4, 2012

This reproduces it or a similar error w/o the dependency:

var Fiber = require('fibers');

Fiber(function F(ii) {
    console.log(ii);
    try {
        F(ii + 1);
    } catch(e) {
        console.log('caught');
    }
}).run(0);

Seems to be related to the presence of the try/catch block. I'm out of time to debug right now but I'll pick this up later and see if it's an issue with fibers or v8. Thanks!

scriby commented Sep 4, 2012

Nice work distilling it down to the root issue! Definitely a weird one!

Owner

laverdet commented Sep 4, 2012

Can you try out this change and let me know if that fixes it on your end? Just apply the patch, and then run node build -f from the fibers directory.

diff --git a/src/fibers.cc b/src/fibers.cc
index ad79415..13d30a2 100644
--- a/src/fibers.cc
+++ b/src/fibers.cc
@@ -336,7 +336,7 @@ class Fiber {
                                // to run
                                ResourceConstraints constraints;
                                constraints.set_stack_limit(reinterpret_cast<uint32_t*>(
-                                       (char*)that.this_fiber->bottom() + 2 * 1024));
+                                       (char*)that.this_fiber->bottom() + 4 * 1024));
                                SetResourceConstraints(&constraints);

                                TryCatch try_catch;

scriby commented Sep 4, 2012

node build -f
execvp(): No such file or directory
Build failed

scriby commented Sep 4, 2012

Looks like I don't have node-gyp... I'll see if I can figure it out.

Owner

laverdet commented Sep 4, 2012

node-gyp comes with node. You can also do: rm -rf bin/*; npm install . from inside the fibers directory and that will work with no system changes needed.

scriby commented Sep 4, 2012

Ok, got it built.

The error has gone away, but the stack size seems really small. I get 116 levels of recursion before stack overflow.

Your example code goes to 571 levels before stack overflow. Without the try / catch, your example goes to 929.

And being run outside a fiber it recurses 10,469 times. Without the try / catch, it recurses 17,948 times.

Owner

laverdet commented Sep 5, 2012

This is intentional. Each fiber is a new stack, and you have to allocate the whole stack in advance. I think most platforms default to a stack space of 8mb for each thread. With fibers you might reasonably have thousands active at time, so they only allocate 64kb each. I haven't heard of anyone legitimately running out of stack space on a fiber; it seems to be pretty good size for most applications.

Now I did come up with the 64kb stack size when Node was still defaulting to 32-bit builds even on 64-bit systems. It seemed feasible that with a larger stack size you might run out of address space. The system should generally only page out memory in 8kb chunks as you write to it, so allocating something like 8mb for each fiber shouldn't be a problem when it comes to running out of physical memory. Perhaps I should consider raising this on 64-bit platforms?

scriby commented Sep 5, 2012

Ok, that makes sense.

I agree it's rare that the stack size will cause a problem in the real world. We can work around it if the issue comes up in development / testing, but if it shows up in production due to different data it could be a bummer.

As long as there's no big drawbacks, I think it makes sense to reconsider the stack size for 64-bit. Or at least make it configurable.

Thanks for looking into this!

Chris

@laverdet laverdet closed this in a92076a Jan 6, 2013

glasser commented Jan 8, 2013

We've been seeing some reports of Meteor crashes that might be caused by this. Any chance of pushing a release with this fix in it?

Owner

laverdet commented Jan 8, 2013

Do you have more information on the crashes?

glasser commented Jan 9, 2013

Heh, actually, having tested this, it doesn't fix our issues :(

Unfortunately I haven't gotten a particularly minified execution of them yet. One issue works like this:

$ curl https://install.meteor.com/ | sh    # install meteor
$ git clone https://github.com/glasser/fiber-crash
$ cd fiber-crash
$ meteor
[[[[[ /private/tmp/fiber-crash ]]]]]

Running on: http://localhost:3000/
start
Exited from signal: SIGSEGV

It doesn't happen consistently, maybe about 1 in 3 times. When I apply this patch (ie, replacing the 0.6.9 fibers installed with Meteor with the one here, and making a few fixes to deal with the lack of global Fiber) I get (sometimes, not all the time)

[[[[[ /private/tmp/fiber-crash ]]]]]

Running on: http://localhost:3000/
start
node(51241,0x7fff78fc8180) malloc: *** error for object 0x101831c08: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Exited from signal: SIGABRT

The other crash that may or may not be related involves UglifyJS. That should be more minimal in the sense that it won't require Meteor, I hope. Working on that now.

glasser commented Jan 9, 2013

Or OK, here's the other one. Clone https://github.com/glasser/fiber-crash-uglify and npm install (which uses 0.6.9, but the problem happens with HEAD too) and run "node crash.js". Maybe 1 time in 10 (on my OSX 10.8 machine, Node 0.8.16) there'll be some sort of segfault or malloc modified-after-free or whatever. Nothing if the use of fibers is removed.

glasser commented Jan 9, 2013

(I should probably have opened a new bug, sorry!)

glasser commented Jan 9, 2013

Here are stack traces. From the sugar/Meteor one:

(gdb) bt
#0  0x000000010016fb47 in v8::internal::InnerPointerToCodeCache::GcSafeFindCodeForInnerPointer ()
#1  0x000000010016ecc5 in v8::internal::InnerPointerToCodeCache::GetCacheEntry ()
#2  0x00000001001bd5c3 in v8::internal::IC::PostPatching ()
#3  0x00000001001c2d8f in v8::internal::ToBoolean_Patch ()
#4  0x00000f589ec0618e in ?? ()
#5  0x00000f589f035386 in ?? ()
#6  0x00000f589f039cae in ?? ()
#7  0x00000f589ec098ce in ?? ()
#8  0x00000f589f04e9ed in ?? ()
#9  0x00000f589ec098ce in ?? ()
#10 0x00000f589f0b4e09 in ?? ()
#11 0x00000f589ec9638f in ?? ()
#12 0x00000f589ec098ce in ?? ()
#13 0x00000f589f08d37f in ?? ()
#14 0x00000f589eeb356d in ?? ()
#15 0x00000f589ec098ce in ?? ()
#16 0x00000f589ec967e2 in ?? ()
#17 0x00000f589ec248c7 in ?? ()
#18 0x00000f589ec11317 in ?? ()
#19 0x000000010015d251 in v8::internal::Invoke ()
#20 0x000000010011b970 in v8::Function::Call ()
#21 0x00000001007f9414 in Fiber::RunFiber ()
#22 0x00000001007f9bdd in Coroutine::trampoline ()
#23 0x00000001007fac22 in coro_init ()
#24 0x00000001007fabd5 in trampoline ()
#25 <signal handler called>
#26 0x00007fff91626552 in __sigsuspend ()

From the Uglify one:

(gdb) bt
#0  0x000000010021bfa9 in v8::internal::ConsString::ConsStringReadBlockIntoBuffer ()
#1  0x000000010021bfae in v8::internal::ConsString::ConsStringReadBlockIntoBuffer ()
#2  0x000000010021bfae in v8::internal::ConsString::ConsStringReadBlockIntoBuffer ()
#3  0x000000010021bfae in v8::internal::ConsString::ConsStringReadBlockIntoBuffer ()
[721 lines elided]
#725 0x000000010021bfae in v8::internal::ConsString::ConsStringReadBlockIntoBuffer ()
#726 0x00000001002202fc in v8::internal::ConsString::ConsStringReadBlock ()
#727 0x0000000100220d52 in unibrow::InputBuffer<v8::internal::String, v8::internal::String*, 1024u>::InputBuffer ()
#728 0x000000010020b511 in v8::internal::String::ComputeAndSetHash ()
#729 0x000000010021d022 in v8::internal::RegExpKey::Hash ()
#730 0x0000000100218480 in v8::internal::HashTable<v8::internal::CompilationCacheShape, v8::internal::HashTableKey*>::FindEntry ()
#731 0x00000001002151a0 in v8::internal::CompilationCacheTable::LookupRegExp ()
#732 0x000000010013fdc2 in v8::internal::CompilationCacheRegExp::Lookup ()
#733 0x00000001001c6370 in v8::internal::RegExpImpl::Compile ()
#734 0x0000000100248020 in v8::internal::Runtime_RegExpCompile ()
#735 0x0000029bc530618e in ?? ()
#736 0x0000029bc53544d2 in ?? ()
#737 0x0000029bc5315326 in ?? ()
#738 0x0000029bc531520e in ?? ()
#739 0x0000029bc55476e7 in ?? ()
#740 0x0000029bc5545fc3 in ?? ()
#741 0x0000029bc55459a1 in ?? ()
#742 0x0000029bc557971d in ?? ()
#743 0x0000029bc53098ce in ?? ()
#744 0x0000029bc557b9de in ?? ()
#745 0x0000029bc553a5fa in ?? ()
#746 0x0000029bc553e042 in ?? ()
#747 0x0000029bc53098ce in ?? ()
#748 0x0000029bc553d3de in ?? ()
#749 0x0000029bc5559c78 in ?? ()
#750 0x0000029bc55395b2 in ?? ()
#751 0x0000029bc553d934 in ?? ()
#752 0x0000029bc553dea0 in ?? ()
#753 0x0000029bc53098ce in ?? ()
#754 0x0000029bc553eb31 in ?? ()
#755 0x0000029bc5576aa1 in ?? ()
#756 0x0000029bc557f84b in ?? ()
#757 0x0000029bc554070a in ?? ()
#758 0x0000029bc5540a4a in ?? ()
#759 0x0000029bc5540dc9 in ?? ()
#760 0x0000029bc53098ce in ?? ()
#761 0x0000029bc553e8e1 in ?? ()
#762 0x0000029bc5576aa1 in ?? ()
#763 0x0000029bc557f84b in ?? ()
#764 0x0000029bc554070a in ?? ()
#765 0x0000029bc5540a4a in ?? ()
#766 0x0000029bc5540bf2 in ?? ()
#767 0x0000029bc5540dc9 in ?? ()
#768 0x0000029bc53098ce in ?? ()
#769 0x0000029bc55381dd in ?? ()
#770 0x0000029bc53098ce in ?? ()
#771 0x0000029bc553c680 in ?? ()
#772 0x0000029bc5537aa5 in ?? ()
#773 0x0000029bc55410ae in ?? ()
#774 0x0000029bc55424c2 in ?? ()
#775 0x0000029bc5398bcb in ?? ()
#776 0x0000029bc53098ce in ?? ()
#777 0x0000029bc53aa448 in ?? ()
#778 0x0000029bc53098ce in ?? ()
#779 0x0000029bc5398eb9 in ?? ()
#780 0x0000029bc53098ce in ?? ()
#781 0x0000029bc538e86c in ?? ()
#782 0x0000029bc53248c7 in ?? ()
#783 0x0000029bc5311317 in ?? ()
#784 0x000000010015d251 in v8::internal::Invoke ()
#785 0x000000010011b970 in v8::Function::Call ()
#786 0x00000001007f93d2 in Fiber::RunFiber ()
#787 0x00000001007f9b8d in Coroutine::trampoline ()
#788 0x00000001007fabe2 in coro_init ()
#789 0x00000001007fab95 in trampoline ()
#790 <signal handler called>
#791 0x00007fff91626552 in __sigsuspend ()
Owner

laverdet commented Jan 11, 2013

@glasser thanks for that repro it's super helpful. I've narrowed this issue down.

Consider this code:

var foo = '';
for (var ii = 0; ii < 1024; ++ii) {
  foo += 'a';
}
new RegExp(foo, 'g');

This will chew up 240k (!) of stack space. I will have to make some adjustments taking this into consideration.

tmeasday pushed a commit to meteor/docs that referenced this issue May 9, 2016

Upgrade Fibers to 1.0.0 and UglifyJS to 2.2.3.
This definitely won't "work" because both have API changes.

UglifyJS 1 was having issues minifying certain code (eg lodash).  But UglifyJS 2
was occasionally segfaulting (as was other code using large regexps).  See the
bottom of laverdet/node-fibers#89

Hopefully this will fix it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment