Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when loading many small rules files #906

Closed
mstemm opened this issue Oct 29, 2019 · 1 comment · Fixed by #907
Closed

Crash when loading many small rules files #906

mstemm opened this issue Oct 29, 2019 · 1 comment · Fixed by #907
Labels

Comments

@mstemm
Copy link
Contributor

@mstemm mstemm commented Oct 29, 2019

What happened:

I found a crash bug in Falco that occurs when Falco loads lots of relatively small files with lots of macros. It doesn’t seem to be a bug in falco itself, a stack trace from a debug build shows the crash is in the lua garbage collector:

Core was generated by `./userspace/falco/falco -c /mnt/sf_mstemm/work/src/falco/falco.yaml -r /tmp/cus'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000013395d3 in lj_alloc_free (msp=0x40e87010, ptr=0x408737c0) at lj_alloc.c:1305
1305            unlink_chunk(fm, next, nsize);
(gdb) where
#0  0x00000000013395d3 in lj_alloc_free (msp=0x40e87010, ptr=0x408737c0) at lj_alloc.c:1305
#1  0x0000000001339f59 in lj_alloc_f (msp=0x40e87010, ptr=0x408737c0, osize=32, nsize=0) at lj_alloc.c:1382
#2  0x00000000013456b6 in lj_mem_free (osize=32, p=0x408737c0, g=0x40e873b8) at lj_gc.h:120
#3  lj_tab_free (g=0x40e873b8, t=0x408737c0) at lj_tab.c:211
#4  0x0000000001343980 in gc_sweep (g=0x40e873b8, p=0x4143fc70, lim=10) at lj_gc.c:405
#5  0x0000000001344470 in gc_onestep (L=0x40e87378) at lj_gc.c:633
#6  0x0000000001344625 in lj_gc_step (L=0x40e87378) at lj_gc.c:684
#7  0x00000000013233cd in lua_newuserdata (L=0x40e87378, size=8176) at lj_api.c:695
#8  0x00000000013a3e8c in newtree ()
#9  0x00000000013a53b8 in getpatt ()
#10 0x00000000013a6083 in lp_match ()
#11 0x0000000001340528 in lj_BC_FUNCC ()
#12 0x0000000001324c2d in lua_pcall (L=0x40e87378, nargs=9, nresults=2, errfunc=0) at lj_api.c:1052
#13 0x0000000000c18ce7 in falco_rules::load_rules (this=0x30cd920, rules_content=..., verbose=false, all_events=false, extra=..., replace_container_info=false, min_priority=falco_common::PRIORITY_DEBUG, required_engine_version=@0x7ffdbd0463d0: 0)
    at /mnt/sf_mstemm/work/src/falco/userspace/engine/rules.cpp:463
#14 0x0000000000be1859 in falco_engine::load_rules (this=0x30ad7d0, rules_content=..., verbose=false, all_events=false, required_engine_version=@0x7ffdbd0463d0: 0) at /mnt/sf_mstemm/work/src/falco/userspace/engine/falco_engine.cpp:182
#15 0x0000000000be1b0a in falco_engine::load_rules_file (this=0x30ad7d0, rules_filename=..., verbose=false, all_events=false, required_engine_version=@0x7ffdbd0463d0: 0) at /mnt/sf_mstemm/work/src/falco/userspace/engine/falco_engine.cpp:207
#16 0x0000000000b92b8a in falco_init (argc=25, argv=0x7ffdbd0477c8) at /mnt/sf_mstemm/work/src/falco/userspace/falco/falco.cpp:818
#17 0x0000000000b9652d in main (argc=25, argv=0x7ffdbd0477c8) at /mnt/sf_mstemm/work/src/falco/userspace/falco/falco.cpp:1262

The full command line that reproduces the crash is falco -c <path to falco.yaml> -r crash_rules_files, zip file attached
crash_rules_files.zip.

You'll need a recent dev build of falco, as that has support for the psp related fields in the rules files. I was using 0.1.2709dev.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Falco version (use falco --version):
  • System info <!-- Falco has a built-in support command you can use "falco --support | jq .system_info" -->
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools (e.g. in kubernetes, rpm, deb, from source):
  • Others:
@mstemm mstemm added the kind/bug label Oct 29, 2019
@mstemm

This comment has been minimized.

Copy link
Contributor Author

@mstemm mstemm commented Oct 30, 2019

After compiling luajit with assertions, I found that the problem was here:

                if(lua_pcall(m_ls, 9, 2, 0) != 0)
                {
                        const char* lerr = lua_tostring(m_ls, -1);

                        string err = "Error loading rules: " + string(lerr);

                        throw falco_exception(err);
                }

                // Either returns (true, required_engine_version), or (false, error string)
                bool successful = lua_toboolean(m_ls, -2);

                if(successful)
                {
                        required_engine_version = lua_tonumber(m_ls, -1);
                }
                else
                {
                        std::string err = lua_tostring(m_ls, -1);
                        throw falco_exception(err);
                }

                lua_pop(m_ls, 4);

I don't know why this pops 4 values from the stack when the function only returns 2 values, but it was introduced in 1711ed0.

It only shows up after a large number of files because there are ~10 tables on the stack from the various calls to luaL_openlib(), plus some time for the stack to be corrupted and the garbage collector to kick in and crash.

Changing the pop to 2 values fixes the problem.

mstemm added a commit that referenced this issue Oct 30, 2019
The call to rule_loader.load_rules only returns 2 values, so only pop
two values from the stack. This fixes #906.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
@fntlnz fntlnz closed this in #907 Oct 30, 2019
fntlnz added a commit that referenced this issue Oct 30, 2019
The call to rule_loader.load_rules only returns 2 values, so only pop
two values from the stack. This fixes #906.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.