Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault while fitting two-compartment model #241

Closed
billdenney opened this issue Sep 28, 2022 · 15 comments
Closed

Segfault while fitting two-compartment model #241

billdenney opened this issue Sep 28, 2022 · 15 comments

Comments

@billdenney
Copy link
Contributor

I was fitting a 2-compartment model with ODEs. The model I'm fitting is pretty standard:

mod_2cmt$ui
 ── rxode2-based free-form 2-cmt ODE model ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 ── Initalization: ──
Fixed Effects ($theta):
         lcl          lvc          lvp           lq     prop.err      add.err
-0.004342655  4.429537617  5.122684798 -0.697489836  0.500000000  1.000000000

Omega ($omega):
       etalvc
etalvc    0.1

States ($state or $stateDf):
  Compartment Number Compartment Name
1                  1           center
2                  2      peripheral1
 ── μ-referencing ($muRefTable): ──
  theta    eta level
1   lvc etalvc    id

 ── Model (Normalized Syntax): ──
function() {
    description <- "Two compartment PK model with linear clearance using differential equations"
    ini({
        lcl <- c(-2.93661770114182, -0.00434265546659215, 3.01380660241922)
        label("Clearance (CL)")
        lvc <- c(1.42540485812497, 4.42953761734493, 7.53786506453078)
        label("Central volume of distribution (V)")
        lvp <- c(2.11855203868492, 5.12268479790488, 8.23101224509072)
        label("Peripheral volume of distribution (Vp)")
        lq <- c(-3.62976488170177, -0.697489836026537, 2.32065942185927)
        label("Intercompartmental clearance (Q)")
        prop.err <- c(0, 0.5)
        label("Proportional residual error (fraction)")
        add.err <- c(0, 1)
        etalvc ~ 0.1
    })
    model({
        cl <- exp(lcl)
        vc <- exp(lvc + etalvc)
        vp <- exp(lvp)
        q <- exp(lq)
        kel <- cl/vc
        k12 <- q/vc
        k21 <- q/vp
        d/dt(center) <- -kel * center - k12 * center + k21 *
            peripheral1
        d/dt(peripheral1) <- k12 * center - k21 * peripheral1
        cp <- 1e+06 * center/vc
        cp ~ add(add.err) + prop(prop.err)
    })
}

And, the estimation is pretty simple saem:

mod_2cmt_est <- nlmixr(object = mod_2cmt$ui, data = data, est = "saem")

unhandled error message: EE:lsoda -- at t = 1.77213 and step size _rxC(h) = 5.55952e-08, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0.0353404 and step size _rxC(h) = 1.16547e-07, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 1.35703 and step size _rxC(h) = 1.46475e-05, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0 and step size _rxC(h) = 8.15893e-07, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0.0274671 and step size _rxC(h) = 4.56926e-07, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0 and step size _rxC(h) = 7.20507e-06, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0 and step size _rxC(h) = 1.53938e-06, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0 and step size _rxC(h) = 1.54452e-06, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0 and step size _rxC(h) = 2.80127e-07, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
unhandled error message: EE:lsoda -- at t = 0.00210878 and step size _rxC(h) = 1.57116e-11, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
Error in tryCatch(evalq((function (x, rules)  : not a BUILTIN fu▒▒TVUU
Error in exists("control", envir = .ui) :
  'rho' must be an environment not pairlist: detected in C-level eval
Error: 'rho' must be an environment not pairlist: detected in C-level eval
Browse[1]>
Thread 7 "R" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe9dbc640 (LWP 221636)]
0x00007ffff7c0c86b in Rf_applyClosure () from /usr/lib/R/lib/libR.so

(gdb) bt
#0  0x00007ffff7c0c86b in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#1  0x00007ffff7c82cf7 in ?? () from /usr/lib/R/lib/libR.so
#2  0x00007ffff7d84c4a in ?? () from /usr/lib/R/lib/libR.so
#3  0x00007ffff7c830b1 in ?? () from /usr/lib/R/lib/libR.so
#4  0x00007ffff7c23e80 in ?? () from /usr/lib/R/lib/libR.so
#5  0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#6  0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#7  0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#8  0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#9  0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#10 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#11 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#12 0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#13 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#14 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#15 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#16 0x00007ffff7c0cd4a in Rf_eval () from /usr/lib/R/lib/libR.so
#17 0x00007ffff7c14b58 in ?? () from /usr/lib/R/lib/libR.so
#18 0x00007ffff7c0d027 in Rf_eval () from /usr/lib/R/lib/libR.so
#19 0x00007ffff7c0d027 in Rf_eval () from /usr/lib/R/lib/libR.so
#20 0x00007ffff7c14b58 in ?? () from /usr/lib/R/lib/libR.so
#21 0x00007ffff7da15b3 in ?? () from /usr/lib/R/lib/libR.so
#22 0x00007ffff7da15b3 in ?? () from /usr/lib/R/lib/libR.so
#23 0x00007ffffff44560 in ?? ()
#24 0x00007ffffff44370 in ?? ()
#25 0x00007ffff79493b6 in __vfprintf_internal (s=0x55555df01048, s@entry=0x7ffffff44370, format=format@entry=0x7ffff7da15b3 "%s", ap=ap@entry=0x7ffffff44560,
    mode_flags=mode_flags@entry=2) at ./stdio-common/vfprintf-internal.c:1602
#26 0x00007ffff795c51a in __vsnprintf_internal (string=0x7ffffff44580 "'rho' must be an environment not pairlist: detected in C-level eval", maxlen=<optimized out>,
    format=0x7ffff7da15b3 "%s", args=0x7ffffff44560, mode_flags=2) at ./libio/vsnprintf.c:114
#27 0x00007ffff7c0bdbf in ?? () from /usr/lib/R/lib/libR.so
#28 0x00007ffff7c0da22 in Rf_errorcall () from /usr/lib/R/lib/libR.so
#29 0x00007ffff7c0dc40 in Rf_error () from /usr/lib/R/lib/libR.so
#30 0x00007ffff7c0d338 in Rf_eval () from /usr/lib/R/lib/libR.so
#31 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#32 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#33 0x00007ffff7c0cd4a in Rf_eval () from /usr/lib/R/lib/libR.so
#34 0x00007ffff7c14c9f in ?? () from /usr/lib/R/lib/libR.so
#35 0x00007ffff7c0d027 in Rf_eval () from /usr/lib/R/lib/libR.so
#36 0x00007ffff7c14b58 in ?? () from /usr/lib/R/lib/libR.so
#37 0x00007ffff7c0d027 in Rf_eval () from /usr/lib/R/lib/libR.so
#38 0x00007ffff7bcef9a in Rf_endcontext () from /usr/lib/R/lib/libR.so
#39 0x00007ffff7c151ac in ?? () from /usr/lib/R/lib/libR.so
#40 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#41 0x00007ffff7c82cf7 in ?? () from /usr/lib/R/lib/libR.so
#42 0x00007ffff7d84cab in ?? () from /usr/lib/R/lib/libR.so
#43 0x00007ffff7c830b1 in ?? () from /usr/lib/R/lib/libR.so
#44 0x00007ffff7c23e80 in ?? () from /usr/lib/R/lib/libR.so
#45 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#46 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#47 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#48 0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#49 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#50 0x00007ffff7c13deb in ?? () from /usr/lib/R/lib/libR.so
#51 0x00007ffff7c1ddf8 in ?? () from /usr/lib/R/lib/libR.so
#52 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#53 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#54 0x00007ffff7c13deb in ?? () from /usr/lib/R/lib/libR.so
#55 0x00007ffff7c1ddf8 in ?? () from /usr/lib/R/lib/libR.so
#56 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#57 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#58 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#59 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#60 0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#61 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#62 0x00007ffff7c13deb in ?? () from /usr/lib/R/lib/libR.so
#63 0x00007ffff7c1ddf8 in ?? () from /usr/lib/R/lib/libR.so
#64 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#65 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#66 0x00007ffff7c13deb in ?? () from /usr/lib/R/lib/libR.so
#67 0x00007ffff7c1ddf8 in ?? () from /usr/lib/R/lib/libR.so
#68 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#69 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#70 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#71 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#72 0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#73 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#74 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#75 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#76 0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#77 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#78 0x00007ffff7c1520f in ?? () from /usr/lib/R/lib/libR.so
#79 0x00007ffff7c0c968 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#80 0x00007ffff7c32db5 in ?? () from /usr/lib/R/lib/libR.so
#81 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#82 0x00007ffff7c13deb in ?? () from /usr/lib/R/lib/libR.so
#83 0x00007ffff7c1ddf8 in ?? () from /usr/lib/R/lib/libR.so
#84 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#85 0x00007ffff7c1dd72 in ?? () from /usr/lib/R/lib/libR.so
#86 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#87 0x00007ffff49e84d0 in ?? ()
#88 0x00005555555dae10 in ?? ()
#89 0x0000555555570e00 in ?? ()
#90 0x00007ffff7e52d90 in R_Home () from /usr/lib/R/lib/libR.so
#91 0x00007ffff7bf90f6 in R_findVarLocInFrame () from /usr/lib/R/lib/libR.so
#92 0x00007ffff7c1696b in ?? () from /usr/lib/R/lib/libR.so
#93 0x00007ffff7c1dd72 in ?? () from /usr/lib/R/lib/libR.so
#94 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#95 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#96 0x000055555558f240 in ?? ()
#97 0x00007ffff7e46afc in ?? () from /usr/lib/R/lib/libR.so
#98 0x000055555583ae90 in ?? ()
#99 0x00005555555b5b03 in ?? ()
#100 0x000055555557ab40 in ?? ()
#101 0x00007ffff7d9b9e7 in ?? () from /usr/lib/R/lib/libR.so
#102 0x00007ffff7c0d0b0 in Rf_eval () from /usr/lib/R/lib/libR.so
#103 0x00007ffff7c21b84 in ?? () from /usr/lib/R/lib/libR.so
#104 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#105 0x00007ffff7c151ac in ?? () from /usr/lib/R/lib/libR.so
#106 0x00007ffff7c0c985 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#107 0x00007ffff7c24994 in ?? () from /usr/lib/R/lib/libR.so
#108 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#109 0x00007ffff7c151ac in ?? () from /usr/lib/R/lib/libR.so
#110 0x00007ffff7c0c985 in Rf_applyClosure () from /usr/lib/R/lib/libR.so
#111 0x00007ffff7c32dc0 in ?? () from /usr/lib/R/lib/libR.so
#112 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#113 0x000000000000000a in ?? ()
#114 0x00005555657651f0 in ?? ()
#115 0x000055555be5c600 in ?? ()
#116 0x00005555556908a8 in ?? ()
#117 0x0000000000000001 in ?? ()
#118 0x0000555555570ce8 in ?? ()
#119 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#120 0x00007ffff7c13e1a in ?? () from /usr/lib/R/lib/libR.so
#121 0x00007ffff7c1ddf8 in ?? () from /usr/lib/R/lib/libR.so
#122 0x00007ffff7c2b729 in ?? () from /usr/lib/R/lib/libR.so
#123 0x00007ffff7c0cc18 in Rf_eval () from /usr/lib/R/lib/libR.so
#124 0x000055555568f798 in ?? ()
#125 0x00007ffff7e52dd0 in ?? () from /usr/lib/R/lib/libR.so
#126 0x0000000000000000 in ?? ()
@mattfidler
Copy link
Contributor

Thank you @billdenney for the backtrace. Unfortunately it doesn't show anything from rxode2 or nlmixr2est

@mattfidler
Copy link
Contributor

Can you give a reproducible example by simulating a dataset with similar nominal times...?

I can't spot rho anywhere

@mattfidler
Copy link
Contributor

Wait, I guess it is nmsimplex

@billdenney
Copy link
Contributor Author

If it helps, I don't get the crash with focei.

@mattfidler
Copy link
Contributor

You probably wouldn't get the crash with

control=saemControl(type = "newuoa")

@mattfidler mattfidler mentioned this issue Sep 28, 2022
Merged
@mattfidler
Copy link
Contributor

In theory using #242 will either fix the issue or give a better back-trace about why it is happening.

@mattfidler
Copy link
Contributor

I merged it so you can check against main

@billdenney
Copy link
Contributor Author

Here is the updated backtrace from main at about 10am Eastern time today:

[New Thread 0x7fffe4a04640 (LWP 245847)]
unhandled error message: EE:lsoda -- at t = 4.04193 and step size _rxC(h) = 4.08807e-05, the
 corrector convergence failed repeatedly or
 with fabs(_rxC(h)) = hmin
 @(lsoda.c:889
Error : C stack usage  358544940 is too close to the limit
Error: C stack usage  358544940 is too close to the limit
>
Thread 3656 "R" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6a08640 (LWP 245846)]
ind_liblsoda0 (rx=rx@entry=0x7fffecd1eca0 <rx_global>, op=op@entry=0x7fffecd1e700 <op_global>, opt=..., solveid=solveid@entry=483, dydt_liblsoda=0x0, u_inis=0x0) at /tmp/Rtmp7lTo2R/R.INSTALL3a304730a6e70/rxode2/src/par_solve.cpp:1272
1272    /tmp/Rtmp7lTo2R/R.INSTALL3a304730a6e70/rxode2/src/par_solve.cpp: No such file or directory.
(gdb) bt
#0  ind_liblsoda0 (rx=rx@entry=0x7fffecd1eca0 <rx_global>, op=op@entry=0x7fffecd1e700 <op_global>, opt=..., solveid=solveid@entry=483, dydt_liblsoda=0x0, u_inis=0x0)
    at /tmp/Rtmp7lTo2R/R.INSTALL3a304730a6e70/rxode2/src/par_solve.cpp:1272
#1  0x00007fffeca5bcca in par_liblsoda._omp_fn.0 () at /tmp/Rtmp7lTo2R/R.INSTALL3a304730a6e70/rxode2/src/par_solve.cpp:1490
#2  0x00007ffff7079b9e in ?? () from /usr/lib/x86_64-linux-gnu/libgomp.so.1
#3  0x00007ffff7968b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007ffff79faa00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

@mattfidler
Copy link
Contributor

This is basically saying it ran out of memory.

You can try:

gc()
rxUnloadAll()

And removing anything un-necessary from your R memory before trying this. I wish I knew a way to catch the memory issue earlier...

This is what I did to make the fix for #199

@billdenney
Copy link
Contributor Author

Running out of memory is surprising here. It's a 2-compartment IV PK model with normal CL/V/Q/VP parameterization and about 100 individuals with mostly sparse sampling (1-2 samples per animal), and the system has 32GB of RAM.

@mattfidler
Copy link
Contributor

That is true, but it is the C stack usage which has a different metric of memory usage. We could revert #199 to see if that helps.

@mattfidler
Copy link
Contributor

I can be triggered by R alone:
https://stackoverflow.com/questions/14719349/error-c-stack-usage-is-too-close-to-the-limit

@mattfidler
Copy link
Contributor

So it could be the recursion depth is too great.

@mattfidler mattfidler mentioned this issue Sep 29, 2022
Closed
@mattfidler
Copy link
Contributor

Probably the recursion depth as measured by R. My guess is the number of environments is too high. You add something like drake or targets to the mix then it is even worse.

@mattfidler mattfidler mentioned this issue Sep 29, 2022
Merged
@billdenney
Copy link
Contributor Author

I cannot reproduce this now. I'll close and open a new issue if I am able to reproduce it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattfidler @billdenney