Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StackOverflowException with large number of Evaluate() invocations #20

Open
drprabhu opened this issue Nov 5, 2015 · 18 comments

Comments

Projects
None yet
6 participants
@drprabhu
Copy link

commented Nov 5, 2015

I am trying to use RDotNet within a larger C# project. The issue that I see is this:
On executing a fairly large number of REngine.evaluate() commands, RDotNet (or R?) almost always throws StackOverflowException. (Of course, on windows, this kills the C# process.)

After some trial and error, I was able to reproduce this issue simply by repeatedly printing a line to a log file; StackOverflowException gets thrown often before completing log of 25-50k lines. The code snippet is below. (Of course, in my regular code, I cache the engine instance and dispose it later when I am done.)

Would appreciate any suggestions/pointers..

Notes:

  • Environment: windows 7 Pro; .Net 4.6 (build target of 4.5); R 3.1.1; R.NET 1.6.5;
  • I was NOT able to reproduce this on a stand-alone project that has only RDotNet even after printing 5 million lines.
  • Occasionally, especially when I write fewer lines, the StackOverflowException gets thrown after the loop below completes and execution moves on to something else. Does this point to any cleanup-related issues in R or R.Net? Possibly related to #8?
  • I have also tried executing gc() every 100 invocations, for example. Still get the StackOverflowException.
[MethodImpl(MethodImplOptions.Synchronized)]
public void Initialize(){

   // get engine instance
   REngine.SetEnvironmentVariables();
   REngine engine = REngine.GetInstance();
   engine.Initialize();

   // open a log file
   var logFileInfo = new FileInfo("r.log");
   var rCmd = @"sink('" + logFileInfo.FullName.Replace("\\", "\\\\") + "')";
   engine.Evaluate(rCmd);

   // write lines to log file
   string newLine = @"""\n""";
   for (int i = 1; i <= 50000; i++)
   {
      engine.Evaluate(string.Format("cat(date(), \" Line # {0}\", {1})", i, newLine));
   }
}

@drprabhu drprabhu changed the title StackoverflowException with large number of evaluate() invocations StackOverflowException with large number of evaluate() invocations Nov 5, 2015

@drprabhu drprabhu changed the title StackOverflowException with large number of evaluate() invocations StackOverflowException with large number of Evaluate() invocations Nov 5, 2015

@drprabhu

This comment has been minimized.

Copy link
Author

commented Nov 5, 2015

The behavior does seem related to R_ReleaseObject api. Stacktrace points to SymbolicExpression.Unpreserve() in R.Net

        public void Unpreserve()
        {
            if (!IsInvalid && IsProtected)
            {
                if (Engine.EnableLock)
                {
                    // ** throws exception in the next line**
                    lock (lockObject) { this.GetFunction<R_ReleaseObject>()(handle); ; } 
                }
                else
                    this.GetFunction<R_ReleaseObject>()(handle);
                this.isProtected = false;
            }
        }

@drprabhu drprabhu closed this Nov 5, 2015

@drprabhu drprabhu reopened this Nov 5, 2015

@skyguy94

This comment has been minimized.

Copy link
Contributor

commented Nov 5, 2015

Ok, I see where you said you couldn't reproduce it in a standalone app. Could it be related to something you are doing with threading, ie: passing an R object to a different thread. That MethImplOptions.Synchronized is very legacy and very suspicious.

@drprabhu

This comment has been minimized.

Copy link
Author

commented Nov 5, 2015

Threading issue is very unlikely.

  • The above code snippet pretty much mirrors what's in my project, in that all of that happens at startup/initialization time. There are no multiple threads at all at this time. The only difference in my actual code is that I cache the engine object for later use. But that's not really relevant since the crash happens in the same method where I create the instance. No R objects are passed around at this time at all; REngine instance is not even used by my real code at this point in the initialization.
  • I also removed the [MethodImpl(MethodImplOptions.Synchronized)] directive; It still crashes after writing out a few thousand lines.
@skyguy94

This comment has been minimized.

Copy link
Contributor

commented Nov 5, 2015

Ok, good to know. Does the call stack have any useful details? Have you tried the mingw debugger to see whats happening on the R side?

@drprabhu

This comment has been minimized.

Copy link
Author

commented Nov 5, 2015

Unfortunately StackOverflowException is not user-catchable on Windows. So, no, not any other details. I presume, it is due to recursive search by R_ReleaseObject()?
I haven't tried debugging R. Could you pl point me to resources on how I can connect to and debug R.dll when I run it via R.Net?

@skyguy94

This comment has been minimized.

Copy link
Contributor

commented Nov 5, 2015

That's somewhat inaccurate. You are correct that you can't get at a first chance exception with a try/ctach block, but you can catch StackOverflow's, but you have to use windbg to do it and not the calling process.

If you've never built R from scratch or run it through the mingw debugger, then its a bigger task than you might realize. Here's a link to the appropriate page in the R Administration manual. https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Building-from-source
You might find the R internals manual useful also. Again, it gets complex quite quickly. Expect to spend some time on it.

@skyguy94

This comment has been minimized.

Copy link
Contributor

commented Nov 5, 2015

@drprabhu

This comment has been minimized.

Copy link
Author

commented Nov 5, 2015

Yeah, haven't built R from scratch. Thanks for the links.

@elgarza8587

This comment has been minimized.

Copy link

commented May 15, 2017

Hi,

is there any update regarding this issue? I have exactly the same issue while trying to load some packages including tm. It works fine on RStudio but on R.NET is giving me the StackOverflowException error.

Any help would be appreciated.

Thanks.

@drprabhu

This comment has been minimized.

Copy link
Author

commented May 19, 2017

@isedwards

This comment has been minimized.

Copy link

commented Jun 17, 2017

@jmp75 is there any chance of this getting fixed? We're experiencing the same problem now that we're using RDotNet in a larger project and it looks like we might have to start looking for other options instead

@skyguy94

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2017

I think that looking into this issue is unlikely given the complexity of getting a debug environment configured and operational (compiled R, windbg, gdb on windows, etc...). We host the R engine with RDotNet in a distributed .net app (we offload processing to multiple instances) and have not run into this issue. My sense is that 'calling the same operation in a tight loop' just isn't a good idea.

As a postmortem to what we've created in our shop, I wouldn't use RDotNet again. Hosting the engine in a .Net process is just incredibly problematic from a runtime and deployment perspective, not to mention the effort required to get distributed/multi-process hosting working. I just don't think that 'accessing R from .Net' is actually something desirable when you just want to pass in scripts and get the results/plots back out.

@drprabhu

This comment has been minimized.

Copy link
Author

commented Jun 17, 2017

@skyguy94

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2017

@drprabhu How well does RServe work for you? I do think that a rserve compatible .net port would be useful for M$ shops that can't quite justify the cost of the (formerly) RevolutionAnalytics stuff M$ is embedding in SqlServer.

@buybackoff

This comment has been minimized.

Copy link

commented Jun 17, 2017

@drprabhu In your particular example you do not dispose the result of Evaluate, which directly leads to the problem described in #8. Cannot reproduce SO exception on my side. Unpreserve is called from ReleaseHandle on object clean-up and your second comment indicates that SO happens on R side, likely when too many objects have been accumulated. Probably a better design would be to make Evaluate() return void and dispose the result internally when it is not needed, and add an overload with out SExp to explicitly keep the result, given that in C# 7.0 out parameters have become very convenient. Such signatures will avoid accidental garbage pile up.

There was a similar bug in RStan: stan-dev/rstan#76

@drprabhu

This comment has been minimized.

Copy link
Author

commented Jun 21, 2017

@drprabhu

This comment has been minimized.

Copy link
Author

commented Jun 21, 2017

@tomekziel

This comment has been minimized.

Copy link

commented Mar 14, 2019

Similar issue with simple repro code using Keras: #90

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.