Still running out of memory

[calebbraun]

Running the query "GHG by subsector" once again gave me the error:
[XPTY0004] RunMIQuery:runMIQuery(element(), xs:string, empty-sequence()): GC overhead limit exceeded.

I was able to run it by upping the memory limit from 4 to 8 GB, but why does one query use so much memory anyway? The entire DB I'm querying is only 1.45 GB to start with.

[pralitp]

@xavier-gutierrez also noticed queries were using all of the cores on the machine just to process a single query. Even after the Model Interface logging indicates that the query is "finished" processing.

My current thinking is something has changed in BaseX since we upgraded versions with regards to how it serializes the output generated from the Model Interface which causes it to use a lot of memory and CPU. I'll dig into this a bit and see if we can't do better.

[russellhz]

any update on this?

@d3y419 and I have a project using iamrpt that relies on lots of GHG emissions queries and we ran into the same problem

[rplzzz]

Is this really an rgcam problem? It seems like the issue is in the Model Interface. Or is the idea here that there is something about the way that rgcam is running the Model Interface that is causing this problem?

[calebbraun]

I thought the issue was with rgcam because running the same query on the same database through Model Interface directly goes quickly and doesn't show large memory use.

[rplzzz]

Interesting. What is the command that you use to run the query through the Model Interface directly?

[calebbraun]

I ran it through the GUI using a batch query file.

[pralitp]

Sorry I haven't filled in any details about this issue yet. Well, it is sort of an rgcam issue. It appears the explosion in memory usage occurs when serializing results to get passed to rgcam.

This process is part Model Interface and part BaseX (When running batch queries in just the Model interface it is all Model Interface thus we do not see this issue) where the model interface must generate output in one of the BaseX structure they support for serializing and from there it will handle converting to CSV (or potentially other formats).

I tried switching to another BaseX structure which seem like it should do better with memory -- didn't help. I tried digging into he BaseX source code to see what I could get away with in terms of avoiding copying memory in the data structure I pass for serialization (a sort of copy on access scheme) -- didn't help.

And there is where I left it as I didn't (still don't?) have time to dig into it any further.

[russellhz]

Can the memory limit be increased again? A client needs to be able to run GHG queries from iamrpt.