Tomcat: Too many open files error #162

cmdcolin · 2015-02-06T20:43:19Z

If a bunch of tracks are opened at once, then many duplicate file handles are opened for config.properties, basically until it hits the system limit.

Normal operation: around 250 file handles open

After opening up about 10 bam tracks and scrolling around: over 800 file handles open, all duplicates of config.properties

It will just increase until the "too many open files error" or the garbage collector picks up the objects

nathandunn · 2015-02-07T01:12:09Z

@cmdcolin It looks like it opens 1-3 files per track loaded and doesn't seem to increase (most of the time) and will eventually release the file handles.

Loading the bam track (Forager RNA-Seq reads), I see that it loads around 14 . . . goes up to 20 or so and then plateaus.

With 3 BAM files I got it to around 80-100 . . and it kind of maxed out (moving back and forth and changing groups).

That being said, in the code there is this line . . can't but help to fix:

    // TODO: move up so not recalculating each time
    String pathTranslated = request.getPathTranslated();
    String rootPath = pathTranslated.substring(0, pathTranslated.length() - request.getPathInfo().length());
    String configPath = rootPath + "/config/config.properties";


    File propertyFile = new File(configPath);
    String filename = null;

cmdcolin · 2015-02-07T01:17:26Z

should probably call propertyFile.close()

I wasn't sure whether we should put it in the try{} finally {} block or just manually close it

cmdcolin · 2015-02-07T01:18:31Z

Actually we should probably just load the file into memory so that it doesn't have to do that on every get request right?

nathandunn · 2015-02-07T01:23:38Z

I think it is working now . . . It probably wouldn't hurt to have it in the init, but performance wise it doesn't make any difference and I am using the "request" explicitly.

nathandunn · 2015-02-07T01:38:37Z

That is what I ended up doing. I had a note to do that eventually.

Nathan

On Feb 6, 2015, at 5:18 PM, Colin Diesh notifications@github.com wrote:

Actually we should probably just load the file into memory so that it doesn't have to do that on every get request right?

—
Reply to this email directly or view it on GitHub #162 (comment).

nathandunn · 2015-02-09T18:21:55Z

@cmdcolin If you think this fixed it I would say go ahead and close it.

cmdcolin · 2015-02-09T18:43:53Z

Thanks it seems to work so i'll close it for now

It probably wouldn't hurt to have it in the init

I would prefer this, it looks a little messy currently, so...whenever we get a chance in our copious amounts of free time...refactor it

cmdcolin · 2015-02-09T18:57:10Z

Reopen:

I thought this was a fluke but the first time you open webapollo after a deploy, it returns

SyntaxError: expected expression, got ')' when parsing data/trackList.json.

This is reproducable on two different servers but I haven't narrowed down the cause yet.

nathandunn · 2015-02-09T19:35:45Z

Hmm . . do you have the trackList.json? It could be something introduced by the JBrowse Servlet, but I suspect its a faulty tracklist.json. I just tested this and it worked great.

cmdcolin · 2015-02-09T20:02:35Z

I'm fairly sure it is independant of trackList.json (i tested two different servers, local dev and production) and both have it

It only happens the first time you access it, after that, presumably, the config.properties is initialized (since it is initialized within the GET request handler)

For completeness though here is a basic trackList.json

==removed==

Again, the error goes away after you refresh again

nathandunn · 2015-02-09T20:05:28Z

I but it is having all of the asynchronous requests coming through. I guess we’ll have to move it to the init servlet method. Give me a few to fix this.

Nathan

On Feb 9, 2015, at 12:02 PM, Colin Diesh notifications@github.com wrote:

I'm fairly sure it is independant of trackList.json (i tested two different servers, local dev and production) and both have it

It only happens the first time you access it, after that, presumably, the config.properties is initialized (since it is initialized within the GET request handler)

For completeness though here is a basic trackList.json

{
"share_link" : 0,
"plugins" : [
{
"location" : "./plugins/WebApollo",
"name" : "WebApollo"
}
],
"tracks" : [
{
"chunkSize" : 20000,
"storeClass" : "JBrowse/Store/Sequence/StaticChunked",
"urlTemplate" : "seq/{refseq_dirpath}/{refseq}-",
"category" : "Reference sequence",
"type" : "SequenceTrack",
"label" : "DNA",
"key" : "Reference sequence"
},
{
"autocomplete" : "none",
"style" : {
"className" : "annot",
"renderClassName" : "annot-render",
"subfeatureClasses" : {
"non_canonical_three_prime_splice_site" : "noncanonical-splice-site",
"wholeCDS" : null,
"exon" : "container-100pct",
"CDS" : "annot-CDS",
"non_canonical_five_prime_splice_site" : "noncanonical-splice-site",
"UTR" : "annot-UTR"
},
"arrowheadClass" : "annot-arrowhead",
"alternateClasses" : {
"transposable_element" : {
"renderClassName" : "blue-ibeam-render",
"className" : "blue-ibeam"
},
"pseudogene" : {
"renderClassName" : "gray-center-30pct",
"className" : "light-purple-80pct"
},
"snRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"rRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"snoRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"repeat_region" : {
"className" : "magenta-80pct"
},
"tRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"ncRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"miRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
}
},
"uniqueIdField" : "id",
"centerSubFeature" : {
"non_canonical_three_prime_splice_site" : false,
"non_canonical_five_prime_splice_site" : false
}
},
"key" : "User-created Annotations",
"storeClass" : "WebApollo/Store/SeqFeature/ScratchPad",
"disableJBrowseMode" : false,
"phase" : 0,
"compress" : 0,
"label" : "Annotations",
"type" : "WebApollo/View/Track/AnnotTrack",
"subfeatures" : 1
}
],
"names" : {
"url" : "names/",
"type" : "Hash"
},
"favicon" : "./plugins/WebApollo/img/webapollo_favicon.ico",
"alwaysOnTracks" : "DNA,Annotations",
"formatVersion" : 1
}
https://cloud.githubusercontent.com/assets/6511937/6114505/3fbfeb1a-b064-11e4-9cc2-970de6742e1f.jpg
Again, the error goes away after you refresh again

—
Reply to this email directly or view it on GitHub #162 (comment).

nathandunn · 2015-02-09T20:27:09Z

There were a few problems . . . and it was serendipity that it was working in the first place (properties was checking for a global, but assigning a local).

Both of these are now fixed (doing in the init servlet).

Let me know if it doesn’t fix it for you. Still not sure why it ended up working for me.

Nathan

On Feb 9, 2015, at 12:02 PM, Colin Diesh notifications@github.com wrote:

I'm fairly sure it is independant of trackList.json (i tested two different servers, local dev and production) and both have it

It only happens the first time you access it, after that, presumably, the config.properties is initialized (since it is initialized within the GET request handler)

For completeness though here is a basic trackList.json

{
"share_link" : 0,
"plugins" : [
{
"location" : "./plugins/WebApollo",
"name" : "WebApollo"
}
],
"tracks" : [
{
"chunkSize" : 20000,
"storeClass" : "JBrowse/Store/Sequence/StaticChunked",
"urlTemplate" : "seq/{refseq_dirpath}/{refseq}-",
"category" : "Reference sequence",
"type" : "SequenceTrack",
"label" : "DNA",
"key" : "Reference sequence"
},
{
"autocomplete" : "none",
"style" : {
"className" : "annot",
"renderClassName" : "annot-render",
"subfeatureClasses" : {
"non_canonical_three_prime_splice_site" : "noncanonical-splice-site",
"wholeCDS" : null,
"exon" : "container-100pct",
"CDS" : "annot-CDS",
"non_canonical_five_prime_splice_site" : "noncanonical-splice-site",
"UTR" : "annot-UTR"
},
"arrowheadClass" : "annot-arrowhead",
"alternateClasses" : {
"transposable_element" : {
"renderClassName" : "blue-ibeam-render",
"className" : "blue-ibeam"
},
"pseudogene" : {
"renderClassName" : "gray-center-30pct",
"className" : "light-purple-80pct"
},
"snRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"rRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"snoRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"repeat_region" : {
"className" : "magenta-80pct"
},
"tRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"ncRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
},
"miRNA" : {
"renderClassName" : "gray-center-30pct",
"className" : "brightgreen-80pct"
}
},
"uniqueIdField" : "id",
"centerSubFeature" : {
"non_canonical_three_prime_splice_site" : false,
"non_canonical_five_prime_splice_site" : false
}
},
"key" : "User-created Annotations",
"storeClass" : "WebApollo/Store/SeqFeature/ScratchPad",
"disableJBrowseMode" : false,
"phase" : 0,
"compress" : 0,
"label" : "Annotations",
"type" : "WebApollo/View/Track/AnnotTrack",
"subfeatures" : 1
}
],
"names" : {
"url" : "names/",
"type" : "Hash"
},
"favicon" : "./plugins/WebApollo/img/webapollo_favicon.ico",
"alwaysOnTracks" : "DNA,Annotations",
"formatVersion" : 1
}
https://cloud.githubusercontent.com/assets/6511937/6114505/3fbfeb1a-b064-11e4-9cc2-970de6742e1f.jpg
Again, the error goes away after you refresh again

—
Reply to this email directly or view it on GitHub #162 (comment).

cmdcolin · 2015-02-09T21:15:13Z

Looks good now

cmdcolin added this to the 1.0.4 milestone Feb 6, 2015

cmdcolin assigned nathandunn Feb 6, 2015

nathandunn added a commit that referenced this issue Feb 7, 2015

#162 - only getting property file once and closign file lookup

bd19eec

nathandunn added the 1 - Ready label Feb 7, 2015

nathandunn assigned cmdcolin and unassigned nathandunn Feb 7, 2015

nathandunn added a commit that referenced this issue Feb 7, 2015

#162 - a minor refactor

421a4f1

cmdcolin closed this as completed Feb 9, 2015

cmdcolin reopened this Feb 9, 2015

nathandunn added a commit that referenced this issue Feb 9, 2015

#162 - moved property file to init servlet

9279a8c

cmdcolin closed this as completed Feb 9, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tomcat: Too many open files error #162

Tomcat: Too many open files error #162

cmdcolin commented Feb 6, 2015

nathandunn commented Feb 7, 2015

cmdcolin commented Feb 7, 2015

cmdcolin commented Feb 7, 2015

nathandunn commented Feb 7, 2015

nathandunn commented Feb 7, 2015

nathandunn commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

nathandunn commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

nathandunn commented Feb 9, 2015

nathandunn commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

Tomcat: Too many open files error #162

Tomcat: Too many open files error #162

Comments

cmdcolin commented Feb 6, 2015

nathandunn commented Feb 7, 2015

cmdcolin commented Feb 7, 2015

cmdcolin commented Feb 7, 2015

nathandunn commented Feb 7, 2015

nathandunn commented Feb 7, 2015

nathandunn commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

nathandunn commented Feb 9, 2015

cmdcolin commented Feb 9, 2015

nathandunn commented Feb 9, 2015

nathandunn commented Feb 9, 2015

cmdcolin commented Feb 9, 2015