Pileup fetch limit for deep coverage loci

[malonge]

Hi @cmdcolin and thanks in advance for your help,

Suppose we have a BAM with sparse and uneven coverage and in some loci there is excessive coverage (e.g. >5M reads). In these situations we tend to see something like the following error

Error: HTTP 500 fetching https://path/to/alns.bam bytes 523632640-628359167

../../../packages/core/util/io/RemoteFileWithRangeCache.ts:97:13 (at c.fetchBinaryRange ()
JBrowse 3.2.0

In these situations I am wondering if there is something like a coverage limit such that JBrowse will fetch up until a certain amount of reads and display what it was able to fetch with some sort of warning. Rather than what I assume can happen otherwise where JBrowse is fetching all the reads but not necessarily displaying all of them depending on the configuration.

Or in general I wonder if you recommend an existing configuration to deal with this. The best I could think of otherwise would be to do something like downsample, which is doable but obviously less convenient.

Thanks,
Mike

[cmdcolin]

Hi Mike,
I am imagining the google cloud which i recall you mentioning is probably having a 100Mb max range request size
I made a new PR here that might be able to help address this issue.

so i made this pr with 32mb max range size by forcing large byte range requests to be split into several smaller ones. #5576

I think it will help with your case

It looks like you are on a pretty old version of jbrowse now, if it is desirable i can potentially backport this PR to v3.2.0 older version or help with upgrade to latest