[bwGetOverlappingIntervalsCore] Got an error

[zoink]

Thanks for the help on the last error!

The error I get now is:
[bwGetOverlappingIntervalsCore] Got an error
An error occurred while fetching values!

I'm fetching dnase values quickly with a PyTorch dataloader (i.e. bw.values(chrm,start,end)). The error doesn't occur when I'm using the main process (num_workers=0) to fetch the values.

When I set num_workers to 8 like below then I get the error.
loader=DataLoader(valdata,batch_size=20,num_workers=8)

num_workers refers to the number of subprocesses fetching the data.
"how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process (default: 0)" from PyTorch doc.

So I think the error might have to do with simultaneously fetching multiple interval values from the same open file.

Is there anyway to circumvent this? Thanks so much!

[dpryan79]

This happens frequently when using multiple threads to fetch data. The cause is usually the remote server limiting the number of concurrent connections. Consequently you're best served to minimizing the number of threads concurrently contacting the same server.

[zoink]

Thanks, I was able to increase num_workers once I requested a node with more cores/sockets/threads.