Looking up file sizes imposes serious lag #40
When the amount was governed by file sizes we needed file sizes on every file. As part of creating a combo instance an expansion was done on all file entries, including adding information about file size. When this is not included in the feed, we resort to pinging each url in turn to gather this information. For a long feed this can take several minutes.
This should only happen once, because the entryinfo.expand function is only run on entries not in jar. However, it seems to be a returning issue in some cases....?
Combo required expanded entry info on all entries in feed. Since only info on kept files were saved to jar, each checkup meant re-pinging each entry. Obscuring the issue was that etag value was regularly renewed despite there being no new content. So if you checked twice, one after the other, there would e no lag (because old etag = new etag) but as soon as there was a new etag a full checkup would be performed, causing a delay.