Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Looking up file sizes imposes serious lag #40

Closed
brokkr opened this issue Dec 21, 2016 · 2 comments

Comments

@brokkr
Copy link
Owner

commented Dec 21, 2016

When the amount was governed by file sizes we needed file sizes on every file. As part of creating a combo instance an expansion was done on all file entries, including adding information about file size. When this is not included in the feed, we resort to pinging each url in turn to gather this information. For a long feed this can take several minutes.

This should only happen once, because the entryinfo.expand function is only run on entries not in jar. However, it seems to be a returning issue in some cases....?

Options:

  • Investigate if it is indeed a returning issue or just a one time thing per feed
  • Remove all references to file size (we aren't using it currently but it might return?)
  • Work around the fact that some entries will not have file size information

@brokkr brokkr added the bug label Dec 21, 2016

@brokkr

This comment has been minimized.

Copy link
Owner Author

commented Jan 4, 2017

Combo required expanded entry info on all entries in feed. Since only info on kept files were saved to jar, each checkup meant re-pinging each entry. Obscuring the issue was that etag value was regularly renewed despite there being no new content. So if you checked twice, one after the other, there would e no lag (because old etag = new etag) but as soon as there was a new etag a full checkup would be performed, causing a delay.

@brokkr brokkr closed this Jan 4, 2017

@brokkr brokkr reopened this Jan 4, 2017

@brokkr

This comment has been minimized.

Copy link
Owner Author

commented Jan 4, 2017

Solution: Skip pinging files for size info altogether. If file size is not included, it's reported as 'Unknown'. This is only used for output.

@brokkr brokkr closed this Jan 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.