Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Podcasts: Pre-caching image and more-info data can bring the server to a crawl #668

Merged
merged 5 commits into from
Sep 28, 2021

Commits on Sep 26, 2021

  1. Podcast: Pre-caching - Guard against fetching same feed multiple times

    When browsing within the podcast menu, pre-caching activity will be
    initiated for each feed where pre-cached image or podcast info data is
    not found.
    
    A guard is needed to prevent launching the pre-cache process multiple
    times for the same feed. This will happen whenever the user is browsing
    within the podcast menu and a pre-existing pre-cache process has been
    launched but has not completed.
    
    This can have an adverse impact on modestly powered servers, and/or
    where there are a large number of feeds to be handled.
    
    This change handles the issue by pre-caching place-holder data each time
    a feed pre-cache process is started. The place-holder data will be
    replaced by real data when the process completes.
    mw9 committed Sep 26, 2021
    Configuration menu
    Copy the full SHA
    450ad75 View commit details
    Browse the repository at this point in the history
  2. Podcast: Pre-caching - Randomize image & moreInfo caching period

    At present, a podcast's 'moreInfo' data is cached for the default period
    of one hour, in contrast to image data which is cached for 90 days.
    
    This generates regular pre-caching activity, at at least hourly intervals,
    whenever the user browses within the podcast menu. This can become a
    significant burden if the user subscribes to more than a handful of feeds.
    
    This change aligns the caching period with image data. Staleness will not
    be a significant issue, as this data rarely changes.
    
    Additionally, the caching period is now randomized to fall within a window
    of +/- 10 days about 90 days. This will reduce the chance that all cached
    data becomes stale at the same time, thereby spreading the impact of
    pre-caching activity over a number of user browse sessions.
    mw9 committed Sep 26, 2021
    Configuration menu
    Copy the full SHA
    676580a View commit details
    Browse the repository at this point in the history
  3. Podcast: Pre-caching - Refresh pre-cached data while browsing a feed

    Take the opportunity to refresh/update a podcast's pre-cached data while
    browsing the feed.
    
    This is, essentially, free, and extends the life of the cached data.
    
    By preventing cached data from becoming stale, we reduce the burden of
    podcast feed pre-caching activity that can take place whenever the user
    browses within the podcast menu.
    mw9 committed Sep 26, 2021
    Configuration menu
    Copy the full SHA
    d0b9c8c View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2021

  1. Podcast: Pre-caching - Schedule activity over an extended period

    This change adapts the podcast image and 'more info' pre-caching
    activity to take place over an extended period of time.
    
    The intention is to avoid overly burdening modestly endowed servers.
    Some users may have a considerable number of subscribed feeds (in excess
    of 100 have been reported), and to fetch and parse this number of feeds
    all at the same time can cause considerable delays (reportedly in the
    10s of seconds) when browsing the podcast UI.
    
    This will be particularly noticeable when using the podcast menu after
    clearing server caches, or re-installing LMS. In those circumstances all
    feeds need to be pre-cached, and this will be attempted on first use of
    the podcast menu.
    
    With this change, fetching and parsing each feed takes place
    sequentially, So the server will only be processing one feed at a time,
    instead of processing all feeds concurrently.
    
    No explicit expiry time need be set for the retrieved feed, we will be
    caching the data we want from it.
    mw9 committed Sep 27, 2021
    Configuration menu
    Copy the full SHA
    fa945f7 View commit details
    Browse the repository at this point in the history
  2. Podcast: Pre-caching - Sanity check on params to preCacheFeedData

    sub 'preCacheFeedData' requires a URL and a feed as parameters.
    Check that we are given something that looks believable.
    
    Also, 'preCacheFeedData' is already called by the parser, we don't need
    to do it again.
    mw9 committed Sep 27, 2021
    Configuration menu
    Copy the full SHA
    f96ff71 View commit details
    Browse the repository at this point in the history