Skip to content
This repository

changed: upgrade memory based cache to filebased after having been full f #86

Closed
wants to merge 1 commit into from

3 participants

Joakim Plate davilla tiben20
Joakim Plate
Collaborator

changed: upgrade memory based cache to filebased after having been full for 2 seconds without anything reading out of it

By starting out with a memory based cache, then replacing that with a file based one on need we avoid the rather large overhead a file based cache introduces. However it can currently not discern the difference between paused playback on a really fast network, and on a slow one. So if you play over http from local network for example and pause playback. The whole file will be downloaded to local disk at maximum network speed.

I'm hesitant to this feature. I suspect it must be possible to turn on and off. On embedded system we may not have diskspace enough to do this. And currently i don't think the file based cache handles the situation of running out of space well at all.

Comments are very welcome to how we should handle a feature like this.

Joakim Plate changed: upgrade memory based cache to filebased after having been fu…
…ll for 2 seconds without anything reading out of it

By starting out with a memory based cache, then replacing that with a file based one on need we avoid the rather large overhead a file based cache introduces. However it can currently not discern the difference between paused playback on a really fast network, and on a slow one. So if you play over http from local network for example and pause playback. The whole file will be downloaded to local disk at maximum network speed.
d633aea
davilla
Collaborator

Seems to work quite well under OSX. No issues that I can see yet. iOS testing tonight.

Joakim Plate
Collaborator

Nice to hear. I've realized a rather major bug with the file based cache thou which must be fixed before this goes in. If the free disk space of the local machine does not support the size of the streamed file, playback will break down.

So two things really needs to be fixed here. One verify available disk space. Define some limit as to how much free space we always must leave. Something like atleast 200 megs (computers do funny things when totally out of disk space).

Fix the SimpleFileCache to wrap around similary to how the new mem buffer does. So it doesn't break down with a limited amount of disk space.

davilla
Collaborator

what's a good way to trigger this ? iOS is fine but I don't really see the file cache getting created.

Joakim Plate
Collaborator

Normally it would just be pause playback on a file larger than 20megs+8*video byterate, wait for the file cache to fill larger than 20+8*videobyterate. (is visible in the codec info data)

davilla
Collaborator

humm, can't seem to force it. will beat on it some more.

Deleted user
ghost commented April 22, 2011

The content you are editing has changed. Reload the page and try again.

hi elupus, tiben20 has also been experimenting with file caches

(for my benefit, because i made an addon that streams .avi, which was having buffering issues on slower connections)
http://forum.xbmc.org/showthread.php?t=98782 http://trac.xbmc.org/ticket/11416 (download links are in the trac ticket)

i've thought of 2 more things that could trigger it:

A) check whether stream supports skipping, if it doesn't, force file cache.
(prevents a bug on streams that don't support skipping where pausing for too long kills the stream)

B) allow addon coders to set in xbmc.player.play a cache file path (which also forces file cache)
(this lets us provide a cool Stream + Download option in addons) ---- tiben's patch does this

*PS for the streams that don't support skipping, another nice thing might be to support skipping within the already cached video, but refusing to try to skip beyond the cache limit because we know we won't be able to.

i hope my suggestions help!

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Joakim Plate
Collaborator

the current filecache logic will not guarantee a complete download thou. if a seek request is made (not neccessarily by user, could be done by demuxer) it will just maintain some part of the file. So it would most likely be corrupt if you try to reuse it.

Deleted user
ghost commented April 23, 2011

The content you are editing has changed. Reload the page and try again.

ah ok, AFAIK tiben's patch completely switches to use the curl download code used elsewhere in xbmc (like for downloading images) to guarantee a complete download. (if the stream is file cached from the very start, and the stream is not skipped around in)

i'll ask him to post on this discussion so he can coordinate.

apologies if i have hijacked this pull request with scope creep, maybe i should ask tiben to make a new pull request

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

tiben20 tiben20 closed this April 25, 2011
tiben20
Collaborator

anarchintosh elupus is right about the current seek that will corrupt the download. My test were done with megaupload free account this is why i didn't see it right away.
I'll see if i can make the seek happen on what as already been cached instead of making it on the input stream directly

tiben20 tiben20 reopened this April 25, 2011
Joakim Plate
Collaborator

This won't merge anymore and i'm not sure i see much point in it anymore.

Joakim Plate elupus closed this April 15, 2012
Deleted user
ghost commented April 19, 2012

The content you are editing has changed. Reload the page and try again.

i still see the point in it... pausing something to load (because it is a slow connection) and then finding it has not is very frustrating.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

ryanroth ryanroth referenced this pull request from a commit in garbear/xbmc March 12, 2013
Garrett Brown Savegames - success b3f95f2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Mar 29, 2011
Joakim Plate changed: upgrade memory based cache to filebased after having been fu…
…ll for 2 seconds without anything reading out of it

By starting out with a memory based cache, then replacing that with a file based one on need we avoid the rather large overhead a file based cache introduces. However it can currently not discern the difference between paused playback on a really fast network, and on a slow one. So if you play over http from local network for example and pause playback. The whole file will be downloaded to local disk at maximum network speed.
d633aea
This page is out of date. Refresh to see the latest.
1  xbmc/filesystem/CacheCircular.h
@@ -43,6 +43,7 @@ class CCacheCircular : public CCacheStrategy
43 43
 
44 44
     virtual int64_t Seek(int64_t pos) ;
45 45
     virtual void Reset(int64_t pos) ;
  46
+    virtual int64_t GetCacheStart() { return m_beg; }
46 47
 
47 48
 protected:
48 49
     uint64_t          m_beg;       /**< index in file (not buffer) of beginning of valid data */
1  xbmc/filesystem/CacheMemBuffer.h
@@ -47,6 +47,7 @@ class CacheMemBuffer : public CCacheStrategy
47 47
 
48 48
     virtual int64_t Seek(int64_t iFilePosition) ;
49 49
     virtual void Reset(int64_t iSourcePosition) ;
  50
+    virtual int64_t GetCacheStart() { return m_nStartPosition; }
50 51
 
51 52
 protected:
52 53
     int64_t m_nStartPosition;
3  xbmc/filesystem/CacheStrategy.h
@@ -57,6 +57,7 @@ class CCacheStrategy{
57 57
   virtual void EndOfInput(); // mark the end of the input stream so that Read will know when to return EOF
58 58
   virtual bool IsEndOfInput();
59 59
   virtual void ClearEndOfInput();
  60
+  virtual int64_t GetCacheStart() = 0;
60 61
 
61 62
   CEvent m_space;
62 63
 protected:
@@ -81,6 +82,8 @@ class CSimpleFileCache : public CCacheStrategy {
81 82
   virtual void Reset(int64_t iSourcePosition);
82 83
   virtual void EndOfInput();
83 84
 
  85
+  virtual int64_t GetCacheStart() { return m_nStartPosition; }
  86
+
84 87
   int64_t  GetAvailableRead();
85 88
 
86 89
 protected:
71  xbmc/filesystem/FileCache.cpp
@@ -29,11 +29,13 @@
29 29
 #include "threads/SingleLock.h"
30 30
 #include "utils/log.h"
31 31
 #include "settings/AdvancedSettings.h"
  32
+#include "utils/TimeUtils.h"
32 33
 
33 34
 using namespace AUTOPTR;
34 35
 using namespace XFILE;
35 36
 
36 37
 #define READ_CACHE_CHUNK_SIZE (64*1024)
  38
+#define WRITE_CACHE_UPGRADE_TIMEOUT 2000
37 39
 
38 40
 CFileCache::CFileCache()
39 41
 {
@@ -122,6 +124,37 @@ bool CFileCache::Open(const CURL& url)
122 124
   return true;
123 125
 }
124 126
 
  127
+static bool CopyCacheStrategy(CCacheStrategy *dst, CCacheStrategy *src)
  128
+{
  129
+  if(dst->Open() != CACHE_RC_OK)
  130
+    return false;
  131
+
  132
+  dst->Reset(src->GetCacheStart());
  133
+  if(src->Seek(src->GetCacheStart()) < 0)
  134
+    return false;
  135
+
  136
+  const size_t    data_size = 32 * 1024;
  137
+  auto_aptr<char> data(new char[data_size]);
  138
+
  139
+  while(1)
  140
+  {
  141
+    int res_r = src->ReadFromCache(data.get(), data_size);
  142
+    if(res_r == 0 || res_r == CACHE_RC_WOULD_BLOCK)
  143
+      break;
  144
+
  145
+    if(res_r < 0)
  146
+      return false;
  147
+
  148
+    for(int pos = 0, res_w = 0; pos < res_r; pos += res_w)
  149
+    {
  150
+      res_w = dst->WriteToCache(data.get()+pos, res_r - pos);
  151
+      if(res_w <= 0)
  152
+        return false;
  153
+    }
  154
+  }
  155
+  return true;
  156
+}
  157
+
125 158
 void CFileCache::Process()
126 159
 {
127 160
   if (!m_pCache) {
@@ -181,6 +214,7 @@ void CFileCache::Process()
181 214
       m_bStop = true;
182 215
 
183 216
     int iTotalWrite=0;
  217
+    unsigned iTimeBlocked = 0;
184 218
     while (!m_bStop && (iTotalWrite < iRead))
185 219
     {
186 220
       int iWrite = 0;
@@ -195,9 +229,40 @@ void CFileCache::Process()
195 229
         break;
196 230
       }
197 231
       else if (iWrite == 0)
198  
-        m_pCache->m_space.WaitMSec(5);
199  
-
200  
-      iTotalWrite += iWrite;
  232
+      {
  233
+        if(!m_pCache->m_space.WaitMSec(5))
  234
+        {
  235
+          if(iTimeBlocked == 0)
  236
+            iTimeBlocked = CTimeUtils::GetTimeMS();
  237
+
  238
+          if(iTimeBlocked + WRITE_CACHE_UPGRADE_TIMEOUT < CTimeUtils::GetTimeMS() && typeid(*m_pCache) != typeid(CSimpleFileCache) )
  239
+          {
  240
+            iTimeBlocked = 0;
  241
+
  242
+            CSingleLock lock(m_sync);
  243
+            CCacheStrategy *cache = new CSimpleFileCache();
  244
+            if(CopyCacheStrategy(cache, m_pCache))
  245
+            {
  246
+              CLog::Log(LOGDEBUG, "CFileCache::Process - dumped memory cache to file");
  247
+              delete m_pCache;
  248
+              m_pCache = cache;
  249
+            }
  250
+            else
  251
+            {
  252
+              CLog::Log(LOGERROR, "CFileCache::Process - failed to dump memory cache to file");
  253
+              delete cache;
  254
+            }
  255
+            /* restore correct position */
  256
+            if(m_pCache->Seek(m_readPos) < 0)
  257
+              CLog::Log(LOGERROR, "CFileCache::Process - failed to restore file position");
  258
+          }
  259
+        }
  260
+      }
  261
+      else
  262
+      {
  263
+        iTimeBlocked = 0;
  264
+        iTotalWrite += iWrite;
  265
+      }
201 266
 
202 267
       // check if seek was asked. otherwise if cache is full we'll freeze.
203 268
       if (m_seekEvent.WaitMSec(0))
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.