Skip to content
This repository

added: housekeeping code for addons/packages #323

Merged
1 commit merged into from over 1 year ago

6 participants

CrystalP Vincent Guerci Zeljko Ametovic Martijn Kaijser Jester NedScott
Deleted user

The content you are editing has changed. Reload the page and try again.

this does housekeeping, trying to never go above a given folder size (200MB by default).

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Deleted user

The content you are editing has changed. Reload the page and try again.

@jmarshallnz; all your comments taken into consideration. ok to pull?

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

xbmc/addons/AddonInstaller.cpp
((21 lines not shown))
  348
+  items.Sort(SORT_METHOD_SIZE,SORT_ORDER_DESC);
  349
+  int i=0;
  350
+  while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
  351
+  {
  352
+    size -= items[i]->m_dwSize;
  353
+    CFileUtils::DeleteItem(items[i++],true);
  354
+  }
  355
+
  356
+  if (size > g_advancedSettings.m_addonPackageFolderSize)
  357
+  {
  358
+    // 2. Remove the oldest packages (leaving at least 1 for each add-on)
  359
+    items.Clear();
  360
+    for (std::map<CStdString,CFileItemList*>::iterator it  = packs.begin();
  361
+                                                       it != packs.end();++it)
  362
+    {
  363
+      if (it->second.Size() > 1)
1
CrystalP Collaborator
CrystalP added a note August 06, 2011

build error, should be it->second->Size()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
xbmc/addons/AddonInstaller.cpp
((4 lines not shown))
  331
+void CAddonInstaller::PrunePackageCache()
  332
+{
  333
+  std::map<CStdString,CFileItemList*> packs;
  334
+  int64_t size = EnumeratePackageFolder(packs);
  335
+  if (size < g_advancedSettings.m_addonPackageFolderSize)
  336
+    return;
  337
+
  338
+  // Prune packages
  339
+  // 1. Remove the largest packages, leaving at least 1 for each add-on
  340
+  CFileItemList items;
  341
+  for (std::map<CStdString,CFileItemList*>::const_iterator it  = packs.begin();
  342
+                                                          it != packs.end();++it)
  343
+  {
  344
+    it->second->Sort(SORT_METHOD_LABEL,SORT_ORDER_DESC);
  345
+    for (int j=2;j<it->second->Size();++j)
  346
+      items.Add(CFileItemPtr(new CFileItem(*it->second->Get(j))));
2
CrystalP Collaborator
CrystalP added a note August 06, 2011

That leaves 2 packages, not one as the comment says. I think it makes more sense with 2 as you did here, to always keep the last version and the previous one.

Deleted user
Unknown added a note August 08, 2011

The content you are editing has changed. Reload the page and try again.

we can't invent packages for stuff we only have one for. granted, comment is probably a bit confusing as it's not what the code itself does so shall change it.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
xbmc/addons/AddonInstaller.cpp
((22 lines not shown))
  349
+  int i=0;
  350
+  while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
  351
+  {
  352
+    size -= items[i]->m_dwSize;
  353
+    CFileUtils::DeleteItem(items[i++],true);
  354
+  }
  355
+
  356
+  if (size > g_advancedSettings.m_addonPackageFolderSize)
  357
+  {
  358
+    // 2. Remove the oldest packages (leaving at least 1 for each add-on)
  359
+    items.Clear();
  360
+    for (std::map<CStdString,CFileItemList*>::iterator it  = packs.begin();
  361
+                                                       it != packs.end();++it)
  362
+    {
  363
+      if (it->second.Size() > 1)
  364
+        items.Add(CFileItemPtr(new CFileItem(*it->second->Get(1))));
1
CrystalP Collaborator
CrystalP added a note August 07, 2011

Shouldn't this be a c/p of lines 345-346 (except with "int j=1"), to catch ALL versions, except for the last one?
edit: nm, they've all been deleted in pass1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
CrystalP
Collaborator

Regardless of disk space, I'd personally like to always preserve the two most recent versions per addon, to always have the ability to rollback. I'm comfortable enough that two broken releases in a row are quite unlikely.

One suggestion: if that's not done somewhere else already, use this function to delete the packages of uninstalled addons?

Deleted user

The content you are editing has changed. Reload the page and try again.

hmm.. i sorta agree and sorta disagree. i guess we could change the space limit to be a 'desired max', while allowing more usage to keep two archives around of all.

i was pondering using it to nuke uninstalled addons. but sometimes quick re-adding is also nice. other opinions on that around? i'm fine with going with the consensus..

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Vincent Guerci

Was guessing why a backup was big&slow, just figured addons/packages grew to 830MB over the time...
I'll give it a try and hope it will be merged some day :)
EDIT: Works perfectly, dropped to 166MB as expected, thanks @cptspiff

Zeljko Ametovic
Collaborator

@cptspiff will this go in at any point, my macmini would love you for it.. it has years of updates :)

Deleted user

The content you are editing has changed. Reload the page and try again.

i have done what I am willing to do but it was rejected as can be seen from the comments here.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Martijn Kaijser

@cptspiff
I can't see any comments on why it couldn't be merged or am i missing something?
With the merging an small form HTPC like rpi and the android boxes this would be a very welcome addition to clean disk space

Deleted user

The content you are editing has changed. Reload the page and try again.

crystal p is not satisfied.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Jester

Maybe an ifdef for certain platforms? look at ATV2, lots of questions on the forum about having no space and us pointing out to clean up (it's on the wiki but still) (same goes for higher database revs on upgrade, the old ones stay behind and fill up fast on big databases, but thats off-topic)

NedScott

We NEED this for Frodo. Even if it defaults to off and is turned on in advancedsettings.xml

Deleted user

The content you are editing has changed. Reload the page and try again.

@CrystalP, popular demand.. can be revised later.

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Deleted user ghost merged commit ed0641f into from October 02, 2012
Deleted user ghost closed this October 02, 2012
Tobias Hieta tru referenced this pull request from a commit in plexinc/plex-home-theater-public November 11, 2012
Tobias Hieta Remove PVR keyboard shortcuts
Fixes #323
b5ee0eb
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Oct 02, 2012
added: housekeeping code for addons/packages ba51f55
This page is out of date. Refresh to see the latest.
72  xbmc/addons/AddonInstaller.cpp
@@ -21,10 +21,12 @@
21 21
 #include "AddonInstaller.h"
22 22
 #include "Service.h"
23 23
 #include "utils/log.h"
  24
+#include "utils/FileUtils.h"
24 25
 #include "utils/URIUtils.h"
25 26
 #include "Util.h"
26 27
 #include "guilib/LocalizeStrings.h"
27 28
 #include "filesystem/Directory.h"
  29
+#include "settings/AdvancedSettings.h"
28 30
 #include "settings/GUISettings.h"
29 31
 #include "settings/Settings.h"
30 32
 #include "ApplicationMessenger.h"
@@ -84,6 +86,7 @@ void CAddonInstaller::OnJobComplete(unsigned int jobID, bool success, CJob* job)
84 86
     JobMap::iterator i = find_if(m_downloadJobs.begin(), m_downloadJobs.end(), bind2nd(find_map(), jobID));
85 87
     if (i != m_downloadJobs.end())
86 88
       m_downloadJobs.erase(i);
  89
+    PrunePackageCache();
87 90
   }
88 91
   lock.Leave();
89 92
 
@@ -381,6 +384,75 @@ bool CAddonInstaller::HasJob(const CStdString& ID) const
381 384
   return m_downloadJobs.find(ID) != m_downloadJobs.end();
382 385
 }
383 386
 
  387
+void CAddonInstaller::PrunePackageCache()
  388
+{
  389
+  std::map<CStdString,CFileItemList*> packs;
  390
+  int64_t size = EnumeratePackageFolder(packs);
  391
+  if (size < g_advancedSettings.m_addonPackageFolderSize)
  392
+    return;
  393
+
  394
+  // Prune packages
  395
+  // 1. Remove the largest packages, leaving at least 2 for each add-on
  396
+  CFileItemList items;
  397
+  for (std::map<CStdString,CFileItemList*>::const_iterator it  = packs.begin();
  398
+                                                          it != packs.end();++it)
  399
+  {
  400
+    it->second->Sort(SORT_METHOD_LABEL,SortOrderDescending);
  401
+    for (int j=2;j<it->second->Size();++j)
  402
+      items.Add(CFileItemPtr(new CFileItem(*it->second->Get(j))));
  403
+  }
  404
+  items.Sort(SORT_METHOD_SIZE,SortOrderDescending);
  405
+  int i=0;
  406
+  while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
  407
+  {
  408
+    size -= items[i]->m_dwSize;
  409
+    CFileUtils::DeleteItem(items[i++],true);
  410
+  }
  411
+
  412
+  if (size > g_advancedSettings.m_addonPackageFolderSize)
  413
+  {
  414
+    // 2. Remove the oldest packages (leaving least 1 for each add-on)
  415
+    items.Clear();
  416
+    for (std::map<CStdString,CFileItemList*>::iterator it  = packs.begin();
  417
+                                                       it != packs.end();++it)
  418
+    {
  419
+      if (it->second->Size() > 1)
  420
+        items.Add(CFileItemPtr(new CFileItem(*it->second->Get(1))));
  421
+    }
  422
+    items.Sort(SORT_METHOD_DATE,SortOrderAscending);
  423
+    i=0;
  424
+    while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
  425
+    {
  426
+      size -= items[i]->m_dwSize;
  427
+      CFileUtils::DeleteItem(items[i++],true);
  428
+    }
  429
+  }
  430
+  // clean up our mess
  431
+  for (std::map<CStdString,CFileItemList*>::iterator it  = packs.begin();
  432
+                                                     it != packs.end();++it)
  433
+    delete it->second;
  434
+}
  435
+
  436
+int64_t CAddonInstaller::EnumeratePackageFolder(std::map<CStdString,CFileItemList*>& result)
  437
+{
  438
+  CFileItemList items;
  439
+  CDirectory::GetDirectory("special://home/addons/packages/",items,".zip",DIR_FLAG_NO_FILE_DIRS);
  440
+  int64_t size=0;
  441
+  for (int i=0;i<items.Size();++i)
  442
+  {
  443
+    if (items[i]->m_bIsFolder)
  444
+      continue;
  445
+    size += items[i]->m_dwSize;
  446
+    CStdString pack,dummy;
  447
+    AddonVersion::SplitFileName(pack,dummy,items[i]->GetLabel());
  448
+    if (result.find(pack) == result.end())
  449
+      result[pack] = new CFileItemList;
  450
+    result[pack]->Add(CFileItemPtr(new CFileItem(*items[i])));
  451
+  }
  452
+
  453
+  return size;
  454
+}
  455
+
384 456
 CAddonInstallJob::CAddonInstallJob(const AddonPtr &addon, const CStdString &hash, bool update, const CStdString &referer)
385 457
 : m_addon(addon), m_hash(hash), m_update(update), m_referer(referer)
386 458
 {
3  xbmc/addons/AddonInstaller.h
@@ -122,6 +122,9 @@ class CAddonInstaller : public IJobCallback
122 122
    */
123 123
   bool DoInstall(const ADDON::AddonPtr &addon, const CStdString &hash = "", bool update = false, const CStdString &referer = "", bool background = true);
124 124
 
  125
+  void PrunePackageCache();
  126
+  int64_t EnumeratePackageFolder(std::map<CStdString,CFileItemList*>& result);
  127
+
125 128
   CCriticalSection m_critSection;
126 129
   JobMap m_downloadJobs;
127 130
   CStopWatch m_repoUpdateWatch;   ///< repository updates are done based on this counter
2  xbmc/settings/AdvancedSettings.cpp
@@ -298,6 +298,7 @@ void CAdvancedSettings::Initialize()
298 298
   m_measureRefreshrate = false;
299 299
 
300 300
   m_cacheMemBufferSize = 1024 * 1024 * 20;
  301
+  m_addonPackageFolderSize = 200*1024*1024;
301 302
 
302 303
   m_jsonOutputCompact = true;
303 304
   m_jsonTcpPort = 9090;
@@ -759,6 +760,7 @@ void CAdvancedSettings::ParseSettingsFile(const CStdString &file)
759 760
   XMLUtils::GetUInt(pRootElement,"restrictcapsmask", m_RestrictCapsMask);
760 761
   XMLUtils::GetFloat(pRootElement,"sleepbeforeflip", m_sleepBeforeFlip, 0.0f, 1.0f);
761 762
   XMLUtils::GetBoolean(pRootElement,"virtualshares", m_bVirtualShares);
  763
+  XMLUtils::GetUInt(pRootElement, "packagefoldersize", m_addonPackageFolderSize);
762 764
 
763 765
   //Tuxbox
764 766
   pElement = pRootElement->FirstChildElement("tuxbox");
1  xbmc/settings/AdvancedSettings.h
@@ -346,6 +346,7 @@ class CAdvancedSettings
346 346
     bool m_guiVisualizeDirtyRegions;
347 347
     int  m_guiAlgorithmDirtyRegions;
348 348
     int  m_guiDirtyRegionNoFlipTimeout;
  349
+    unsigned int m_addonPackageFolderSize;
349 350
 
350 351
     unsigned int m_cacheMemBufferSize;
351 352
 
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.