Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

added: housekeeping code for addons/packages #323

Merged
1 commit merged into from

6 participants

CrystalP Vincent Guerci Zeljko Ametovic Martijn Kaijser Jester NedScott
Deleted user

this does housekeeping, trying to never go above a given folder size (200MB by default).

Deleted user

@jmarshallnz; all your comments taken into consideration. ok to pull?

xbmc/addons/AddonInstaller.cpp
((21 lines not shown))
+ items.Sort(SORT_METHOD_SIZE,SORT_ORDER_DESC);
+ int i=0;
+ while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
+ {
+ size -= items[i]->m_dwSize;
+ CFileUtils::DeleteItem(items[i++],true);
+ }
+
+ if (size > g_advancedSettings.m_addonPackageFolderSize)
+ {
+ // 2. Remove the oldest packages (leaving at least 1 for each add-on)
+ items.Clear();
+ for (std::map<CStdString,CFileItemList*>::iterator it = packs.begin();
+ it != packs.end();++it)
+ {
+ if (it->second.Size() > 1)
CrystalP Collaborator
CrystalP added a note

build error, should be it->second->Size()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
xbmc/addons/AddonInstaller.cpp
((4 lines not shown))
+void CAddonInstaller::PrunePackageCache()
+{
+ std::map<CStdString,CFileItemList*> packs;
+ int64_t size = EnumeratePackageFolder(packs);
+ if (size < g_advancedSettings.m_addonPackageFolderSize)
+ return;
+
+ // Prune packages
+ // 1. Remove the largest packages, leaving at least 1 for each add-on
+ CFileItemList items;
+ for (std::map<CStdString,CFileItemList*>::const_iterator it = packs.begin();
+ it != packs.end();++it)
+ {
+ it->second->Sort(SORT_METHOD_LABEL,SORT_ORDER_DESC);
+ for (int j=2;j<it->second->Size();++j)
+ items.Add(CFileItemPtr(new CFileItem(*it->second->Get(j))));
CrystalP Collaborator
CrystalP added a note

That leaves 2 packages, not one as the comment says. I think it makes more sense with 2 as you did here, to always keep the last version and the previous one.

Deleted user
ghost added a note

we can't invent packages for stuff we only have one for. granted, comment is probably a bit confusing as it's not what the code itself does so shall change it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
xbmc/addons/AddonInstaller.cpp
((22 lines not shown))
+ int i=0;
+ while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
+ {
+ size -= items[i]->m_dwSize;
+ CFileUtils::DeleteItem(items[i++],true);
+ }
+
+ if (size > g_advancedSettings.m_addonPackageFolderSize)
+ {
+ // 2. Remove the oldest packages (leaving at least 1 for each add-on)
+ items.Clear();
+ for (std::map<CStdString,CFileItemList*>::iterator it = packs.begin();
+ it != packs.end();++it)
+ {
+ if (it->second.Size() > 1)
+ items.Add(CFileItemPtr(new CFileItem(*it->second->Get(1))));
CrystalP Collaborator
CrystalP added a note

Shouldn't this be a c/p of lines 345-346 (except with "int j=1"), to catch ALL versions, except for the last one?
edit: nm, they've all been deleted in pass1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
CrystalP
Collaborator

Regardless of disk space, I'd personally like to always preserve the two most recent versions per addon, to always have the ability to rollback. I'm comfortable enough that two broken releases in a row are quite unlikely.

One suggestion: if that's not done somewhere else already, use this function to delete the packages of uninstalled addons?

Deleted user

hmm.. i sorta agree and sorta disagree. i guess we could change the space limit to be a 'desired max', while allowing more usage to keep two archives around of all.

i was pondering using it to nuke uninstalled addons. but sometimes quick re-adding is also nice. other opinions on that around? i'm fine with going with the consensus..

Vincent Guerci

Was guessing why a backup was big&slow, just figured addons/packages grew to 830MB over the time...
I'll give it a try and hope it will be merged some day :)
EDIT: Works perfectly, dropped to 166MB as expected, thanks @cptspiff

Zeljko Ametovic

@cptspiff will this go in at any point, my macmini would love you for it.. it has years of updates :)

Deleted user

i have done what I am willing to do but it was rejected as can be seen from the comments here.

Martijn Kaijser

@cptspiff
I can't see any comments on why it couldn't be merged or am i missing something?
With the merging an small form HTPC like rpi and the android boxes this would be a very welcome addition to clean disk space

Deleted user

crystal p is not satisfied.

Jester

Maybe an ifdef for certain platforms? look at ATV2, lots of questions on the forum about having no space and us pointing out to clean up (it's on the wiki but still) (same goes for higher database revs on upgrade, the old ones stay behind and fill up fast on big databases, but thats off-topic)

NedScott

We NEED this for Frodo. Even if it defaults to off and is turned on in advancedsettings.xml

Deleted user

@CrystalP, popular demand.. can be revised later.

Deleted user ghost merged commit ed0641f into from
Tobias Hieta tru referenced this pull request from a commit in plexinc/plex-home-theater-public
Tobias Hieta tru Remove PVR keyboard shortcuts
Fixes #323
b5ee0eb
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 2, 2012
This page is out of date. Refresh to see the latest.
72 xbmc/addons/AddonInstaller.cpp
View
@@ -21,10 +21,12 @@
#include "AddonInstaller.h"
#include "Service.h"
#include "utils/log.h"
+#include "utils/FileUtils.h"
#include "utils/URIUtils.h"
#include "Util.h"
#include "guilib/LocalizeStrings.h"
#include "filesystem/Directory.h"
+#include "settings/AdvancedSettings.h"
#include "settings/GUISettings.h"
#include "settings/Settings.h"
#include "ApplicationMessenger.h"
@@ -84,6 +86,7 @@ void CAddonInstaller::OnJobComplete(unsigned int jobID, bool success, CJob* job)
JobMap::iterator i = find_if(m_downloadJobs.begin(), m_downloadJobs.end(), bind2nd(find_map(), jobID));
if (i != m_downloadJobs.end())
m_downloadJobs.erase(i);
+ PrunePackageCache();
}
lock.Leave();
@@ -381,6 +384,75 @@ bool CAddonInstaller::HasJob(const CStdString& ID) const
return m_downloadJobs.find(ID) != m_downloadJobs.end();
}
+void CAddonInstaller::PrunePackageCache()
+{
+ std::map<CStdString,CFileItemList*> packs;
+ int64_t size = EnumeratePackageFolder(packs);
+ if (size < g_advancedSettings.m_addonPackageFolderSize)
+ return;
+
+ // Prune packages
+ // 1. Remove the largest packages, leaving at least 2 for each add-on
+ CFileItemList items;
+ for (std::map<CStdString,CFileItemList*>::const_iterator it = packs.begin();
+ it != packs.end();++it)
+ {
+ it->second->Sort(SORT_METHOD_LABEL,SortOrderDescending);
+ for (int j=2;j<it->second->Size();++j)
+ items.Add(CFileItemPtr(new CFileItem(*it->second->Get(j))));
+ }
+ items.Sort(SORT_METHOD_SIZE,SortOrderDescending);
+ int i=0;
+ while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
+ {
+ size -= items[i]->m_dwSize;
+ CFileUtils::DeleteItem(items[i++],true);
+ }
+
+ if (size > g_advancedSettings.m_addonPackageFolderSize)
+ {
+ // 2. Remove the oldest packages (leaving least 1 for each add-on)
+ items.Clear();
+ for (std::map<CStdString,CFileItemList*>::iterator it = packs.begin();
+ it != packs.end();++it)
+ {
+ if (it->second->Size() > 1)
+ items.Add(CFileItemPtr(new CFileItem(*it->second->Get(1))));
+ }
+ items.Sort(SORT_METHOD_DATE,SortOrderAscending);
+ i=0;
+ while (size > g_advancedSettings.m_addonPackageFolderSize && i < items.Size())
+ {
+ size -= items[i]->m_dwSize;
+ CFileUtils::DeleteItem(items[i++],true);
+ }
+ }
+ // clean up our mess
+ for (std::map<CStdString,CFileItemList*>::iterator it = packs.begin();
+ it != packs.end();++it)
+ delete it->second;
+}
+
+int64_t CAddonInstaller::EnumeratePackageFolder(std::map<CStdString,CFileItemList*>& result)
+{
+ CFileItemList items;
+ CDirectory::GetDirectory("special://home/addons/packages/",items,".zip",DIR_FLAG_NO_FILE_DIRS);
+ int64_t size=0;
+ for (int i=0;i<items.Size();++i)
+ {
+ if (items[i]->m_bIsFolder)
+ continue;
+ size += items[i]->m_dwSize;
+ CStdString pack,dummy;
+ AddonVersion::SplitFileName(pack,dummy,items[i]->GetLabel());
+ if (result.find(pack) == result.end())
+ result[pack] = new CFileItemList;
+ result[pack]->Add(CFileItemPtr(new CFileItem(*items[i])));
+ }
+
+ return size;
+}
+
CAddonInstallJob::CAddonInstallJob(const AddonPtr &addon, const CStdString &hash, bool update, const CStdString &referer)
: m_addon(addon), m_hash(hash), m_update(update), m_referer(referer)
{
3  xbmc/addons/AddonInstaller.h
View
@@ -122,6 +122,9 @@ class CAddonInstaller : public IJobCallback
*/
bool DoInstall(const ADDON::AddonPtr &addon, const CStdString &hash = "", bool update = false, const CStdString &referer = "", bool background = true);
+ void PrunePackageCache();
+ int64_t EnumeratePackageFolder(std::map<CStdString,CFileItemList*>& result);
+
CCriticalSection m_critSection;
JobMap m_downloadJobs;
CStopWatch m_repoUpdateWatch; ///< repository updates are done based on this counter
2  xbmc/settings/AdvancedSettings.cpp
View
@@ -298,6 +298,7 @@ void CAdvancedSettings::Initialize()
m_measureRefreshrate = false;
m_cacheMemBufferSize = 1024 * 1024 * 20;
+ m_addonPackageFolderSize = 200*1024*1024;
m_jsonOutputCompact = true;
m_jsonTcpPort = 9090;
@@ -759,6 +760,7 @@ void CAdvancedSettings::ParseSettingsFile(const CStdString &file)
XMLUtils::GetUInt(pRootElement,"restrictcapsmask", m_RestrictCapsMask);
XMLUtils::GetFloat(pRootElement,"sleepbeforeflip", m_sleepBeforeFlip, 0.0f, 1.0f);
XMLUtils::GetBoolean(pRootElement,"virtualshares", m_bVirtualShares);
+ XMLUtils::GetUInt(pRootElement, "packagefoldersize", m_addonPackageFolderSize);
//Tuxbox
pElement = pRootElement->FirstChildElement("tuxbox");
1  xbmc/settings/AdvancedSettings.h
View
@@ -346,6 +346,7 @@ class CAdvancedSettings
bool m_guiVisualizeDirtyRegions;
int m_guiAlgorithmDirtyRegions;
int m_guiDirtyRegionNoFlipTimeout;
+ unsigned int m_addonPackageFolderSize;
unsigned int m_cacheMemBufferSize;
Something went wrong with that request. Please try again.