Skip to content

Commit

Permalink
Resurrect ZODB packing from the ZMI (#627)
Browse files Browse the repository at this point in the history 
* - remove tabs and reformat for better readability

* - Resurrect ZODB packing from the ZMI

* - remove broken and terrible advice involving URL-whacking the ZMI

* - remove unused parameter

* - add test and do value coercion if needed
  • Loading branch information
@dataflake
dataflake committed May 17, 2019
1 parent e674a05 commit 474cd53
Show file tree
Hide file tree
Showing 5 changed files with 158 additions and 130 deletions.
3 changes: 3 additions & 0 deletions CHANGES.rst
View file
Expand Up @@ -14,6 +14,9 @@ https://github.com/zopefoundation/Zope/blob/4.0a6/CHANGES.rst
Features
++++++++

- Resurrect ZODB packing from the ZMI
(`#623 <https://github.com/zopefoundation/Zope/issues/623>`_)

- Optionally control the use of Zope's built-in XML-RPC support for
POST requests with Content-Type ``text/xml`` via the
registration of a ``ZPublisher.interfaces.IXmlrpcChecker`` utility
Expand Down
53 changes: 0 additions & 53 deletions docs/zopebook/MaintainingZope.rst
View file
Expand Up @@ -607,59 +607,6 @@ manually by opening Zopes Control_Panel and clicking on the "Database
Management" link. Zope offers you the option of removing only object version
older than an adjustable amount of days.

If you want to automatically pack the ZODB you could tickle the appropriate URL
with a small python script (the traditional filesystem based kind, not Zopes
"Script (Python)")::

#!/usr/bin/python
import sys, urllib
host = sys.argv[1]
days = sys.argv[2]
url = "%s/Control_Panel/Database/manage_pack?days:float=%s" % (host, days)
try:
f = urllib.urlopen(url).read()
except IOError:
print "Cannot open URL %s, aborting" % url
print "Successfully packed ZODB on host %s" % host

The script takes two arguments, the URL of your server (eg.
http://mymachine.com) and the number of days old an object version has to be to
get discarded.

On Unix, put this in eg. the file::

/usr/local/sbin/zope_pack

and make it executable with::

chmod +x zope_pack

Then you can put in into your crontab with eg.::

5 4 * * sun /usr/local/sbin/zope_pack http://localhost 7

This would instruct your system to pack the ZODB on 4:05 every sunday. It would
connect to the local machine, and leave object versions younger than 7 days in
the ZODB.

Under Windows, you should use the scheduler to periodically start the script.
Put the above script in eg.::

c:\Program Files\zope_pack.py

or whereever you keep custom scripts, and create a batch file::

zope_pack.bat

with contents similar to the following:::

"C:\Program Files\zope\bin\python.exe" "C:\Program Files\zope_pack.py" "http://localhost" 7

The first parameter to python is the path to the python script we just created.
The second is the root URL of the machine you want to pack, and the third is
the maximum age of object versions you want to keep. Now instruct the scheduler
to run this `.bat` file every week.

Zope backup is quite straightforward. If you are using the default storage
(FileStorage), all you need to do is to save the file::

Expand Down
29 changes: 28 additions & 1 deletion src/App/ApplicationManager.py
View file
Expand Up @@ -250,7 +250,34 @@ def manage_minimize(self, value=1, REQUEST=None):
self._getDB().cacheMinimize()

if REQUEST is not None:
REQUEST.RESPONSE.redirect(REQUEST['URL1'] + '/manage_main')
msg = 'ZODB in-memory caches minimized.'
url = '%s/manage_main?manage_tabs_message=%s' % (REQUEST['URL1'],
msg)
REQUEST.RESPONSE.redirect(url)

@requestmethod('POST')
def manage_pack(self, days=0, REQUEST=None):
"""Pack the database"""
if not isinstance(days, (int, float)):
try:
days = float(days)
except ValueError:
days = None

if days is not None:
t = time.time() - (days * 86400)
self._getDB().pack(t)
msg = 'Database packed to %s days' % str(days)
else:
t = None
msg = 'Invalid days value %s' % str(days)

if REQUEST is not None:
url = '%s/manage_main?manage_tabs_message=%s' % (REQUEST['URL1'],
msg)
REQUEST['RESPONSE'].redirect(url)

return t


InitializeClass(AltDatabaseManager)
169 changes: 93 additions & 76 deletions src/App/dtml/dbMain.dtml
View file
Expand Up @@ -4,86 +4,103 @@

<main class="container-fluid">

<p class="form-help mt-4">
The Database Manager allows you to view database status information.
</p>
<p class="form-help mt-4">
The Database Manager allows you to view database status information.
</p>

<table id="zmi-db_info" class="table table-striped">
<thead>
<tr>
<th colspan="2">Database information <i class="fa fa-database text-secondary ml-1 mr-1"></i> <em>&dtml-id;</em></th>
</tr>
</thead>
<tbody>
<tr>
<td>Database Location</td>
<td class="code">&dtml-db_name;</td>
</tr>
<tr>
<td>Database Size</td>
<td class="code">&dtml-db_size;</td>
</tr>
<tr>
<td>Total number of objects in the database</td>
<td class="code">&dtml-database_size;</td>
</tr>
<tr>
<td>Total number of objects in memory from all caches</td>
<td class="code">&dtml-cache_length;</td>
</tr>
<tr>
<td>Target number of objects in memory per cache</td>
<td class="code">&dtml-cache_size;</td>
</tr>
<tr>
<td>Target memory size per cache in bytes</td>
<td class="code">&dtml-cache_length_bytes;</td>
</tr>
</tbody>
</table>
<table id="zmi-db_info" class="table table-striped">
<thead>
<tr>
<th colspan="2">
Database information
<i class="fa fa-database text-secondary ml-1 mr-1"></i>
<em>&dtml-id;</em>
</th>
</tr>
</thead>
<tbody>
<tr>
<td>Database Location</td>
<td class="code">&dtml-db_name;</td>
</tr>
<tr>
<td>Database Size</td>
<td class="code">&dtml-db_size;</td>
</tr>
<tr>
<td>Total number of objects in the database</td>
<td class="code">&dtml-database_size;</td>
</tr>
<tr>
<td>Total number of objects in memory from all caches</td>
<td class="code">&dtml-cache_length;</td>
</tr>
<tr>
<td>Target number of objects in memory per cache</td>
<td class="code">&dtml-cache_size;</td>
</tr>
<tr>
<td>Target memory size per cache in bytes</td>
<td class="code">&dtml-cache_length_bytes;</td>
</tr>
</tbody>
</table>

<div class="zmi-controls mb-5">
<form action="&dtml-URL1;/manage_pack" method="post">
<input class="btn btn-primary" id="pack" type="submit"
name="submit" value="Pack" />
<input type="text" name="days:float" value="0" size="3"> days
<small class="form-text text-muted">
Pack: Remove previous object revisions older than the selected
number of days.
</small>
</form>
</div>

<table id="zmi-db_cache" class="table table-striped mt-5">
<thead>
<tr>
<th colspan="3">Total number of objects in each cache</th>
</tr>
<tr>
<th>
<em>Cache Name</em></th>
<th>
<em class="d-none d-md-block">Number of active objects</em>
<em class="d-sm-block d-md-none">Active</em>
</th>
<th>
<em class="d-none d-md-block">Total active and non-active objects</em>
<em class="d-sm-block d-md-none">Total</em>
</th>
</tr>
</thead>
<tbody>
<dtml-in cache_detail_length mapping>
<tr>
<td>&dtml-connection;</td>
<td class="code">&dtml-ngsize;</td>
<td class="code">&dtml-size;</td>
</tr>
</dtml-in>
<tr>
<th>Total</th>
<th>&dtml-cache_length;</th>
<th>&nbsp;</th>
</tr>
</tbody>
</table>
<table id="zmi-db_cache" class="table table-striped mt-5">
<thead>
<tr>
<th colspan="3">Total number of objects in each cache</th>
</tr>
<tr>
<th>
<em>Cache Name</em></th>
<th>
<em class="d-none d-md-block">Number of active objects</em>
<em class="d-sm-block d-md-none">Active</em>
</th>
<th>
<em class="d-none d-md-block">Total active and non-active objects</em>
<em class="d-sm-block d-md-none">Total</em>
</th>
</tr>
</thead>
<tbody>
<dtml-in cache_detail_length mapping>
<tr>
<td>&dtml-connection;</td>
<td class="code">&dtml-ngsize;</td>
<td class="code">&dtml-size;</td>
</tr>
</dtml-in>
<tr>
<th>Total</th>
<th>&dtml-cache_length;</th>
<th>&nbsp;</th>
</tr>
</tbody>
</table>


<div class="zmi-controls mb-5">
<form action="&dtml-URL1;/manage_minimize" method="post">
<input class="btn btn-primary" id="minimize" type="submit" name="submit" value="Minimize" />
<small class="form-text text-muted">Minimize: Remove all objects from all ZODB in-memory caches.</small>
</form>
</div>
<div class="zmi-controls mb-5">
<form action="&dtml-URL1;/manage_minimize" method="post">
<input class="btn btn-primary" id="minimize" type="submit"
name="submit" value="Minimize" />
<small class="form-text text-muted">
Minimize: Remove all objects from all ZODB in-memory caches.
</small>
</form>
</div>

</main>

Expand Down
34 changes: 34 additions & 0 deletions src/App/tests/test_ApplicationManager.py
View file
Expand Up @@ -331,6 +331,40 @@ def test_db_size_gt_1_meg(self):
am._p_jar = self._makeJar('foo', (2048 * 1024) + 123240)
self.assertEqual(am.db_size(), '2.1M')

def test_manage_pack(self):
am = self._makeOne()
am._p_jar = self._makeJar('foo', '')

# The default value for days is 0, meaning pack to now
pack_to = time.time()
am.manage_pack()
self.assertAlmostEqual(am._getDB()._packed, pack_to, delta=1)

# Try a float value
pack_to = time.time() - 10800 # 3 hrs, 0.125 days
packed_to = am.manage_pack(days=.125)
self.assertAlmostEqual(am._getDB()._packed, pack_to, delta=1)
self.assertAlmostEqual(packed_to, pack_to, delta=1)

# Try an integer
pack_to = time.time() - 86400 # 1 day
packed_to = am.manage_pack(days=1)
self.assertAlmostEqual(am._getDB()._packed, pack_to, delta=1)
self.assertAlmostEqual(packed_to, pack_to, delta=1)

# Pass a string
pack_to = time.time() - 97200 # 27 hrs, 1.125 days
packed_to = am.manage_pack(days='1.125')
self.assertAlmostEqual(am._getDB()._packed, pack_to, delta=1)
self.assertAlmostEqual(packed_to, pack_to, delta=1)

# Set the dummy storage pack indicator manually
am._getDB()._packed = None
# Pass an invalid value
self.assertIsNone(am.manage_pack(days='foo'))
# The dummy storage value should not change because pack was not called
self.assertIsNone(am._getDB()._packed)


class MenuDtmlTests(ConfigTestBase, Testing.ZopeTestCase.FunctionalTestCase):
"""Browser testing ..dtml.menu.dtml."""
Expand Down

0 comments on commit 474cd53

Please sign in to comment.