Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

All TODO.txt docs for alpha1 are done. Only 3 weeks after promised, yay!

  • Loading branch information...
commit c05b42243d5557df925de5c7df51f24d88e7fc0f 1 parent 4d3a7cd
@pauleveritt pauleveritt authored
Showing with 184 additions and 14 deletions.
  1. +5 −14 TODO.txt
  2. +178 −0 docs/dump.rst
  3. +1 −0  docs/index.rst
19 TODO.txt
@@ -39,20 +39,6 @@ Features
- Add an autoevolve setting.
-Dump and Load
-- Explain ``__dump__`` and ``__load__`` methods of resources.
-- Explain how to add a dumper.
-- Add API docs for dump/load, and dump contexts.
-- Document ``sd_dump`` console script. Explain that there is no ``sd_load``,
- as loading is more a development effort rather than an administrative one.
Can wait until after alpha 1
@@ -92,6 +78,11 @@ Misc
- Explain the objectmap get_extents API.
+Dump and Load
+- Add API docs for dump/load, and dump contexts.
Can Wait Forever (Nice-to-Have)
178 docs/dump.rst
@@ -12,3 +12,181 @@ important data stored in a long-term format. For this,
Substance D provides a dump facility for content types to be serialized
in a `YAML <>`_ representation on disk.
+.. note::
+ You'll note in the following the absence of docs on *loading* data.
+ This is intentional. The process of loading data into a new,
+ or semi-new, or newer-than-new site has many policy implications.
+ Too many to fit into a single loading script. Substance D considers
+ the particularls of loading data to be in the province of the
+ application developer.
+Dumping Resources Using ``sd_dump``
+The ``sd_dump`` console script loads your Substance D application,
+connects to your object database, and writes serialized representations
+of resources to disk in a directory hierarchy::
+ $ ../bin/sd_dump --help
+ Usage: sd_dump [options]
+ Dump an object (and its subobjects) to the filesystem: sd_dump [--source
+ =ZODB-PATH] [--dest=FILESYSTEM-PATH] config_uri Dumps the object at ZODB-
+ PATH and all of its subobjects to a filesystem path. Such a dump can be
+ loaded (programmatically) by using the substanced.dump.load function e.g.
+ sd_dump --source=/ --dest=/my/dump etc/development.ini
+ Options:
+ -h, --help show this help message and exit
+ -s ZODB-PATH, --source=ZODB-PATH
+ The ZODB source path to dump (e.g. /foo/bar or /)
+ The destination filesystem path to dump to.
+For example::
+ $ ../bin/sd_dump ../etc/development.ini
+ 2013-01-07 13:27:03,939 INFO [ZEO.ClientStorage][MainThread] ('localhost', 9963) ClientStorage (pid=93148) created RW/normal for storage: 'main'
+ 2013-01-07 13:27:03,941 INFO [ZEO.cache][MainThread] created temporary cache file '<fdopen>'
+ 2013-01-07 13:27:03,981 WARNI [ZEO.zrpc][Connect([(2, ('localhost', 9963))])] (93148) CW: error connecting to ('fe80::1%lo0', 9963): EHOSTUNREACH
+ 2013-01-07 13:27:03,982 WARNI [ZEO.zrpc][Connect([(2, ('localhost', 9963))])] (93148) CW: error connecting to ('fe80::1%lo0', 9963): EHOSTUNREACH
+ 2013-01-07 13:27:04,002 WARNI [ZEO.zrpc][Connect([(2, ('localhost', 9963))])] (93148) CW: error connecting to ('::1', 9963): EINVAL
+ 2013-01-07 13:27:04,003 INFO [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) Testing connection <ManagedClientConnection ('', 9963)>
+ 2013-01-07 13:27:04,004 INFO [ZEO.zrpc.Connection(C)][('localhost', 9963) zeo client networking thread] ( received handshake 'Z3101'
+ 2013-01-07 13:27:04,105 INFO [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) Server authentication protocol None
+ 2013-01-07 13:27:04,106 INFO [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) Connected to storage: ('localhost', 9963)
+ 2013-01-07 13:27:04,108 INFO [ZEO.ClientStorage][Connect([(2, ('localhost', 9963))])] ('localhost', 9963) No verification necessary -- empty cache
+ 2013-01-07 13:27:04,727 INFO [substanced.catalog][MainThread] system update_indexes: no indexes added or removed
+ 2013-01-07 13:27:04,730 INFO [substanced.catalog][MainThread] sdidemo update_indexes: no indexes added or removed
+ 2013-01-07 13:27:04,732 INFO [substanced.dump][MainThread] Dumping /
+ 2013-01-07 13:27:04,749 INFO [substanced.dump][MainThread] Dumping /principals
+ 2013-01-07 13:27:04,754 INFO [substanced.dump][MainThread] Dumping /principals/users
+ 2013-01-07 13:27:04,760 INFO [substanced.dump][MainThread] Dumping /principals/users/admin
+ 2013-01-07 13:27:04,779 INFO [substanced.dump][MainThread] Dumping /principals/resets
+ 2013-01-07 13:27:04,783 INFO [substanced.dump][MainThread] Dumping /principals/groups
+...with logging messages being emitted until all known content is
+dumped. A ``dump`` subdirectory in the current directory is created (if
+no argument is provided) containing::
+ $ ls
+ acl.yaml propsheets references.yaml resource.yaml resources
+.. note::
+ To correctly encode as much meaning as possible,
+ the dump files contain some advanced and custom YAML constructs
+ when needed.
+``acl.yaml`` For Security Settings
+This YAML file contains security settings for this resource. For
+ - !!python/tuple [Allow, 1644064392535565429, !all_permissions '']
+``references.yaml`` for Reference Information
+Data about references aren't stored on the resources involved in the
+reference. Instead, they are stored in the objectmap. This file
+contains the reference information for the resource identified at the
+current dump directory. For example::
+ !interface 'substanced.interfaces.PrincipalToACLBearing':
+ sources: [1644064392535565429]
+``workflow.yaml`` for Workflow Settings
+The workflow engine can contain information about resource state. For
+ !!python/object:persistent.mapping.PersistentMapping
+ data: {document: draft}
+``propsheets`` Directory for Property Sheet Data
+Resources can have multiple system-defined or application-defined
+property sheets on resources. These are serialized as subdirectories
+under ``propsheets``, with a directory for each property sheet. For
+example, a resources ``propsheets/Basic/properties.yaml`` might contain::
+ {body: !!python/unicode 'The quick brown fox jumps over the lazy dog. The quick brown
+ fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog.
+ The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the
+ lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps
+ over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown
+ fox jumps over the lazy dog. ', name: !!python/unicode 'document_0', title: !!python/unicode 'Document
+ 0 Binder 0'}
+``resource.yaml`` for Content Type Information
+Each directory after the top corresponds to a resource in the database.
+As such, the resource likely has content type information. The dump
+script encodes this into a YAML file in the resource's dump directory::
+ {content_type: Root, created: !!timestamp '2013-01-07 14:23:23.133436', is_service: false,
+ name: null, oid: 1644064392535565415}
+``resources`` for Contained Resources in Containers
+If the resource at a current dump directory is a ``Folder`` or some
+other kind of container, it will contain a ``resources`` subdirectory.
+This might contain more subfolders and thus subdirectories. It might
+also contain individual resources, as a subdirectory named with the
+resource name.
+Custom Dumping with ``__dump__``
+The built-in facilities allow automatic dumping of most information for
+your content, including information in your property sheets,
+the content type, security settings, references, workflows, etc.
+If you do need extra information dumped to YAML about your content type,
+Substance D has a Python protocol using an ``__dump__`` on your
+``@content`` class. As an example,
+:py:meth:``substanced.principal.User.dump`` is a callable which returns
+a mapping of simple Python objects. The dumper checks to see if a
+resource has a ``__dump__`` method. If so, it calls the method,
+encodes the result to YAML, and writes it to an ``adhoc.yaml`` file in
+the dumped-resource's directory.
+The inverse is also true. If a content type has a ``__load__`` method,
+information from that method is added to the state that is loaded.
+Adding New Dumpers
+The ``adhoc.yaml`` file that we just saw is an example of the
+``AdhocAttrDumper``. There are seven other dumpers built-in: acl,
+workflow, references, sdiproperties, interfaces, order, and propsheets.
+If you would like a custom dumper, you can register it with
+``config.add_dumper``. For example,
+:py:func:`substanced.dump.includeme` registers the existing dumpers and
+their dumper factories:
+.. code-block:: python
+ def includeme(config):
+ ('acl', ACLDumper),
+ ('workflow', WorkflowDumper),
+ ('references', ReferencesDumper),
+ ('sdiproperties', SDIPropertiesDumper),
+ ('interfaces', DirectlyProvidedInterfacesDumper),
+ ('order', FolderOrderDumper),
+ ('propsheets', PropertySheetDumper),
+ ('adhoc', AdhocAttrDumper),
+ ]
+ config.add_directive('add_dumper', add_dumper)
+ for dumper_name, dumper_factory in DEFAULT_DUMPERS:
+ config.add_dumper(dumper_name, dumper_factory)
1  docs/index.rst
@@ -69,6 +69,7 @@ Narrative Documentation
+ dump

1 comment on commit c05b422


There is a note about not taking care of load however the load method is mentioned. I guess this reference needs to go away.

Please sign in to comment.
Something went wrong with that request. Please try again.