doc/SchevoInternalDatabaseStructures.txt doc/SchevoTutorialMovieReviews2.txt schevo/counter.py schevo/database1.py schevo/database2.py schevo/example/moviereviews/__init__.py schevo/example/moviereviews/schema/__init__.py schevo/example/moviereviews/schema/schema_001.py schevo/mt/dummy.py schevo/placeholder.py schevo/script/db_convert.py tests/test_convert_format.py tests/test_field_entity.py tests/test_field_entitylist.py tests/test_field_factory_aliases.py @@ -148,7 +148,7 @@ such a program is covered only if its contents constitute a work based on the Library (independent of the use of the Library in a tool for writing it). Whether that is true depends on what the Library does and what the program that uses the Library does. - + 1. You may copy and distribute verbatim copies of the Library's complete source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an @@ -242,7 +242,7 @@ compelled to copy the source along with the object code. 5. A program that contains no derivative of any portion of the Library, but is designed to work with the Library by being compiled or linked with it, is called a "work that uses the Library". Such a -work, in isolation, is not a derivative work of the Library, and +work, in isolation, is not a derivative work of the Library, and therefore falls outside the scope of this License. However, linking a "work that uses the Library" with the Library COPYING.txt @@ -1 +1 @@ -For information and progress on future Schevo changes and features, please see http://schevo.org/trac/report/1 \ No newline at end of file +For information and progress on future Schevo changes and features, please see http://schevo.org/trac/report/1 ROADMAP.txt @@ -8,10 +8,10 @@ schevo.schema.prep(locals()) # class definition: # ## class SchevoIcon(E.Entity): -## +## ## _hidden = True -## +## ## name = f.unicode() ## data = f.image() -## +## ## _key(name) SCHEMA-PREAMBLE.txt @@ -70,7 +70,7 @@ In a command prompt, use *EasyInstall* to install the latest release of Schevo:: easy_install Schevo - + What to do after Schevo is installed ==================================== doc/SchevoGettingStartedMacOsx.txt @@ -63,7 +63,7 @@ In a command prompt, use *EasyInstall* to install the latest release of Schevo:: easy_install Schevo - + What to do after Schevo is installed ==================================== doc/SchevoGettingStartedUbuntu.txt @@ -17,7 +17,7 @@ Install Python `2.4.4 release page <http://www.python.org/download/releases/2.4.4/>`__ and downloading the ``python-2.4.4.msi`` file. - + 2. Open the ``msi`` file to start the installation process. 3. Step through the Python installer, accepting the default values for @@ -37,7 +37,7 @@ Install Python Python 2.4.4 (#71, Oct 18 2006, 08:34:43) ... Type "help", "copyright", "credits", or "license" ... - >>> + >>> To exit the Python shell, press Ctrl-Z and press Enter. @@ -91,7 +91,7 @@ In a command prompt, use *EasyInstall* to install the latest release of Schevo:: easy_install Schevo - + What to do after Schevo is installed ==================================== doc/SchevoGettingStartedWindows.txt @@ -2,10 +2,14 @@ Schevo Tutorial: Movie Reviews ================================ -In this tutorial, we will show you how to create a skeleton Schevo -application and a simple database schema. As you update the schema, we -will show you how to explore the database using the Python -interpreter. Here is what the final schema will look like: +--------------- + Installment 1 +--------------- + +In this installment of this tutorial, we will show you how to create a +skeleton Schevo application and a simple database schema. As you update the +schema, we will show you how to explore the database using the Python +interpreter. Here is what the final schema will look like: .. code-block:: python @@ -28,32 +32,32 @@ interpreter. Here is what the final schema will look like: class Actor(E.Entity): name = f.unicode() - + _key(name) def x_movies(self): - return [casting.movie + return [casting.movie for casting in self.m.movie_castings()] class Director(E.Entity): name = f.unicode() - + _key(name) class Movie(E.Entity): title = f.unicode() - description = f.memo() release_date = f.date() director = f.entity('Director') + description = f.memo(required=False) _key(title) def x_actors(self): - return [casting.actor + return [casting.actor for casting in self.m.movie_castings()] def __unicode__(self): @@ -67,6 +71,35 @@ interpreter. Here is what the final schema will look like: _key(movie, actor) + + E.Actor._sample = [ + ('Keanu Reeves', ), + ('Winona Ryder', ), + ] + + E.Director._sample = [ + ('Richard Linklater', ), + ('Stephen Herek', ), + ('Tim Burton', ), + ] + + E.Movie._sample = [ + ('A Scanner Darkly', '2006-07-28', ('Richard Linklater', ), + DEFAULT), + ("Bill & Ted's Excellent Adventure", '1989-02-17', + ('Stephen Herek', ), DEFAULT), + ('Edward Scissorhands', '1990-12-14', ('Tim Burton', ), + DEFAULT), + ] + + E.MovieCasting._sample = [ + (('A Scanner Darkly', ), ('Keanu Reeves', )), + (('A Scanner Darkly', ), ('Winona Ryder', )), + (("Bill & Ted's Excellent Adventure", ), ('Keanu Reeves', )), + (('Edward Scissorhands', ), ('Winona Ryder', )), + ] + + If you have not done so already, set up a Schevo environment as described in `SchevoGettingStarted`:trac:. @@ -137,12 +170,12 @@ classes you'll add to your schema:: >>> schema += ''' ... class SchevoIcon(E.Entity): # [1] ... """Stores icons for Schevo database and application objects.""" - ... + ... ... _hidden = True # [2] - ... + ... ... name = f.unicode() # [3] ... data = f.image() # [4] - ... + ... ... _key(name) # [5] ... ''' @@ -200,11 +233,11 @@ Add the following class to the database schema:: >>> schema += ''' ... class Movie(E.Entity): - ... + ... ... title = f.unicode() ... release_date = f.date() ... description = f.memo(required=False) # [1], [2] - ... + ... ... _key(title) # [3] ... ''' @@ -229,7 +262,7 @@ Save the ``schema_001.py`` file. At a shell prompt, use the ``schevo`` tool to create a new database with the application's schema:: $ schevo db create --app=moviereviews sample.db - + That's it! At this point, the ``sample.db`` file contains your new database, @@ -335,11 +368,11 @@ Exit the Python session, then edit ``schema_001.py`` again and add the >>> schema += ''' ... class Movie(E.Entity): - ... + ... ... title = f.unicode() ... release_date = f.date() ... description = f.memo(required=False) - ... + ... ... _key(title) ... ... def __unicode__(self): @@ -405,63 +438,63 @@ schema by assigning ``_sample`` data lists to each entity class:: >>> schema += ''' ... class Actor(E.Entity): - ... + ... ... name = f.unicode() - ... + ... ... _key(name) - ... - ... + ... + ... ... class Director(E.Entity): - ... + ... ... name = f.unicode() - ... + ... ... _key(name) - ... - ... + ... + ... ... class Movie(E.Entity): - ... + ... ... title = f.unicode() ... release_date = f.date() ... director = f.entity('Director') ... description = f.memo(required=False) - ... + ... ... _key(title) - ... + ... ... def __unicode__(self): ... return u'%s (%i)' % (self.title, self.release_date.year) - ... - ... + ... + ... ... class MovieCasting(E.Entity): - ... + ... ... movie = f.entity('Movie', CASCADE) ... actor = f.entity('Actor') - ... + ... ... _key(movie, actor) - ... + ... ... def __unicode__(self): ... return u'%s :: %s' (self.movie, self.actor) - ... - ... + ... + ... ... E.Actor._sample = [ ... ('Keanu Reeves', ), ... ('Winona Ryder', ), ... ] - ... + ... ... E.Director._sample = [ ... ('Richard Linklater', ), ... ('Stephen Herek', ), ... ('Tim Burton', ), ... ] - ... + ... ... E.Movie._sample = [ - ... ('A Scanner Darkly', '2006-07-28', ('Richard Linklater', ), + ... ('A Scanner Darkly', '2006-07-28', ('Richard Linklater', ), ... DEFAULT), - ... ("Bill & Ted's Excellent Adventure", '1989-02-17', + ... ("Bill & Ted's Excellent Adventure", '1989-02-17', ... ('Stephen Herek', ), DEFAULT), - ... ('Edward Scissorhands', '1990-12-14', ('Tim Burton', ), + ... ('Edward Scissorhands', '1990-12-14', ('Tim Burton', ), ... DEFAULT), ... ] - ... + ... ... E.MovieCasting._sample = [ ... (('A Scanner Darkly', ), ('Keanu Reeves', )), ... (('A Scanner Darkly', ), ('Winona Ryder', )), @@ -479,7 +512,7 @@ then open the database in a Python session:: $ schevo db create --app=moviereviews --delete --sample sample.db $ schevo shell sample.db - + For the doctest, let us do the equivalent:: >>> t.done() @@ -549,13 +582,13 @@ model. Add a new method called ``x_movies`` to the bottom of the >>> schema += ''' ... class Actor(E.Entity): - ... + ... ... name = f.unicode() - ... + ... ... _key(name) ... ... def x_movies(self): - ... return [casting.movie + ... return [casting.movie ... for casting in self.m.movie_castings()] ... ''' @@ -563,16 +596,16 @@ Add a new method to the ``Movie`` class:: >>> schema += ''' ... class Movie(E.Entity): - ... + ... ... title = f.unicode() ... release_date = f.date() ... director = f.entity('Director') ... description = f.memo(required=False) - ... + ... ... _key(title) - ... + ... ... def x_actors(self): - ... return [casting.actor + ... return [casting.actor ... for casting in self.m.movie_castings()] ... ... def __unicode__(self): doc/SchevoTutorialMovieReviews1.txt @@ -6,40 +6,97 @@ For copyright, license, and warranty, see bottom of file. import sys from schevo.lib import optimize -import random -import louie - -from schevo import base -from schevo import change -from schevo.change import CREATE, UPDATE, DELETE -from schevo.constant import UNASSIGNED -from schevo import error -from schevo.entity import Entity -from schevo.extent import Extent -from schevo.field import Entity as EntityField -from schevo.field import not_fget +from schevo import database1 +from schevo import database2 from schevo import icon -from schevo.lib import module -from schevo.namespace import NamespaceExtension -import schevo.schema -from schevo.signal import TransactionExecuted -from schevo.store.btree import BTree from schevo.store.connection import Connection from schevo.store.file_storage import FileStorage -from schevo.store.persistent_dict import PersistentDict as PDict -from schevo.store.persistent_list import PersistentList as PList -from schevo.trace import log -from schevo.transaction import ( - CallableWrapper, Combination, Initialize, Populate, Transaction) + + +format_dbclass = { + # Default database class. + None: database2.Database, + + # Format-specific database classes. + 1: database1.Database, + 2: database2.Database, + } + + +format_converter = { + 2: database2.convert_from_format1, + } + + +def convert_format(filename_or_fp, format): + """Convert database to a new storage format. + + - `filename_or_fp`: The filename or file pointer to convert. + - `format`: The format to convert to. + """ + if isinstance(filename_or_fp, basestring): + # If it's a string, assume it's a filename. + fs = FileStorage(filename_or_fp) + # Since it's an actual filesystem file, close the FileStorage after + # the entire conversion is done. + close_when_done = True + else: + # Otherwise, assume it's a file-like object. + fs = FileStorage(fp=filename_or_fp) + # Since StringIO and similar file-liike objects' values cannot be + # inspected after closing, do not explicitly close the FileStorage + # object when conversion is done. + close_when_done = False + # Check the format of the database. + conn = Connection(fs) + root = conn.get_root() + # XXX: Better error checking might be handy. + schevo = root['SCHEVO'] + original_format = schevo['format'] + # Convert one version at a time, ensuring that failures result in a + # rollback to the database's original state. + try: + try: + for new_format in xrange(original_format + 1, format + 1): + converter = format_converter[new_format] + converter(conn) + except: + conn.abort() + raise + else: + conn.commit() + finally: + if close_when_done: + fs.close() def evolve(db, schema_source, version): + """Evolve database to new version and new schema source. + + - `db`: The database to evolve. + - `schema_source`: The new schema source to evolve to. + - `version`: The new version number of the database schema. + """ db._evolve(schema_source, version) db._on_open() def inject(filename, schema_source, version): + """Inject a new schema and schema version into a database file. DANGEROUS! + + PLEASE USE WITH CAUTION; this is not intended to be used in normal course + of Schevo operation, but can be useful in some corner cases and during + application development. + + Inject in **no way shape or form evolves data or updates internal + structures** to reflect changes between the database's current schema and + that provided in the `schema_source` argument. + + - `filename`: Filename of database to inject new schema into. + - `schema_source`: The new schema source to inject into the database. + - `version`: The new version number of the database schema to inject. + """ fs = FileStorage(filename) conn = Connection(fs) root = conn.get_root() @@ -51,1693 +108,57 @@ def inject(filename, schema_source, version): def open(filename=None, schema_source=None, initialize=True, label='', - fp=None, cache_size=100000): - """Return an open database.""" + fp=None, cache_size=100000, format_for_new=None): + """Return an open database by opening an existing database or creating + a new one. + + If a new database is created, it is assumed that the `schema_source` given + is schema version 1. + + - `filename`: Filename of database to create or open, or None if using `fp`. + - `schema_source`: Schema source to create a new database with, or None + if not creating a new database. + - `initialize`: If True, will create initial values in the database if + creating a new database. + - `label`: The label of the database, to be used for a return value when + the open database instance is passed to `schevo.label.label`. + - `fp`: A file-like object, such as a StringIO instance, to use instead of + opening a file in the filesystem. None if using `filename`. + - `cache_size`: The number of Python objects that the stoage backend + will cache in memory before re-ghosting old objects. + - `format_for_new`: The version number of the database format to use when + creating a new database, or None to use the default format. + """ + # Create a FileStorage instance for the database based on `fp` or + # `filename` arguments. if fp is not None: fs = FileStorage(fp=fp) else: fs = FileStorage(filename) + # Create a Connection instance for the file storage, with a specific + # cache size. conn = Connection(fs, cache_size) + # Determine the version of the database. + root = conn.get_root() + if 'SCHEVO' in root: + schevo = root['SCHEVO'] + format = schevo['format'] + else: + format = format_for_new + # Determine database class based on format number. + Database = format_dbclass[format] + # Create the Database instance. db = Database(conn) if label: db.label = label + # Synchronize it with the given schema source. db._sync(schema_source, initialize) - # Install icon support. + # Install icon support and finalize opening of database. icon.install(db) db._on_open() return db -class dummy_lock(object): - """Dummy class for read_lock and write_lock objects in a database, - so that code can be written to be multi-thread-ready but still be - run in cases where the schevo.mt plugin is not installed.""" - - def release(self): - pass - - -class schema_counter(object): - """Schema counter singleton. - - This is a class instead of a global, because globals won't work - because of the binding done by optimize.bind_all. - """ - - _current = 0 - - @classmethod - def next(cls): - c = cls._current - cls._current += 1 - return c - - @classmethod - def next_schema_name(cls): - return 'schevo-db-schema-%i' % cls.next() - - -class Database(base.Database): - """Schevo database, using Durus as an object store. - - See doc/reference/database.txt for detailed information on data - structures, or visit http://docs.schevo.org/trunk/reference/database.html - """ - - label = 'Schevo Database' - - # By default, don't dispatch signals. Set to True to dispatch - # TransactionExecuted signals. - dispatch = False - - # See dummy_lock documentation. - read_lock = dummy_lock - write_lock = dummy_lock - - def __init__(self, connection): - """Create a database. - - - `connection`: The Durus connection to use. - """ - self.connection = connection - self._root = connection.get_root() - # Shortcuts to coarse-grained commit and rollback. - self._commit = connection.commit - self._rollback = connection.abort - # Keep track of schema modules remembered. - self._remembered = [] - # Initialization. - self._create_schevo_structures() - self._commit() - # Index to extent instances assigned by _sync. - self._extents = {} - # Index to entity classes assigned by _sync. - self._entity_classes = {} - # Vars used in transaction processing. - self._bulk_mode = False - self._executing = [] - # Shortcuts. - schevo = self._root['SCHEVO'] - self._extent_name_id = schevo['extent_name_id'] - self._extent_maps_by_id = schevo['extents'] - self._update_extent_maps_by_name() - # Plugin support. - self._plugins = [] - - def __repr__(self): - return '<Database %r :: V %r>' % (self.label, self.version) - - @property - def _extent_id_name(self): - return dict((v, k) for k, v in self._extent_name_id.items()) - - def close(self): - """Close the database.""" - assert log(1, 'Stopping plugins.') - p = self._plugins - while p: - assert log(2, 'Stopping', p) - p.pop().close() - assert log(1, 'Closing storage.') - self.connection.storage.close() - del self.connection - remembered = self._remembered - while remembered: - module.forget(remembered.pop()) - - def execute(self, *transactions, **kw): - """Execute transaction(s).""" - strict = kw.get('strict', True) - executing = self._executing - if len(transactions) == 0: - raise RuntimeError('Must supply at least one transaction.') - if len(transactions) > 1: - if not executing: - raise RuntimeError( - 'Must supply only one top-level transaction.') - else: - # Multiple transactions are treated as a single - # transaction containing subtransactions. - tx = Combination(transactions) - else: - tx = transactions[0] - if tx._executed: - raise error.TransactionAlreadyExecuted('%r already executed.' % tx) - if not executing: - # Bulk mode can only be set on an outermost transaction - # and effects all inner transactions. - self._bulk_mode = kw.get('bulk_mode', False) - # Outermost transaction must be executed strict. - strict = True - # Bulk mode minimizes transaction metadata. - bulk_mode = self._bulk_mode - executing.append(tx) - assert log(1, 'Begin executing [%i]' % len(executing), tx) - try: - retval = tx._execute(self) - assert log(2, 'Result was', repr(retval)) - # Enforce any indices relaxed by the transaction. - for extent_name, index_spec in tx._relaxed: - assert log(2, 'Enforcing index', extent_name, index_spec) - self._enforce_index(extent_name, index_spec) - # If the transaction must be executed with strict - # validation, perform that validation now. - if strict: - c = tx._changes_requiring_validation - assert log( - 2, 'Validating', len(c), 'changes requiring validation') - self._validate_changes(c) - except Exception, e: - assert log(1, e, 'was raised; undoing side-effects.') - if bulk_mode: - assert log(2, 'Bulk Mode transaction; storage rollback.') - self._rollback() - elif len(executing) == 1: - assert log(2, 'Outer transaction; storage rollback.') - self._rollback() - else: - assert log(2, 'Inner transaction; inverting.') - inversions = tx._inversions - while len(inversions): - method, args, kw = inversions.pop() - # Make sure the inverse operation doesn't append - # an inversion itself. - self._executing = None - # Perform the inversion. - method(*args, **kw) - # Restore state. - self._executing = executing - # Get rid of the current transaction on the stack since - # we're done undoing it. - executing.pop() - # Allow exception to bubble up. - raise - assert log(1, ' Done executing [%i]' % len(executing), tx) - tx._executed = True - # Post-transaction - if bulk_mode and len(executing) > 1: - assert log(2, 'Bulk Mode inner transaction.') - e2 = executing[-2] - e1 = executing[-1] - if not strict: - e2._changes_requiring_validation.extend( - e1._changes_requiring_validation) - elif bulk_mode: - assert log(2, 'Bulk Mode outer transaction; storage commit.') - # Done executing the outermost transaction. Use - # Durus-based commit. - self._commit() - elif len(executing) > 1: - assert log(2, 'Inner transaction; record inversions and changes.') - # Append the inversions from this transaction to the next - # outer transaction. - e2 = executing[-2] - e1 = executing[-1] - e2._inversions.extend(e1._inversions) - # Also append the changes made from this transaction. - e2._changes_requiring_notification.extend( - e1._changes_requiring_notification) - if not strict: - e2._changes_requiring_validation.extend( - e1._changes_requiring_validation) - else: - assert log(2, 'Outer transaction; storage commit.') - # Done executing the outermost transaction. Use - # Durus-based commit. - self._commit() - # Send a signal if told to do so. - if self.dispatch: - assert log(2, 'Dispatching TransactionExecuted signal.') - louie.send(TransactionExecuted, sender=self, transaction=tx) - executing.pop() - return retval - - def extent(self, extent_name): - """Return the named extent instance.""" - return self._extents[extent_name] - - def extent_names(self): - """Return a sorted list of extent names.""" - return sorted(self._extent_maps_by_name.keys()) - - def extents(self): - """Return a list of extent instances sorted by name.""" - extent = self.extent - return [extent(name) for name in self.extent_names()] - - def pack(self): - """Pack the database.""" - self.connection.pack() - - def populate(self, sample_name=''): - """Populate the database with sample data.""" - tx = Populate(sample_name) - self.execute(tx) - - @property - def format(self): - return self._root['SCHEVO']['format'] - - @property - def schema_source(self): - return self._root['SCHEVO']['schema_source'] - - @property - def version(self): - return self._root['SCHEVO']['version'] - - def _append_change(self, typ, extent_name, oid): - executing = self._executing - if executing: - info = (typ, extent_name, oid) - tx = executing[-1] - tx._changes_requiring_validation.append(info) - if not self._bulk_mode: - tx._changes_requiring_notification.append(info) - - def _append_inversion(self, method, *args, **kw): - """Append an inversion to a transaction if one is being - executed.""" - if self._bulk_mode: - return - executing = self._executing - if executing: - executing[-1]._inversions.append((method, args, kw)) - - def _by_entity_oids(self, extent_name, *index_spec): - """Return a list of OIDs from an extent sorted by index_spec.""" - extent_map = self._extent_map(extent_name) - indices = extent_map['indices'] - index_map = extent_map['index_map'] - # Separate index_spec into two tuples, one containing field - # names and one containing 'ascending' bools. - field_names = [] - ascending = [] - for field_name in index_spec: - if field_name.startswith('-'): - field_names.append(field_name[1:]) - ascending.append(False) - else: - field_names.append(field_name) - ascending.append(True) - index_spec = _field_ids(extent_map, field_names) - if index_spec not in indices: - # Specific index not found; look for an index where - # index_spec matches the beginning of that index's spec. - if index_spec not in index_map: - # None found. - raise error.IndexDoesNotExist( - 'Index %r not found in extent %r.' - % (_field_names(extent_map, index_spec), extent_name)) - # Use the first index found. - index_spec = index_map[index_spec][0] - oids = [] - unique, branch = indices[index_spec] - _walk_index(branch, ascending, oids) - return oids - - def _create_entity(self, extent_name, fields, oid=None, rev=None): - """Create a new entity in an extent; return the oid.""" - extent_map = self._extent_map(extent_name) - entities = extent_map['entities'] - old_next_oid = extent_map['next_oid'] - field_name_id = extent_map['field_name_id'] - entity_field_ids = extent_map['entity_field_ids'] - extent_name_id = self._extent_name_id - extent_maps_by_id = self._extent_maps_by_id - indices_added = [] - ia_append = indices_added.append - links_created = [] - lc_append = links_created.append - try: - if oid is None: - oid = extent_map['next_oid'] - extent_map['next_oid'] += 1 - if rev is None: - rev = 0 - if oid in entities: - raise error.EntityExists( - 'OID %r already exists in %r' % (oid, extent_name)) - # Create dict with field-id:field-value items. - fields_by_id = PDict() - new_links = [] - nl_append = new_links.append - for name, value in fields.iteritems(): - field_id = field_name_id[name] - # Handle entity reference fields. - if (field_id in entity_field_ids - and isinstance(value, Entity)): - # Dereference entity. - other_extent_id = extent_name_id[value._extent.name] - other_oid = value._oid - value = (other_extent_id, other_oid) - nl_append((field_id, other_extent_id, other_oid)) - fields_by_id[field_id] = value - # Make sure fields that weren't specified are set to - # UNASSIGNED. - setdefault = fields_by_id.setdefault - for field_id in field_name_id.itervalues(): - setdefault(field_id, UNASSIGNED) - # Update index mappings. - indices = extent_map['indices'] - for index_spec in indices.iterkeys(): - field_values = tuple(fields_by_id[field_id] - for field_id in index_spec) - # Find out if the index has been relaxed. - relaxed_specs = self._relaxed[extent_name] - if index_spec in relaxed_specs: - txns, relaxed = relaxed_specs[index_spec] - else: - relaxed = None - _index_add(extent_map, index_spec, relaxed, oid, field_values) - ia_append((extent_map, index_spec, oid, field_values)) - # Update links from this entity to another entity. - referrer_extent_id = extent_name_id[extent_name] - for referrer_field_id, other_extent_id, other_oid in new_links: - other_extent_map = extent_maps_by_id[other_extent_id] - try: - other_entity_map = other_extent_map['entities'][other_oid] - except KeyError: - field_id_name = extent_map['field_id_name'] - field_name = field_id_name[referrer_field_id] - raise error.EntityDoesNotExist( - 'Entity referenced in %r does not exist.' - % field_name) - # Add a link to the other entity. - links = other_entity_map['links'] - link_key = (referrer_extent_id, referrer_field_id) - if link_key not in links: # XXX Should already be there. - links[link_key] = BTree() - links[link_key][oid] = None - other_entity_map['link_count'] += 1 - lc_append((other_entity_map, links, link_key, oid)) - # Create the actual entity. - entity_map = entities[oid] = PDict() - entity_map['rev'] = rev - entity_map['fields'] = fields_by_id - # XXX flesh out links based on who is capable of linking - # to this one. - entity_map['link_count'] = 0 - entity_map['links'] = PDict() - extent_map['len'] += 1 - # Allow inversion of this operation. - self._append_inversion(self._delete_entity, extent_name, oid) - # Keep track of changes. - append_change = self._append_change - append_change(CREATE, extent_name, oid) - return oid - except: - # Revert changes made during create attempt. - for _e, _i, _o, _f in indices_added: - _index_remove(_e, _i, _o, _f) - for other_entity_map, links, link_key, oid in links_created: - del links[link_key][oid] - other_entity_map['link_count'] -= 1 - extent_map['next_oid'] = old_next_oid - raise - - def _delete_entity(self, extent_name, oid): - entity_map, extent_map = self._entity_extent_map(extent_name, oid) - extent_id = extent_map['id'] - extent_name_id = self._extent_name_id - extent_maps_by_id = self._extent_maps_by_id - entity_field_ids = extent_map['entity_field_ids'] - field_name_id = extent_map['field_name_id'] - link_count = entity_map['link_count'] - links = entity_map['links'] - # Disallow deletion if other entities refer to this one, - # unless all references are merely from ourself or an entity - # that will be deleted. - deletes = set() - executing = self._executing - if executing: - tx = executing[-1] - deletes.update([(extent_name_id[del_entity_cls.__name__], del_oid) - for del_entity_cls, del_oid in tx._deletes]) - deletes.update([(extent_name_id[del_entity_cls.__name__], del_oid) - for del_entity_cls, del_oid in tx._known_deletes]) - for (other_extent_id, other_field_id), others in links.items(): - for other_oid in others: - if (other_extent_id, other_oid) in deletes: - continue - # Give up as soon as we find one outside reference. - if (other_extent_id, other_oid) != (extent_id, oid): - msg = 'Cannot delete; other entities depend on this one.' - raise error.DeleteRestricted(msg) - # Get old values for use in a potential inversion. - old_fields = self._entity_fields(extent_name, oid) - old_rev = entity_map['rev'] - # Remove index mappings. - indices = extent_map['indices'] - fields_by_id = entity_map['fields'] - for index_spec in indices.iterkeys(): - field_values = tuple(fields_by_id.get(f_id, UNASSIGNED) - for f_id in index_spec) - _index_remove(extent_map, index_spec, oid, field_values) - # Delete links from this entity to other entities. - referrer_extent_id = extent_name_id[extent_name] - for referrer_field_id in entity_field_ids: - other_value = fields_by_id.get(referrer_field_id, UNASSIGNED) - if isinstance(other_value, tuple): - # Remove the link to the other entity. - other_extent_id, other_oid = other_value - link_key = (referrer_extent_id, referrer_field_id) - other_extent_map = extent_maps_by_id[other_extent_id] - if other_oid in other_extent_map['entities']: - other_entity_map = other_extent_map['entities'][other_oid] - links = other_entity_map['links'] - other_links = links[link_key] - del other_links[oid] - other_entity_map['link_count'] -= 1 - del extent_map['entities'][oid] - extent_map['len'] -= 1 - # Allow inversion of this operation. - self._append_inversion(self._create_entity, extent_name, old_fields, - oid, old_rev) - # Keep track of changes. - append_change = self._append_change - append_change(DELETE, extent_name, oid) - - def _enforce_index(self, extent_name, *index_spec): - """Validate and begin enforcing constraints on the specified - index if it was relaxed within the currently-executing - transaction.""" - executing = self._executing - if not executing: - # No-op if called outside a transaction. - return - # ID-ify the index_spec. - extent_map = self._extent_map(extent_name) - index_spec = _field_ids(extent_map, index_spec) - # Find the index to re-enforce. - indices = extent_map['indices'] - if index_spec not in indices: - raise error.IndexDoesNotExist( - 'Index %r not found in extent %r.' - % (_field_names(extent_map, index_spec), extent_name)) - # Find out if it has been relaxed. - current_txn = executing[-1] - relaxed = self._relaxed[extent_name] - txns, added = relaxed.get(index_spec, ([], [])) - if not txns: - # Was never relaxed; no-op. - return - if current_txn in txns: - current_txn._relaxed.remove((extent_name, index_spec)) - txns.remove(current_txn) - # If no more transactions have relaxed this index, enforce it. - if not txns: - for _extent_map, _index_spec, _oid, _field_values in added: - _index_validate(_extent_map, _index_spec, _oid, _field_values) - - def _entity(self, extent_name, oid): - """Return the entity instance.""" - EntityClass = self._entity_classes[extent_name] - return EntityClass(oid) - - def _entity_field(self, extent_name, oid, name): - """Return the value of a field in an entity in named extent - with given OID.""" - entity_map, extent_map = self._entity_extent_map(extent_name, oid) - field_name_id = extent_map['field_name_id'] - entity_field_ids = extent_map['entity_field_ids'] - field_id = field_name_id[name] - value = entity_map['fields'][field_id] - if field_id in entity_field_ids and isinstance(value, tuple): - extent_id, oid = value - EntityClass = self._entity_classes[extent_id] - value = EntityClass(oid) - return value - - def _entity_field_rev(self, extent_name, oid, name): - """Return a tuple of (value, rev) of a field in an entity in - named extent with given OID.""" - value = self._entity_field(extent_name, oid, name) - rev = self._entity_rev(extent_name, oid) - return value, rev - - def _entity_fields(self, extent_name, oid): - """Return a dictionary of field values for an entity in - `extent` with given OID.""" - entity_classes = self._entity_classes - entity_map, extent_map = self._entity_extent_map(extent_name, oid) - field_id_name = extent_map['field_id_name'] - entity_field_ids = extent_map['entity_field_ids'] - fields = {} - for field_id, value in entity_map['fields'].iteritems(): - if field_id in entity_field_ids and isinstance(value, tuple): - extent_id, oid = value - EntityClass = entity_classes[extent_id] - value = EntityClass(oid) - # During database evolution, it may turn out that fields - # get removed. For time efficiency reasons, Schevo does - # not iterate through all entities to remove existing - # data. Therefore, when getting entity fields from the - # database here, ignore fields that exist in the entity - # but no longer exist in the extent. - field_name = field_id_name.get(field_id, None) - if field_name: - fields[field_name] = value - return fields - - def _entity_links(self, extent_name, oid, other_extent_name=None, - other_field_name=None, return_count=False): - """Return dictionary of (extent_name, field_name): entity_list - pairs, or list of linking entities if `other_extent_name` and - `other_field_name` are supplied; return link count instead if - `return_count` is True.""" - assert log(1, '_entity_links', extent_name, oid, other_extent_name, - other_field_name, return_count) - entity_classes = self._entity_classes - entity_map = self._entity_map(extent_name, oid) - entity_links = entity_map['links'] - extent_maps_by_id = self._extent_maps_by_id - if other_extent_name is not None and other_field_name is not None: - # Both extent name and field name were provided. - other_extent_map = self._extent_map(other_extent_name) - other_extent_id = other_extent_map['id'] - try: - other_field_id = other_extent_map['field_name_id'][ - other_field_name] - except KeyError: - raise error.FieldDoesNotExist( - 'Field %r does not exist in extent %r' % ( - other_field_name, other_extent_name)) - key = (other_extent_id, other_field_id) - # Default to a dict since it has the same API as a BTree - # for our use but is faster and will stay empty anyway. - btree = entity_links.get(key, {}) - if return_count: - count = len(btree.keys()) # XXX Optimization opportunity. - assert log(2, 'returning len(btree.keys())', count) - return count - else: - EntityClass = entity_classes[other_extent_name] - others = [EntityClass(oid) for oid in btree] - return others - # Shortcut if we only care about the count, with no specificity. - link_count = entity_map['link_count'] - if return_count and other_extent_name is None: - assert log(2, 'returning link_count', link_count) - return link_count - # Build links tree. - specific_extent_name = other_extent_name - if return_count: - links = 0 - else: - links = {} - if link_count == 0: - # No links; no need to traverse. - assert log(2, 'no links - returning', links) - return links - for key, btree in entity_links.iteritems(): - other_extent_id, other_field_id = key - other_extent_map = extent_maps_by_id[other_extent_id] - other_extent_name = other_extent_map['name'] - if (specific_extent_name - and specific_extent_name != other_extent_name - ): - assert log(2, 'Skipping', other_extent_name) - continue - if return_count: - links += len(btree.keys()) # XXX: Optimization opportunity. - else: - other_field_name = other_extent_map['field_id_name'][ - other_field_id] - if specific_extent_name: - link_key = other_field_name - else: - link_key = (other_extent_name, other_field_name) - EntityClass = entity_classes[other_extent_name] - others = [EntityClass(oid) for oid in btree] - if others: - links[link_key] = others - if return_count: - assert log(2, 'returning links', links) - return links - - def _entity_rev(self, extent_name, oid): - """Return the revision of an entity in `extent` with given - OID.""" - entity_map = self._entity_map(extent_name, oid) - return entity_map['rev'] - - def _extent_contains_oid(self, extent_name, oid): - extent_map = self._extent_map(extent_name) - return oid in extent_map['entities'] - - def _extent_len(self, extent_name): - """Return the number of entities in the named extent.""" - extent_map = self._extent_map(extent_name) - return extent_map['len'] - - def _extent_next_oid(self, extent_name): - """Return the next OID to be assigned in the named extent.""" - extent_map = self._extent_map(extent_name) - return extent_map['next_oid'] - - def _find_entity_oids(self, extent_name, **criteria): - """Return list of entity OIDs matching given field value(s).""" - assert log(1, extent_name, criteria) - extent_map = self._extent_map(extent_name) - entity_maps = extent_map['entities'] - EntityClass = self._entity_classes[extent_name] - if not criteria: - # Return all of them. - assert log(2, 'Return all oids.') - return entity_maps.keys() - extent_name_id = self._extent_name_id - indices = extent_map['indices'] - assert log(3, 'indices.keys()', indices.keys()) - normalized_index_map = extent_map['normalized_index_map'] - assert log(3, 'normalized_index_map.keys()', - normalized_index_map.keys()) - entity_field_ids = extent_map['entity_field_ids'] - field_name_id = extent_map['field_name_id'] - # Convert from field_name:value to field_id:value. - field_id_value = {} - field_spec = EntityClass._field_spec - for field_name, value in criteria.iteritems(): - try: - field_id = field_name_id[field_name] - except KeyError: - raise error.FieldDoesNotExist( - 'Field %r does not exist for %r' % ( - field_name, extent_name)) - # Dereference if it's an entity field and not UNASSIGNED. - if field_id in entity_field_ids and isinstance(value, Entity): - # Dereference entity. - other_extent_id = extent_name_id[value._extent.name] - other_oid = value._oid - value = (other_extent_id, other_oid) - else: - # Create a field to convert the value. - FieldClass = field_spec[field_name] - field = FieldClass(None, None) - value = field.convert(value) - field_id_value[field_id] = value - # First, see if the fields given can be found in an index. If - # so, use the index to return matches. - # - # XXX: Should be updated to use partial search via an index, - # and brute-force on the subset found via that index. - field_ids = tuple(sorted(field_id_value)) - assert log(3, 'field_ids', field_ids) - len_field_ids = len(field_ids) - index_spec = None - if field_ids in normalized_index_map: - for spec in normalized_index_map[field_ids]: - if len(spec) == len_field_ids: - index_spec = spec - break - results = [] - if index_spec is not None: - # We found an index to use. - assert log(2, 'Use index spec:', index_spec) - unique, branch = indices[index_spec] - match = True - for field_id in index_spec: - field_value = field_id_value[field_id] - if field_value not in branch: - # No matches found. - assert log(3, field_value, 'not found in', branch.keys()) - match = False - break - branch = branch[field_value] - if match: - # Now we're at a leaf that matches all of the - # criteria, so return the OIDs in that leaf. - results = branch.keys() - else: - # Fields aren't indexed, so use brute force. - assert log(2, 'Use brute force.') - append = results.append - for oid, entity_map in entity_maps.iteritems(): - fields = entity_map['fields'] - match = True - for field_id, value in field_id_value.iteritems(): - if fields[field_id] != value: - match = False - break - if match: - append(oid) - assert log(2, 'Result count', len(results)) - return results - - def _relax_index(self, extent_name, *index_spec): - """Relax constraints on the specified index until a matching - enforce_index is called, or the currently-executing - transaction finishes, whichever occurs first.""" - executing = self._executing - if not executing: - raise RuntimeError('Indexes can only be relaxed inside ' - 'transaction execution.') - # ID-ify the index_spec. - extent_map = self._extent_map(extent_name) - index_spec = _field_ids(extent_map, index_spec) - # Find the index to relax. - indices = extent_map['indices'] - if index_spec not in indices: - raise error.IndexDoesNotExist( - 'Index %r not found in extent %r.' - % (_field_names(extent_map, index_spec), extent_name)) - # Keep track of the relaxation. - current_txn = executing[-1] - relaxed = self._relaxed[extent_name] - txns, added = relaxed.setdefault(index_spec, ([], [])) - txns.append(current_txn) - current_txn._relaxed.add((extent_name, index_spec)) - - def _set_extent_next_oid(self, extent_name, next_oid): - extent_map = self._extent_map(extent_name) - extent_map['next_oid'] = next_oid - - def _update_entity(self, extent_name, oid, fields, rev=None): - """Update an existing entity in an extent.""" - # XXX: Could be optimized to update mappings only when - # necessary. - entity_classes = self._entity_classes - entity_map, extent_map = self._entity_extent_map(extent_name, oid) - field_name_id = extent_map['field_name_id'] - entity_field_ids = extent_map['entity_field_ids'] - extent_name_id = self._extent_name_id - extent_maps_by_id = self._extent_maps_by_id - indices_added = [] - indices_removed = [] - new_links = [] - links_created = [] - links_deleted = [] - ia_append = indices_added.append - ir_append = indices_removed.append - nl_append = new_links.append - lc_append = links_created.append - ld_append = links_deleted.append - try: - # Get old values for use in a potential inversion. - old_fields = self._entity_fields(extent_name, oid) - old_rev = entity_map['rev'] - # Dereference entities. - for name, value in fields.items(): - field_id = field_name_id[name] - if (field_id in entity_field_ids and - isinstance(value, Entity) - ): - # Dereference entity. - other_extent_id = extent_name_id[value._extent.name] - other_oid = value._oid - value = (other_extent_id, other_oid) - nl_append((field_id, other_extent_id, other_oid)) - fields[name] = value - # Get fields, and set UNASSIGNED for any fields that are - # new since the last time the entity was stored. - fields_by_id = entity_map['fields'] - all_field_ids = set(extent_map['field_id_name'].iterkeys()) - new_fields = all_field_ids - set(fields_by_id.iterkeys()) - fields_by_id.update(dict( - (field_id, UNASSIGNED) for field_id in new_fields)) - # Remove existing index mappings. - indices = extent_map['indices'] - for index_spec in indices.iterkeys(): - field_values = tuple(fields_by_id[field_id] - for field_id in index_spec) - # Find out if the index has been relaxed. - relaxed_specs = self._relaxed[extent_name] - if index_spec in relaxed_specs: - txns, relaxed = relaxed_specs[index_spec] - else: - relaxed = None - _index_remove(extent_map, index_spec, oid, field_values) - ir_append((extent_map, index_spec, relaxed, oid, field_values)) - # Delete links from this entity to other entities. - referrer_extent_id = extent_name_id[extent_name] - for referrer_field_id in entity_field_ids: - other_value = fields_by_id[referrer_field_id] - if isinstance(other_value, tuple): - # Remove the link to the other entity. - other_extent_id, other_oid = other_value - link_key = (referrer_extent_id, referrer_field_id) - other_extent_map = extent_maps_by_id[other_extent_id] - other_entity_map = other_extent_map['entities'][other_oid] - links = other_entity_map['links'] - other_links = links[link_key] - del other_links[oid] - other_entity_map['link_count'] -= 1 - ld_append((other_entity_map, links, link_key, oid)) - # Create ephemeral fields for creating new index mappings. - new_fields = dict(fields_by_id) - for name, value in fields.iteritems(): - new_fields[field_name_id[name]] = value - # Create new index mappings. - for index_spec in indices.iterkeys(): - field_values = tuple(new_fields[field_id] - for field_id in index_spec) - # Find out if the index has been relaxed. - relaxed_specs = self._relaxed[extent_name] - if index_spec in relaxed_specs: - txns, relaxed = relaxed_specs[index_spec] - else: - relaxed = None - _index_add(extent_map, index_spec, relaxed, oid, field_values) - ia_append((extent_map, index_spec, oid, field_values)) - # Update links from this entity to another entity. - referrer_extent_id = extent_name_id[extent_name] - for referrer_field_id, other_extent_id, other_oid in new_links: - other_extent_map = extent_maps_by_id[other_extent_id] - try: - other_entity_map = other_extent_map['entities'][other_oid] - except KeyError: - field_id_name = extent_map['field_id_name'] - field_name = field_id_name[referrer_field_id] - raise error.EntityDoesNotExist( - 'Entity referenced in %r does not exist.' - % field_name) - # Add a link to the other entity. - links = other_entity_map['links'] - link_key = (referrer_extent_id, referrer_field_id) - if link_key not in links: # XXX Should already be there. - links[link_key] = BTree() - links[link_key][oid] = None - other_entity_map['link_count'] += 1 - lc_append((other_entity_map, links, link_key, oid)) - # Update actual fields. - for name, value in fields.iteritems(): - fields_by_id[field_name_id[name]] = value - if rev is None: - entity_map['rev'] += 1 - else: - entity_map['rev'] = rev - # Allow inversion of this operation. - self._append_inversion(self._update_entity, extent_name, oid, - old_fields, old_rev) - # Keep track of changes. - append_change = self._append_change - append_change(UPDATE, extent_name, oid) - except: - # Revert changes made during update attempt. - for _e, _i, _o, _f in indices_added: - _index_remove(_e, _i, _o, _f) - for _e, _i, _r, _o, _f in indices_removed: - _index_add(_e, _i, _r, _o, _f) - for other_entity_map, links, link_key, oid in links_created: - del links[link_key][oid] - other_entity_map['link_count'] -= 1 - for other_entity_map, links, link_key, oid in links_deleted: - links[link_key][oid] = None - other_entity_map['link_count'] += 1 - raise - - def _create_extent(self, extent_name, field_names, entity_field_names, - key_spec=None, index_spec=None): - """Create a new extent with a given name.""" - if extent_name in self._extent_maps_by_name: - raise error.ExtentExists('%r already exists.' % extent_name) - if key_spec is None: - key_spec = [] - if index_spec is None: - index_spec = [] - extent_map = PDict() - extent_id = self._unique_extent_id() - indices = extent_map['indices'] = PDict() - extent_map['index_map'] = PDict() - normalized_index_map = extent_map['normalized_index_map'] = PDict() - extent_map['entities'] = BTree() - field_id_name = extent_map['field_id_name'] = PDict() - field_name_id = extent_map['field_name_id'] = PDict() - extent_map['id'] = extent_id - extent_map['len'] = 0 - extent_map['name'] = extent_name - extent_map['next_oid'] = 1 - self._extent_name_id[extent_name] = extent_id - self._extent_maps_by_id[extent_id] = extent_map - self._extent_maps_by_name[extent_name] = extent_map - # Give each field name a unique ID. - for name in field_names: - field_id = self._unique_field_id(extent_name) - field_id_name[field_id] = name - field_name_id[name] = field_id - # Convert field names to field IDs in key spec and create - # index structures. - for field_names in key_spec: - i_spec = _field_ids(extent_map, field_names) - _create_index(extent_map, i_spec, True) - # Convert field names to field IDs in index spec and create - # index structures. - for field_names in index_spec: - i_spec = _field_ids(extent_map, field_names) - # Although we tell it unique=False, it may find a subset - # key, which will cause this superset to be unique=True. - _create_index(extent_map, i_spec, False) - # Convert field names to field IDs for entity field names. - extent_map['entity_field_ids'] = _field_ids( - extent_map, entity_field_names) - - def _delete_extent(self, extent_name): - """Remove a named extent.""" - # XXX: Need to check for links to any entity in this extent, - # and fail to remove it if so. - extent_map = self._extent_map(extent_name) - extent_id = extent_map['id'] - del self._extent_name_id[extent_name] - del self._extent_maps_by_id[extent_id] - del self._extent_maps_by_name[extent_name] - - def _create_schevo_structures(self): - """Create or update Schevo structures in the database.""" - root = self._root - if 'SCHEVO' not in root.keys(): - schevo = root['SCHEVO'] = PDict() - schevo['format'] = 1 - schevo['version'] = 0 - schevo['extent_name_id'] = PDict() - schevo['extents'] = PDict() - schevo['schema_source'] = None - - def _entity_map(self, extent_name, oid): - """Return an entity PDict corresponding to named - `extent` and OID.""" - extent_map = self._extent_map(extent_name) - try: - entity_map = extent_map['entities'][oid] - except KeyError: - raise error.EntityDoesNotExist( - 'OID %r does not exist in %r' % (oid, extent_name)) - return entity_map - - def _entity_extent_map(self, extent_name, oid): - """Return an (entity PDict, extent PDict) - tuple corresponding to named `extent` and OID.""" - extent_map = self._extent_map(extent_name) - try: - entity_map = extent_map['entities'][oid] - except KeyError: - raise error.EntityDoesNotExist( - 'OID %r does not exist in %r' % (oid, extent_name)) - return entity_map, extent_map - - def _evolve(self, schema_source, version): - """Evolve the database to a new schema definition. - - - `schema_source`: String containing the source code for the - schema to be evolved to. - - - `version`: Integer with the version number of the new schema - source. Must be the current database version, plus 1. - """ - if version != self.version + 1: - raise error.DatabaseVersionMismatch( - 'Current version is %i, expected %i, got %i' - % (self.version, self.version + 1, version)) - def call(module, name): - fn = getattr(module, name, None) - if callable(fn): - tx = CallableWrapper(fn) - # Trick the database into not performing a - # storage-level commit. - self._executing = [Transaction()] - try: - self.execute(tx) - finally: - self._executing = [] - # Load the new schema. - schema_name = schema_counter.next_schema_name() - schema_module = self._import_from_source(schema_source, schema_name) - try: - # Execute `before_evolve` function if defined. - call(schema_module, 'before_evolve') - # Perform first pass of evolution. - self._sync(schema_source, initialize=False, commit=False, - evolving=True) - # Execute `during_evolve` function if defined. - call(self._schema_module, 'during_evolve') - # Perform standard schema synchronization. - self._sync(schema_source, initialize=False, commit=False, - evolving=False) - # Execute `after_evolve` function if defined. - call(self._schema_module, 'after_evolve') - except: - self._rollback() - # Re-raise exception. - raise - else: - self._root['SCHEVO']['version'] = version - self._commit() - - def _extent_map(self, extent_name): - """Return an extent PDict corresponding to `extent_name`.""" - try: - return self._extent_maps_by_name[extent_name] - except KeyError: - raise error.ExtentDoesNotExist('%r does not exist.' % extent_name) - - def _import_from_source(self, source, module_name): - """Import a schema module from a string containing source code.""" - # Now that prerequisites are loaded, load this schema. - schema_module = module.from_string(source, module_name) - # Remember the schema module. - module.remember(schema_module) - self._remembered.append(schema_module) - # Expose this database to the schema module. - schema_module.db = self - # Return the schema module. - return schema_module - - def _initialize(self): - """Populate the database with initial data.""" - tx = Initialize() - self.execute(tx) - - def _on_open(self): - """Allow schema to run code after the database is opened.""" - if hasattr(self, '_schema_module'): - # An empty database created without a schema source will - # not have a schema module. - fn = getattr(self._schema_module, 'on_open', None) - if callable(fn): - fn(self) - - def _remove_stale_links(self, extent_id, field_id, FieldClass): - # Remove links from this field to other - # entities that are held in the structures for - # those other entities. - allow = FieldClass.allow - for other_name in allow: - other_extent_map = self._extent_map(other_name) - other_entities = other_extent_map['entities'] - for other_entity in other_entities.itervalues(): - other_link_count = other_entity['link_count'] - other_links = other_entity['links'] - referrer_key = (extent_id, field_id) - if referrer_key in other_links: - referrers = other_links[referrer_key] - other_link_count -= len(referrers.keys()) - del other_links[referrer_key] - other_entity['link_count'] = other_link_count - - def _sync(self, schema_source=None, initialize=True, commit=True, - evolving=False): - """Synchronize the database with a schema definition. - - - `schema_source`: String containing the source code for a - schema. If `None`, the schema source contained in the - database itself will be used. - - - `initialize`: True if a new database should be populated - with initial values defined in the schema. - - - `commit`: True if a successful synchronization should commit - to the storage backend. False if the caller of `_sync` will - handle this task. - - - `evolving`: True if the synchronization is occuring during a - database evolution. - """ - sync_schema_changes = True - locked = False - try: - SCHEVO = self._root['SCHEVO'] - # Import old schema. - old_schema_source = SCHEVO['schema_source'] - if old_schema_source is not None: - old_schema_module = None - schevo.schema.start(self, evolving) - locked = True - schema_name = schema_counter.next_schema_name() - try: - old_schema_module = self._import_from_source( - old_schema_source, schema_name) - finally: - old_schema = schevo.schema.finish(self, old_schema_module) - locked = False - self._old_schema = old_schema - self._old_schema_module = old_schema_module - else: - old_schema = self._old_schema = None - old_schema_module = self._old_schema_module = None - # Import current schema. - if schema_source is None: - schema_source = old_schema_source - if schema_source is None: - # No schema source was specified and this is a new - # database, so _sync becomes a no-op. - return - else: - # No schema source was specified and this is an - # existing database with a defined schema. - sync_schema_changes = False - if schema_source == old_schema_source: - # If the same source, it'll be the same schema. - schema = old_schema - schema_module = old_schema_module - else: - # XXX - # Temporary code to deal with older schemata. - schema_source = schema_source.replace("""\ -from schevo import * -from schevo.namespace import schema_prep -schema_prep(locals()) -""", """\ -from schevo.schema import * -schevo.schema.prep(locals()) -""") - # /XXX - schema_module = None - schevo.schema.start(self, evolving) - locked = True - schema_name = schema_counter.next_schema_name() - try: - schema_module = self._import_from_source( - schema_source, schema_name) - finally: - schema = schevo.schema.finish(self, schema_module) - locked = False - self.schema = schema - self._schema_module = schema_module - # Expose database-level namespaces. - self.q = schema.q - self.t = schema.t - self.Q = schema.Q - # Create an extenders namespace. - self.x = DatabaseExtenders(self._schema_module) - # If the schema has changed then sync with it. - if sync_schema_changes: - # Update schema source stored in database. - SCHEVO['schema_source'] = schema_source - self._sync_extents(schema, evolving) - # Create extent instances. - E = schema.E - extents = self._extents = {} - relaxed = self._relaxed = {} - entity_classes = self._entity_classes = {} - extent_name_id = self._extent_name_id - for e_name in self.extent_names(): - e_id = extent_name_id[e_name] - EntityClass = E[e_name] - extent = Extent(self, e_name, EntityClass) - extents[e_id] = extents[e_name] = extent - relaxed[e_name] = {} - entity_classes[e_id] = entity_classes[e_name] = EntityClass - # Decorate this Database instance to support the - # following syntax within schema code, for example: - # tx = db.Foo.t.create() - setattr(self, e_name, extent) - # Initialize a new database. - if SCHEVO['version'] == 0: - SCHEVO['version'] = 1 - # Populate with initial data, unless overridden such as - # when importing from an XML file. - if initialize: - self._initialize() - except: - if locked: - schevo.schema.import_lock.release() - if commit: - self._rollback() - raise - else: - if commit: - self._commit() - - def _sync_extents(self, schema, evolving): - """Synchronize the extents based on the schema.""" - E = schema.E - old_schema = self._old_schema - # Rename extents in the database whose entity class definition - # has a `_was` attribute. - in_schema = set(iter(E)) - if evolving: - for extent_name in in_schema: - EntityClass = E[extent_name] - was_named = EntityClass._was - if was_named is not None: - # Change the name of the existing extent in the - # database. - extent_name_id = self._extent_name_id - extent_map = self._extent_map(was_named) - extent_id = extent_map['id'] - extent_map['name'] = extent_name - del extent_name_id[was_named] - extent_name_id[extent_name] = extent_id - self._update_extent_maps_by_name() - # Create extents that are in schema but not in db. - in_db = set(self.extent_names()) - to_create = in_schema - in_db - for extent_name in to_create: - if extent_name.startswith('_'): - # Do not bother with hidden classes. - continue - EntityClass = E[extent_name] - field_spec = EntityClass._field_spec - field_names = field_spec.keys() - entity_field_names = [] - for name in field_names: - if issubclass(field_spec[name], EntityField): - entity_field_names.append(name) -## if issubclass(field_spec[name][0], EntityField): -## entity_field_names.append(name) - key_spec = EntityClass._key_spec - index_spec = EntityClass._index_spec - self._create_extent( - extent_name, field_names, entity_field_names, - key_spec, index_spec) - # Remove extents that are in the db but not in the schema. - in_db = set(self.extent_names()) - to_remove = in_db - in_schema - for extent_name in to_remove: - if extent_name.startswith('_'): - # Do not bother with hidden classes. - continue - # Check for links made from entities in this extent to - # other entities, where the other entities maintain those - # link structures. - if old_schema: - extent_map = self._extent_map(extent_name) - field_name_id = extent_map['field_name_id'] - extent_id = extent_map['id'] - # The old extent name will not exist in the old schema - # if it was an evolve_only class definition, and we - # are not in the process of evolving. - if extent_name in old_schema.E: - for old_field_name, FieldClass in ( - old_schema.E[extent_name]._field_spec.iteritems() - ): - old_field_id = field_name_id[old_field_name] - if issubclass(FieldClass, EntityField): - self._remove_stale_links( - extent_id, old_field_id, FieldClass) - # Delete the extent. XXX: Need to skip system extents? - self._delete_extent(extent_name) - # Update entity_field_ids, field_id_name, and field_name_id - # for all extents. - for extent_name in self.extent_names(): - EntityClass = E[extent_name] - field_spec = EntityClass._field_spec - extent_map = self._extent_map(extent_name) - extent_id = extent_map['id'] - entity_field_ids = set(extent_map['entity_field_ids']) - field_id_name = extent_map['field_id_name'] - field_name_id = extent_map['field_name_id'] - # Rename fields with 'was' attribute. - existing_field_names = set(field_name_id.keys()) - new_field_names = set(field_spec.keys()) - if evolving: - for field_name in new_field_names: - field_class = field_spec[field_name] - was_named = field_class.was - if was_named is not None: - if was_named not in existing_field_names: - raise error.FieldDoesNotExist( - 'Field %s.%s was being renamed from ' - '%s, but that field does not exist ' - 'in the previous schema.' - % (extent_name, field_name, was_named)) - # Rename the field. - field_id = field_name_id[was_named] - del field_name_id[was_named] - field_name_id[field_name] = field_id - field_id_name[field_id] = field_name - # Remove from the set of existing field names. - existing_field_names.remove(was_named) - # Remove fields that no longer exist. - old_field_names = existing_field_names - new_field_names - for old_field_name in old_field_names: - old_field_id = field_name_id[old_field_name] - if old_schema: - # Get the field spec for the field being deleted. - # It may not exist in the old schema, if it was only - # there in an _evolve_only class definition. - if extent_name in old_schema.E: - FieldClass = old_schema.E[extent_name]._field_spec.get( - old_field_name, None) - if (FieldClass is not None and - issubclass(FieldClass, EntityField)): - self._remove_stale_links( - extent_id, old_field_id, FieldClass) - if old_field_id in entity_field_ids: - entity_field_ids.remove(old_field_id) - del field_name_id[old_field_name] - del field_id_name[old_field_id] - # Create fields IDs for new fields. - existing_field_names = set(field_name_id.keys()) - fields_to_create = new_field_names - existing_field_names - for field_name in fields_to_create: - field_id = self._unique_field_id(extent_name) - field_name_id[field_name] = field_id - field_id_name[field_id] = field_name - # Check for entity field. - if issubclass(field_spec[field_name], EntityField): - entity_field_ids.add(field_id) - extent_map['entity_field_ids'] = tuple(entity_field_ids) - # Update index specs for all extents. - for extent_name in self.extent_names(): - # Skip system extents. - EntityClass = E[extent_name] - key_spec = EntityClass._key_spec - index_spec = EntityClass._index_spec - self._update_extent_key_spec(extent_name, key_spec, index_spec) - - def _unique_extent_id(self): - """Return an unused random extent ID.""" - extent_name_id = self._extent_name_id - while True: - extent_id = random.randint(0, 2**31) - if extent_id not in extent_name_id: - return extent_id - - def _unique_field_id(self, extent_name): - """Return an unused random field ID.""" - field_id_name = self._extent_map(extent_name)['field_id_name'] - while True: - field_id = random.randint(0, 2**31) - if field_id not in field_id_name: - return field_id - -## This is how Zope does it: - -## def _generateId(self): -## """Generate an id which is not yet taken. - -## This tries to allocate sequential ids so they fall into the -## same BTree bucket, and randomizes if it stumbles upon a -## used one. -## """ -## while True: -## if self._v_nextid is None: -## self._v_nextid = random.randint(0, 2**31) -## uid = self._v_nextid -## self._v_nextid += 1 -## if uid not in self.refs: -## return uid -## self._v_nextid = None - - def _update_extent_maps_by_name(self): - extent_maps_by_name = self._extent_maps_by_name = {} - for extent in self._extent_maps_by_id.itervalues(): - extent_maps_by_name[extent['name']] = extent - - def _update_extent_key_spec(self, extent_name, key_spec, index_spec): - """Update an existing extent to match given key spec.""" - extent_map = self._extent_map(extent_name) - entities = extent_map['entities'] - indices = extent_map['indices'] - key_spec_ids = [_field_ids(extent_map, field_names) - for field_names in key_spec] - index_spec_ids = [_field_ids(extent_map, field_names) - for field_names in index_spec] - # Convert key indices that have been changed to non-unique - # incides. - for i_spec in index_spec_ids: - if i_spec not in key_spec and i_spec in indices: - unique, branch = indices[i_spec] - indices[i_spec] = (False, branch) - # Create new key indices for those that don't exist. - for i_spec in key_spec_ids: - if i_spec not in indices: - # Create a new unique index and populate it. - _create_index(extent_map, i_spec, True) - for oid in entities: - fields_by_id = entities[oid]['fields'] - field_values = tuple(fields_by_id.get(field_id, UNASSIGNED) - for field_id in i_spec) - _index_add(extent_map, i_spec, None, oid, field_values) - # Create new non-unique indices for those that don't exist. - for i_spec in index_spec_ids: - if i_spec not in indices: - # Create a new non-unique index and populate it. - _create_index(extent_map, i_spec, False) - for oid in entities: - fields_by_id = entities[oid]['fields'] - field_values = tuple(fields_by_id.get(field_id, UNASSIGNED) - for field_id in i_spec) - _index_add(extent_map, i_spec, None, oid, field_values) - # Remove key indices that no longer exist. - to_remove = set(indices.keys()) - set(key_spec_ids + index_spec_ids) - for i_spec in to_remove: - _delete_index(extent_map, i_spec) - # Check non-unique indices to see if any are supersets of - # unique indices. If any found, change them to 'unique' and - # validate them. - # - # XXX: Needs testing. - for i_spec, (unique, branch) in indices.items(): - # Look for unique index supersets of this index, and make - # it unique if any exist. - if not unique: - spec_set = set(index_spec) - for i_spec in indices: - compare_set = set(i_spec) - if compare_set.issuperset(spec_set): - unique = True - break - if unique: - # Should be unique but isn't; alter and validate. - indices[i_spec] = (unique, branch) - for oid in entities: - fields_by_id = entities[oid]['fields'] - field_values = tuple(fields_by_id[field_id] - for field_id in i_spec) - _index_validate(extent_map, i_spec, oid, field_values) - - def _validate_changes(self, changes): - entity_classes = self._entity_classes - changes = change.normalize(changes) - for typ, extent_name, oid in changes: - if typ in (CREATE, UPDATE): - EntityClass = entity_classes[extent_name] - entity = EntityClass(oid) - field_map = entity.sys.field_map(not_fget) - for field in field_map.itervalues(): - field.validate(field._value) - - def _reset_all(self): - """Clear all entities, indices, etc. in the database. - - FOR USE WITH SINGLE-SCHEMA UNIT TESTS. - - NOT INDENDED FOR GENERAL USE. - """ - for extent_name in self.extent_names(): - extent_map = self._extent_map(extent_name) - extent_map['entities'] = BTree() - extent_map['len'] = 0 - extent_map['next_oid'] = 1 - indices = extent_map['indices'] - for index_spec, (unique, index_tree) in indices.items(): - indices[index_spec] = (unique, BTree()) - self._commit() - self.dispatch = Database.dispatch - self.label = Database.label - self._initialize() - self._on_open() - - -def _create_index(extent_map, index_spec, unique): - """Create a new index in the extent with the given spec and - uniqueness flag.""" - assert log(1, extent_map['name']) - assert log(1, 'index_spec', index_spec) - indices = extent_map['indices'] - index_map = extent_map['index_map'] - normalized_index_map = extent_map['normalized_index_map'] - # Look for unique index subsets of this index, and make it unique - # if any exist. - if not unique: - spec_set = set(index_spec) - for i_spec in indices: - compare_set = set(i_spec) - if compare_set.issubset(spec_set): - unique = True - break - # Continue with index creation. - assert log(2, 'unique', unique) - assert log(2, 'normalized_index_map.keys()', normalized_index_map.keys()) - partial_specs = _partial_index_specs(index_spec) - assert log(3, 'partial_specs', partial_specs) - normalized_specs = _normalized_index_specs(partial_specs) - assert log(3, 'normalized_specs', normalized_specs) - index_root = BTree() - indices[index_spec] = (unique, index_root) - for partial_spec in partial_specs: - L = index_map.setdefault(partial_spec, PList()) - L.append(index_spec) - for normalized_spec in normalized_specs: - L = normalized_index_map.setdefault(normalized_spec, PList()) - L.append(index_spec) - assert log(2, 'normalized_index_map.keys()', normalized_index_map.keys()) - - -def _delete_index(extent_map, index_spec): - indices = extent_map['indices'] - index_map = extent_map['index_map'] - normalized_index_map = extent_map['normalized_index_map'] - partial_specs = _partial_index_specs(index_spec) - normalized_specs = _normalized_index_specs(partial_specs) - del indices[index_spec] - for partial_spec in partial_specs: - L = index_map[partial_spec] - L.remove(index_spec) - if not L: - del index_map[partial_spec] - for normalized_spec in normalized_specs: - if normalized_spec in normalized_index_map: - L = normalized_index_map[normalized_spec] - L.remove(index_spec) - if not L: - del normalized_index_map[normalized_spec] - - -def _field_ids(extent_map, field_names): - """Convert a (field-name, ...) tuple to a (field-id, ...) - tuple for the given extent map.""" - field_name_id = extent_map['field_name_id'] - return tuple(field_name_id[name] for name in field_names) - - -def _field_names(extent_map, field_ids): - """Convert a (field-id, ...) tuple to a (field-name, ...) tuple - for the given extent map.""" - field_id_name = extent_map['field_id_name'] - return tuple(field_id_name[id] for id in field_ids) - - -def _index_add(extent_map, index_spec, relaxed, oid, field_values): - """Add an entry to the specified index, of entity oid having the - given values in order of the index spec.""" - indices = extent_map['indices'] - unique, branch = indices[index_spec] - # Traverse branches to find a leaf. - for field_id, field_value in zip(index_spec, field_values): - if field_value in branch: - branch = branch[field_value] - else: - new_branch = BTree() - branch[field_value] = new_branch - branch = new_branch - # Raise error if unique index and not an empty leaf. - if unique and branch.keys() and relaxed is None: - _index_clean(extent_map, index_spec, field_values) - raise error.KeyCollision( - 'Duplicate value %r for key %r on %r' - % (field_values, _field_names(extent_map, index_spec), - extent_map['name']), - branch.keys()[0]) - # Inject the OID into the leaf. - branch[oid] = True - # Keep track of the addition if relaxed. - if relaxed is not None: - relaxed.append((extent_map, index_spec, oid, field_values)) - - -def _index_clean(extent_map, index_spec, field_values): - """Remove stale branches from the specified index.""" - indices = extent_map['indices'] - unique, branch = indices[index_spec] - _index_clean_branch(branch, field_values) - - -def _index_clean_branch(branch, field_values): - """Recursively clean a branch of stale child branches.""" - branch_value = field_values[0] - child_values = field_values[1:] - if branch_value in branch: - if child_values: - # Clean children first. - _index_clean_branch(branch[branch_value], child_values) - # Clean ourself if empty. - if not branch[branch_value].keys(): - del branch[branch_value] - - -def _index_remove(extent_map, index_spec, oid, field_values): - """Remove an entry from the specified index, of entity oid having - the given values in order of the index spec.""" - indices = extent_map['indices'] - unique, branch = indices[index_spec] - # Traverse branches to find a leaf. - for field_id, field_value in zip(index_spec, field_values): - if field_value not in branch: - # Was never indexed for some reason, so stop traversing. - break - branch = branch[field_value] - if oid in branch: - del branch[oid] - _index_clean(extent_map, index_spec, field_values) - - -def _index_validate(extent_map, index_spec, oid, field_values): - """Validate the index entry for uniqueness.""" - indices = extent_map['indices'] - unique, branch = indices[index_spec] - # Traverse branches to find a leaf. - for field_id, field_value in zip(index_spec, field_values): - if field_value in branch: - branch = branch[field_value] - else: - new_branch = BTree() - branch[field_value] = new_branch - branch = new_branch - # Raise error if unique index and not an empty leaf. - if unique and len(branch.keys()) > 1: - _index_clean(extent_map, index_spec, field_values) - raise error.KeyCollision( - 'Duplicate value %r for key %r' - % (field_values, _field_names(extent_map, index_spec)), - None) - - -def _normalized_index_specs(index_specs): - """Return normalized index specs based on index_specs.""" - return [tuple(sorted(spec)) for spec in index_specs] - - -def _partial_index_specs(index_spec): - """Return a list of partial index specs based on index_spec.""" - return [tuple(index_spec[:x+1]) for x in xrange(len(index_spec))] - - -def _walk_index(branch, ascending_seq, result_list): - """Recursively walk a branch of an index, appending OIDs found to - result_list. - - - `branch`: The branch to start at. - - `ascending_seq`: The sequence of ascending flags corresponding - to the current branch. - - `result_list`: List to append OIDs to. - """ - if len(ascending_seq): - # We are at a branch. - ascending, inner_ascending = ascending_seq[0], ascending_seq[1:] - if ascending: - for key, inner_branch in branch.iteritems(): - _walk_index(inner_branch, inner_ascending, result_list) - else: - keys = reversed(branch.keys()) - for key in keys: - inner_branch = branch[key] - _walk_index(inner_branch, inner_ascending, result_list) - else: - # We are at a leaf. - result_list.extend(branch.iterkeys()) - - -class DatabaseExtenders(NamespaceExtension): - """Methods that extend the functionality of a database.""" - - __slots__ = NamespaceExtension.__slots__ - - _readonly = False - - def __init__(self, schema_module): - NamespaceExtension.__init__(self) - # Expose functions through this namespace. - for name in dir(schema_module): - # Extender functions always have x_ prefix. - if name.startswith('x_'): - function = getattr(schema_module, name) - # Drop the 'x_' prefix. - name = name[2:] - self._set(name, function) - - optimize.bind_all(sys.modules[__name__]) # Last line of module. schevo/database.py @@ -165,6 +165,7 @@ class EntityMeta(type): except KeyError: # XXX This needs to be more specific. value = UNASSIGNED field._value = value + field._restore(db) return field.get() setattr(cls, field_name, property(fget=get_field_value)) cls._fget_fields = tuple(fget_fields) @@ -273,7 +274,7 @@ class Entity(base.Entity, LabelMixin): # The actual class/extent name to use for this Entity type. _actual_name = None - + # The database instance associated with this Entity type. _db = None @@ -650,6 +651,7 @@ class EntitySys(NamespaceExtension): """Return field_map for the entity, filtered by optional callable objects specified in `filters`.""" e = self._entity + db = e._db stored_values = e._db._entity_fields(e._extent.name, e._oid) entity_field_map = e._field_spec.field_map(e, stored_values) # Remove fields that should not be included. @@ -658,13 +660,14 @@ class EntitySys(NamespaceExtension): new_fields = [field for field in new_fields if filt(field)] entity_field_map = FieldMap( (field.name, field) for field in new_fields) - # Update fields that have fget callables. for field in entity_field_map.itervalues(): if field.fget is not None: + # Update fields that have fget callables. value = field.fget[0](e) else: - value = field._value - field._value = field.convert(value) + # Allow fields to restore themselves from a stored + # value. + field._restore(db) return entity_field_map def links(self, other_extent_name=None, other_field_name=None): @@ -745,19 +748,6 @@ class EntityViews(NamespaceExtension): if k not in self._e._hidden_views) -class EntityRef(object): - """Reference to an Entity via its extent name and OID.""" - - def __init__(self, extent_name, oid): - """Create an EntityRef instance. - - - `extent_name`: The name of the extent. - - `oid`: The OID of the entity. - """ - self.extent_name = extent_name - self.oid = oid - - optimize.bind_all(sys.modules[__name__]) # Last line of module. schevo/entity.py @@ -94,6 +94,10 @@ class AmbiguousFieldDefinition(SchemaError): """A field defition's attributes were ambiguous.""" +class UnsupportedFieldType(SchemaError): + """The field type is not supported by the database engine in use.""" + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo schevo/error.py @@ -75,7 +75,7 @@ class Priority(E.Entity): @f.integer(label=u'# Done Items') def done(self): return len([item for item in self.m.items() if item.done]) - + _key(code) _key(name) @@ -88,7 +88,7 @@ class Priority(E.Entity): def __str__(self): return '%s %s' % (self.code, self.name) - + class Topic(E.Entity): """Subject area for todo items.""" schevo/example/todo/schema/schema_001.py @@ -8,14 +8,18 @@ from schevo.lib import optimize import datetime import md5 import random +import string import sys import time +from warnings import warn from schevo import base +from schevo.base import Entity as EntityActual from schevo.constant import ANY, RESTRICT, UNASSIGNED import schevo.error import schevo.fieldspec import schevo.namespace +from schevo.placeholder import Placeholder # Filters for calls to field_map methods. @@ -30,6 +34,20 @@ def not_hidden(field): return not field.hidden +def lowercase_from_camelcase(name): + """Return a lowercase, underscore-separated name based on a CamelCase one. + + For example, if 'ClassName' is given as input, 'class_name' is returned. + """ + new_name = '' + for char in name: + if char in string.uppercase: + if new_name != '': + new_name += '_' + new_name += char.lower() + return new_name + + class FieldMeta(type): """Create field constuctors for every Field class.""" @@ -43,18 +61,27 @@ class FieldMeta(type): def __init__(cls, class_name, bases, class_dict): type.__init__(cls, class_name, bases, class_dict) # Create a field constructor. - def_name = class_name[0].lower() + class_name[1:] + def_name = lowercase_from_camelcase(class_name) + deprecated_def_name = class_name[0].lower() + class_name[1:] class def_class(schevo.fieldspec.FieldDefinition): BaseFieldClass = cls + class deprecated_def_class(def_class): + _deprecated_name = True + _preferred_name = def_name def_class.__name__ = def_name + deprecated_def_class.__name__ = deprecated_def_name cls._def_class = def_class + cls._deprecated_def_class = def_class cls._def_name = def_name + cls._deprecated_def_name = deprecated_def_name # Only if this global schema definition variable exists. if schevo.namespace.SCHEMADEF is not None: # Add this class to the field classes namespace. schevo.namespace.SCHEMADEF.F._set(cls.__name__, cls) # Add a field constructor to the field constructors namespace. schevo.namespace.SCHEMADEF.f._set(def_name, def_class) + schevo.namespace.SCHEMADEF.f._set( + deprecated_def_name, deprecated_def_class) class Field(base.Field): @@ -97,13 +124,16 @@ class Field(base.Field): label: Descriptive label for the field that can be used for reporting, the GUI field label, the column heading, etc. + min_size: Minimum size allowed, or None for no limit. + max_size: Maximum size allowed, or None for no limit. - min_size: Minimum size allowed, or None for no limit. + min_value: Minimum value allowed, or None for no limit. max_value: Maximum value allowed, or None for no limit. - min_value: Minimum value allowed, or None for no limit. + may_store_entities: True if the field's value may store references + to entities. notice: A tuple of (notice type, notice text) describing a notice for this field, or None for no notice. @@ -141,10 +171,11 @@ class Field(base.Field): fget = None hidden = False label = None - max_size = None min_size = None - max_value = None + max_size = None min_value = None + max_value = None + may_store_entities = False notice = None preferred_values = None readonly = False @@ -175,7 +206,7 @@ class Field(base.Field): """Create a Field instance for an instance with a given value. instance: usually an Entity, Transaction or Query instance. - + value: optional initial value, without validation checking. rev: revision of the instance containing the value, if @@ -200,6 +231,11 @@ class Field(base.Field): # Otherwise a field is created with an initial rev of -1. self._rev = -1 + def _entities_in_value(self): + """Return a set or frozenset of Placeholders for entities contained in + the field's value.""" + return frozenset() + def _initialize(self, value): """Initialize the field with a value.""" self._initial = value @@ -264,7 +300,11 @@ class Field(base.Field): new_field = FieldClass(None, None) new_field.__dict__.update(self.__dict__) return new_field - + + def _dump(self): + """Return a value suitable for storage in a database.""" + return self._value + def get(self): """Return the field value.""" if self.fget is not None: @@ -272,6 +312,11 @@ class Field(base.Field): else: return self._value + def _restore(self, db): + """Restore field's true value by converting it from the value stored + in the database.""" + pass + def reversible(self, value=None): """Return a reversible string representation of the field value, or a different value if ``value`` is not None. @@ -582,14 +627,14 @@ class Blob(Field): return Field.__str__(self) else: return '(Binary data)' - + def __unicode__(self): v = self.get() if v is UNASSIGNED: return Field.__unicode__(self) else: return u'(Binary data)' - + class Image(Blob): """Image field class.""" @@ -904,8 +949,8 @@ class Boolean(Field): # -------------------------------------------------------------------- -class Entity(Field): - """Entity instance field class. +class _EntityBase(Field): + """Private base class for fields that may store entity references. allow_create: set to True if UI should give users the option of creating new instances when displaying this field. @@ -929,6 +974,7 @@ class Entity(Field): allow_create = True allow_update = False allow = set() + may_store_entities = True on_delete = {} on_delete_default = RESTRICT @@ -991,6 +1037,25 @@ class Entity(Field): else: return value + def _dump(self): + """Return a value suitable for storage in a database.""" + value = self._value + if isinstance(value, EntityActual): + value = Placeholder(value) + return value + + def _entities_in_value(self): + value = self._value + if isinstance(value, EntityActual): + return frozenset([Placeholder(value)]) + return frozenset() + + def _restore(self, db): + value = self._value + if isinstance(value, Placeholder): + value = value.restore(db) + self._value = value + def reversible(self, value=None): if value is None: value = self.get() @@ -998,7 +1063,7 @@ class Entity(Field): return u'' else: return u'%s-%i' % (value.sys.extent.name, value.sys.oid) - + def reversible_valid_values(self, db): """Returns a list of (reversible, value) tuples for the valid values of this field.""" @@ -1054,6 +1119,62 @@ class Entity(Field): self._raise(TypeError, msg) +class Entity(_EntityBase): + """Entity instance field class.""" + pass + + +class EntityList(_EntityBase): + """List of Entity instances field class.""" + + def convert(self, value, db=None): + if isinstance(value, list): + new_values = [] + for item in value: + new_values.append(_EntityBase.convert(self, item, db)) + value = new_values + else: + value = _EntityBase.convert(self, value, db) + return value + + def _dump(self): + value = self._value + if isinstance(value, list): + value = [Placeholder(entity) for entity in value] + return value + + def _entities_in_value(self): + value = self._value + if isinstance(value, list): + return frozenset(Placeholder(entity) for entity in value) + return frozenset() + + def _restore(self, db): + value = self._value + if isinstance(value, list): + value = [placeholder.restore(db) for placeholder in value] + self._value = value + + def reversible(self, value=None): + return None + + def validate(self, value): + """Validate the value, raising an error on failure.""" + if isinstance(value, list): + for item in value: + _EntityBase.validate(self, item) + else: + _EntityBase.validate(self, value) + + def verify(self, value): + """Verify the value, raising an error on failure.""" + if isinstance(value, list): + for item in value: + _EntityBase.verify(self, item) + else: + _EntityBase.verify(self, value) + + optimize.bind_all(sys.modules[__name__]) # Last line of module. schevo/field.py @@ -6,6 +6,8 @@ For copyright, license, and warranty, see bottom of file. import sys from schevo.lib import optimize +from warnings import warn + from schevo.constant import UNASSIGNED from schevo.label import label_from_name from schevo.lib.odict import odict @@ -16,6 +18,21 @@ class FieldMap(odict): __slots__ = ['_keys'] + def dump_map(self): + """Return a dictionary of field_name:dumped_value pairs.""" + d = odict() + for name, field in self.items(): + d[name] = field._dump() + return d + + def related_entity_map(self): + """Return a dictionary of field_name:related_entity_set pairs.""" + d = odict() + for name, field in self.items(): + if field.may_store_entities: + d[name] = field._entities_in_value() + return d + def update_values(self, other): """Update field values based on field values in other FieldMap.""" for name, field in other.iteritems(): @@ -26,13 +43,16 @@ class FieldMap(odict): f.set(field.get()) def value_map(self): + """Return a dictionary of field_name:field_value pairs.""" d = odict() for name, field in self.items(): # Do not use field.get() here because we want the value to # be stored in the database, not the value that is exposed # to users. - value = field._value - d[name] = value + # + # XXX: Is this comment and implementation accurate + # anymore, now that we use dump_map for that purpose? + d[name] = field._value return d @@ -58,7 +78,7 @@ class FieldSpecMap(odict): value=values.get(name, UNASSIGNED))) for name, FieldClass in self.iteritems()] return FieldMap(pairs) - + def field_spec_from_class(cls, class_dict, slots=False): field_spec = FieldSpecMap() @@ -110,9 +130,21 @@ class FieldDefinition(object): __do_not_optimize__ = True + # The field class that this field definition class is based on. BaseFieldClass = None + + # "Global", class-level counter used for sorting after a series of + # fields are defined. _counter = 0 + # Whether or not the name of this field definition class is + # deprecated. If it is deprecated, the user is given a DeprecationWarning + # about the use of this name. + _deprecated_name = False + + # If this class's name is deprecated, the preferred class name. + _preferred_name = None + def __init__(self, *args, **kw): self.name = None # Set by field_spec_from_class(). BaseFieldClass = self.BaseFieldClass @@ -126,6 +158,14 @@ class FieldDefinition(object): self.FieldClass = _Field self.counter = FieldDefinition._counter FieldDefinition._counter += 1 + # Warn about name deprecation if this class's name is deprecated. + if self._deprecated_name: + msg = ( + '%r is a deprecated field definition name. ' + 'Please replace with %r in your schemata.' + % (self.__class__.__name__, self._preferred_name) + ) + warn(msg, DeprecationWarning, 2) def __call__(self, fn): """For use as a decorator.""" schevo/fieldspec.py @@ -33,7 +33,7 @@ preamble = textwrap.dedent( from warnings import warn as _warn _warn('See http://schevo.org/lists/archives/schevo-devel/' '2006-March/000568.html', DeprecationWarning) - _import('Schevo', 'icon', 1) + _import('Schevo', 'icon', 1) """ ) # /XXX schevo/icon/schema/schema_001.py @@ -122,7 +122,7 @@ class SchevoIdUser(E.Entity): display_name = f.unicode(required=False) password = f.password() created = f.datetime(default=datetime.datetime.now) - + _key(name) _initial = [ @@ -162,7 +162,7 @@ preamble = textwrap.dedent( from warnings import warn as _warn _warn('See http://schevo.org/lists/archives/schevo-devel/' '2006-March/000568.html', DeprecationWarning) - _import('Schevo', 'identity', 1) + _import('Schevo', 'identity', 1) """ ) # /XXX schevo/identity/schema/schema_001.py @@ -20,9 +20,9 @@ class odict(dict): def __init__(self, seq=None): """dict() -> new empty dictionary. - + dict(seq) -> new dictionary initialized as if via:: - + d = {} for k, v in seq: d[k] = v @@ -114,7 +114,7 @@ class odict(dict): def pop(self, key, *failobj): """D.pop(k[,d]) -> v, remove specified key and return the corresponding value - + If key is not found, d is returned if given, otherwise KeyError is raised """ schevo/lib/odict.py @@ -3,7 +3,7 @@ For copyright, license, and warranty, see bottom of file. """ -from schevo.database import dummy_lock +from schevo.mt.dummy import dummy_lock from schevo.mt import mrow @@ -16,7 +16,7 @@ def install(db): class Plugin(object): """A plugin giving a Schevo database a multiple-reader, one-writer locking mechanism. - + Usage:: from schevo.database import open schevo/mt/plugin.py @@ -572,7 +572,7 @@ class Max(Query): self.query = query self.field_name = field_name self.FieldClass = FieldClass - + def __call__(self): def generator(): field_name = self.field_name schevo/query.py @@ -33,7 +33,7 @@ class Command(object): def help(self): pass - + def main(self, arg0, args): if ((self.requires_args and not args) or (args and (args[0] in ('--help'))) @@ -60,7 +60,7 @@ class CommandSet(Command): for command_name, command in sorted(commands.items()): print format % (command_name, command.description) print - + def main(self, arg0, args): if not Command.main(self, arg0, args): # Replace arg0 with the command specified and take it off @@ -72,7 +72,7 @@ class CommandSet(Command): return 1 command = self.commands[command_name] return command()(*args) - + # Copyright (C) 2001-2006 Orbtech, L.L.C. # schevo/script/command.py @@ -5,6 +5,7 @@ For copyright, license, and warranty, see bottom of file. from schevo.script.command import CommandSet from schevo.script import ( + db_convert, db_create, db_inject, db_evolve, @@ -19,6 +20,7 @@ class Database(CommandSet): def __init__(self): self.commands = { + 'convert': db_convert.start, 'create': db_create.start, 'inject': db_inject.start, 'evolve': db_evolve.start, schevo/script/db.py @@ -54,7 +54,7 @@ def start_hotshot(): stats.print_stats(schevo_package_path, 50) stats.sort_stats('time', 'calls') stats.print_stats(schevo_package_path, 50) - + # Copyright (C) 2001-2006 Orbtech, L.L.C. # schevo/script/main.py @@ -1,36 +1,36 @@ -Durus was inspired by ZODB and ZEO, the database and client/server -code produced by our friends at Zope, Inc. Like ZODB, Durus -implements a pickle storage system and manages instances of a -Persistent class in a transactional way. Like ZEO, Durus supports -access to a shared storage from connections in multiple processes. -Unlike ZODB/ZEO, Durus does not support access to earlier object versions -or object modification times. - -The PersistentList class of Durus is derived from the python standard -UserList, following the same pattern as the ZODB4 PersistentList -class. - -The PersistentDict class of Durus is derived from the python standard -UserDict, following the same pattern as the ZODB4 PersistentMapping -class. - -The code of Durus shares some small fragments and some names with code -in ZODB/ZEO, but we do not consider Durus to be a modification of -ZODB/ZEO. - -Durus was developed by the MEMS Exchange software team: - -Anton Benard -David Binger -Roger Masse -Neil Schemenauer - -Neil initiated work on Durus and made it fast. - --- - -Other Contributors: - -Andrew Kuchling -Patrick K. O'Brien -Jesus Cea Avion +Durus was inspired by ZODB and ZEO, the database and client/server +code produced by our friends at Zope, Inc. Like ZODB, Durus +implements a pickle storage system and manages instances of a +Persistent class in a transactional way. Like ZEO, Durus supports +access to a shared storage from connections in multiple processes. +Unlike ZODB/ZEO, Durus does not support access to earlier object versions +or object modification times. + +The PersistentList class of Durus is derived from the python standard +UserList, following the same pattern as the ZODB4 PersistentList +class. + +The PersistentDict class of Durus is derived from the python standard +UserDict, following the same pattern as the ZODB4 PersistentMapping +class. + +The code of Durus shares some small fragments and some names with code +in ZODB/ZEO, but we do not consider Durus to be a modification of +ZODB/ZEO. + +Durus was developed by the MEMS Exchange software team: + +Anton Benard +David Binger +Roger Masse +Neil Schemenauer + +Neil initiated work on Durus and made it fast. + +-- + +Other Contributors: + +Andrew Kuchling +Patrick K. O'Brien +Jesus Cea Avion schevo/store/ACKS.txt @@ -1,299 +1,299 @@ -3.5: (2006-08-16): - - * Fix a bug introduced in version 3.4 that could, under certain conditions, - allow conflicts to be missed. In particular, if the last strong reference - to a Persistent instance was removed, a conflict involving that instance - would be missed. The fix involves changing the Persistent.__getattr__ - so that it calls the 'note_access' method on the Connection. - This method creates a strong reference to the Persistent instance by - adding it to a 'recent_objects' set. Objects are eventually removed from - the recent_objects set when they are converted into ghosts in the cache's - shrink() method. Since a set() is used for recent_objects, all Persistent - instances *must* now be hashable. PersistentSet instances were not, and - that has been changed. - - * Revise the cache code. It now uses a WeakValueDict instead of a plain dict - to hold the references. This simplifies the code because we no longer need - to call the weakref instances directly. It also helps the cache shrinking - loop because the weakref callbacks have an immediate impact on the size of - the mapping. - - * Remove the ghost_fraction attribute from the Cache. It is no longer used - in shrink(). - - * Rename the Persistent '_p_touched' attribute to '_p_serial'. This is - set, on every attribute access, to the value of the Connection's - 'transaction_serial' attribute (which was formerly named 'sync_count'). - - * Add a bulk_load() method to Storage. The ClientStorage implementation - reduces avoids latency delays be retrieving many object records with - a single request to the server. - - * Add Connection.get_crawler(). This returns a generator for the sequence - of objects reachable from a given start OID. The crawler uses the - new bulk_load() method for speed. It can be used, with some care, to - initialize the object cache. - - * Remove Connection.cache_get() and Connection.cache_set(). - - * Use set instead of Set throughout. This means that Durus now requires - Python version >= 2.4. - - * Add the ability to set the owner, group, and umask when a unix domain - socket is used for a server. - - * Attempt to clean up stale socket files when starting a server on a - unix domain socket. - - * Move some repeated code related to addresses and sockets into a - SocketAddress class and subclasses HostPortAddress and - UnixDomainSocketAddress. - - * In the server, add support for a protocol verification command. - Use this in the client constructor to allow orderly behavior if - the client and the server do not implement the same protocol. - - * Add a server command for allocating blocks of oids. - - * Add client support for maintaining and drawing from a pool of oids - allocated by the server. This reduces the number of commands that - must be sent to the server during a commit. - - * Add support for recycling allocated oids when there is a conflict during - a commit. - - * Make sure that the FileStorage constructor can work if the named file - exists, but is empty. - - * Initialize sync_count to 1 so that new ghosts, for which _p_touched - is initialized to 0, never appear to have been accessed since the last - transaction. - - * Move some logic used for unpickling references to the connection cache - so that it can be faster. Add Cache.get_instance() for this purpose. - Add Connection.get_cache() so that the ObjectReader can use it. - - -3.4.1: (2006-05-18): - - * Fix a memory leak that was in the 3.4 tarball until 2006-05-12. - - * Fix doc string errors. - - * Fix initialization of _p_touched in python version of PersistentBase - to agree with the C implementation. - -3.4: (2006-05-11): 28347 - - * Refine the conflict avoidance and cache aging behavior. Now conflicts - don't occur unless there is an invalid object for which this Connection - has actually accessed an attribute since the last call to commit() or - abort(). The Connection saves a "sync_count" on every commit() or abort(). - On every access to (an ordinary) attribute of a Persistent instance, - the _p_touched is set to be the Connection's sync_count. - To make this possible without any significant performance penalty, - the _p_connection and other '_p_' attributes are moved from - Persistent to PersistentBase and implemented in C. - Also, a ConnectionBase class is implemented in C so that the - sync_count, which is needed so frequently, can be accessed directly - in the C implementation of PersistentBase. - - Since we now know which instances have actually been accessed since the - last commit() or abort(), the Connection no longer need to maintain the - set of loaded_oids. The cache manager can use the _p_touched to - distinguish less recently used instances. - - The Cache class has a new ghost_fraction attribute. The value of - this attribute defaults to 0.5 and can be any number between 0 and 1. - Higher values make the cache more aggressive about ghosting objects - as it tries to reduce the cache size. - - The Cache "held" attribute is removed, along with the hold() method. - - * Added a history.py module that defines HistoryConnection, a Connection - subclass that supports time-travel in a read-only FileStorage file. - The class provides next() and previous() methods for stepping - among the stored transactions. It also provides next_instance(obj) - and previous_instance(obj) for moving to to a transaction where - obj has a state that is different from the current state. - Note that packing a FileStorage consolidates the transactions, so - the HistoryConnection can only move among the transactions since - the last pack. - - * Make the durus client run in a __console__ module. This makes it - behave a little more like the regular Python interpreter. - - * Add support for running the durus client/server connections through - unix domain sockets. The ClientStorage and StorageServer constructors - accept an "address" keyword argument. If the address value can be - a (host, port) tuple or else a string giving a path use for the - unix domain socket. The separate "host" and "port" keyword parameters - are still supported, but they may be removed in future releases. If your - code calls these constructors, please change it to use the "address" - keyword. - - * Change the recv() function used in the client/server to read in chunks - of at most 1 million bytes. This avoids a malloc error observed when - running test/stress.py on an OS X machine with python 2.4.2. - - * Make the durus server a little tougher. If it gets an unknown command, - it now logs the error and closes the connection instead of crashing. - - * Add Storage.pack() and Storage.get_size(). - - -3.3: (2006-03-15): 28065 - - * Keep strong references to objects until we decide that they haven't - been recently touched. This limits the impact of the Python - garbage collector. - - * Change the FileStorage class as needed to agree with the magic - header string found in an existing file. Do this no matter which - of the constructors (FileStorage, FileStorage1, or FileStorage2) - is called to create the instance. Before this change, opening an - an existing file with the FileStorage2 constructor (instead of - the generic FileStorage constructor), raised an exception. - - * Adjust logging. - - Before this change, the server logs the class of each object - when it is loaded if the logginglevel is at level 5 or below. - This changes that threshhold to level 4. - - Now, when the logging level is 5 or below, the server prints - a census of objects loaded since the last commit or abort. - This can make it easier to understand patterns of cache misses. - - * Add dummy new_oid() and get_packer() methods to the Storage class - for documentation. - -3.2: (2006-02-01): r27892 - - * Add 5 iteration methods to BTree. - __reversed__() for reverse iteration of keys. - items_backward() for reverse iteration of items. - items_from() for iterations starting at a given key. - items_backward_from() for reverse iterations from a given key. - items_range() for iterations of items with keys in a given range. - - * Add __nonzero__ and setdefault methods to BTree - - * Change the name of BTree's get_count method to __len__. - - * Add setuptools support (when installed) to setup.py. - - * Remove convert_zodb.py script, rather than fix/maintain it. - -3.1: (2005-10-18): r27556 - - * Add PersistentSet. (Applications that use the persistent_set - must use Python versions >= 2.4). - - * Add MemoryStorage, an in-memory storage for testing purposes. - -3.0: (2005-09-08): r27334 - - * Fix bug in utility function (touch_every_reference()) added in 3.0a. - - * Replace ._p_changed = 1 to ._p_note_change() in btree.py. - -3.0a: (2005-08-09): r27118 - - * Revise packed record format to write records where the instance state is - compressed using zlib. This reduces the size of stored files and the - number of bytes of data transmitted during load/store operations. - - * Add a FileStorage2 file format. The new format does not need or store - transaction identifiers. It also includes a pre-built index of the - object records written at the time of the last pack. This results in - a faster start-up. Conversion is not automatic, though. - A convert_file_storage.py is included with this release to make it simple - to convert a file to either format. - - * Add stress testing client stress.py. - - * When the state of a ghost can't be loaded because it has been - removed by a pack (from another connection), make the error be a - ReadConflictError instead of a DurusKeyError. - - * Implement an incremental pack. The storage server can now serve - clients while packing the database. - - * Add gen_every_instance() utility to durus.connection. - - * Add touch_every_reference() utility function to durus.connection. - - * Remove the use of tids from Connection, ClientStorage, and - StorageServer. The StorageServer now sends STATUS_INVALID for - requests to load object records that are known to be invalid for - that client. Change Storage.load() so that the return value is - a tid-less object-record. The client/server protocol also - changes to stop transmitting tids as part of the response for - sync and commit requests. Storage.sync() now returns only a - list of oids. Storage.end() now returns None. - -2.0 (2005-4-26) r26653: - - * The only change is in the license. The new license is the GPL-compatible - version of the CNRI Open Source License. - -1.5 (2005-3-07) r26296: - - * A small change makes Persistent instances pickle-able. - -1.4 (2005-1-14) r25851: - - * Revise serialize.py to avoid using cPickle.noload(), which can't handle - a dict subclass. - - * Add a --startup option to the durus command line tool. - -1.3 (2004-12-10) r25746: - - * Use 'b' in file modes. - - * Continue to rename the open packed file on POSIX systems. - - * Convert tests to sancho's new utest format. - - * Improve test coverage for FileStorage. - - * Lower the priority of Sync logging. - - * Show host and port when client fails to connect. - -1.2 (2004-09-08) r25044: - - * Add durus command line tool. - - * Add btree module. - - * Update pack() so that it works on Windows and so that it gets a lock - again after the pack is completed. - -1.1 (2004-08-05) r24872: - - * Provide a close() method for FileStorage. Call appropriate win32 - unlock function. - - * Remember to lock the new file created by pack(). Unlock the - old file. - - -1.0 (2004-07-31) r24846: - - * Fix obscure bug in storage_server during logging. - - * Repaired example in README.txt. - - * Added FAQ. - - * Made ProtocolError inherit from DurusError. - - * Added fsync() after writing transaction data. - - -0.1 (2004-07-27) r24791: - - * Initial Release +3.5: (2006-08-16): + + * Fix a bug introduced in version 3.4 that could, under certain conditions, + allow conflicts to be missed. In particular, if the last strong reference + to a Persistent instance was removed, a conflict involving that instance + would be missed. The fix involves changing the Persistent.__getattr__ + so that it calls the 'note_access' method on the Connection. + This method creates a strong reference to the Persistent instance by + adding it to a 'recent_objects' set. Objects are eventually removed from + the recent_objects set when they are converted into ghosts in the cache's + shrink() method. Since a set() is used for recent_objects, all Persistent + instances *must* now be hashable. PersistentSet instances were not, and + that has been changed. + + * Revise the cache code. It now uses a WeakValueDict instead of a plain dict + to hold the references. This simplifies the code because we no longer need + to call the weakref instances directly. It also helps the cache shrinking + loop because the weakref callbacks have an immediate impact on the size of + the mapping. + + * Remove the ghost_fraction attribute from the Cache. It is no longer used + in shrink(). + + * Rename the Persistent '_p_touched' attribute to '_p_serial'. This is + set, on every attribute access, to the value of the Connection's + 'transaction_serial' attribute (which was formerly named 'sync_count'). + + * Add a bulk_load() method to Storage. The ClientStorage implementation + reduces avoids latency delays be retrieving many object records with + a single request to the server. + + * Add Connection.get_crawler(). This returns a generator for the sequence + of objects reachable from a given start OID. The crawler uses the + new bulk_load() method for speed. It can be used, with some care, to + initialize the object cache. + + * Remove Connection.cache_get() and Connection.cache_set(). + + * Use set instead of Set throughout. This means that Durus now requires + Python version >= 2.4. + + * Add the ability to set the owner, group, and umask when a unix domain + socket is used for a server. + + * Attempt to clean up stale socket files when starting a server on a + unix domain socket. + + * Move some repeated code related to addresses and sockets into a + SocketAddress class and subclasses HostPortAddress and + UnixDomainSocketAddress. + + * In the server, add support for a protocol verification command. + Use this in the client constructor to allow orderly behavior if + the client and the server do not implement the same protocol. + + * Add a server command for allocating blocks of oids. + + * Add client support for maintaining and drawing from a pool of oids + allocated by the server. This reduces the number of commands that + must be sent to the server during a commit. + + * Add support for recycling allocated oids when there is a conflict during + a commit. + + * Make sure that the FileStorage constructor can work if the named file + exists, but is empty. + + * Initialize sync_count to 1 so that new ghosts, for which _p_touched + is initialized to 0, never appear to have been accessed since the last + transaction. + + * Move some logic used for unpickling references to the connection cache + so that it can be faster. Add Cache.get_instance() for this purpose. + Add Connection.get_cache() so that the ObjectReader can use it. + + +3.4.1: (2006-05-18): + + * Fix a memory leak that was in the 3.4 tarball until 2006-05-12. + + * Fix doc string errors. + + * Fix initialization of _p_touched in python version of PersistentBase + to agree with the C implementation. + +3.4: (2006-05-11): 28347 + + * Refine the conflict avoidance and cache aging behavior. Now conflicts + don't occur unless there is an invalid object for which this Connection + has actually accessed an attribute since the last call to commit() or + abort(). The Connection saves a "sync_count" on every commit() or abort(). + On every access to (an ordinary) attribute of a Persistent instance, + the _p_touched is set to be the Connection's sync_count. + To make this possible without any significant performance penalty, + the _p_connection and other '_p_' attributes are moved from + Persistent to PersistentBase and implemented in C. + Also, a ConnectionBase class is implemented in C so that the + sync_count, which is needed so frequently, can be accessed directly + in the C implementation of PersistentBase. + + Since we now know which instances have actually been accessed since the + last commit() or abort(), the Connection no longer need to maintain the + set of loaded_oids. The cache manager can use the _p_touched to + distinguish less recently used instances. + + The Cache class has a new ghost_fraction attribute. The value of + this attribute defaults to 0.5 and can be any number between 0 and 1. + Higher values make the cache more aggressive about ghosting objects + as it tries to reduce the cache size. + + The Cache "held" attribute is removed, along with the hold() method. + + * Added a history.py module that defines HistoryConnection, a Connection + subclass that supports time-travel in a read-only FileStorage file. + The class provides next() and previous() methods for stepping + among the stored transactions. It also provides next_instance(obj) + and previous_instance(obj) for moving to to a transaction where + obj has a state that is different from the current state. + Note that packing a FileStorage consolidates the transactions, so + the HistoryConnection can only move among the transactions since + the last pack. + + * Make the durus client run in a __console__ module. This makes it + behave a little more like the regular Python interpreter. + + * Add support for running the durus client/server connections through + unix domain sockets. The ClientStorage and StorageServer constructors + accept an "address" keyword argument. If the address value can be + a (host, port) tuple or else a string giving a path use for the + unix domain socket. The separate "host" and "port" keyword parameters + are still supported, but they may be removed in future releases. If your + code calls these constructors, please change it to use the "address" + keyword. + + * Change the recv() function used in the client/server to read in chunks + of at most 1 million bytes. This avoids a malloc error observed when + running test/stress.py on an OS X machine with python 2.4.2. + + * Make the durus server a little tougher. If it gets an unknown command, + it now logs the error and closes the connection instead of crashing. + + * Add Storage.pack() and Storage.get_size(). + + +3.3: (2006-03-15): 28065 + + * Keep strong references to objects until we decide that they haven't + been recently touched. This limits the impact of the Python + garbage collector. + + * Change the FileStorage class as needed to agree with the magic + header string found in an existing file. Do this no matter which + of the constructors (FileStorage, FileStorage1, or FileStorage2) + is called to create the instance. Before this change, opening an + an existing file with the FileStorage2 constructor (instead of + the generic FileStorage constructor), raised an exception. + + * Adjust logging. + + Before this change, the server logs the class of each object + when it is loaded if the logginglevel is at level 5 or below. + This changes that threshhold to level 4. + + Now, when the logging level is 5 or below, the server prints + a census of objects loaded since the last commit or abort. + This can make it easier to understand patterns of cache misses. + + * Add dummy new_oid() and get_packer() methods to the Storage class + for documentation. + +3.2: (2006-02-01): r27892 + + * Add 5 iteration methods to BTree. + __reversed__() for reverse iteration of keys. + items_backward() for reverse iteration of items. + items_from() for iterations starting at a given key. + items_backward_from() for reverse iterations from a given key. + items_range() for iterations of items with keys in a given range. + + * Add __nonzero__ and setdefault methods to BTree + + * Change the name of BTree's get_count method to __len__. + + * Add setuptools support (when installed) to setup.py. + + * Remove convert_zodb.py script, rather than fix/maintain it. + +3.1: (2005-10-18): r27556 + + * Add PersistentSet. (Applications that use the persistent_set + must use Python versions >= 2.4). + + * Add MemoryStorage, an in-memory storage for testing purposes. + +3.0: (2005-09-08): r27334 + + * Fix bug in utility function (touch_every_reference()) added in 3.0a. + + * Replace ._p_changed = 1 to ._p_note_change() in btree.py. + +3.0a: (2005-08-09): r27118 + + * Revise packed record format to write records where the instance state is + compressed using zlib. This reduces the size of stored files and the + number of bytes of data transmitted during load/store operations. + + * Add a FileStorage2 file format. The new format does not need or store + transaction identifiers. It also includes a pre-built index of the + object records written at the time of the last pack. This results in + a faster start-up. Conversion is not automatic, though. + A convert_file_storage.py is included with this release to make it simple + to convert a file to either format. + + * Add stress testing client stress.py. + + * When the state of a ghost can't be loaded because it has been + removed by a pack (from another connection), make the error be a + ReadConflictError instead of a DurusKeyError. + + * Implement an incremental pack. The storage server can now serve + clients while packing the database. + + * Add gen_every_instance() utility to durus.connection. + + * Add touch_every_reference() utility function to durus.connection. + + * Remove the use of tids from Connection, ClientStorage, and + StorageServer. The StorageServer now sends STATUS_INVALID for + requests to load object records that are known to be invalid for + that client. Change Storage.load() so that the return value is + a tid-less object-record. The client/server protocol also + changes to stop transmitting tids as part of the response for + sync and commit requests. Storage.sync() now returns only a + list of oids. Storage.end() now returns None. + +2.0 (2005-4-26) r26653: + + * The only change is in the license. The new license is the GPL-compatible + version of the CNRI Open Source License. + +1.5 (2005-3-07) r26296: + + * A small change makes Persistent instances pickle-able. + +1.4 (2005-1-14) r25851: + + * Revise serialize.py to avoid using cPickle.noload(), which can't handle + a dict subclass. + + * Add a --startup option to the durus command line tool. + +1.3 (2004-12-10) r25746: + + * Use 'b' in file modes. + + * Continue to rename the open packed file on POSIX systems. + + * Convert tests to sancho's new utest format. + + * Improve test coverage for FileStorage. + + * Lower the priority of Sync logging. + + * Show host and port when client fails to connect. + +1.2 (2004-09-08) r25044: + + * Add durus command line tool. + + * Add btree module. + + * Update pack() so that it works on Windows and so that it gets a lock + again after the pack is completed. + +1.1 (2004-08-05) r24872: + + * Provide a close() method for FileStorage. Call appropriate win32 + unlock function. + + * Remember to lock the new file created by pack(). Unlock the + old file. + + +1.0 (2004-07-31) r24846: + + * Fix obscure bug in storage_server during logging. + + * Repaired example in README.txt. + + * Added FAQ. + + * Made ProtocolError inherit from DurusError. + + * Added fsync() after writing transaction data. + + +0.1 (2004-07-27) r24791: + + * Initial Release schevo/store/CHANGES.txt @@ -1,66 +1,66 @@ -CNRI OPEN SOURCE LICENSE AGREEMENT FOR Durus-3.5 - -IMPORTANT: PLEASE READ THE FOLLOWING AGREEMENT CAREFULLY. BY COPYING, -INSTALLING OR OTHERWISE USING Durus-3.5 SOFTWARE, YOU ARE DEEMED TO -HAVE AGREED TO BE BOUND BY THE TERMS AND CONDITIONS OF THIS LICENSE -AGREEMENT. - -1. This LICENSE AGREEMENT is between Corporation for National Research -Initiatives, having an office at 1895 Preston White Drive, Reston, VA -20191 ("CNRI"), and the Individual or Organization ("Licensee") -copying, installing or otherwise using Durus-3.5 software in source -or binary form and its associated documentation ("Durus-3.5"). - -2. Subject to the terms and conditions of this License Agreement, CNRI -hereby grants Licensee a nonexclusive, royalty-free, world-wide -license to reproduce, analyze, test, perform and/or display publicly, -prepare derivative works, distribute, and otherwise use Durus-3.5 -alone or in any derivative version, provided, however, that CNRI's -License Agreement and CNRI's notice of copyright, i.e., "Copyright © -2005 Corporation for National Research Initiatives; All Rights -Reserved" are retained in Durus-3.5 alone or in any derivative -version prepared by Licensee. - -3. In the event Licensee prepares a derivative work that is based on -or incorporates Durus-3.5, or any part thereof, and wants to make -the derivative work available to others as provided herein, then -Licensee hereby agrees to include in any such work a brief summary of -the changes made to Durus-3.5. - -4. CNRI is making Durus-3.5 available to Licensee on an "AS IS" -basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR -IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND -DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS -FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF Durus-3.5 WILL NOT -INFRINGE ANY THIRD PARTY RIGHTS. - -5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF -Durus-3.5 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR -LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING -Durus-3.5, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE -POSSIBILITY THEREOF. - -6. This License Agreement will automatically terminate upon a material -breach of its terms and conditions. - -7. This License Agreement shall be governed by the federal -intellectual property law of the United States, including without -limitation the federal copyright law, and, to the extent such -U.S. federal law does not apply, by the law of the Commonwealth of -Virginia, excluding Virginia's conflict of law -provisions. Notwithstanding the foregoing, with regard to derivative -works based on Durus-3.5 that incorporate non-separable material -that was previously distributed under the GNU General Public License -(GPL), the law of the Commonwealth of Virginia shall govern this -License Agreement only as to issues arising under or with respect to -Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this -License Agreement shall be deemed to create any relationship of -agency, partnership, or joint venture between CNRI and Licensee. This -License Agreement does not grant permission to use CNRI trademarks or -trade name in a trademark sense to endorse or promote products or -services of Licensee, or any third party. - -8. By copying, installing or otherwise using Durus-3.5, Licensee -agrees to be bound by the terms and conditions of this License -Agreement. - +CNRI OPEN SOURCE LICENSE AGREEMENT FOR Durus-3.5 + +IMPORTANT: PLEASE READ THE FOLLOWING AGREEMENT CAREFULLY. BY COPYING, +INSTALLING OR OTHERWISE USING Durus-3.5 SOFTWARE, YOU ARE DEEMED TO +HAVE AGREED TO BE BOUND BY THE TERMS AND CONDITIONS OF THIS LICENSE +AGREEMENT. + +1. This LICENSE AGREEMENT is between Corporation for National Research +Initiatives, having an office at 1895 Preston White Drive, Reston, VA +20191 ("CNRI"), and the Individual or Organization ("Licensee") +copying, installing or otherwise using Durus-3.5 software in source +or binary form and its associated documentation ("Durus-3.5"). + +2. Subject to the terms and conditions of this License Agreement, CNRI +hereby grants Licensee a nonexclusive, royalty-free, world-wide +license to reproduce, analyze, test, perform and/or display publicly, +prepare derivative works, distribute, and otherwise use Durus-3.5 +alone or in any derivative version, provided, however, that CNRI's +License Agreement and CNRI's notice of copyright, i.e., "Copyright © +2005 Corporation for National Research Initiatives; All Rights +Reserved" are retained in Durus-3.5 alone or in any derivative +version prepared by Licensee. + +3. In the event Licensee prepares a derivative work that is based on +or incorporates Durus-3.5, or any part thereof, and wants to make +the derivative work available to others as provided herein, then +Licensee hereby agrees to include in any such work a brief summary of +the changes made to Durus-3.5. + +4. CNRI is making Durus-3.5 available to Licensee on an "AS IS" +basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR +IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND +DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS +FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF Durus-3.5 WILL NOT +INFRINGE ANY THIRD PARTY RIGHTS. + +5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF +Durus-3.5 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR +LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING +Durus-3.5, OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE +POSSIBILITY THEREOF. + +6. This License Agreement will automatically terminate upon a material +breach of its terms and conditions. + +7. This License Agreement shall be governed by the federal +intellectual property law of the United States, including without +limitation the federal copyright law, and, to the extent such +U.S. federal law does not apply, by the law of the Commonwealth of +Virginia, excluding Virginia's conflict of law +provisions. Notwithstanding the foregoing, with regard to derivative +works based on Durus-3.5 that incorporate non-separable material +that was previously distributed under the GNU General Public License +(GPL), the law of the Commonwealth of Virginia shall govern this +License Agreement only as to issues arising under or with respect to +Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this +License Agreement shall be deemed to create any relationship of +agency, partnership, or joint venture between CNRI and Licensee. This +License Agreement does not grant permission to use CNRI trademarks or +trade name in a trademark sense to endorse or promote products or +services of Licensee, or any third party. + +8. By copying, installing or otherwise using Durus-3.5, Licensee +agrees to be bound by the terms and conditions of this License +Agreement. + schevo/store/LICENSE.txt @@ -1,184 +1,184 @@ -This is Durus, a persistent object system for applications written -in the Python programming language. - -Durus was written by the MEMS Exchange software development team at -the Corporation for National Research Initiatives (CNRI). Durus is -designed to be the storage component for the Python-powered web sites -operated by the MEMS Exchange, and it provides the features that we -need for this purpose, and no more. If you are looking for a -language-independent, long term persistent storage system for digital -objects of all kinds, designed to provide secure, managed access to -extensible server-side information services, you should read more -about CNRI's Digital Object Store® at -http://www.cnri.reston.va.us/digital_object_store.html. - - -* Overview: - -Durus offers an easy way to use and maintain a consistent collection -of object instances used by one or more processes. Access and change -of a persistent instances is managed through a cached Connection -instance which includes commit() and abort() methods so that changes -are transactional. Durus is best suited to collections of less than a -million instances with relatively stable state. - - -* Quick Demo: - -Run "durus -s" in one window. This starts a durus storage server -using a temporary file and listening for clients on localhost port -2972. Run "durus -c" in another window. This connects to the storage -server on the port 2972 on the localhost. When you start, you have -access to only one persistent object, "root". If you make changes to -attributes of root and run "connection.commit()", the changes are -written to the (temporary) file. If you make changes to attributes of -root, and then run "connection.abort()", the attributes revert back to -the values they had at the last commit. - -Run *another* "durus -c" in a third window, and you can see how -committed changes to root in the first client are available in -the second client when it starts. Subsequent changes committed in -any client are visible in any other client that synchronizes by calling -either "connection.abort()" or "connection.commit()". - -To stop the running durus server, run "durus -s --stop". - -This demonstrates transactional behavior, but not persistence, since -the temporary file is removed as soon as the durus server is stopped. - -To see how persistence works, do the same thing, except add -"--file test.durus" to the command that starts the server. Make -changes to attributes of root, run "connection.commit()", and -"durus -s --stop", and the changes to root will be stored in -test.durus, so that you"ll see the changes again if you restart again -with the "--file test.durus" option. - -Finally, note that you can run "durus -c --file test.durus" (after -stopping the durus server) to use the file storage directly and -exclusively. Everything works the same way as before, except that no -server is involved. - -Both the "durus -s" and "durus -c" commands accept "--help" command -line options that explain more about their usage. - - -* Using Durus in a Program: - -To use Durus, a Python program needs to make a Storage instance and a -Connection instance. For the Storage instance, you have two choices: -FileStorage or ClientStorage. If your program is to be one of several -processes accessing a shared collection of objects, then you want -ClientStorage. If your program has no competition, then choose -FileStorage. There is only one Connection class, and the constructor -takes a storage instance as an argument. - -Example using FileStorage to open a Connection to a file: - - from durus.file_storage import FileStorage - from durus.connection import Connection - connection = Connection(FileStorage("test.durus")) - -Example using ClientStorage to open a Connection to a Durus server: - - from durus.client_storage import ClientStorage - from durus.connection import Connection - connection = Connection(ClientStorage()) - -Note that the ClientStorage constructor supports the "address" keyword -that you can use to specify the address to use. The value must be either -a (host, port) tuple or a string giving a path to use for a unix domain -socket. If you provide the address you should be sure to start the -storage server the same way. The "durus" command line tool supports -options to specify the address. - -The connection instance has a get_root() method that you can use to -obtain the root object. - -In your program, you can make changes to the root object attributes, -and call connection.commit() or connection.abort() to lock in or -revert changes made since the last commit. The root object is -actually an instance of durus.persistent_dict.PersistentDict, which -means that it can be used like a regular dict, except that changes -will be managed by the Connection. There is a similar class, -durus.persistent_list.PersistentList that provides list-like behavior, -except managed by the Connection. - -PersistentList and PersistentDict both inherit from -durus.persistent.Persistent, and this is the key to making your own -classes participate in the Durus persistence system. Just add -Persistent class A"s list of bases, and your instances will know how -to manage changes to their attributes through a Connection. To -actually store an instance x of A in the storage, though, you need to -commit a reference to x in some object that is already stored in the -database. The root object is always there, for example, so you can do -something like this: - - # Assume mymodule defines A as a subclass of Persistent. - from mymodule import A - x = A() - root = connection.get_root() # connection set as shown above. - root["sample"] = x # root is dict-like - connection.commit() # Now x is stored. - -Subsequent changes to x, or to new A instances put on attributes of X, -and so on, will all be managed by the Connection just as for the root -object. This management of the Persistent instance continues as long -as the instance is in the storage. Sometimes, though, we wish to -remove Persistent instances from the storage so that the file can be -smaller. This is done through an occasional call the Connection"s -pack() method. Packing a storage removes all stored data except what -is required to store the state of Persistent instances that are -reachable from the root object. To remove an instance, therefore, you -need to remove it from all instances that are reachable from the root -object, and then you need to call the Connection's pack() method. - - -* Non-Persistent Containers. - -When you change an attribute of a Persistent instance, the fact that -the instance has been changed is noted with the Connection, so that -the Connection knows what instances need to be stored on the next -commit(). The same change-tracking occurs automatically when you make -dict-like changes to PersistentDict instances or list-like changes to -PersistentList instances. If, however, you make changes to a -non-persistent container, even if it is the value of an attribute of a -Persistent instance, the changes are *not* automatically noted with -the Connection. To make sure that your changes do get saved, you must -call the _p_note_change() method of the Persistent instance that -refers to the changed non-persistent container. You can see an -example of this by looking at the source code of PersistentDict and -PersistentList, both of which maintain a non-persistent container on a -"data" attribute, shadow the methods of the underlying container, and -add calls to self._p_note_change() in every method that makes changes. - - -* Computed Attributes: - -Durus includes (in durus.persistent) ComputedAttribute, a subclass of -Persistent that allows computed values to be cached (for speed) in -such a way that the cache can be flushed by changes committed in other -Connections. This invalidation of instances changed in another -Connection works the way it does for other Persistent instances. The -difference is that changes to ComputedAttributes are noted, but not -stored. - -ComputedAttribute objects have two useful methods: get() and -invalidate(). The get() method requires as an argument a function that -takes no arguments. If there is a value cached then it is returned -otherwise the function is called and it's return value is both cached -and returned. The invalidate() method must be called when the cached -value is no longer valid. The cached value will be discarded in all -connections. - - -* Copyright: - -Copyright (c) Corporation for National Research Initiatives 2006. All -Rights Reserved. - - -* Source of Support: - -This work was supported by DARPA/MTO under Contract MDA972-03-1-0022. - - +This is Durus, a persistent object system for applications written +in the Python programming language. + +Durus was written by the MEMS Exchange software development team at +the Corporation for National Research Initiatives (CNRI). Durus is +designed to be the storage component for the Python-powered web sites +operated by the MEMS Exchange, and it provides the features that we +need for this purpose, and no more. If you are looking for a +language-independent, long term persistent storage system for digital +objects of all kinds, designed to provide secure, managed access to +extensible server-side information services, you should read more +about CNRI's Digital Object Store® at +http://www.cnri.reston.va.us/digital_object_store.html. + + +* Overview: + +Durus offers an easy way to use and maintain a consistent collection +of object instances used by one or more processes. Access and change +of a persistent instances is managed through a cached Connection +instance which includes commit() and abort() methods so that changes +are transactional. Durus is best suited to collections of less than a +million instances with relatively stable state. + + +* Quick Demo: + +Run "durus -s" in one window. This starts a durus storage server +using a temporary file and listening for clients on localhost port +2972. Run "durus -c" in another window. This connects to the storage +server on the port 2972 on the localhost. When you start, you have +access to only one persistent object, "root". If you make changes to +attributes of root and run "connection.commit()", the changes are +written to the (temporary) file. If you make changes to attributes of +root, and then run "connection.abort()", the attributes revert back to +the values they had at the last commit. + +Run *another* "durus -c" in a third window, and you can see how +committed changes to root in the first client are available in +the second client when it starts. Subsequent changes committed in +any client are visible in any other client that synchronizes by calling +either "connection.abort()" or "connection.commit()". + +To stop the running durus server, run "durus -s --stop". + +This demonstrates transactional behavior, but not persistence, since +the temporary file is removed as soon as the durus server is stopped. + +To see how persistence works, do the same thing, except add +"--file test.durus" to the command that starts the server. Make +changes to attributes of root, run "connection.commit()", and +"durus -s --stop", and the changes to root will be stored in +test.durus, so that you"ll see the changes again if you restart again +with the "--file test.durus" option. + +Finally, note that you can run "durus -c --file test.durus" (after +stopping the durus server) to use the file storage directly and +exclusively. Everything works the same way as before, except that no +server is involved. + +Both the "durus -s" and "durus -c" commands accept "--help" command +line options that explain more about their usage. + + +* Using Durus in a Program: + +To use Durus, a Python program needs to make a Storage instance and a +Connection instance. For the Storage instance, you have two choices: +FileStorage or ClientStorage. If your program is to be one of several +processes accessing a shared collection of objects, then you want +ClientStorage. If your program has no competition, then choose +FileStorage. There is only one Connection class, and the constructor +takes a storage instance as an argument. + +Example using FileStorage to open a Connection to a file: + + from durus.file_storage import FileStorage + from durus.connection import Connection + connection = Connection(FileStorage("test.durus")) + +Example using ClientStorage to open a Connection to a Durus server: + + from durus.client_storage import ClientStorage + from durus.connection import Connection + connection = Connection(ClientStorage()) + +Note that the ClientStorage constructor supports the "address" keyword +that you can use to specify the address to use. The value must be either +a (host, port) tuple or a string giving a path to use for a unix domain +socket. If you provide the address you should be sure to start the +storage server the same way. The "durus" command line tool supports +options to specify the address. + +The connection instance has a get_root() method that you can use to +obtain the root object. + +In your program, you can make changes to the root object attributes, +and call connection.commit() or connection.abort() to lock in or +revert changes made since the last commit. The root object is +actually an instance of durus.persistent_dict.PersistentDict, which +means that it can be used like a regular dict, except that changes +will be managed by the Connection. There is a similar class, +durus.persistent_list.PersistentList that provides list-like behavior, +except managed by the Connection. + +PersistentList and PersistentDict both inherit from +durus.persistent.Persistent, and this is the key to making your own +classes participate in the Durus persistence system. Just add +Persistent class A"s list of bases, and your instances will know how +to manage changes to their attributes through a Connection. To +actually store an instance x of A in the storage, though, you need to +commit a reference to x in some object that is already stored in the +database. The root object is always there, for example, so you can do +something like this: + + # Assume mymodule defines A as a subclass of Persistent. + from mymodule import A + x = A() + root = connection.get_root() # connection set as shown above. + root["sample"] = x # root is dict-like + connection.commit() # Now x is stored. + +Subsequent changes to x, or to new A instances put on attributes of X, +and so on, will all be managed by the Connection just as for the root +object. This management of the Persistent instance continues as long +as the instance is in the storage. Sometimes, though, we wish to +remove Persistent instances from the storage so that the file can be +smaller. This is done through an occasional call the Connection"s +pack() method. Packing a storage removes all stored data except what +is required to store the state of Persistent instances that are +reachable from the root object. To remove an instance, therefore, you +need to remove it from all instances that are reachable from the root +object, and then you need to call the Connection's pack() method. + + +* Non-Persistent Containers. + +When you change an attribute of a Persistent instance, the fact that +the instance has been changed is noted with the Connection, so that +the Connection knows what instances need to be stored on the next +commit(). The same change-tracking occurs automatically when you make +dict-like changes to PersistentDict instances or list-like changes to +PersistentList instances. If, however, you make changes to a +non-persistent container, even if it is the value of an attribute of a +Persistent instance, the changes are *not* automatically noted with +the Connection. To make sure that your changes do get saved, you must +call the _p_note_change() method of the Persistent instance that +refers to the changed non-persistent container. You can see an +example of this by looking at the source code of PersistentDict and +PersistentList, both of which maintain a non-persistent container on a +"data" attribute, shadow the methods of the underlying container, and +add calls to self._p_note_change() in every method that makes changes. + + +* Computed Attributes: + +Durus includes (in durus.persistent) ComputedAttribute, a subclass of +Persistent that allows computed values to be cached (for speed) in +such a way that the cache can be flushed by changes committed in other +Connections. This invalidation of instances changed in another +Connection works the way it does for other Persistent instances. The +difference is that changes to ComputedAttributes are noted, but not +stored. + +ComputedAttribute objects have two useful methods: get() and +invalidate(). The get() method requires as an argument a function that +takes no arguments. If there is a value cached then it is returned +otherwise the function is called and it's return value is both cached +and returned. The invalidate() method must be called when the cached +value is no longer valid. The cached value will be discarded in all +connections. + + +* Copyright: + +Copyright (c) Corporation for National Research Initiatives 2006. All +Rights Reserved. + + +* Source of Support: + +This work was supported by DARPA/MTO under Contract MDA972-03-1-0022. + + schevo/store/README.txt @@ -420,7 +420,7 @@ class BTree(Persistent): return self.root.get_count() def items_backward(self): - """() -> generator + """() -> generator Generate all items in reverse order. """ for item in reversed(self.root): schevo/store/btree.py @@ -82,7 +82,7 @@ class Connection(ConnectionBase): def set_cache_size(self, size): """(size:int) - Set the target size for the cache. + Set the target size for the cache. """ self.cache.set_size(size) schevo/store/connection.py @@ -134,7 +134,7 @@ A: This is basically the same as changing a class name. A useful trick is to assign to ``sys.modules`` directly. For example, in your update DB script you could do something like:: - + import newmodule sys.modules["oldmodule"] = newmodule schevo/store/doc/FAQ.txt @@ -28,7 +28,7 @@ class InvalidObjectReference(DurusError): is the object for which the reference is invalid. connection: Connection the connection that attempted to store it. - + obj._p_connection != connection """ schevo/store/error.py @@ -408,7 +408,7 @@ class FileStorage2(FileStorage): 1) the number of bytes in rest of the record (u64) 2) a zlib compressed pickle of a dictionary. The dictionary maps - oids to file offsets for all transactions that preceed the + oids to file offsets for all transactions that preceed the index in the file. A transaction record consists of: schevo/store/file_storage.py @@ -1,143 +1,143 @@ -""" -$URL: svn+ssh://svn/repos/trunk/durus/history.py $ -$Id: history.py 28338 2006-05-04 21:45:03Z dbinger $ -""" -from schevo.store.file_storage import FileStorage2 -from schevo.store.connection import Connection -from schevo.store.persistent import Persistent - -class _HistoryIndex (dict): - """ - A substitute offset index used in HistoryFileStorage. - This keeps a history of updates and provides methods for - moving backward and forward in that history. - - """ - def __init__(self, other_dict): - dict.__init__(self) - self.history = [] - self.future = [] - self.update(other_dict) - - def update(self, arg): - self.history.append(arg.copy()) - dict.update(self, arg) - - def previous_transaction(self): - if not self.history: - return None - transaction_offsets = self.history.pop() - self.future.append(transaction_offsets) - for oid in transaction_offsets: - del self[oid] - for offsets in reversed(self.history): - if oid in offsets: - self[oid] = offsets[oid] - break - return transaction_offsets.copy() - - def next_transaction(self): - if not self.future: - return None - transaction_offsets = self.future.pop() - self.history.append(transaction_offsets) - dict.update(self, transaction_offsets) - return transaction_offsets.copy() - - -class HistoryFileStorage (FileStorage2): - """ - This variant of storage allows stepping forward and backward - among the transaction records. - """ - def __init__(self, filename=None, readonly=True, repair=False): - assert readonly and not repair - FileStorage2.__init__(self, - filename=filename, readonly=True, repair=False) - self.invalid = {} - - def _set_concrete_class_for_magic(self): - pass - - def get_history_index(self): - return self.history_index - - def set_history_index(self, value): - self.history_index = _HistoryIndex(value) - - index = property(get_history_index, set_history_index) - - def previous(self): - invalidations = self.get_history_index().previous_transaction() - if invalidations is None: - return False - else: - self.invalid.update(invalidations) - return True - - def next(self): - invalidations = self.get_history_index().next_transaction() - if invalidations is None: - return False - else: - self.invalid.update(invalidations) - return True - - def sync(self): - result = self.invalid.keys() - self.invalid.clear() - return result - - -class HistoryConnection (Connection): - """ - A Connection that provides (read-only) access to a FileStorage with - the ability reverse and advance transactions. - """ - def __init__(self, filename): - Connection.__init__(self, HistoryFileStorage(filename)) - - def previous(self): - """() -> bool - Move to the previous transaction. - Returns False when there is no previous transaction. - """ - result = self.get_storage().previous() - self.abort() - return result - - def next(self): - """() -> bool - Move to the next transaction. - Returns False when there is no next transaction. - """ - result = self.get_storage().next() - self.abort() - return result - - def previous_instance(self, obj): - """ - Reverse transactions until there is some change in obj. - When you reach a transaction in which obj no longer exists, - the obj will be a ghost, but it will no longer have a __dict__ - attribute. - """ - assert isinstance(obj, Persistent) - while True: - self.previous() - if obj._p_is_ghost(): - return obj - if not hasattr(obj, '__dict__'): - return obj - - def next_instance(self, obj): - """ - Advance transactions until there is some change in obj. - If obj is not a ghost after calling this, it means we are - at the current version. - """ - assert isinstance(obj, Persistent) - while self.next(): - if obj._p_is_ghost(): - break - return obj +""" +$URL: svn+ssh://svn/repos/trunk/durus/history.py $ +$Id: history.py 28338 2006-05-04 21:45:03Z dbinger $ +""" +from schevo.store.file_storage import FileStorage2 +from schevo.store.connection import Connection +from schevo.store.persistent import Persistent + +class _HistoryIndex (dict): + """ + A substitute offset index used in HistoryFileStorage. + This keeps a history of updates and provides methods for + moving backward and forward in that history. + + """ + def __init__(self, other_dict): + dict.__init__(self) + self.history = [] + self.future = [] + self.update(other_dict) + + def update(self, arg): + self.history.append(arg.copy()) + dict.update(self, arg) + + def previous_transaction(self): + if not self.history: + return None + transaction_offsets = self.history.pop() + self.future.append(transaction_offsets) + for oid in transaction_offsets: + del self[oid] + for offsets in reversed(self.history): + if oid in offsets: + self[oid] = offsets[oid] + break + return transaction_offsets.copy() + + def next_transaction(self): + if not self.future: + return None + transaction_offsets = self.future.pop() + self.history.append(transaction_offsets) + dict.update(self, transaction_offsets) + return transaction_offsets.copy() + + +class HistoryFileStorage (FileStorage2): + """ + This variant of storage allows stepping forward and backward + among the transaction records. + """ + def __init__(self, filename=None, readonly=True, repair=False): + assert readonly and not repair + FileStorage2.__init__(self, + filename=filename, readonly=True, repair=False) + self.invalid = {} + + def _set_concrete_class_for_magic(self): + pass + + def get_history_index(self): + return self.history_index + + def set_history_index(self, value): + self.history_index = _HistoryIndex(value) + + index = property(get_history_index, set_history_index) + + def previous(self): + invalidations = self.get_history_index().previous_transaction() + if invalidations is None: + return False + else: + self.invalid.update(invalidations) + return True + + def next(self): + invalidations = self.get_history_index().next_transaction() + if invalidations is None: + return False + else: + self.invalid.update(invalidations) + return True + + def sync(self): + result = self.invalid.keys() + self.invalid.clear() + return result + + +class HistoryConnection (Connection): + """ + A Connection that provides (read-only) access to a FileStorage with + the ability reverse and advance transactions. + """ + def __init__(self, filename): + Connection.__init__(self, HistoryFileStorage(filename)) + + def previous(self): + """() -> bool + Move to the previous transaction. + Returns False when there is no previous transaction. + """ + result = self.get_storage().previous() + self.abort() + return result + + def next(self): + """() -> bool + Move to the next transaction. + Returns False when there is no next transaction. + """ + result = self.get_storage().next() + self.abort() + return result + + def previous_instance(self, obj): + """ + Reverse transactions until there is some change in obj. + When you reach a transaction in which obj no longer exists, + the obj will be a ghost, but it will no longer have a __dict__ + attribute. + """ + assert isinstance(obj, Persistent) + while True: + self.previous() + if obj._p_is_ghost(): + return obj + if not hasattr(obj, '__dict__'): + return obj + + def next_instance(self, obj): + """ + Advance transactions until there is some change in obj. + If obj is not a ghost after calling this, it means we are + at the current version. + """ + assert isinstance(obj, Persistent) + while self.next(): + if obj._p_is_ghost(): + break + return obj schevo/store/history.py @@ -25,7 +25,7 @@ class Storage(object): def begin(self): """ - Begin a commit. + Begin a commit. """ raise NotImplementedError schevo/store/storage.py @@ -1,390 +1,390 @@ -""" -$URL: svn+ssh://svn/repos/trunk/durus/storage_server.py $ -$Id: storage_server.py 28319 2006-05-02 19:01:44Z rmasse $ -""" -from datetime import datetime -from schevo.store.logger import log, is_logging -from schevo.store.serialize import extract_class_name, split_oids -from schevo.store.utils import p32, u32, u64 -from os.path import exists -from time import sleep -import errno -import select -import socket - - -STATUS_OKAY = 'O' -STATUS_KEYERROR = 'K' -STATUS_INVALID = 'I' - -TIMEOUT = 10 -DEFAULT_HOST = '127.0.0.1' -DEFAULT_PORT = 2972 - - -def recv(s, n): - """(s:socket, n:int) -> str - Call the recv() method on the socket, repeating as required until n bytes - are received. - """ - data = [] - while n > 0: - hunk = s.recv(min(n, 1000000)) - if not hunk: - raise IOError, 'connection reset by peer' - n -= len(hunk) - data.append(hunk) - return ''.join(data) - -class _Client: - - def __init__(self, s, addr): - self.s = s - self.addr = addr - self.invalid = set() - -class ClientError(Exception): - pass - - -class SocketAddress (object): - - def new(address, **kwargs): - if isinstance(address, SocketAddress): - return address - elif type(address) is tuple: - host, port = address - return HostPortAddress(host=host, port=port) - elif type(address) is str: - return UnixDomainSocketAddress(address, **kwargs) - else: - raise ValueError(address) - new = staticmethod(new) - - def get_listening_socket(self): - sock = socket.socket(self.get_address_family(), socket.SOCK_STREAM) - sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - self.bind_socket(sock) - sock.listen(40) - return sock - -class HostPortAddress (SocketAddress): - - def __init__(self, host=DEFAULT_HOST, port=DEFAULT_PORT): - self.host = host - self.port = port - - def __str__(self): - return "%s:%s" % (self.host, self.port) - - def get_address_family(self): - return socket.AF_INET - - def bind_socket(self, socket): - socket.bind( (self.host, self.port)) - - def get_connected_socket(self): - sock = socket.socket(self.get_address_family(), socket.SOCK_STREAM) - sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) - try: - sock.connect((self.host, self.port)) - except socket.error, exc: - error = exc.args[0] - if error == errno.ECONNREFUSED: - return None - else: - raise - return sock - - def set_connection_options(self, s): - s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) - s.settimeout(TIMEOUT) - - def close(self, s): - s.close() - - -import sys -if sys.platform != 'win32': - - from grp import getgrnam, getgrgid - from os import unlink, stat, chown, geteuid, getegid, umask - from pwd import getpwnam, getpwuid - - class UnixDomainSocketAddress (SocketAddress): - - def __init__(self, filename, owner=None, group=None, umask=None): - self.filename = filename - self.owner = owner - self.group = group - self.umask = umask - - def __str__(self): - result = self.filename - if exists(self.filename): - filestat = stat(self.filename) - uid = filestat.st_uid - gid = filestat.st_gid - rwx = ['---', '--x', '-w-', '-wx', 'r--', 'r-x', 'rw-', 'rwx'] - owner = getpwuid(uid).pw_name - group = getgrgid(gid).gr_name - result += ' (%s%s%s %s %s)' % ( - rwx[filestat.st_mode >> 6 & 7], - rwx[filestat.st_mode >> 3 & 7], - rwx[filestat.st_mode & 7], - owner, - group) - return result - - def get_address_family(self): - return socket.AF_UNIX - - def bind_socket(self, s): - if self.umask is not None: - old_umask = umask(self.umask) - try: - s.bind(self.filename) - except socket.error, exc: - error = exc.args[0] - if not exists(self.filename): - raise - if stat(self.filename).st_size > 0: - raise - if error == errno.EADDRINUSE: - connected = self.get_connected_socket() - if connected: - connected.close() - raise - unlink(self.filename) - s.bind(self.filename) - else: - raise - uid = geteuid() - if self.owner is not None: - if type(self.owner) is int: - uid = self.owner - else: - uid = getpwnam(self.owner).pw_uid - gid = getegid() - if self.group is not None: - if type(self.group) is int: - gid = self.group - else: - gid = getgrnam(self.group).gr_gid - if self.owner is not None or self.group is not None: - chown(self.filename, uid, gid) - if self.umask is not None: - umask(old_umask) - - def get_connected_socket(self): - sock = socket.socket(self.get_address_family(), socket.SOCK_STREAM) - try: - sock.connect(self.filename) - except socket.error, exc: - error = exc.args[0] - if error in (errno.ENOENT, errno.ENOTSOCK, errno.ECONNREFUSED): - return None - else: - raise - return sock - - def set_connection_options(self, s): - s.settimeout(TIMEOUT) - - def close(self, s): - s.close() - if exists(self.filename): - unlink(self.filename) - - -class StorageServer: - - protocol = p32(1) - - def __init__(self, storage, host=DEFAULT_HOST, - port=DEFAULT_PORT, address=None): - self.storage = storage - self.clients = [] - self.sockets = [] - self.packer = None - self.address = SocketAddress.new(address or (host, port)) - self.load_record = {} - - def serve(self): - sock = self.address.get_listening_socket() - log(20, 'Ready on %s with %s objects', self.address, - self.storage.get_size()) - self.sockets.append(sock) - try: - while 1: - if self.packer is not None: - timeout = 0.0 - else: - timeout = None - r, w, e = select.select(self.sockets, [], [], timeout) - for s in r: - if s is sock: - # new connection - conn, addr = s.accept() - self.address.set_connection_options(conn) - self.clients.append(_Client(conn, addr)) - self.sockets.append(conn) - else: - # command from client - try: - self.handle(s) - except (ClientError, socket.error, socket.timeout), exc: - log(10, '%s', ''.join(map(str, exc.args))) - self.sockets.remove(s) - self.clients.remove(self._find_client(s)) - if self.packer is not None: - try: - self.packer.next() - except StopIteration: - log(20, 'Pack finished at %s' % datetime.now()) - self.packer = None # done packing - finally: - self.address.close(sock) - - def handle(self, s): - command_code = s.recv(1) - if not command_code: - raise ClientError('EOF from client') - handler = getattr(self, 'handle_%s' % command_code, None) - if handler is None: - raise ClientError('No such command code: %r' % command_code) - handler(s) - - def _find_client(self, s): - for client in self.clients: - if client.s is s: - return client - assert 0 - - def handle_N(self, s): - # new OID - s.sendall(self.storage.new_oid()) - - def handle_M(self, s): - # new OIDs - count = ord(recv(s, 1)) - log(10, "oids: %s", count) - s.sendall(''.join([self.storage.new_oid() for j in xrange(count)])) - - def handle_L(self, s): - # load - oid = recv(s, 8) - self._send_load_response(s, oid) - - def _send_load_response(self, s, oid): - if oid in self._find_client(s).invalid: - s.sendall(STATUS_INVALID) - else: - try: - record = self.storage.load(oid) - except KeyError: - log(10, 'KeyError %s', u64(oid)) - s.sendall(STATUS_KEYERROR) - else: - if is_logging(5): - class_name = extract_class_name(record) - if class_name in self.load_record: - self.load_record[class_name] += 1 - else: - self.load_record[class_name] = 1 - log(4, 'Load %-7s %s', u64(oid), class_name) - s.sendall(STATUS_OKAY + p32(len(record)) + record) - - def handle_C(self, s): - # commit - client = self._find_client(s) - s.sendall(p32(len(client.invalid)) + ''.join(client.invalid)) - client.invalid.clear() - tlen = u32(recv(s, 4)) - if tlen == 0: - return # client decided not to commit (e.g. conflict) - tdata = recv(s, tlen) - logging_debug = is_logging(10) - logging_debug and log(10, 'Committing %s bytes', tlen) - self.storage.begin() - i = 0 - oids = [] - while i < len(tdata): - rlen = u32(tdata[i:i+4]) - i += 4 - oid = tdata[i:i+8] - record = tdata[i+8:i+rlen] - i += rlen - if logging_debug: - class_name = extract_class_name(record) - log(10, ' oid=%-6s rlen=%-6s %s', u64(oid), rlen, class_name) - self.storage.store(oid, record) - oids.append(oid) - assert i == len(tdata) - self.storage.end() - self._report_load_record() - log(20, 'Committed %3s objects %s bytes at %s', - len(oids), tlen, datetime.now()) - s.sendall(STATUS_OKAY) - for c in self.clients: - if c is not client: - c.invalid.update(oids) - - def _report_load_record(self): - if self.load_record and is_logging(5): - log(5, '\n'.join( - "%8s: %s" % (item[1], item[0]) - for item in sorted(self.load_record.items()))) - self.load_record.clear() - - def handle_S(self, s): - # sync - client = self._find_client(s) - self._report_load_record() - log(8, 'Sync %s', len(client.invalid)) - invalid = self.storage.sync() - assert not invalid # should have exclusive access - s.sendall(p32(len(client.invalid)) + ''.join(client.invalid)) - client.invalid.clear() - - def handle_P(self, s): - # pack - log(20, 'Pack started at %s' % datetime.now()) - if self.packer is None: - self.packer = self.storage.get_packer() - s.sendall(STATUS_OKAY) - - def handle_B(self, s): - # bulk read of objects - number_of_oids = u32(recv(s, 4)) - oid_str = recv(s, 8 * number_of_oids) - oids = split_oids(oid_str) - for oid in oids: - self._send_load_response(s, oid) - - def handle_Q(self, s): - # graceful quit - log(20, 'Quit') - raise SystemExit - - def handle_V(self, s): - # Verify protocol version match. - client_protocol = recv(s, 4) - log(10, 'Client Protocol: %s', u32(client_protocol)) - assert len(self.protocol) == 4 - s.sendall(self.protocol) - if client_protocol != self.protocol: - raise ClientError("Protocol not supported.") - -def wait_for_server(host=DEFAULT_HOST, port=DEFAULT_PORT, maxtries=30, - sleeptime=2, address=None): - # Wait for the server to bind to the port. - server_address = SocketAddress.new(address or (host, port)) - for attempt in range(maxtries): - connected = server_address.get_connected_socket() - if connected: - connected.close() - break - sleep(sleeptime) - else: - raise SystemExit('Timeout waiting for address.') +""" +$URL: svn+ssh://svn/repos/trunk/durus/storage_server.py $ +$Id: storage_server.py 28319 2006-05-02 19:01:44Z rmasse $ +""" +from datetime import datetime +from schevo.store.logger import log, is_logging +from schevo.store.serialize import extract_class_name, split_oids +from schevo.store.utils import p32, u32, u64 +from os.path import exists +from time import sleep +import errno +import select +import socket + + +STATUS_OKAY = 'O' +STATUS_KEYERROR = 'K' +STATUS_INVALID = 'I' + +TIMEOUT = 10 +DEFAULT_HOST = '127.0.0.1' +DEFAULT_PORT = 2972 + + +def recv(s, n): + """(s:socket, n:int) -> str + Call the recv() method on the socket, repeating as required until n bytes + are received. + """ + data = [] + while n > 0: + hunk = s.recv(min(n, 1000000)) + if not hunk: + raise IOError, 'connection reset by peer' + n -= len(hunk) + data.append(hunk) + return ''.join(data) + +class _Client: + + def __init__(self, s, addr): + self.s = s + self.addr = addr + self.invalid = set() + +class ClientError(Exception): + pass + + +class SocketAddress (object): + + def new(address, **kwargs): + if isinstance(address, SocketAddress): + return address + elif type(address) is tuple: + host, port = address + return HostPortAddress(host=host, port=port) + elif type(address) is str: + return UnixDomainSocketAddress(address, **kwargs) + else: + raise ValueError(address) + new = staticmethod(new) + + def get_listening_socket(self): + sock = socket.socket(self.get_address_family(), socket.SOCK_STREAM) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.bind_socket(sock) + sock.listen(40) + return sock + +class HostPortAddress (SocketAddress): + + def __init__(self, host=DEFAULT_HOST, port=DEFAULT_PORT): + self.host = host + self.port = port + + def __str__(self): + return "%s:%s" % (self.host, self.port) + + def get_address_family(self): + return socket.AF_INET + + def bind_socket(self, socket): + socket.bind( (self.host, self.port)) + + def get_connected_socket(self): + sock = socket.socket(self.get_address_family(), socket.SOCK_STREAM) + sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + try: + sock.connect((self.host, self.port)) + except socket.error, exc: + error = exc.args[0] + if error == errno.ECONNREFUSED: + return None + else: + raise + return sock + + def set_connection_options(self, s): + s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + s.settimeout(TIMEOUT) + + def close(self, s): + s.close() + + +import sys +if sys.platform != 'win32': + + from grp import getgrnam, getgrgid + from os import unlink, stat, chown, geteuid, getegid, umask + from pwd import getpwnam, getpwuid + + class UnixDomainSocketAddress (SocketAddress): + + def __init__(self, filename, owner=None, group=None, umask=None): + self.filename = filename + self.owner = owner + self.group = group + self.umask = umask + + def __str__(self): + result = self.filename + if exists(self.filename): + filestat = stat(self.filename) + uid = filestat.st_uid + gid = filestat.st_gid + rwx = ['---', '--x', '-w-', '-wx', 'r--', 'r-x', 'rw-', 'rwx'] + owner = getpwuid(uid).pw_name + group = getgrgid(gid).gr_name + result += ' (%s%s%s %s %s)' % ( + rwx[filestat.st_mode >> 6 & 7], + rwx[filestat.st_mode >> 3 & 7], + rwx[filestat.st_mode & 7], + owner, + group) + return result + + def get_address_family(self): + return socket.AF_UNIX + + def bind_socket(self, s): + if self.umask is not None: + old_umask = umask(self.umask) + try: + s.bind(self.filename) + except socket.error, exc: + error = exc.args[0] + if not exists(self.filename): + raise + if stat(self.filename).st_size > 0: + raise + if error == errno.EADDRINUSE: + connected = self.get_connected_socket() + if connected: + connected.close() + raise + unlink(self.filename) + s.bind(self.filename) + else: + raise + uid = geteuid() + if self.owner is not None: + if type(self.owner) is int: + uid = self.owner + else: + uid = getpwnam(self.owner).pw_uid + gid = getegid() + if self.group is not None: + if type(self.group) is int: + gid = self.group + else: + gid = getgrnam(self.group).gr_gid + if self.owner is not None or self.group is not None: + chown(self.filename, uid, gid) + if self.umask is not None: + umask(old_umask) + + def get_connected_socket(self): + sock = socket.socket(self.get_address_family(), socket.SOCK_STREAM) + try: + sock.connect(self.filename) + except socket.error, exc: + error = exc.args[0] + if error in (errno.ENOENT, errno.ENOTSOCK, errno.ECONNREFUSED): + return None + else: + raise + return sock + + def set_connection_options(self, s): + s.settimeout(TIMEOUT) + + def close(self, s): + s.close() + if exists(self.filename): + unlink(self.filename) + + +class StorageServer: + + protocol = p32(1) + + def __init__(self, storage, host=DEFAULT_HOST, + port=DEFAULT_PORT, address=None): + self.storage = storage + self.clients = [] + self.sockets = [] + self.packer = None + self.address = SocketAddress.new(address or (host, port)) + self.load_record = {} + + def serve(self): + sock = self.address.get_listening_socket() + log(20, 'Ready on %s with %s objects', self.address, + self.storage.get_size()) + self.sockets.append(sock) + try: + while 1: + if self.packer is not None: + timeout = 0.0 + else: + timeout = None + r, w, e = select.select(self.sockets, [], [], timeout) + for s in r: + if s is sock: + # new connection + conn, addr = s.accept() + self.address.set_connection_options(conn) + self.clients.append(_Client(conn, addr)) + self.sockets.append(conn) + else: + # command from client + try: + self.handle(s) + except (ClientError, socket.error, socket.timeout), exc: + log(10, '%s', ''.join(map(str, exc.args))) + self.sockets.remove(s) + self.clients.remove(self._find_client(s)) + if self.packer is not None: + try: + self.packer.next() + except StopIteration: + log(20, 'Pack finished at %s' % datetime.now()) + self.packer = None # done packing + finally: + self.address.close(sock) + + def handle(self, s): + command_code = s.recv(1) + if not command_code: + raise ClientError('EOF from client') + handler = getattr(self, 'handle_%s' % command_code, None) + if handler is None: + raise ClientError('No such command code: %r' % command_code) + handler(s) + + def _find_client(self, s): + for client in self.clients: + if client.s is s: + return client + assert 0 + + def handle_N(self, s): + # new OID + s.sendall(self.storage.new_oid()) + + def handle_M(self, s): + # new OIDs + count = ord(recv(s, 1)) + log(10, "oids: %s", count) + s.sendall(''.join([self.storage.new_oid() for j in xrange(count)])) + + def handle_L(self, s): + # load + oid = recv(s, 8) + self._send_load_response(s, oid) + + def _send_load_response(self, s, oid): + if oid in self._find_client(s).invalid: + s.sendall(STATUS_INVALID) + else: + try: + record = self.storage.load(oid) + except KeyError: + log(10, 'KeyError %s', u64(oid)) + s.sendall(STATUS_KEYERROR) + else: + if is_logging(5): + class_name = extract_class_name(record) + if class_name in self.load_record: + self.load_record[class_name] += 1 + else: + self.load_record[class_name] = 1 + log(4, 'Load %-7s %s', u64(oid), class_name) + s.sendall(STATUS_OKAY + p32(len(record)) + record) + + def handle_C(self, s): + # commit + client = self._find_client(s) + s.sendall(p32(len(client.invalid)) + ''.join(client.invalid)) + client.invalid.clear() + tlen = u32(recv(s, 4)) + if tlen == 0: + return # client decided not to commit (e.g. conflict) + tdata = recv(s, tlen) + logging_debug = is_logging(10) + logging_debug and log(10, 'Committing %s bytes', tlen) + self.storage.begin() + i = 0 + oids = [] + while i < len(tdata): + rlen = u32(tdata[i:i+4]) + i += 4 + oid = tdata[i:i+8] + record = tdata[i+8:i+rlen] + i += rlen + if logging_debug: + class_name = extract_class_name(record) + log(10, ' oid=%-6s rlen=%-6s %s', u64(oid), rlen, class_name) + self.storage.store(oid, record) + oids.append(oid) + assert i == len(tdata) + self.storage.end() + self._report_load_record() + log(20, 'Committed %3s objects %s bytes at %s', + len(oids), tlen, datetime.now()) + s.sendall(STATUS_OKAY) + for c in self.clients: + if c is not client: + c.invalid.update(oids) + + def _report_load_record(self): + if self.load_record and is_logging(5): + log(5, '\n'.join( + "%8s: %s" % (item[1], item[0]) + for item in sorted(self.load_record.items()))) + self.load_record.clear() + + def handle_S(self, s): + # sync + client = self._find_client(s) + self._report_load_record() + log(8, 'Sync %s', len(client.invalid)) + invalid = self.storage.sync() + assert not invalid # should have exclusive access + s.sendall(p32(len(client.invalid)) + ''.join(client.invalid)) + client.invalid.clear() + + def handle_P(self, s): + # pack + log(20, 'Pack started at %s' % datetime.now()) + if self.packer is None: + self.packer = self.storage.get_packer() + s.sendall(STATUS_OKAY) + + def handle_B(self, s): + # bulk read of objects + number_of_oids = u32(recv(s, 4)) + oid_str = recv(s, 8 * number_of_oids) + oids = split_oids(oid_str) + for oid in oids: + self._send_load_response(s, oid) + + def handle_Q(self, s): + # graceful quit + log(20, 'Quit') + raise SystemExit + + def handle_V(self, s): + # Verify protocol version match. + client_protocol = recv(s, 4) + log(10, 'Client Protocol: %s', u32(client_protocol)) + assert len(self.protocol) == 4 + s.sendall(self.protocol) + if client_protocol != self.protocol: + raise ClientError("Protocol not supported.") + +def wait_for_server(host=DEFAULT_HOST, port=DEFAULT_PORT, maxtries=30, + sleeptime=2, address=None): + # Wait for the server to bind to the port. + server_address = SocketAddress.new(address or (host, port)) + for attempt in range(maxtries): + connected = server_address.get_connected_socket() + if connected: + connected.close() + break + sleep(sleeptime) + else: + raise SystemExit('Timeout waiting for address.') schevo/store/storage_server.py @@ -9,6 +9,7 @@ import sys from schevo import database from schevo.lib import module import schevo.schema +import schevo.trace from schevo.script.path import package_path from textwrap import dedent @@ -34,8 +35,24 @@ def raises(exc_type, fn, *args, **kw): raise AssertionError('Call did not raise an exception') +def tron(monitor_level=3): + """Turn trace monitoring on. + + Automatically injected into global namespace of tests when run. + """ + schevo.trace.monitor_level = monitor_level + + +def troff(): + """Turn trace monitoring off. + + Automatically injected into global namespace of tests when run. + """ + schevo.trace.monitor_level = 0 + + _db_cache = { - # (schema_source, suffix): (db, fp, connection), + # (format, schema_source, suffix): (db, fp, connection), } _cached_dbs = set( # db, @@ -68,14 +85,28 @@ class CreatesDatabase(BaseTest): """A mixin to provide test directory and Durus database creation and deletion per-method.""" + format = None + def setUp(self): self.suffixes = set() self.open() + # Also set the global 'tron' and 'troff' variables in our + # class's module namespace for convenience. + modname = self.__class__.__module__ + mod = sys.modules[modname] + mod.tron = tron + mod.troff = troff def tearDown(self): for suffix in list(self.suffixes): self.close(suffix) + def db_contents(self, suffix=''): + value = '' + if hasattr(self, 'fpv' + suffix): + value = getattr(self, 'fpv' + suffix) + return value + def close(self, suffix=''): """Close the database.""" db_name = 'db' + suffix @@ -91,6 +122,16 @@ class CreatesDatabase(BaseTest): delattr(mod, db_name) self.suffixes.remove(suffix) + def convert_format(self, suffix, format): + # Get the contents of the database from the fpv attribute. + contents = self.db_contents(suffix) + # Turn it into a fresh StringIO. + fp = StringIO(contents) + # Convert it to the requested format. + database.convert_format(fp, format) + # Turn it back into a fpv attribute. + setattr(self, 'fpv' + suffix, fp.getvalue()) + def evolve(self, schema_source, version): db = self.reopen() database.evolve(db, schema_source, version) @@ -102,11 +143,10 @@ class CreatesDatabase(BaseTest): May be called more than once in a series of open/close calls. """ # Create database. - value = '' - if hasattr(self, 'fpv' + suffix): - value = getattr(self, 'fpv' + suffix) - fp = StringIO(value) - db = database.open(fp=fp, schema_source=schema_source) + contents = self.db_contents(suffix) + fp = StringIO(contents) + db = database.open( + fp=fp, schema_source=schema_source, format_for_new=self.format) db_name = 'db' + suffix setattr(self, db_name, db) setattr(self, 'fp' + suffix, fp) @@ -131,10 +171,17 @@ class CreatesDatabase(BaseTest): """ return self._open() - def reopen(self, suffix=''): - """Close and reopen self.db and return its new incarnation.""" + def reopen(self, suffix='', format=None): + """Close and reopen self.db and return its new incarnation. + + - `suffix`: The suffix to append to the name of variables; for testing + multiple databases open at once. + - `format`: The format to convert to before reopening. + """ setattr(self, 'fpv' + suffix, getattr(self, 'fp' + suffix).getvalue()) self.close(suffix) + if format is not None: + self.convert_format(suffix, format) db = self._open(suffix) delattr(self, 'fpv' + suffix) return db @@ -154,6 +201,7 @@ class CreatesSchema(CreatesDatabase): _use_db_cache = True def _open(self, suffix=''): + format = self.format if self.body: schema = self.schema = PREAMBLE + dedent(self.body) else: @@ -163,10 +211,10 @@ class CreatesSchema(CreatesDatabase): ex_name = 'ex' + suffix fpv_name = 'fpv' + suffix if (use_db_cache - and (schema, suffix) in _db_cache + and (format, schema, suffix) in _db_cache and not hasattr(self, fpv_name) ): - db, fp, connection = _db_cache[(schema, suffix)] + db, fp, connection = _db_cache[(format, schema, suffix)] setattr(self, 'fp' + suffix, fp) setattr(self, 'connection' + suffix, connection) if not hasattr(self, db_name): @@ -180,7 +228,7 @@ class CreatesSchema(CreatesDatabase): if use_db_cache: fp = getattr(self, 'fp' + suffix) connection = getattr(self, 'connection' + suffix) - _db_cache[(schema, suffix)] = (db, fp, connection) + _db_cache[(format, schema, suffix)] = (db, fp, connection) _cached_dbs.add(db) # Also set the module-level global. modname = self.__class__.__module__ @@ -203,16 +251,17 @@ class EvolvesSchemata(CreatesDatabase): _use_db_cache = True def _open(self, suffix=''): + format = self.format use_db_cache = self._use_db_cache db_name = 'db' + suffix ex_name = 'ex' + suffix fpv_name = 'fpv' + suffix schema = self.schemata[-1] if (use_db_cache - and (schema, suffix) in _db_cache + and (format, schema, suffix) in _db_cache and not hasattr(self, fpv_name) ): - db, fp, connection = _db_cache[(schema, suffix)] + db, fp, connection = _db_cache[(format, schema, suffix)] setattr(self, 'fp' + suffix, fp) setattr(self, 'connection' + suffix, connection) if not hasattr(self, db_name): @@ -231,7 +280,7 @@ class EvolvesSchemata(CreatesDatabase): if use_db_cache: fp = getattr(self, 'fp' + suffix) connection = getattr(self, 'connection' + suffix) - _db_cache[(schema, suffix)] = (db, fp, connection) + _db_cache[(format, schema, suffix)] = (db, fp, connection) _cached_dbs.add(db) # Also set the module-level global. modname = self.__class__.__module__ @@ -264,14 +313,14 @@ class EvolvesSchemata(CreatesDatabase): raise schemata.append(source) return schemata - - + + class DocTest(CreatesSchema): """Doctest-helping test class. Call directly to override body for one test:: - >>> from schevo.test.base import DocTest + >>> from schevo.test import DocTest >>> t = DocTest(''' ... class Foo(E.Entity): ... bar = f.integer() @@ -297,17 +346,18 @@ class DocTest(CreatesSchema): """ body = '' - - def __init__(self, body=None): + + def __init__(self, body=None, format=None): super(DocTest, self).__init__() if body: self.body = body + self.format = format self.setUp() - + def done(self): """Test case is done; free up resources.""" self.tearDown() - + def update(self, body): """Update database with new schema, keeping same schema version.""" self.body = body schevo/test/base.py @@ -52,7 +52,7 @@ class AlphaAlpha(E.Entity): class AlphaBravo(E.Entity): """Has a reference to an AlphaAlpha, such that when that AlphaAlpha is deleted, this AlphaBravo will also be deleted.""" - + alpha_alpha = f.entity('AlphaAlpha', on_delete=CASCADE) @@ -60,7 +60,7 @@ class AlphaCharlie(E.Entity): """Has a reference to an AlphaAlpha, such that when that AlphaAlpha is deleted, the operation will fail because the deletion of this AlphaCharlie is restricted.""" - + alpha_alpha = f.entity('AlphaAlpha', on_delete=RESTRICT) @@ -68,7 +68,7 @@ class AlphaDelta(E.Entity): """Has a reference to an AlphaAlpha, such that when that AlphaAlpha is deleted, this field on this AlphaDelta will be set to UNASSIGNED.""" - + alpha_alpha = f.entity('AlphaAlpha', on_delete=UNASSIGN, required=False) @@ -85,7 +85,7 @@ class AlphaEcho(E.Entity): class Bravo(E.Entity): """Contains one of every class of field possible.""" - hashed_value = f.hashedValue(required=False) + hashed_value = f.hashed_value(required=False) string = f.string(required=False) memo = f.memo(required=False) password = f.password(required=False) @@ -136,7 +136,7 @@ class DeltaBravo(E.Entity): class DeltaCharlie(E.Entity): """An extent that has a custom query.""" - hashed_value = f.hashedValue() + hashed_value = f.hashed_value() @extentmethod def q_hashes(extent, **kw): @@ -159,7 +159,7 @@ class DeltaCharlie(E.Entity): class EchoAlpha(E.Entity): """A plain extent that has a default view like any other.""" - + unicode = f.unicode(required=False) integer = f.integer(required=False) float = f.float(required=False) @@ -317,7 +317,7 @@ class Hotel(E.Entity): class Avatar(E.Entity): - + realm = f.entity('Realm') user = f.entity('User') name = f.unicode() @@ -349,7 +349,7 @@ class Batch_Job(E.Entity): class Realm(E.Entity): - + name = f.unicode() _key(name) @@ -361,10 +361,10 @@ class Realm(E.Entity): class User(E.Entity): - + name = f.unicode() age = f.integer(required=False) - + _key(name) _index(age) schevo/test/schema/schema_001.py @@ -54,7 +54,7 @@ def print_history(max_level): for m in messages: print >>TRACE_TO, m, print >>TRACE_TO - + @do_not_optimize def log(level, *messages): schevo/trace.py @@ -10,7 +10,7 @@ from schevo import base from schevo.change import summarize from schevo.constant import CASCADE, DEFAULT, UNASSIGN, UNASSIGNED from schevo.error import (DatabaseMismatch, DeleteRestricted, - TransactionExpired, TransactionFieldsNotChanged, + TransactionExpired, TransactionFieldsNotChanged, TransactionNotExecuted) from schevo import field from schevo.field import not_fget @@ -19,6 +19,7 @@ from schevo.label import label from schevo.meta import schema_metaclass import schevo.namespace from schevo.namespace import NamespaceExtension +from schevo.trace import log class Transaction(base.Transaction): @@ -87,7 +88,7 @@ class Transaction(base.Transaction): # The default implementation is to return the inverse of this # transaction. return Inverse(self) - + def _update_all_fields(self, name, value): """Update the attribute `name` to `value` on all fields.""" for field in self._field_map.values(): @@ -198,11 +199,14 @@ class Create(Transaction): pass def _execute(self, db): - # Attempt to resolve entity fields to the current database. + field_map = self._field_map + # If any fields contain values that are entities not in `db`, + # attempt to find each equivalent entity in `db` based on the + # information contained in the foreign entity. msg = '"%s" field of "%s" cannot be resolved to the current database' - for field_name, field in self._field_map.iteritems(): + for field_name, field in field_map.iteritems(): entity = field._value - if isinstance(entity, base.Entity) and entity.sys.db is not db: + if isinstance(entity, base.Entity) and entity._db is not db: resolved = False if entity._default_key is not None: extent_name = entity.sys.extent.name @@ -219,20 +223,25 @@ class Create(Transaction): # Before execute callback. self._before_execute(db) # Validate individual fields. - for field in self._field_map.itervalues(): + for field in field_map.itervalues(): if field.fget is None: field.validate(field._value) - style = self._style - extent_name = self._extent_name - field_value_map = self._field_map.value_map() # Strip out unwanted fields. + field_dump_map = field_map.dump_map() + field_related_entity_map = field_map.related_entity_map() fget_fields = self._fget_fields field_spec = self._EntityClass._field_spec - for name in field_value_map.keys(): + for name in field_dump_map.keys(): if name in fget_fields or name not in field_spec: - del field_value_map[name] + del field_dump_map[name] + if name in field_related_entity_map: + del field_related_entity_map[name] + # Proceed with execution based on the create style requested. + extent_name = self._extent_name + style = self._style if style == _Create_Standard: - oid = db._create_entity(extent_name, field_value_map) + oid = db._create_entity( + extent_name, field_dump_map, field_related_entity_map) else: oid = None extent = db.extent(extent_name) @@ -241,6 +250,7 @@ class Create(Transaction): msg = '%s does not have a default key.' % (extent_name,) raise RuntimeError(msg) criteria = {} + field_value_map = field_map.value_map() for name in default_key: criteria[name] = field_value_map[name] entity = extent.findone(**criteria) @@ -252,7 +262,8 @@ class Create(Transaction): else: raise RuntimeError('_style is not set correctly.') else: - oid = db._create_entity(extent_name, field_value_map) + oid = db._create_entity( + extent_name, field_dump_map, field_related_entity_map) self._oid = oid entity = db._entity(extent_name, oid) # After execute callback. @@ -355,10 +366,16 @@ class Delete(Transaction): other = EntityClass(oid) others.append((EntityClass.__name__, oid, other)) field_map = other.sys.field_map(not_fget) - field_value_map = dict(field_map.value_map()) - new_value_map = dict((name, UNASSIGNED) for name in field_names) - field_value_map.update(new_value_map) - db._update_entity(other._extent.name, oid, field_value_map) + field_dump_map = dict(field_map.dump_map()) + field_related_entity_map = dict(field_map.related_entity_map()) + new_dump_map = dict((name, UNASSIGNED) for name in field_names) + new_related_entity_map = dict( + (name, frozenset()) for name in field_names) + field_dump_map.update(new_dump_map) + field_related_entity_map.update(new_related_entity_map) + db._update_entity( + other._extent.name, oid, field_dump_map, + field_related_entity_map) # Delete entities in a deterministic (sorted) fashion. for name, oid, other in sorted(others): tx = other.t.delete() @@ -408,13 +425,14 @@ class Update(Transaction): def _execute(self, db): entity = self._entity + field_map = self._field_map if entity._rev != self._rev: raise TransactionExpired( 'Original entity revision was %i, is now %i' % (self._rev, entity._rev)) if self._require_changes: nothing_changed = True - for field in self._field_map.itervalues(): + for field in field_map.itervalues(): if field.was_changed(): nothing_changed = False break @@ -423,19 +441,23 @@ class Update(Transaction): raise TransactionFieldsNotChanged(msg) self._before_execute(db, entity) # Validate individual fields. - for field in self._field_map.itervalues(): + for field in field_map.itervalues(): if field.fget is None: field.validate(field._value) extent_name = self._extent_name oid = self._oid # Strip out unwanted fields. - field_value_map = self._field_map.value_map() + field_dump_map = field_map.dump_map() + field_related_entity_map = field_map.related_entity_map() fget_fields = self._fget_fields field_spec = self._EntityClass._field_spec - for name in field_value_map.keys(): + for name in field_dump_map.keys(): if name in fget_fields or name not in field_spec: - del field_value_map[name] - db._update_entity(extent_name, oid, field_value_map) + del field_dump_map[name] + if name in field_related_entity_map: + del field_related_entity_map[name] + db._update_entity( + extent_name, oid, field_dump_map, field_related_entity_map) entity = db._entity(extent_name, oid) self._after_execute(db, entity) return entity schevo/transaction.py @@ -7,11 +7,11 @@ import textwrap setup( name="Schevo", - + version=__version__, - + description="Next-generation DBMS", - + long_description=textwrap.dedent(""" Schevo is a next-generation DBMS that focuses on the following: @@ -42,7 +42,7 @@ setup( Schevo to ensure that your database is left in a consistent state at all times. - * **User Interface Generation**. + * **User Interface Generation**. User interface code takes advantage of the richness of your database schema. Use a full-featured database navigator to interact with your @@ -52,7 +52,7 @@ setup( The latest development version is available in a `Subversion repository <http://schevo.org/svn/trunk/Schevo#egg=Schevo-dev>`__. """), - + classifiers=[ 'Development Status :: 4 - Beta', 'Environment :: Console', @@ -63,16 +63,16 @@ setup( 'Topic :: Database :: Database Engines/Servers', 'Topic :: Software Development :: Libraries :: Application Frameworks', ], - + keywords='database dbms', - + author='Orbtech, L.L.C. and contributors', author_email='schevo@googlegroups.com', url='http://schevo.org/wiki/Schevo', - + license='LGPL', - + platforms=['UNIX', 'Windows'], packages=find_packages(exclude=['doc', 'tests']), @@ -80,24 +80,24 @@ setup( include_package_data=True, zip_safe=False, - + install_requires=[ 'Louie >= 1.0', 'PasteScript == dev, >= 1.1.1dev-r6221', ], - + tests_require=[ 'nose >= 0.9.0', ], test_suite='nose.collector', - + extras_require={ }, - + dependency_links = [ 'http://schevo.org/files/thirdparty/', ], - + ext_modules = [ Extension('schevo.store._persistent', ['schevo/store/_persistent.c']), ], setup.py @@ -6,10 +6,10 @@ For copyright, license, and warranty, see bottom of file. from schevo.test import CreatesSchema, raises -class TestBank(CreatesSchema): - +class BaseBank(CreatesSchema): + body = ''' - + class Account(E.Entity): """Bank account.""" @@ -162,7 +162,17 @@ class TestBank(CreatesSchema): tx.to_account = family tx.amount = 0.01 assert raises(Exception, db.execute, tx) - + + +class TestBank1(BaseBank): + + format = 1 + + +class TestBank2(BaseBank): + + format = 2 + # Copyright (C) 2001-2006 Orbtech, L.L.C. # tests/test_bank.py @@ -8,10 +8,10 @@ from textwrap import dedent from schevo.test import CreatesSchema, PREAMBLE -class TestQuery(CreatesSchema): +class BaseCalculatedUnicode(CreatesSchema): body = ''' - + class Thing(E.Entity): image = f.image() password = f.password() @@ -41,6 +41,16 @@ class TestQuery(CreatesSchema): assert unicode(thing_view.f.calc_password) == u'(Hidden)' +class TestCalculatedUnicode1(BaseCalculatedUnicode): + + format = 1 + + +class TestCalculatedUnicode2(BaseCalculatedUnicode): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_calculated_field_unicode.py @@ -7,18 +7,18 @@ import louie from schevo.change import CREATE, DELETE, UPDATE, Distributor, normalize from schevo.constant import UNASSIGNED -from schevo.database import TransactionExecuted from schevo import error +from schevo.signal import TransactionExecuted from schevo.test import CreatesSchema, raises from schevo.transaction import Transaction BODY = ''' class User(E.Entity): - + name = f.unicode() age = f.integer(required=False) - + _key(name) _index(age) @@ -141,7 +141,7 @@ class Transfer(T.Transaction): ''' -class TestChangeset(CreatesSchema): +class BaseChangeset(CreatesSchema): """Upon execution of a transaction, a list of changes that it made becomes available regarding Create, Delete, and Update operations that occurred in each extent. @@ -197,7 +197,7 @@ class TestChangeset(CreatesSchema): assert summary.creates == dict(User=set([oid])) assert summary.deletes == dict() assert summary.updates == dict() - + def test_delete(self): tx = db.User.t.create(name='foo') user = db.execute(tx) @@ -345,7 +345,7 @@ class TestDistributor(CreatesSchema): body = BODY class Watcher(object): - + def __init__(self): self.received = [] @@ -441,6 +441,16 @@ class TestDistributor(CreatesSchema): ] +class TestChangeset1(BaseChangeset): + + format = 1 + + +class TestChangeset2(BaseChangeset): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_change.py @@ -7,18 +7,13 @@ from schevo import error from schevo.test import CreatesDatabase, raises -class TestDatabase(CreatesDatabase): +class BaseDatabase(CreatesDatabase): """Subclass and set self.db to the database instance to test.""" def test_schevo_key_in_root(self): """A Durus database has a `SCHEVO` key in its root.""" assert 'SCHEVO' in self.connection.get_root() - def test_format_1(self): - """A newly-created database will be in database format version - 1.""" - assert self.db.format == 1 - def test_empty(self): """A database should start out with very little.""" assert self.db.version == 0 @@ -71,7 +66,8 @@ class TestDatabase(CreatesDatabase): db = self.db db._create_extent('Some_Extent', ['name', 'age'], []) fields = dict(name='Foo', age=33) - oid = db._create_entity('Some_Extent', fields) + related_entities = dict() + oid = db._create_entity('Some_Extent', fields, related_entities) db._commit() # OID starts out at 1 and rev starts out as 0. assert oid == 1 @@ -87,10 +83,11 @@ class TestDatabase(CreatesDatabase): db = self.db db._create_extent('Some_Extent', ['name', 'age'], []) fields = dict(name='Foo', age=33) - oid = db._create_entity('Some_Extent', fields) + related_entities = dict() + oid = db._create_entity('Some_Extent', fields, related_entities) db._commit() fields = dict(name='Bar') - db._update_entity('Some_Extent', oid, fields) + db._update_entity('Some_Extent', oid, fields, related_entities) db._commit() # Rev increments. assert db._entity_rev('Some_Extent', oid) == 1 @@ -104,7 +101,8 @@ class TestDatabase(CreatesDatabase): db = self.db db._create_extent('Some_Extent', ['name', 'age'], []) fields = dict(name='Foo', age=33) - oid = db._create_entity('Some_Extent', fields) + related_entities = dict() + oid = db._create_entity('Some_Extent', fields, related_entities) db._commit() db._delete_entity('Some_Extent', oid) db._commit() @@ -115,11 +113,29 @@ class TestDatabase(CreatesDatabase): assert db._extent_len('Some_Extent') == 0 # Creating a new entity never uses a deleted oid. fields = dict(name='Foo', age=33) - oid2 = db._create_entity('Some_Extent', fields) + oid2 = db._create_entity('Some_Extent', fields, related_entities) db._commit() assert oid2 == oid + 1 +class TestDatabase1(BaseDatabase): + + format = 1 + + def test_format_1(self): + """A newly-created database will be in database format version 1.""" + assert self.db.format == 1 + + +class TestDatabase2(BaseDatabase): + + format = 2 + + def test_format_2(self): + """A newly-created database will be in database format version 2.""" + assert self.db.format == 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_database.py @@ -6,7 +6,7 @@ For copyright, license, and warranty, see bottom of file. from schevo.test import CreatesSchema -class TestDatabaseNamespaces(CreatesSchema): +class BaseDatabaseNamespaces(CreatesSchema): body = ''' @@ -27,6 +27,16 @@ class TestDatabaseNamespaces(CreatesSchema): assert foo.bar == 'baz' +class TestDatabaseNamespaces1(BaseDatabaseNamespaces): + + format = 1 + + +class TestDatabaseNamespaces2(BaseDatabaseNamespaces): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_database_namespace.py @@ -8,7 +8,7 @@ from schevo import error from schevo.test import CreatesSchema -class TestDefaultValues(CreatesSchema): +class BaseDefaultValues(CreatesSchema): body = ''' @@ -40,6 +40,16 @@ class TestDefaultValues(CreatesSchema): assert charlie.gamma == gamma +class TestDefaultValues1(BaseDefaultValues): + + format = 1 + + +class TestDefaultValues2(BaseDefaultValues): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_default_values.py @@ -13,7 +13,7 @@ from schevo.test import CreatesSchema, raises from schevo.transaction import Transaction -class TestEntityExtent(CreatesSchema): +class BaseEntityExtent(CreatesSchema): body = ''' @@ -88,7 +88,7 @@ class TestEntityExtent(CreatesSchema): def t_trigger_key_collision(extent): return T.Trigger_Key_Collision() - + class Account(E.Entity): """Bank account.""" @@ -751,7 +751,7 @@ class TestEntityExtent(CreatesSchema): assert name_ages == sorted(name_ages) name_ages = [(u.name, u.age) for u in db.User.by('-name', '-age')] assert name_ages == list(reversed(sorted(name_ages))) - + def test_extent_iter(self): # Create several users with random names and ages. tx = self.LotsOfUsers() @@ -762,7 +762,7 @@ class TestEntityExtent(CreatesSchema): count += 1 assert count == user.sys.oid assert count == total - + def test_entity_equality(self): """Entity instances referring to the same entity always have the same OID, revision, and field values, and are also equal.""" @@ -789,6 +789,16 @@ class TestEntityExtent(CreatesSchema): assert extent_names == expected +class TestEntityExtent1(BaseEntityExtent): + + format = 1 + + +class TestEntityExtent2(BaseEntityExtent): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_entity_extent.py @@ -12,7 +12,7 @@ from schevo.test import CreatesSchema from schevo.transaction import Transaction -class TestHiddenBases(CreatesSchema): +class BaseHiddenBases(CreatesSchema): body = ''' @@ -73,7 +73,7 @@ class TestHiddenBases(CreatesSchema): assert tx._field_spec.keys() == ['beta', 'gamma'] -class TestSameNameSubclasses(CreatesSchema): +class BaseSameNameSubclasses(CreatesSchema): body = ''' @@ -103,20 +103,20 @@ class TestSameNameSubclasses(CreatesSchema): assert plural(db.Something) == u'Somethingys' -class TestSubclassTransactionCorrectness(CreatesSchema): +class BaseSubclassTransactionCorrectness(CreatesSchema): body = ''' - + class _Super(E.Entity): pass - + class First(E._Super): pass - + class Second(E._Super): pass ''' - + def test_tx_correctness(self): tx = db.First.t.create() assert tx.sys.extent_name == 'First' @@ -124,6 +124,36 @@ class TestSubclassTransactionCorrectness(CreatesSchema): assert tx.sys.extent_name == 'Second' +class TestHiddenBases1(BaseHiddenBases): + + format = 1 + + +class TestHiddenBases2(BaseHiddenBases): + + format = 2 + + +class TestSameNameSubclasses1(BaseSameNameSubclasses): + + format = 1 + + +class TestSameNameSubclasses2(BaseSameNameSubclasses): + + format = 2 + + +class TestSubclassTransactionCorrectness1(BaseSubclassTransactionCorrectness): + + format = 1 + + +class TestSubclassTransactionCorrectness2(BaseSubclassTransactionCorrectness): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_entity_subclass.py @@ -22,7 +22,7 @@ def fix(schema): return BOILERPLATE + dedent(schema) -class TestEvolveIntraVersion(CreatesDatabase): +class BaseEvolveIntraVersion(CreatesDatabase): """Test evolution of schema within the same version, as occurs during app development.""" @@ -325,7 +325,7 @@ class TestEvolveIntraVersion(CreatesDatabase): db.execute(db.Foo.t.create(bar='baz')) -class TestEvolveInterVersion(CreatesDatabase): +class BaseEvolveInterVersion(CreatesDatabase): """Test evolution of schema from version to version, as occurs with upgrading deployed production apps.""" @@ -652,7 +652,7 @@ class TestEvolveInterVersion(CreatesDatabase): e2 = db.Foo[e2_oid] assert e1.baz == 'abc' assert e2.baz == 'def' - + def test_rename_field_broken(self): schema1 = fix(""" class Foo(E.Entity): @@ -692,7 +692,7 @@ class TestEvolveInterVersion(CreatesDatabase): e2 = db.Foo[e2_oid] assert e1.baz == 'abc' assert e2.baz == 'def' - + def test_rename_field_reused(self): schema1 = fix(""" class Foo(E.Entity): @@ -713,7 +713,7 @@ class TestEvolveInterVersion(CreatesDatabase): e2 = db.Foo[e2_oid] assert e1.baz == 'abc' assert e2.baz == 'def' - + def test_rename_extent(self): schema1 = fix(""" class Foo(E.Entity): @@ -735,7 +735,7 @@ class TestEvolveInterVersion(CreatesDatabase): assert e1.bar == 'abc' assert e2.bar == 'def' assert db.extent_names() == ['Baz'] - + def test_rename_extent_broken(self): schema1 = fix(""" class Foo(E.Entity): @@ -964,6 +964,26 @@ class TestEvolveInterVersion(CreatesDatabase): assert db.x.get_bar() == 44 +class TestEvolveIntraVersion1(BaseEvolveIntraVersion): + + format = 1 + + +class TestEvolveIntraVersion2(BaseEvolveIntraVersion): + + format = 2 + + +class TestEvolveInterVersion1(BaseEvolveInterVersion): + + format = 1 + + +class TestEvolveInterVersion2(BaseEvolveInterVersion): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_evolve.py @@ -7,7 +7,7 @@ For copyright, license, and warranty, see bottom of file. from schevo.test import CreatesSchema -class TestOverride(CreatesSchema): +class BaseOverride(CreatesSchema): body = ''' @@ -27,6 +27,16 @@ class TestOverride(CreatesSchema): assert result == expected +class TestOverride1(BaseOverride): + + format = 1 + + +class TestOverride2(BaseOverride): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. and contributors. # # Schevo tests/test_extent_name_override.py @@ -8,7 +8,7 @@ from textwrap import dedent from schevo.test import CreatesSchema, PREAMBLE -class TestExtentWithoutFields(CreatesSchema): +class BaseExtentWithoutFields(CreatesSchema): body = ''' @@ -49,7 +49,17 @@ class TestExtentWithoutFields(CreatesSchema): assert entity.calc == entity.sys.oid entity = exe(db.Subclassed.t.create()) assert entity.calc == entity.sys.oid - + + +class TestExtentWithoutFields1(BaseExtentWithoutFields): + + format = 1 + + +class TestExtentWithoutFields2(BaseExtentWithoutFields): + + format = 2 + # Copyright (C) 2001-2006 Orbtech, L.L.C. # tests/test_extent_without_fields.py @@ -6,7 +6,7 @@ For copyright, license, and warranty, see bottom of file. from schevo.test import CreatesSchema -class TestExtentMethod(CreatesSchema): +class BaseExtentMethod(CreatesSchema): body = ''' @@ -23,6 +23,16 @@ class TestExtentMethod(CreatesSchema): assert result is extent +class TestExtentMethod1(BaseExtentMethod): + + format = 1 + + +class TestExtentMethod2(BaseExtentMethod): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_extentmethod.py @@ -11,7 +11,7 @@ import sys from schevo.constant import UNASSIGNED from schevo import field from schevo import fieldspec -from schevo.test import BaseTest, CreatesSchema, raises +from schevo.test import BaseTest, raises class TestField(BaseTest): @@ -61,7 +61,7 @@ class Base: bad_values_default = [] convert_values_default = [] - FieldClass = None + FieldClass = None good_values_default = [] str_values_default = [] min_max_values = [] @@ -188,7 +188,7 @@ class TestString(Base, BaseTest): ('test', 5, None, False, False), ('test_test', None, 8, False, False), ] - + class TestUnicode(Base, BaseTest): @@ -258,7 +258,7 @@ class TestInteger(Base, BaseTest): (100, -200, 200, True), (300, -200, 200, False), ] - + class TestFloat(Base, BaseTest): @@ -342,7 +342,7 @@ class TestFloat(Base, BaseTest): class TestDate(Base, BaseTest): class FieldClass(field.Date): pass - + good_values_default = [ datetime.date(2004, 5, 5), datetime.date(1765, 4, 3), @@ -384,14 +384,14 @@ class TestDate(Base, BaseTest): (datetime.date(2004, 1, 1), datetime.date(2004, 2, 2), None, False), (datetime.date(2004, 1, 1), None, datetime.date(2004, 2, 2), True), (datetime.date(2004, 3, 3), None, datetime.date(2004, 2, 2), False), - (datetime.date(2004, 3, 3), - datetime.date(2004, 2, 2), datetime.date(2004, 4, 4), + (datetime.date(2004, 3, 3), + datetime.date(2004, 2, 2), datetime.date(2004, 4, 4), True), - (datetime.date(2004, 1, 1), - datetime.date(2004, 2, 2), datetime.date(2004, 4, 4), + (datetime.date(2004, 1, 1), + datetime.date(2004, 2, 2), datetime.date(2004, 4, 4), False), - (datetime.date(2004, 5, 5), - datetime.date(2004, 2, 2), datetime.date(2004, 4, 4), + (datetime.date(2004, 5, 5), + datetime.date(2004, 2, 2), datetime.date(2004, 4, 4), False), ] @@ -399,7 +399,7 @@ class TestDate(Base, BaseTest): class TestDatetime(Base, BaseTest): class FieldClass(field.Datetime): pass - + good_values_default = [ datetime.datetime(2004, 5, 5, 22, 32, 5), datetime.datetime(1765, 4, 3, 10, 11, 12), @@ -426,28 +426,28 @@ class TestDatetime(Base, BaseTest): ] min_max_values = [ # (value, min, max, is_valid), - (datetime.datetime(2004, 5, 5), None, None, + (datetime.datetime(2004, 5, 5), None, None, True), - (datetime.datetime(2004, 3, 3), - datetime.datetime(2004, 2, 2), None, + (datetime.datetime(2004, 3, 3), + datetime.datetime(2004, 2, 2), None, True), - (datetime.datetime(2004, 1, 1), - datetime.datetime(2004, 2, 2), None, + (datetime.datetime(2004, 1, 1), + datetime.datetime(2004, 2, 2), None, False), - (datetime.datetime(2004, 1, 1), None, - datetime.datetime(2004, 2, 2), + (datetime.datetime(2004, 1, 1), None, + datetime.datetime(2004, 2, 2), True), - (datetime.datetime(2004, 3, 3), None, - datetime.datetime(2004, 2, 2), + (datetime.datetime(2004, 3, 3), None, + datetime.datetime(2004, 2, 2), False), - (datetime.datetime(2004, 3, 3), - datetime.datetime(2004, 2, 2), datetime.datetime(2004, 4, 4), + (datetime.datetime(2004, 3, 3), + datetime.datetime(2004, 2, 2), datetime.datetime(2004, 4, 4), True), - (datetime.datetime(2004, 1, 1), - datetime.datetime(2004, 2, 2), datetime.datetime(2004, 4, 4), + (datetime.datetime(2004, 1, 1), + datetime.datetime(2004, 2, 2), datetime.datetime(2004, 4, 4), False), - (datetime.datetime(2004, 5, 5), - datetime.datetime(2004, 2, 2), datetime.datetime(2004, 4, 4), + (datetime.datetime(2004, 5, 5), + datetime.datetime(2004, 2, 2), datetime.datetime(2004, 4, 4), False), ] @@ -538,72 +538,6 @@ class TestPassword(object): assert unicode(f) == u'(Hidden)' -class TestEntity(CreatesSchema): - - body = ''' - - class Foo(E.Entity): - - thing = f.string() - - _sample_unittest = [ - ('a', ), - ('b', ), - ('c', ), - ] - - class Bar(E.Entity): - - stuff = f.integer() - - _sample_unittest = [ - (1, ), - (2, ), - (3, ), - ] - - class Baz(E.Entity): - - foo = f.entity('Foo') - bar = f.entity('Bar') - foobar = f.entity('Foo', 'Bar', required=False) - ''' - - def test_allow(self): - tx = db.Baz.t.create() - assert tx.f.foo.allow == set(['Foo']) - assert tx.f.bar.allow == set(['Bar']) - assert tx.f.foobar.allow == set(['Foo', 'Bar']) - - def test_convert(self): - tx = db.Baz.t.create() - f = tx.f.foo - assert f.convert('Foo-1', db) == db.Foo[1] - assert f.convert(u'Foo-1', db) == db.Foo[1] - - def test_reversible_valid_values(self): - tx = db.Baz.t.create() - assert tx.f.foo.reversible_valid_values(db) == [ - ('Foo-1', db.Foo[1]), - ('Foo-2', db.Foo[2]), - ('Foo-3', db.Foo[3]), - ] - assert tx.f.bar.reversible_valid_values(db) == [ - ('Bar-1', db.Bar[1]), - ('Bar-2', db.Bar[2]), - ('Bar-3', db.Bar[3]), - ] - assert tx.f.foobar.reversible_valid_values(db) == [ - ('', UNASSIGNED), - ('Bar-1', db.Bar[1]), - ('Bar-2', db.Bar[2]), - ('Bar-3', db.Bar[3]), - ('Foo-1', db.Foo[1]), - ('Foo-2', db.Foo[2]), - ('Foo-3', db.Foo[3]), - ] - - # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_field.py @@ -13,12 +13,12 @@ from schevo.test import CreatesSchema from schevo.field import not_expensive, not_fget, not_hidden -class TestFieldMaps(CreatesSchema): +class BaseFieldMaps(CreatesSchema): body = ''' class Foo(E.Entity): - + aaa = f.unicode() bbb = f.unicode(hidden=True) @f.unicode(hidden=True) @@ -61,7 +61,7 @@ class TestFieldMaps(CreatesSchema): expected = 'aaa', keys = fkeys(not_fget, not_hidden) assert expected == keys - + def test_extent_field_spec(self): extent = db.Foo def fkeys(*filters): @@ -91,7 +91,16 @@ class TestFieldMaps(CreatesSchema): expected = 'aaa', keys = fkeys(not_fget, not_hidden) assert expected == keys - + + +class TestFieldMaps1(BaseFieldMaps): + + format = 1 + + +class TestFieldMaps2(BaseFieldMaps): + + format = 2 # Copyright (C) 2001-2006 Orbtech, L.L.C. tests/test_field_maps.py @@ -19,7 +19,7 @@ SPROCKET_PNG = file(os.path.join(TEST_ICONS, 'db.Sprocket.png'), 'rb').read() CONFIGURE_PNG = file(os.path.join(TEST_ICONS, 'configure.png'), 'rb').read() -class TestFsIconMap(CreatesSchema): +class BaseFsIconMap(CreatesSchema): body = ''' @@ -47,6 +47,16 @@ class TestFsIconMap(CreatesSchema): assert loopsegment_png == DEFAULT_PNG +class TestFsIconMap1(BaseFsIconMap): + + format = 1 + + +class TestFsIconMap2(BaseFsIconMap): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_icon.py @@ -8,7 +8,7 @@ from schevo import label from schevo import transaction -class TestDecoration(CreatesSchema): +class BaseDecoration(CreatesSchema): body = ''' @@ -461,6 +461,16 @@ class TestDecoration(CreatesSchema): assert label.label('some string') == 'some string' +class TestDecoration1(BaseDecoration): + + format = 1 + + +class TestDecoration2(BaseDecoration): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_label.py @@ -9,7 +9,7 @@ from schevo.label import label, plural from schevo.test import CreatesSchema -class TestLinks(CreatesSchema): +class BaseLinks(CreatesSchema): # XXX: Todo: test_links @@ -79,21 +79,21 @@ class TestLinks(CreatesSchema): (4, DEFAULT), (5, DEFAULT), ] - - + + class Goauld(E.Entity): """Fictional characters from a TV series, to test plural usage on entity m namespace.""" - + something = f.entity('Something') - + _label = u"Goa\u2032uld" _plural = u"Goa\u2032ulds" - - + + class Something(E.Entity): pass - + ''' def test_many(self): @@ -120,7 +120,7 @@ class TestLinks(CreatesSchema): alphas = bravo.m.foxtrot_alphas('foxtrot_any') assert len(alphas) == 1 assert db.FoxtrotAlpha[4] in alphas - + def test_many_pluralization(self): assert label(db.Goauld) == u"Goa\u2032uld" assert plural(db.Goauld) == u"Goa\u2032ulds" @@ -194,6 +194,16 @@ class TestLinks(CreatesSchema): assert rset == eset +class TestLinks1(BaseLinks): + + format = 1 + + +class TestLinks2(BaseLinks): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_links.py @@ -180,7 +180,7 @@ For copyright, license, and warranty, see bottom of file. ## assert 'bar' in W ## assert TR1[0] >= TW1[1] # Read 1 started after write 1. ## assert TW2[0] >= TR1[1] # Write 2 started after read 1. - + # Copyright (C) 2001-2006 Orbtech, L.L.C. # tests/test_mrow.py @@ -5,10 +5,11 @@ For copyright, license, and warranty, see bottom of file. from schevo.constant import UNASSIGNED from schevo import error +from schevo.placeholder import Placeholder from schevo.test import CreatesSchema, raises -class TestOnDelete(CreatesSchema): +class BaseOnDelete(CreatesSchema): body = ''' @@ -92,7 +93,31 @@ class TestOnDelete(CreatesSchema): class Boo(E.Entity): """The parent of two children, one of which references the other. We want the cascade to succeed on both children in - spite of the field reference that has the default RESTRICT.""" + spite of the field reference that has the default RESTRICT. + + The relationships created upon a creation of a Boo entity + can be visualized as follows:: + + .---------------. + | Boo[1] | <-----------------. + | | <-----. | + | .name = 'BOO' | | | + '---------------' | | + |CASCADE | + .---------------. | | + | Bar[1] | | | + .-> | | | |CASCADE + | | .boo = Boo[1]---------' | + | | .baz = Baz[1]---------. | + | '---------------' | | + | | | + RESTRICT| .---------------. |RESTRICT | + | | Baz[1] | | | + | | | <-----' | + | | .boo = Boo[1]---------------------' + '-----.bar = Bar[1] | + '---------------' + """ name = f.unicode() @@ -155,7 +180,7 @@ class TestOnDelete(CreatesSchema): faz = f.entity(('Faz', CASCADE)) far = f.entity('Far') - + ''' def _alpha_alpha(self): @@ -186,14 +211,37 @@ class TestOnDelete(CreatesSchema): assert alpha_bravo not in db.AlphaBravo def test_cascade_complex(self): + # Create the complex structure. boo = db.execute(db.Boo.t.create(name='BOO')) assert len(db.Boo) == 1 assert len(db.Bar) == 1 assert len(db.Baz) == 1 + bar = db.Bar[1] + baz = db.Baz[1] + assert bar.boo == boo + assert bar.baz == baz + assert baz.boo == boo + assert baz.bar == bar + assert bar.m.bazs() == [baz] + assert baz.m.bars() == [bar] + assert boo.m.bars() == [bar] + assert boo.m.bazs() == [baz] + assert boo.sys.count() == 2 + assert bar.sys.count() == 1 + assert baz.sys.count() == 1 + self.internal_cascade_complex_1() + # Delete it. db.execute(boo.t.delete()) assert len(db.Boo) == 0 assert len(db.Bar) == 0 assert len(db.Baz) == 0 + self.internal_cascade_complex_2() + + def internal_cascade_complex_1(self): + raise NotImplementedError() + + def internal_cascade_complex_2(self): + raise NotImplementedError() ## def test_cascade_complex_hierarchy(self): ## foo = db.execute(db.Foo.t.create(name='FOO')) @@ -245,6 +293,168 @@ class TestOnDelete(CreatesSchema): assert alpha_echo not in db.AlphaEcho +class TestOnDelete1(BaseOnDelete): + + format = 1 + + def internal_cascade_complex_1(self): + root = db._root + schevo = root['SCHEVO'] + extent_name_id = schevo['extent_name_id'] + extents = schevo['extents'] + Boo_extent_id = extent_name_id['Boo'] + Bar_extent_id = extent_name_id['Bar'] + Baz_extent_id = extent_name_id['Baz'] + Boo_extent = extents[Boo_extent_id] + Bar_extent = extents[Bar_extent_id] + Baz_extent = extents[Baz_extent_id] + Boo_field_name_id = Boo_extent['field_name_id'] + Bar_field_name_id = Bar_extent['field_name_id'] + Baz_field_name_id = Baz_extent['field_name_id'] + assert len(Boo_extent['entities']) == 1 + assert len(Bar_extent['entities']) == 1 + assert len(Baz_extent['entities']) == 1 + # Check for Boo[1] having backlinks to Bar[1].boo and Baz[1].boo + Boo1 = Boo_extent['entities'][1] + assert Boo1['link_count'] == 2 + Bar_boo_field_id = Bar_field_name_id['boo'] + Baz_boo_field_id = Baz_field_name_id['boo'] + Boo1_links_keys = set(Boo1['links'].keys()) + expected_Boo1_links_keys = set([ + (Bar_extent_id, Bar_boo_field_id), + (Baz_extent_id, Baz_boo_field_id), + ]) + assert Boo1_links_keys == expected_Boo1_links_keys + assert Boo1['links'][(Bar_extent_id, Bar_boo_field_id)].keys() == [1] + assert Boo1['links'][(Baz_extent_id, Baz_boo_field_id)].keys() == [1] + # Check for Bar[1] having backlink to Baz[1].bar + Bar1 = Bar_extent['entities'][1] + assert Bar1['link_count'] == 1 + Baz_bar_field_id = Baz_field_name_id['bar'] + Bar1_links_keys = set(Bar1['links'].keys()) + expected_Bar1_links_keys = set([ + (Baz_extent_id, Baz_bar_field_id), + ]) + assert Bar1_links_keys == expected_Bar1_links_keys + assert Bar1['links'][(Baz_extent_id, Baz_bar_field_id)].keys() == [1] + # Check for Baz[1] having backlink to Bar[1].bar + Baz1 = Baz_extent['entities'][1] + assert Baz1['link_count'] == 1 + Bar_baz_field_id = Bar_field_name_id['baz'] + Baz1_links_keys = set(Baz1['links'].keys()) + expected_Baz1_links_keys = set([ + (Bar_extent_id, Bar_baz_field_id), + ]) + assert Baz1_links_keys == expected_Baz1_links_keys + assert Baz1['links'][(Bar_extent_id, Bar_baz_field_id)].keys() == [1] + # Check for Bar[1].boo and Bar[1].baz having correct related entity + # structures. + Bar1_fields = Bar1['fields'] + Bar_boo_field_id = Bar_field_name_id['boo'] + Bar1_boo = Bar1_fields[Bar_boo_field_id] + assert Bar1_boo == (Boo_extent_id, 1) + Bar_baz_field_id = Bar_field_name_id['baz'] + Bar1_baz = Bar1_fields[Bar_baz_field_id] + assert Bar1_baz == (Baz_extent_id, 1) + + def internal_cascade_complex_2(self): + root = db._root + schevo = root['SCHEVO'] + extent_name_id = schevo['extent_name_id'] + extents = schevo['extents'] + Boo_extent_id = extent_name_id['Boo'] + Bar_extent_id = extent_name_id['Bar'] + Baz_extent_id = extent_name_id['Baz'] + Boo_extent = extents[Boo_extent_id] + Bar_extent = extents[Bar_extent_id] + Baz_extent = extents[Baz_extent_id] + assert len(Boo_extent['entities']) == 0 + assert len(Bar_extent['entities']) == 0 + assert len(Baz_extent['entities']) == 0 + + +class TestOnDelete2(BaseOnDelete): + + format = 2 + + def internal_cascade_complex_1(self): + root = db._root + schevo = root['SCHEVO'] + extent_name_id = schevo['extent_name_id'] + extents = schevo['extents'] + Boo_extent_id = extent_name_id['Boo'] + Bar_extent_id = extent_name_id['Bar'] + Baz_extent_id = extent_name_id['Baz'] + Boo_extent = extents[Boo_extent_id] + Bar_extent = extents[Bar_extent_id] + Baz_extent = extents[Baz_extent_id] + Boo_field_name_id = Boo_extent['field_name_id'] + Bar_field_name_id = Bar_extent['field_name_id'] + Baz_field_name_id = Baz_extent['field_name_id'] + assert len(Boo_extent['entities']) == 1 + assert len(Bar_extent['entities']) == 1 + assert len(Baz_extent['entities']) == 1 + # Check for Boo[1] having backlinks to Bar[1].boo and Baz[1].boo + Boo1 = Boo_extent['entities'][1] + assert Boo1['link_count'] == 2 + Bar_boo_field_id = Bar_field_name_id['boo'] + Baz_boo_field_id = Baz_field_name_id['boo'] + Boo1_links_keys = set(Boo1['links'].keys()) + expected_Boo1_links_keys = set([ + (Bar_extent_id, Bar_boo_field_id), + (Baz_extent_id, Baz_boo_field_id), + ]) + assert Boo1_links_keys == expected_Boo1_links_keys + assert Boo1['links'][(Bar_extent_id, Bar_boo_field_id)].keys() == [1] + assert Boo1['links'][(Baz_extent_id, Baz_boo_field_id)].keys() == [1] + # Check for Bar[1] having backlink to Baz[1].bar + Bar1 = Bar_extent['entities'][1] + assert Bar1['link_count'] == 1 + Baz_bar_field_id = Baz_field_name_id['bar'] + Bar1_links_keys = set(Bar1['links'].keys()) + expected_Bar1_links_keys = set([ + (Baz_extent_id, Baz_bar_field_id), + ]) + assert Bar1_links_keys == expected_Bar1_links_keys + assert Bar1['links'][(Baz_extent_id, Baz_bar_field_id)].keys() == [1] + # Check for Baz[1] having backlink to Bar[1].bar + Baz1 = Baz_extent['entities'][1] + assert Baz1['link_count'] == 1 + Bar_baz_field_id = Bar_field_name_id['baz'] + Baz1_links_keys = set(Baz1['links'].keys()) + expected_Baz1_links_keys = set([ + (Bar_extent_id, Bar_baz_field_id), + ]) + assert Baz1_links_keys == expected_Baz1_links_keys + assert Baz1['links'][(Bar_extent_id, Bar_baz_field_id)].keys() == [1] + # Check for Bar[1].boo and Bar[1].baz having correct related entity + # structures. + Bar1_related_entities = Bar1['related_entities'] + Bar_boo_field_id = Bar_field_name_id['boo'] + Bar1_related_boos = Bar1_related_entities[Bar_boo_field_id] + expected_Bar1_related_boos = frozenset([Placeholder(db.Boo[1])]) + assert Bar1_related_boos == expected_Bar1_related_boos + Bar_baz_field_id = Bar_field_name_id['baz'] + Bar1_related_bazs = Bar1_related_entities[Bar_baz_field_id] + expected_Bar1_related_bazs = frozenset([Placeholder(db.Baz[1])]) + assert Bar1_related_bazs == expected_Bar1_related_bazs + + def internal_cascade_complex_2(self): + root = db._root + schevo = root['SCHEVO'] + extent_name_id = schevo['extent_name_id'] + extents = schevo['extents'] + Boo_extent_id = extent_name_id['Boo'] + Bar_extent_id = extent_name_id['Bar'] + Baz_extent_id = extent_name_id['Baz'] + Boo_extent = extents[Boo_extent_id] + Bar_extent = extents[Bar_extent_id] + Baz_extent = extents[Baz_extent_id] + assert len(Boo_extent['entities']) == 0 + assert len(Bar_extent['entities']) == 0 + assert len(Baz_extent['entities']) == 0 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_on_delete.py @@ -8,7 +8,7 @@ from schevo.test import CreatesSchema from schevo.constant import UNASSIGNED -class TestPopulateSimple(CreatesSchema): +class BasePopulateSimple(CreatesSchema): body = ''' @@ -34,7 +34,7 @@ class TestPopulateSimple(CreatesSchema): assert db.Foo.as_datalist() == db.Foo._EntityClass._sample_unittest -class TestPopulateComplex(CreatesSchema): +class BasePopulateComplex(CreatesSchema): body = ''' @@ -117,7 +117,7 @@ class TestPopulateComplex(CreatesSchema): db.Multi._EntityClass._sample_unittest) -class TestPopulateHidden(CreatesSchema): +class BasePopulateHidden(CreatesSchema): body = ''' @@ -150,6 +150,41 @@ class TestPopulateHidden(CreatesSchema): assert db.Foo[2].bof == 5.6 +BasePopulateHidden +BasePopulateComplex +BasePopulateSimple + + +class TestPopulateSimple1(BasePopulateSimple): + + format = 1 + + +class TestPopulateSimple2(BasePopulateSimple): + + format = 2 + + +class TestPopulateComplex1(BasePopulateComplex): + + format = 1 + + +class TestPopulateComplex2(BasePopulateComplex): + + format = 2 + + +class TestPopulateHidden1(BasePopulateHidden): + + format = 1 + + +class TestPopulateHidden2(BasePopulateHidden): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. and contributors. # # Schevo tests/test_populate.py @@ -10,7 +10,7 @@ from schevo.label import label from schevo.test import CreatesSchema, raises -class TestQuery(CreatesSchema): +class BaseQuery(CreatesSchema): # XXX: tests for schevo.query.Links @@ -36,7 +36,7 @@ class TestQuery(CreatesSchema): class DeltaCharlie(E.Entity): """An extent that has a custom query.""" - hashed_value = f.hashedValue() + hashed_value = f.hashed_value() @extentmethod def q_hashes(extent, **kw): @@ -113,7 +113,7 @@ class TestQuery(CreatesSchema): q = qmethod(integer=12, string='foo') results = list(sorted(q())) assert len(results) == 0 - + def test_exact_query(self): # This is based on test_by_example_query, but uses the find query # instead, which is more specific than the by_example query. @@ -195,7 +195,7 @@ class TestQuery(CreatesSchema): q = hashes() assert isinstance(q.f.compare_with, field.String) assert list(q.f) == ['compare_with'] - + def test_parameterized_query_defaults_and_names(self): q = db.DeltaAlpha.q.exact() assert q.string is UNASSIGNED @@ -215,7 +215,17 @@ class TestQuery(CreatesSchema): assert q.match_names == ['string', 'float', 'entity'] assert len(q.queries) == 3 assert raises(error.FieldDoesNotExist, q.remove_match, 'integer') - + + +class TestQuery1(BaseQuery): + + format = 1 + + +class TestQuery2(BaseQuery): + + format = 2 + # Copyright (C) 2001-2006 Orbtech, L.L.C. # tests/test_query.py @@ -50,7 +50,7 @@ class Person(E.Entity): # Declare this out of order to test sorting. class Animal(E.Entity): - name = f.customString() + name = f.custom_string() wild = f.boolean() @@ -77,14 +77,14 @@ def t_tx_with_fields(): """ -class TestSchema(CreatesSchema): +class BaseSchema(CreatesSchema): body = BODY def test_entity_dict(self): E = db.schema.E assert len(E) == 4 - + def test_entity_entity(self): E = db.schema.E assert 'Entity' not in E @@ -166,7 +166,7 @@ class TestSchema(CreatesSchema): assert hasattr(d.t, 'tx_with_fields') tx = d.t.tx_with_fields() assert isinstance(tx, d.T.TxWithFields) - + def test_duplicate_key_and_index_declaration_is_a_schema_error(self): body = ''' class Foo(E.Entity): @@ -180,7 +180,7 @@ class TestSchema(CreatesSchema): class Foo(E.Entity): bar = f.integer() _key(bar) - + class Foo(E.Foo): _index('bar') ''' @@ -189,13 +189,23 @@ class TestSchema(CreatesSchema): class Foo(E.Entity): bar = f.integer() _index(bar) - + class Foo(E.Foo): _key('bar') ''' assert raises(SchemaError, DocTest, body) +class TestSchema1(BaseSchema): + + format = 1 + + +class TestSchema2(BaseSchema): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_schema.py @@ -2,6 +2,9 @@ $URL: svn+ssh://svn/repos/trunk/durus/test/utest_btree.py $ $Id: utest_btree.py 27868 2006-01-25 14:55:43Z dbinger $ """ + +import os + from schevo.store.btree import BTree, BNode from schevo.store.connection import Connection from schevo.store.file_storage import TempFileStorage @@ -189,36 +192,37 @@ class TestCoverage(object): assert not bt.has_key(2) assert bt.keys() == [] -class TestSlow(object): - - def test_slow(self): - bt = BTree() - print 'bt = BTree()' - d = {} - number = 0 - limit = 10000 - for k in xrange(limit*10): - number = randint(0, limit) - if number in bt: - assert number in d - if randint(0, 1) == 1: - del bt[number] - del d[number] - print 'del bt[%s]' % number - else: - if number in d: - print number - print number in bt - print number in d - assert number not in d - bt[number] = 1 - d[number] = 1 - print 'bt[%s] = 1' % number - if k % limit == 0: - d_items = d.items() - d_items.sort() - assert d_items == bt.items() - assert len(d_items) == len(bt) +if not 'SKIP_SLOW' in os.environ: + class TestSlow(object): + + def test_slow(self): + bt = BTree() + print 'bt = BTree()' + d = {} + number = 0 + limit = 10000 + for k in xrange(limit*10): + number = randint(0, limit) + if number in bt: + assert number in d + if randint(0, 1) == 1: + del bt[number] + del d[number] + print 'del bt[%s]' % number + else: + if number in d: + print number + print number in bt + print number in d + assert number not in d + bt[number] = 1 + d[number] = 1 + print 'bt[%s] = 1' % number + if k % limit == 0: + d_items = d.items() + d_items.sort() + assert d_items == bt.items() + assert len(d_items) == len(bt) class TestDurus(object): tests/test_store_btree.py @@ -2,6 +2,10 @@ $URL: svn+ssh://svn/repos/trunk/durus/test/utest_client_storage.py $ $Id: utest_client_storage.py 28137 2006-04-04 14:34:21Z dbinger $ """ + +import os +import sys + from schevo.store import run_durus from schevo.store.client_storage import ClientStorage from schevo.store.error import ReadConflictError, DurusKeyError @@ -11,60 +15,60 @@ from schevo.store.utils import p64 from popen2 import popen4 from schevo.test import raises from time import sleep -import sys -class Test(object): +if not 'SKIP_SLOW' in os.environ: + class Test(object): - address = (DEFAULT_HOST, 9123) + address = (DEFAULT_HOST, 9123) - def setUp(self): - if type(self.address) is tuple: - self.server = popen4('python %s --port=%s' % ( - run_durus.__file__, self.address[1])) - else: - self.server = popen4('python %s --address=%s' % ( - run_durus.__file__, self.address)) - sleep(3) # wait for bind + def setUp(self): + if type(self.address) is tuple: + self.server = popen4('python %s --port=%s' % ( + run_durus.__file__, self.address[1])) + else: + self.server = popen4('python %s --address=%s' % ( + run_durus.__file__, self.address)) + sleep(3) # wait for bind - def tearDown(self): - run_durus.stop_durus(self.address) + def tearDown(self): + run_durus.stop_durus(self.address) - def test_check_client_storage(self): - b = ClientStorage(address=self.address) - c = ClientStorage(address=self.address) - print self.address - oid = b.new_oid() - assert oid == p64(1), repr(oid) - oid = b.new_oid() - assert oid == p64(2), repr(oid) - try: + def test_check_client_storage(self): + b = ClientStorage(address=self.address) + c = ClientStorage(address=self.address) + print self.address + oid = b.new_oid() + assert oid == p64(1), repr(oid) + oid = b.new_oid() + assert oid == p64(2), repr(oid) + try: + b.load(p64(0)) + assert 0 + except KeyError: pass + record = pack_record(p64(0), 'ok', '') + b.begin() + b.store(p64(0), record) + assert b.end() is None b.load(p64(0)) - assert 0 - except KeyError: pass - record = pack_record(p64(0), 'ok', '') - b.begin() - b.store(p64(0), record) - assert b.end() is None - b.load(p64(0)) - assert b.sync() == [] - b.begin() - b.store(p64(1), pack_record(p64(1), 'no', '')) - b.end() - assert len(list(b.gen_oid_record())) == 2 - records = b.bulk_load([p64(0), p64(1)]) - assert len(list(records)) == 2 - records = b.bulk_load([p64(0), p64(1), p64(2)]) - assert raises(DurusKeyError, list, records) - b.pack() - assert len(list(b.gen_oid_record())) == 1 - assert raises(ReadConflictError, c.load, p64(0)) - assert raises(ReadConflictError, c.load, p64(0)) - assert set(c.sync()) == set([p64(0), p64(1)]) - assert record == c.load(p64(0)) + assert b.sync() == [] + b.begin() + b.store(p64(1), pack_record(p64(1), 'no', '')) + b.end() + assert len(list(b.gen_oid_record())) == 2 + records = b.bulk_load([p64(0), p64(1)]) + assert len(list(records)) == 2 + records = b.bulk_load([p64(0), p64(1), p64(2)]) + assert raises(DurusKeyError, list, records) + b.pack() + assert len(list(b.gen_oid_record())) == 1 + assert raises(ReadConflictError, c.load, p64(0)) + assert raises(ReadConflictError, c.load, p64(0)) + assert set(c.sync()) == set([p64(0), p64(1)]) + assert record == c.load(p64(0)) -if sys.platform != 'win32': + if sys.platform != 'win32': - class UnixDomainSocketTest(Test): - address = "test.durus_server" + class UnixDomainSocketTest(Test): + address = "test.durus_server" tests/test_store_client_storage.py @@ -2,6 +2,10 @@ $URL: svn+ssh://svn/repos/trunk/durus/test/utest_connection.py $ $Id: utest_connection.py 28275 2006-04-28 17:44:20Z dbinger $ """ + +import os +import sys + from schevo.store import run_durus from schevo.store.client_storage import ClientStorage from schevo.store.connection import Connection, touch_every_reference @@ -16,7 +20,6 @@ from schevo.store.utils import p64 from popen2 import popen4 from schevo.test import raises from time import sleep -import sys class TestConnection(object): @@ -117,124 +120,125 @@ class TestConnection(object): assert not root._p_is_unsaved() -class TestConnectionClientStorage (TestConnection): - - def setUp(self): - self.port = 9123 - self.server = popen4('python %s --port=%s' % ( - run_durus.__file__, self.port)) - sleep(3) # wait for bind - - def tearDown(self): - run_durus.stop_durus((DEFAULT_HOST, self.port)) - - def _get_storage(self): - return ClientStorage(port=self.port) - - def test_check_conflict(self): - b = Connection(self._get_storage()) - c = Connection(self._get_storage()) - rootb = b.get(p64(0)) - rootb['b'] = Persistent() - rootc = c.get(p64(0)) - rootc['c'] = Persistent() - c.commit() - assert raises(ConflictError, b.commit) - assert raises(KeyError, rootb.__getitem__, 'c') - transaction_serial = b.transaction_serial - b.abort() - assert b.get_transaction_serial() > transaction_serial - assert rootb._p_is_ghost() - rootc['d'] = Persistent() - c.commit() - rootb['d'] - - def test_check_fine_conflict(self): - c1 = Connection(self._get_storage()) - c2 = Connection(self._get_storage()) - c1.get_root()['A'] = Persistent() - c1.get_root()['A'].a = 1 - c1.get_root()['B'] = Persistent() - c1.commit() - c2.abort() - # c1 has A loaded. - assert not c1.get_root()['A']._p_is_ghost() - c1.get_root()['B'].b = 1 - c2.get_root()['A'].a = 2 - c2.commit() - # Even though A has been changed by c2, - # c1 has not accessed an attribute of A since - # the last c1.commit(), so we don't want a ConflictError. - c1.commit() - assert not c1.get_root()['A']._p_is_ghost() - c1.get_root()['A'].a # accessed! - c1.get_root()['B'].b = 1 - c2.get_root()['A'].a = 2 - c2.commit() - assert raises(ConflictError, c1.commit) - - def _scenario(self): - c1 = Connection(self._get_storage()) - c2 = Connection(self._get_storage()) - c1.get_root()['A'] = Persistent() - c1.get_root()['B'] = Persistent() - c1.get_root()['A'].a = 1 - c1.commit() - c2.abort() - c1.cache.recent_objects.discard(c1.get_root()['A']) - # Imagine c1 has been running for a while, and - # cache management, for example, has caused the - # cache reference to be weak. - return c1, c2 - - def test_conflict_from_invalid_removable_previously_accessed(self): - c1, c2 = self._scenario() - A = c1.get_root()['A'] - A.a # access A in c1. This will lead to conflict. - A_oid = A._p_oid - assert A in c1.cache.recent_objects - A = None # forget about it - print - # Lose the reference to A. - c1.get_root()._p_set_status_ghost() - # Commit a new A in c2. - c2.get_root()['A'].a = 2 - c2.commit() - c1.get_root()['B'].b = 1 # touch B in c1 - # Conflict because A has been accessed in c1, but it is invalid. - assert raises(ConflictError, c1.commit) - - def test_no_conflict_from_invalid_removable_not_previously_accessed(self): - c1, c2 = self._scenario() - c1.get_root()._p_set_status_ghost() - # Commit a new A in c2. - c2.get_root()['A'].a = 2 - c2.commit() - c1.get_root()['B'].b = 1 # touch B in c1 - # A was not accessed before the reference was lost, - # so there is no conflict. - c1.commit() - - def test_check_persistentbase_refs(self): - refs = getattr(sys, 'gettotalrefcount', None) - if refs is None: - return - before = 0 - after = 0 - before = refs() - PersistentBase() - after = refs() - assert after - before == 0, after - before - - def test_check_connectionbase_refs(self): - refs = getattr(sys, 'gettotalrefcount', None) - if refs is None: - return - before = 0 - after = 0 - before = refs() - ConnectionBase() - after = refs() - assert after - before == 0, after - before +if not 'SKIP_SLOW' in os.environ: + class TestConnectionClientStorage (TestConnection): + + def setUp(self): + self.port = 9123 + self.server = popen4('python %s --port=%s' % ( + run_durus.__file__, self.port)) + sleep(3) # wait for bind + + def tearDown(self): + run_durus.stop_durus((DEFAULT_HOST, self.port)) + + def _get_storage(self): + return ClientStorage(port=self.port) + + def test_check_conflict(self): + b = Connection(self._get_storage()) + c = Connection(self._get_storage()) + rootb = b.get(p64(0)) + rootb['b'] = Persistent() + rootc = c.get(p64(0)) + rootc['c'] = Persistent() + c.commit() + assert raises(ConflictError, b.commit) + assert raises(KeyError, rootb.__getitem__, 'c') + transaction_serial = b.transaction_serial + b.abort() + assert b.get_transaction_serial() > transaction_serial + assert rootb._p_is_ghost() + rootc['d'] = Persistent() + c.commit() + rootb['d'] + + def test_check_fine_conflict(self): + c1 = Connection(self._get_storage()) + c2 = Connection(self._get_storage()) + c1.get_root()['A'] = Persistent() + c1.get_root()['A'].a = 1 + c1.get_root()['B'] = Persistent() + c1.commit() + c2.abort() + # c1 has A loaded. + assert not c1.get_root()['A']._p_is_ghost() + c1.get_root()['B'].b = 1 + c2.get_root()['A'].a = 2 + c2.commit() + # Even though A has been changed by c2, + # c1 has not accessed an attribute of A since + # the last c1.commit(), so we don't want a ConflictError. + c1.commit() + assert not c1.get_root()['A']._p_is_ghost() + c1.get_root()['A'].a # accessed! + c1.get_root()['B'].b = 1 + c2.get_root()['A'].a = 2 + c2.commit() + assert raises(ConflictError, c1.commit) + + def _scenario(self): + c1 = Connection(self._get_storage()) + c2 = Connection(self._get_storage()) + c1.get_root()['A'] = Persistent() + c1.get_root()['B'] = Persistent() + c1.get_root()['A'].a = 1 + c1.commit() + c2.abort() + c1.cache.recent_objects.discard(c1.get_root()['A']) + # Imagine c1 has been running for a while, and + # cache management, for example, has caused the + # cache reference to be weak. + return c1, c2 + + def test_conflict_from_invalid_removable_previously_accessed(self): + c1, c2 = self._scenario() + A = c1.get_root()['A'] + A.a # access A in c1. This will lead to conflict. + A_oid = A._p_oid + assert A in c1.cache.recent_objects + A = None # forget about it + print + # Lose the reference to A. + c1.get_root()._p_set_status_ghost() + # Commit a new A in c2. + c2.get_root()['A'].a = 2 + c2.commit() + c1.get_root()['B'].b = 1 # touch B in c1 + # Conflict because A has been accessed in c1, but it is invalid. + assert raises(ConflictError, c1.commit) + + def test_no_conflict_from_invalid_removable_not_previously_accessed(self): + c1, c2 = self._scenario() + c1.get_root()._p_set_status_ghost() + # Commit a new A in c2. + c2.get_root()['A'].a = 2 + c2.commit() + c1.get_root()['B'].b = 1 # touch B in c1 + # A was not accessed before the reference was lost, + # so there is no conflict. + c1.commit() + + def test_check_persistentbase_refs(self): + refs = getattr(sys, 'gettotalrefcount', None) + if refs is None: + return + before = 0 + after = 0 + before = refs() + PersistentBase() + after = refs() + assert after - before == 0, after - before + + def test_check_connectionbase_refs(self): + refs = getattr(sys, 'gettotalrefcount', None) + if refs is None: + return + before = 0 + after = 0 + before = refs() + ConnectionBase() + after = refs() + assert after - before == 0, after - before tests/test_store_connection.py @@ -1,342 +1,342 @@ -""" -$URL: svn+ssh://svn/repos/trunk/durus/test/utest_persistent_set.py $ -$Id: utest_persistent_set.py 27549 2005-10-12 18:45:22Z dbinger $ -""" -from schevo.test import raises -from schevo.store.persistent_set import PersistentSet - -set_type = None -other_type = None - -class Base(object): - - def setUp(self): - global set_type, other_type - set_type = self.set_type - other_type = self.other_type - - def tearDown(self): - global set_type, other_type - del set_type - del other_type - - def test__and__(self): - s1 = set_type() - s2 = other_type() - assert set(s1 & s2) == set() - s2 = other_type(['k']) - assert set(s1 & s2) == set(s1) - s1 = set_type(['j', 'k']) - assert set(s1 & s2) == set(s2) - - def test__cmp__(self): - s1 = set_type() - s2 = other_type() - assert raises(TypeError, s1.__cmp__, s2) - - def test__contains__(self): - s1 = set_type() - s2 = other_type(['j']) - assert s2.__contains__('j') - assert 'j' in s2 - - def test__eq__(self): - s1 = set_type() - s2 = other_type() - if set_type != other_type: - assert not s1.__eq__(s2) - return - assert s1 == s2 - assert s1.__eq__(s2) - s2 = other_type(['k']) - assert not (s1 == s2) - s1 = set_type(['k']) - assert s1 == s2 - s2 = other_type(['j', 'k']) - assert not (s1 == s2) - - def test__ge__(self): - s1 = set_type() - s2 = other_type() - if set_type != other_type: - assert raises(TypeError, s1.__ge__, s2) - return - assert s1 >= s2 - s2 = other_type(['k']) - assert not (s1 >= s2) - assert s2 >= s1 - s1 = set_type(['k']) - assert s1 >= s2 - assert s1.__ge__(s2) - assert s2 >= s1 - s2 = other_type(['j', 'k']) - assert not (s1 >= s2) - assert s2 >= s1 - - def test__gt__(self): - s1 = set_type() - s2 = other_type() - if set_type != other_type: - assert raises(TypeError, s1.__gt__, s2) - return - assert not (s1 > s2) - s2 = other_type(['k']) - assert not (s1 > s2) - assert s2 > s1 - assert s2.__gt__(s1) - s1 = set_type(['k']) - assert not (s1 > s2) - assert not (s2 > s1) - s2 = other_type(['j', 'k']) - assert not (s1 > s2) - assert s2 > s1 - - def test__iand__(self): - s1 = set_type() - s2 = other_type() - print s1, s2, s1.__iand__(s2) - assert s1.__iand__(s2) == set_type() - s2 = other_type(['k']) - assert s1.__iand__(s2) == s1 - assert s1 == set_type() - s1 = set_type(['j', 'k']) - s1 &= s2 - assert set(s1) == set(s2) - assert set(s1.__iand__(s2)) == set(s2) - assert set(s1) == set(s2) - - def test__ior__(self): - s1 = set_type() - s2 = other_type() - assert s1.__ior__(s2) == set_type() - s2 = other_type(['k']) - s1 |= s2 - assert set(s1) == set(s2) - s1 = set_type(['j', 'k']) - assert s1.__ior__(s2) == s1 - s3 = set_type(['g']) - assert s1.__ior__(s3) == set_type(['j', 'k', 'g']) - assert s1 == set_type(['j', 'k', 'g']) - - def test__isub__(self): - s1 = set_type() - s2 = other_type() - assert s1.__isub__(s2) == s1 - assert s1 == set_type() - s2 = other_type(['k']) - s1 -= s2 - assert s1 == set_type() - s1 = set_type(['j', 'k']) - s3 = set_type(['j']) - assert s1.__isub__(s2) == s3 - assert s1 == s3 - - def test__iter__(self): - s1 = set_type() - assert set_type(list(s1)) == s1 - s1 = set_type('abc') - assert set_type(list(s1)) == s1 - - def test__ixor__(self): - s1 = set_type('abc') - s2 = other_type('cfg') - s1.__ixor__(s2) - assert s1 == set_type('abfg') - s1 = set_type('abc') - s2 = other_type('cfg') - s1 ^= s2 - assert s1 == set_type('abfg') - s1 ^= set_type() - assert s1 == set_type('abfg') - - def test__le__(self): - s1 = set_type() - s2 = other_type() - if set_type != other_type: - assert raises(TypeError, s1.__le__, s2) - return - assert s1 <= s2 - s2 = other_type(['k']) - assert not (s2 <= s1) - assert s1 <= s2 - s1 = set_type(['k']) - assert s1 <= s2 - assert s2 <= s1 - s2 = other_type(['j', 'k']) - assert not (s2 <= s1) - assert s1 <= s2 - - def test_len(self): - s = set_type() - assert len(s) == 0 - s = set_type([]) - assert len(s) == 0 - s = set_type(['a']) - assert len(s) == 1 - - def test__lt__(self): - s1 = set_type() - s2 = other_type() - if set_type != other_type: - assert raises(TypeError, s1.__lt__, s2) - return - assert not (s1 < s2) - s2 = other_type(['k']) - assert not (s2 < s1) - assert s1 < s2 - s1 = set_type('k') - assert not (s1 < s2) - s2 = other_type('jk') - assert not (s2 < s1) - assert s1 < s2 - - def test__ne__(self): - s1 = set_type() - s2 = other_type() - if set_type != other_type: - assert s1.__ne__(s2) - return - assert not s1.__ne__(s2) - s3 = set_type('a') - assert s1 != s3 - assert s3 != s1 - assert s1 <> s3 - - def test__or__(self): - s1 = set_type() - s2 = other_type() - assert s1 | s2 == set_type() - s3 = set_type('ab') - assert s1 | s3 == s3 - s4 = set_type('bc') - assert s3 | s4 == set_type('abc') - - def test__ror__(self): - s1 = set_type() - s2 = other_type() - assert s1.__ror__(s2) == set_type() - s3 = set_type('ab') - assert s1.__ror__(s3) == s3 - s4 = set_type('bc') - assert s3.__ror__(s4) == set_type('abc') - - def test__rand__(self): - s1 = set_type() - s2 = other_type() - assert s1.__rand__(s2) == set_type() - s3 = set_type('ab') - assert s1.__rand__(s3) == set_type() - s4 = set_type('bc') - assert s3.__rand__(s4) == set_type('b') - - def test__rsub__(self): - s1 = set_type() - s2 = other_type() - assert s1.__rsub__(s2) == set_type() - s3 = set_type('ab') - assert s1.__rsub__(s3) == s3 - assert s3.__rsub__(s1) == set_type() - s4 = set_type('bc') - assert s3.__rsub__(s4) == set_type('c') - - def test__rxor__(self): - s1 = set_type() - s2 = other_type() - assert s1.__rxor__(s2) == set_type() - s3 = set_type('ab') - assert s1.__rxor__(s3) == s3 - assert s3.__rxor__(s1) == s3 - s4 = set_type('bc') - assert s3.__rxor__(s4) == set_type('ac') - - def test__sub__(self): - s1 = set_type() - s2 = other_type() - assert s1 - s2 == set_type() - s3 = set_type('ab') - assert s3 - s1 == s3 - assert s1 - s3 == set_type() - s4 = set_type('bc') - assert s4 - s3 == set_type('c') - - def test__xor__(self): - s1 = set_type() - s2 = other_type() - assert s1 ^ s2 == set_type() - s3 = set_type('ab') - assert s1 ^ s3 == s3 - assert s3 ^ s1 == s3 - s4 = set_type('bc') - assert s3 ^ s4 == set_type('ac') - - def test_add(self): - s1 = set_type() - s1.add(1) - assert s1 == set_type([1]) - s1.add(1) - assert s1 == set_type([1]) - s1.add(2) - assert s1 == set_type([1, 2]) - - def test_clear(self): - s1 = set_type() - s1.clear() - assert s1 == set_type() - s1 = set_type('asdf') - s1.clear() - assert s1 == set_type() - - def test_copy(self): - s1 = set_type() - s2 = s1.copy() - assert s1 is not s2 - assert s1 == s2 - assert type(s1) is type(s2) - s1 = set_type('asdf') - s2 = s1.copy() - assert s1 is not s2 - assert s1 == s2 - assert type(s1) is type(s2) - - def test_discard(self): - s1 = set_type() - s1.discard(1) - assert s1 == set_type() - s1 = set_type('asdf') - s1.discard(1) - s1.discard('a') - assert s1 == set_type('sdf') - - def test_pop(self): - assert raises(KeyError, set_type().pop) - s1 = set_type('asdf') - x = s1.pop() - assert x not in s1 - assert len(s1) == 3 - assert (s1 | set_type(x)) == set_type('asdf') - - def test_remove(self): - s1 = set_type() - assert raises(KeyError, s1.remove, 1) - assert s1 == set_type() - s1 = set_type('asdf') - s1.remove('a') - assert s1 == set_type('sdf') - - -class TestSet_Set(Base): - - set_type = set - other_type = set - - -class TestPersistentSet_Set(Base): - - set_type = PersistentSet - other_type = set - - -class TestPersistentSet_PersistentSet(Base): - - set_type = PersistentSet - other_type = PersistentSet +""" +$URL: svn+ssh://svn/repos/trunk/durus/test/utest_persistent_set.py $ +$Id: utest_persistent_set.py 27549 2005-10-12 18:45:22Z dbinger $ +""" +from schevo.test import raises +from schevo.store.persistent_set import PersistentSet + +set_type = None +other_type = None + +class Base(object): + + def setUp(self): + global set_type, other_type + set_type = self.set_type + other_type = self.other_type + + def tearDown(self): + global set_type, other_type + del set_type + del other_type + + def test__and__(self): + s1 = set_type() + s2 = other_type() + assert set(s1 & s2) == set() + s2 = other_type(['k']) + assert set(s1 & s2) == set(s1) + s1 = set_type(['j', 'k']) + assert set(s1 & s2) == set(s2) + + def test__cmp__(self): + s1 = set_type() + s2 = other_type() + assert raises(TypeError, s1.__cmp__, s2) + + def test__contains__(self): + s1 = set_type() + s2 = other_type(['j']) + assert s2.__contains__('j') + assert 'j' in s2 + + def test__eq__(self): + s1 = set_type() + s2 = other_type() + if set_type != other_type: + assert not s1.__eq__(s2) + return + assert s1 == s2 + assert s1.__eq__(s2) + s2 = other_type(['k']) + assert not (s1 == s2) + s1 = set_type(['k']) + assert s1 == s2 + s2 = other_type(['j', 'k']) + assert not (s1 == s2) + + def test__ge__(self): + s1 = set_type() + s2 = other_type() + if set_type != other_type: + assert raises(TypeError, s1.__ge__, s2) + return + assert s1 >= s2 + s2 = other_type(['k']) + assert not (s1 >= s2) + assert s2 >= s1 + s1 = set_type(['k']) + assert s1 >= s2 + assert s1.__ge__(s2) + assert s2 >= s1 + s2 = other_type(['j', 'k']) + assert not (s1 >= s2) + assert s2 >= s1 + + def test__gt__(self): + s1 = set_type() + s2 = other_type() + if set_type != other_type: + assert raises(TypeError, s1.__gt__, s2) + return + assert not (s1 > s2) + s2 = other_type(['k']) + assert not (s1 > s2) + assert s2 > s1 + assert s2.__gt__(s1) + s1 = set_type(['k']) + assert not (s1 > s2) + assert not (s2 > s1) + s2 = other_type(['j', 'k']) + assert not (s1 > s2) + assert s2 > s1 + + def test__iand__(self): + s1 = set_type() + s2 = other_type() + print s1, s2, s1.__iand__(s2) + assert s1.__iand__(s2) == set_type() + s2 = other_type(['k']) + assert s1.__iand__(s2) == s1 + assert s1 == set_type() + s1 = set_type(['j', 'k']) + s1 &= s2 + assert set(s1) == set(s2) + assert set(s1.__iand__(s2)) == set(s2) + assert set(s1) == set(s2) + + def test__ior__(self): + s1 = set_type() + s2 = other_type() + assert s1.__ior__(s2) == set_type() + s2 = other_type(['k']) + s1 |= s2 + assert set(s1) == set(s2) + s1 = set_type(['j', 'k']) + assert s1.__ior__(s2) == s1 + s3 = set_type(['g']) + assert s1.__ior__(s3) == set_type(['j', 'k', 'g']) + assert s1 == set_type(['j', 'k', 'g']) + + def test__isub__(self): + s1 = set_type() + s2 = other_type() + assert s1.__isub__(s2) == s1 + assert s1 == set_type() + s2 = other_type(['k']) + s1 -= s2 + assert s1 == set_type() + s1 = set_type(['j', 'k']) + s3 = set_type(['j']) + assert s1.__isub__(s2) == s3 + assert s1 == s3 + + def test__iter__(self): + s1 = set_type() + assert set_type(list(s1)) == s1 + s1 = set_type('abc') + assert set_type(list(s1)) == s1 + + def test__ixor__(self): + s1 = set_type('abc') + s2 = other_type('cfg') + s1.__ixor__(s2) + assert s1 == set_type('abfg') + s1 = set_type('abc') + s2 = other_type('cfg') + s1 ^= s2 + assert s1 == set_type('abfg') + s1 ^= set_type() + assert s1 == set_type('abfg') + + def test__le__(self): + s1 = set_type() + s2 = other_type() + if set_type != other_type: + assert raises(TypeError, s1.__le__, s2) + return + assert s1 <= s2 + s2 = other_type(['k']) + assert not (s2 <= s1) + assert s1 <= s2 + s1 = set_type(['k']) + assert s1 <= s2 + assert s2 <= s1 + s2 = other_type(['j', 'k']) + assert not (s2 <= s1) + assert s1 <= s2 + + def test_len(self): + s = set_type() + assert len(s) == 0 + s = set_type([]) + assert len(s) == 0 + s = set_type(['a']) + assert len(s) == 1 + + def test__lt__(self): + s1 = set_type() + s2 = other_type() + if set_type != other_type: + assert raises(TypeError, s1.__lt__, s2) + return + assert not (s1 < s2) + s2 = other_type(['k']) + assert not (s2 < s1) + assert s1 < s2 + s1 = set_type('k') + assert not (s1 < s2) + s2 = other_type('jk') + assert not (s2 < s1) + assert s1 < s2 + + def test__ne__(self): + s1 = set_type() + s2 = other_type() + if set_type != other_type: + assert s1.__ne__(s2) + return + assert not s1.__ne__(s2) + s3 = set_type('a') + assert s1 != s3 + assert s3 != s1 + assert s1 <> s3 + + def test__or__(self): + s1 = set_type() + s2 = other_type() + assert s1 | s2 == set_type() + s3 = set_type('ab') + assert s1 | s3 == s3 + s4 = set_type('bc') + assert s3 | s4 == set_type('abc') + + def test__ror__(self): + s1 = set_type() + s2 = other_type() + assert s1.__ror__(s2) == set_type() + s3 = set_type('ab') + assert s1.__ror__(s3) == s3 + s4 = set_type('bc') + assert s3.__ror__(s4) == set_type('abc') + + def test__rand__(self): + s1 = set_type() + s2 = other_type() + assert s1.__rand__(s2) == set_type() + s3 = set_type('ab') + assert s1.__rand__(s3) == set_type() + s4 = set_type('bc') + assert s3.__rand__(s4) == set_type('b') + + def test__rsub__(self): + s1 = set_type() + s2 = other_type() + assert s1.__rsub__(s2) == set_type() + s3 = set_type('ab') + assert s1.__rsub__(s3) == s3 + assert s3.__rsub__(s1) == set_type() + s4 = set_type('bc') + assert s3.__rsub__(s4) == set_type('c') + + def test__rxor__(self): + s1 = set_type() + s2 = other_type() + assert s1.__rxor__(s2) == set_type() + s3 = set_type('ab') + assert s1.__rxor__(s3) == s3 + assert s3.__rxor__(s1) == s3 + s4 = set_type('bc') + assert s3.__rxor__(s4) == set_type('ac') + + def test__sub__(self): + s1 = set_type() + s2 = other_type() + assert s1 - s2 == set_type() + s3 = set_type('ab') + assert s3 - s1 == s3 + assert s1 - s3 == set_type() + s4 = set_type('bc') + assert s4 - s3 == set_type('c') + + def test__xor__(self): + s1 = set_type() + s2 = other_type() + assert s1 ^ s2 == set_type() + s3 = set_type('ab') + assert s1 ^ s3 == s3 + assert s3 ^ s1 == s3 + s4 = set_type('bc') + assert s3 ^ s4 == set_type('ac') + + def test_add(self): + s1 = set_type() + s1.add(1) + assert s1 == set_type([1]) + s1.add(1) + assert s1 == set_type([1]) + s1.add(2) + assert s1 == set_type([1, 2]) + + def test_clear(self): + s1 = set_type() + s1.clear() + assert s1 == set_type() + s1 = set_type('asdf') + s1.clear() + assert s1 == set_type() + + def test_copy(self): + s1 = set_type() + s2 = s1.copy() + assert s1 is not s2 + assert s1 == s2 + assert type(s1) is type(s2) + s1 = set_type('asdf') + s2 = s1.copy() + assert s1 is not s2 + assert s1 == s2 + assert type(s1) is type(s2) + + def test_discard(self): + s1 = set_type() + s1.discard(1) + assert s1 == set_type() + s1 = set_type('asdf') + s1.discard(1) + s1.discard('a') + assert s1 == set_type('sdf') + + def test_pop(self): + assert raises(KeyError, set_type().pop) + s1 = set_type('asdf') + x = s1.pop() + assert x not in s1 + assert len(s1) == 3 + assert (s1 | set_type(x)) == set_type('asdf') + + def test_remove(self): + s1 = set_type() + assert raises(KeyError, s1.remove, 1) + assert s1 == set_type() + s1 = set_type('asdf') + s1.remove('a') + assert s1 == set_type('sdf') + + +class TestSet_Set(Base): + + set_type = set + other_type = set + + +class TestPersistentSet_Set(Base): + + set_type = PersistentSet + other_type = set + + +class TestPersistentSet_PersistentSet(Base): + + set_type = PersistentSet + other_type = PersistentSet tests/test_store_persistent_set.py @@ -4,14 +4,14 @@ For copyright, license, and warranty, see bottom of file. """ from schevo.test import CreatesSchema, raises +from schevo.constant import UNASSIGNED from schevo import error from schevo import field from schevo.label import label +from schevo.placeholder import Placeholder from schevo import transaction -from schevo.constant import UNASSIGNED - -class TestTransaction(CreatesSchema): +class BaseTransaction(CreatesSchema): body = ''' @@ -628,7 +628,7 @@ class TestTransaction(CreatesSchema): assert result1.sys.rev == 1 assert result1.name == 'bar' assert result1 == result2 - + def test_update_cannot_skip_revisions(self): tx = db.User.t.create(name='foo') user = db.execute(tx) @@ -653,6 +653,19 @@ class TestTransaction(CreatesSchema): assert result.name == 'Female' assert result.count == 0 + def test_update_entities(self): + male = db.execute(db.Gender.t.create(code='M', name='Male')) + female = db.execute(db.Gender.t.create(code='F', name='Female')) + p = db.execute(db.Person.t.create(gender=male, name='Some person')) + assert p.gender == male + self.internal_update_entities_1(expected=male) + db.execute(p.t.update(gender=female)) + assert p.gender == female + self.internal_update_entities_1(expected=female) + + def internal_update_entities_1(self, expected): + raise NotImplementedError() + def test_delete_simple(self): # Create something that we can delete. tx = db.User.t.create(name='foo') @@ -777,7 +790,7 @@ class TestTransaction(CreatesSchema): assert len(db.User) == 2 assert db.User[1].name == 'foo' assert db.User[2].name == 'bar' - + def test_nested_rollback(self): assert len(db.User) == 0 # Execute a transaction that fails. @@ -797,7 +810,7 @@ class TestTransaction(CreatesSchema): # Executing the transaction again is not allowed. assert raises(error.TransactionAlreadyExecuted, db.execute, tx) assert len(db.User) == 1 - + def test_sys_executed(self): """You can tell if a transaction has been executed yet by checking its sys.executed flag.""" @@ -1081,7 +1094,7 @@ class TestTransaction(CreatesSchema): assert delete.sys.count == realm.sys.count update = realm.t.update() assert update.sys.count == realm.sys.count - + def test_callable_wrapper(self): exe = db.execute assert len(db.User) == 0 @@ -1096,7 +1109,56 @@ class TestTransaction(CreatesSchema): exe(db.User.t.create(name='bar')) exe(fn2) assert len(db.User) == 2 - + + +class TestTransaction1(BaseTransaction): + + format = 1 + + def internal_update_entities_1(self, expected): + person_entity = db.Person.findone(name='Some person') + root = db._root + schevo = root['SCHEVO'] + extent_name_id = schevo['extent_name_id'] + extents = schevo['extents'] + Gender_extent_id = extent_name_id['Gender'] + Person_extent_id = extent_name_id['Person'] + Gender_extent = extents[Gender_extent_id] + Person_extent = extents[Person_extent_id] + Gender_field_name_id = Gender_extent['field_name_id'] + Person_field_name_id = Person_extent['field_name_id'] + # Check for p.gender having correct field values. + p = Person_extent['entities'][person_entity.sys.oid] + p_fields = p['fields'] + Person_gender_field_id = Person_field_name_id['gender'] + p_gender = p_fields[Person_gender_field_id] + assert p_gender == (Gender_extent_id, expected.sys.oid) + + +class TestTransaction2(BaseTransaction): + + format = 2 + + def internal_update_entities_1(self, expected): + person_entity = db.Person.findone(name='Some person') + root = db._root + schevo = root['SCHEVO'] + extent_name_id = schevo['extent_name_id'] + extents = schevo['extents'] + Gender_extent_id = extent_name_id['Gender'] + Person_extent_id = extent_name_id['Person'] + Gender_extent = extents[Gender_extent_id] + Person_extent = extents[Person_extent_id] + Gender_field_name_id = Gender_extent['field_name_id'] + Person_field_name_id = Person_extent['field_name_id'] + # Check for p.gender having correct related entity structures. + p = Person_extent['entities'][person_entity.sys.oid] + p_related_entities = p['related_entities'] + Person_gender_field_id = Person_field_name_id['gender'] + p_related_genders = p_related_entities[Person_gender_field_id] + expected_p_related_genders = frozenset([Placeholder(expected)]) + assert p_related_genders == expected_p_related_genders + # Copyright (C) 2001-2006 Orbtech, L.L.C. # tests/test_transaction.py @@ -6,7 +6,7 @@ For copyright, license, and warranty, see bottom of file. from schevo.test import CreatesSchema -class TestTransaction(CreatesSchema): +class BaseTransaction(CreatesSchema): body = ''' @@ -31,6 +31,16 @@ class TestTransaction(CreatesSchema): assert list(tx.f) == ['field1', 'field3', 'field2'] +class TestTransaction1(BaseTransaction): + + format = 1 + + +class TestTransaction2(BaseTransaction): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_transaction_field_reorder.py @@ -7,7 +7,7 @@ from schevo.test import CreatesSchema, raises from schevo import error -class TestTransactionRequireChanges(CreatesSchema): +class BaseTransactionRequireChanges(CreatesSchema): body = ''' @@ -36,7 +36,7 @@ class TestTransactionRequireChanges(CreatesSchema): # No error is raised when changes are made. b = db.execute(b.t.update(bar=2)) assert b.bar == 2 - + def test_do_not_require_changes(self): """Update transaction subclasses that set _require_changes to False succeed if no changes have been made to any fields.""" @@ -45,6 +45,16 @@ class TestTransactionRequireChanges(CreatesSchema): assert a.foo == 1 +class TestTransactionRequireChanges1(BaseTransactionRequireChanges): + + format = 1 + + +class TestTransactionRequireChanges2(BaseTransactionRequireChanges): + + format = 2 + + # Copyright (C) 2001-2006 Orbtech, L.L.C. # # Schevo tests/test_transaction_require_changes.py @@ -9,7 +9,7 @@ from schevo.label import label from schevo.test import CreatesSchema -class TestView(CreatesSchema): +class BaseView(CreatesSchema): body = ''' @@ -104,7 +104,17 @@ class TestView(CreatesSchema): assert ea_view.f.unicode.readonly assert ea_view.f.integer.readonly assert ea_view.f.float.readonly - + + +class TestView1(BaseView): + + format = 1 + + +class TestView2(BaseView): + + format = 2 + # Copyright (C) 2001-2006 Orbtech, L.L.C. # tests/test_view.py schevo/example/videostore/__init__.py schevo/example/videostore/schema/__init__.py schevo/example/videostore/schema/schema_001.py tests/test_doctests.py tests/test_export_all.py 195a8fc3b78b3c6c371df58de9b031e9215f6f1a Matthew R. Scott gldnspud@gmail.com http://github.com/11craft/schevo/commit/607d176f53436dc55ee74ce66e9413b51435b9bc 607d176f53436dc55ee74ce66e9413b51435b9bc 2007-04-09T18:11:05-07:00 2007-04-09T18:11:05-07:00 Merging ticket #43 - fieldstates branch - to trunk. Field classes: * Reorder min_- and max_-prefixed attributes of fields to prefer min/max ordering rather than alphabetical max/min ordering. * Add a new may_store_entities attribute that indicates whether a field may store references to entities in its value. * Add _dump method to return value suitable for database storage. * Add _restore method to convert value from database storage to true field value. * Add _entities_in_value method to return a set of Placeholder instances for entities contained in the field's value. * New field class _EntityBase with code common to the Entity and EntityList field class, as well as any custom field classes that may store entities. * New field class EntityList stores a list of entity instances. Bidirectional link and referential integrity features are handled by the database as with single-value Entity fields. * Code that calls the `convert` method on field instances now always passes the database in the method call when available in the call context. Entities: * Calculated fields are now expected to return the proper data type; return values are no longer filtered through a behind-the-scenes call to the field class's convert method. Schema definition: * schevo.field.FieldMeta now attaches a lowercase_with_underscore version of a field's class name to the 'f' field factory namespace, rather than a lowercaseCamelCase version. This means that if you have a custom field class called {{{CustomField}}}, you will need to create fields with that by using {{{fieldname = f.custom_field()}}}, instead of the now-deprecated {{{fieldname = f.customField()}}} manner. This is to provide consistency with lowercase names in other parts of the Schevo API, such as 'create_if_necessary'. * lowercaseCamelCase names of field factories are still allowed in Schevo 3.1, but their use is discouraged and shows a DeprecationWarning on each use. New format 2 database engine: * Default database engine for new databases, if no format is specified. * Entity field values are no longer stored as tuples; rather, they are stored as schevo.placeholder.Placeholder instances. This takes place in each entity's 'fields' structure, and also in the index trees in each extent. * Each entity now has a related_entities structure to store sets of Placeholder instances returned by calling fields' _entities_in_value methods. * entity_field_ids internal structure is maintained for backward-compatibility, but is no longer used for special-case checking for entity fields. Backward-compatible format 1 database engine: * Subclasses the format 2 engine. * Does not support any may_store_entities field classes other than the Entity class itself. UnsupportedFieldType error is raised if use of one is attempted. Database format conversion: * schevo.database.convert_format for programmatically converting a database file or file-like object from format 1 to format 2. * "schevo db convert" for doing format conversions on database files using the command line. Common database operations: * schevo.database.open inspects the format of an existing database, and chooses the appropriate engine based on what it finds. * schevo.database.open allows you to specify a format for new databases, in case you want to create a new format 1 database. * Database classes in general now have a new private method, _schema_format_compatibility_check, which takes a SchemaDefinition instance as its argument and ensures compatibility between the schema and the database engine. This will be useful in the future for further style and correctness checking, and perhaps to allow more simplified schemata to be used with backends that use something other than an object database, such as an SQLite-based backend. New Placeholder class: * Based on the now-defunct schevo.entity.EntityRef class. * Used to refer to an entity in a persisted field value. * Stores the ID of the entity's extent and the OID of the entity. Unit tests: * 'tron' and 'troff' convenience functions are now injected into a test module's namespace, to make it easy to turn schevo.trace monitoring on and off while writing and debugging test cases. * Specify SKIP_SLOW=1 in your environment to skip slow schevo.store unit tests. This is useful during development of Schevo core. Before committing with the assumption that all tests pass, remember to run "nosetests" without using SKIP_SLOW=1. * More tests of internal database structures, along with the usual high-level public API tests. * For each test that is applicable to both format 1 and format 2 databases, run the test against *each* database engine. Documentation: * Docstring expansions in a few areas. * Tweaks to movie review tutorial installment 1. * Beginning of movie review tutorial installment 2. (currently incomplete) Movie reviews tutorial: * Tweaks to installment 1. * Beginning of installment 2. SchevoXml: * SchevoXml is now defunct and has been removed; it was not used in any production scenario, and required too much change to be compatible with Schevo 3.1. The effort required to make it compatible would not have resulted in any productive gain. Code style: * Whitespace cleanup across the entire code base. 3bc2696a9095d09c6807e4c1b9e551ac0c31e4cc Matthew R. Scott gldnspud@gmail.com