JSON decoding/encoding, dumpers/loaders, extensions and system fields

[lnielsen]

• Adds support for JSON encoding/decoding to/from the database. This allows e.g. have records with complex data types such as datetime objects. JSONSchema validation happens on the JSON encoded version of the record.

• Adds dumpers to support dumping and loading records from secondary copies (e.g. records stored in an Elasticsearch index).

• Adds support record extensions as a more strict replacement of signals. Allows writing extensions (like the system fields), that integrate into the Records API.

• Adds support for system fields, that are Python data descriptors on the Record which allows for managed access to the Record's dictionary.

• Adds support for disabling signals. Some signals can never have been used as they don't make sense (e.g. for revert()).

Backward compatibility note

• All changes are optional add-ons. The Record class behaves as previously. If you have subclassed Record, and in particular if you have overwritten the validate() method, you may need to make slight changes to your record.

A TL;DR example:

# Don't try to run it :-) 
# - Imports missing 
# - Some classes are imaginary (e.g. GeoLocationsExt)

class DateEncoder:
    def __init__(self, *keys):
        self.keys = keys

    def encode(self, data):
        for k in self.keys:
            if k in data:
                data[k] = data[k].isoformat()

    def decode(self, data):
        for k in self.keys:
            if k in data:
                s = data[k]
                year, month, day = int(s[0:4]), int(s[5:7]), int(s[8:10])
                data[k] = date(year, month, date)

class MyRecordMetadata(RecordMetadataBase):
     __table_name__ = 'myrecord_metadata'
     encoder = DateEncoder('pubdate')
     expires_at = expires_at = db.Column(db.DateTime(), nullable=False)

class MyRecord(Record, SystemFieldsMixin):
    schema = ConstantField('$schema', 'https://.../records/record-v1.0.0.json')
    expires_at = ModelField()
    metadata = DictField(clear_none=True)
    model_cls = MyRecordMetadata
    send_signals = False
    enable_jsonref = False
    dumper = ElasticsearchDumper(
        extensions=[
            GeoLocationExt('locations')
        ],
        models_fields={
            'expires_at': ('expires_at', datetime),
        }
    )

# JSON Encoding
record = MyRecord.create({'pubdate': date(2020, 9, 7)})

# Dumping/loading
dump = MyRecord({}).dumps(dumper=ElasticsearchDumper())
MyRecord.loads(dump, loader=ElasticsearchDumper())

# System fields:
record = MyRecord({})
record.schema == record['$schema']

r = MyRecord.create({}, metadata={'title': 'a', 'a':None}, expires_at=datetime(2020,9,7,0,0))
r.expires_at == r.model.expires_at
r == {'metadata': {'title': 'a'}}

# Cleaning a record:
record.clear_none(key='metadata')

[lnielsen]

Merging as discussed in the architects meeting. Releasing as Invenio 1.4.0a1

[ntarocco]

Is this also for backward compat? Otherwise I would personally try to abandon signals.

[ntarocco]

Why not a post_commit? In ILS, we have a use case that we want to re-fetch book covers from the external provider after a record has been updated.
I guess any way that such use cases will be implemented in invenio-records-resources...

[ntarocco]

can it be that both are None?