Feat/update triggers

[whatnick]

Reason for this pull request

Add updated column + triggers to core postgresql tables to ease incremental Datacube federation.

Proposed changes

• Add updated column to all tables

• Add triggers on these columns to capture update query timestamps.

• Closes Datacube Database Federation and ETL using timestamps #947

• Tests added / passed

• Fully documented, including docs/about/whats_new.rst for all changes

[codecov]

Codecov Report

Merging #951 into develop will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #951      +/-   ##
===========================================
+ Coverage    92.96%   92.98%   +0.01%     
===========================================
  Files           96       97       +1     
  Lines         9630     9648      +18     
===========================================
+ Hits          8953     8971      +18     
  Misses         677      677              

Impacted Files	Coverage Δ
datacube/drivers/postgres/_core.py	95.60% <100.00%> (+0.31%)	⬆️
datacube/drivers/postgres/_triggers.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 155bc5c...3fbe7f3. Read the comment docs.

[jeremyh]

Nice work 👍

Minor optional comment below

[jeremyh]

This is going to log every time they run init, even if they already have applied this update.

By convention, I usually wrapped them in a check previously:

if not pg_column_exists(engine, schema_qualified('dataset'), 'updated'):
    _LOG.info("Adding Update Triggers")
    with engine.connect() as c:
        c.execute('begin')
        install_timestamp_trigger(c)
        .... etc

(of course, no need to check other columns exist as they're all applied together in one transaction.)

The actual SQL looks idempotent so this is all an extremely minor gripe.

[jeremyh]

For these sort of db-specific changes I prefer raw sql. Not a hack in my view 🙂 (it's also what we did in the past)

[Kirill888]

I'm not familiar with DB side of things, so can't really review in a meaningfull way. But what does this mean from user point of view?

• Is this some manual step users need to run?
• What happens when new code runs against old DB?
• What happens when new code runs against old DB and you are in read-only mode?
• What happens when OLD code runs against NEW DB?

[whatnick]

I'm not familiar with DB side of things, so can't really review in a meaningfull way. But what does this mean from user point of view?

• Is this some manual step users need to run?
• What happens when new code runs against old DB?
• What happens when new code runs against old DB and you are in read-only mode?
• What happens when OLD code runs against NEW DB?

• Manual step as part of datacube system init
• New Datacube load code is unaffected, when datacube system init is run against old db, new columns here get added.
• Running against old DB in read-only mode for datacube system init fails.
• Old code reading datacube still works against new DB.

[Kirill888]

If it's just for init then write permissions are assumed anyway