Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/update triggers #951

Merged
merged 5 commits into from
May 25, 2020
Merged

Conversation

whatnick
Copy link
Member

@whatnick whatnick commented May 22, 2020

Reason for this pull request

Add updated column + triggers to core postgresql tables to ease incremental Datacube federation.

Proposed changes

@whatnick whatnick requested review from jeremyh and omad May 22, 2020 03:35
@whatnick whatnick self-assigned this May 22, 2020
@whatnick whatnick requested a review from Kirill888 May 22, 2020 03:37
@codecov
Copy link

codecov bot commented May 22, 2020

Codecov Report

Merging #951 into develop will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #951      +/-   ##
===========================================
+ Coverage    92.96%   92.98%   +0.01%     
===========================================
  Files           96       97       +1     
  Lines         9630     9648      +18     
===========================================
+ Hits          8953     8971      +18     
  Misses         677      677              
Impacted Files Coverage Δ
datacube/drivers/postgres/_core.py 95.60% <100.00%> (+0.31%) ⬆️
datacube/drivers/postgres/_triggers.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 155bc5c...3fbe7f3. Read the comment docs.

Copy link
Contributor

@jeremyh jeremyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work 👍

Minor optional comment below


# Post 1.8 DB Federation triggers
from datacube.drivers.postgres._triggers import install_timestamp_trigger
_LOG.info("Adding Update Triggers")
Copy link
Contributor

@jeremyh jeremyh May 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to log every time they run init, even if they already have applied this update.

By convention, I usually wrapped them in a check previously:

if not pg_column_exists(engine, schema_qualified('dataset'), 'updated'):
    _LOG.info("Adding Update Triggers")
    with engine.connect() as c:
        c.execute('begin')
        install_timestamp_trigger(c)
        .... etc

(of course, no need to check other columns exist as they're all applied together in one transaction.)

The actual SQL looks idempotent so this is all an extremely minor gripe.


for name in TABLE_NAMES:
# Add update_at columns
# HACK: Make this more SQLAlchemy with add_column on Table objects
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For these sort of db-specific changes I prefer raw sql. Not a hack in my view 🙂 (it's also what we did in the past)

@Kirill888
Copy link
Member

I'm not familiar with DB side of things, so can't really review in a meaningfull way. But what does this mean from user point of view?

  • Is this some manual step users need to run?
  • What happens when new code runs against old DB?
  • What happens when new code runs against old DB and you are in read-only mode?
  • What happens when OLD code runs against NEW DB?

@whatnick
Copy link
Member Author

I'm not familiar with DB side of things, so can't really review in a meaningfull way. But what does this mean from user point of view?

  • Is this some manual step users need to run?
  • What happens when new code runs against old DB?
  • What happens when new code runs against old DB and you are in read-only mode?
  • What happens when OLD code runs against NEW DB?
  • Manual step as part of datacube system init
  • New Datacube load code is unaffected, when datacube system init is run against old db, new columns here get added.
  • Running against old DB in read-only mode for datacube system init fails.
  • Old code reading datacube still works against new DB.

Copy link
Member

@Kirill888 Kirill888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's just for init then write permissions are assumed anyway

@whatnick whatnick merged commit 0bf503f into opendatacube:develop May 25, 2020
@whatnick whatnick deleted the feat/update_triggers branch May 25, 2020 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Datacube Database Federation and ETL using timestamps
3 participants