Collapse database migrations #3524

ocket8888 · 2019-04-25T18:56:07Z

What does this PR (Pull Request) do?

Collapses all previous migrations into seeds.sql and/or create_tables.sql (as appropriate). It does leave all migrations written since 3.0.x was released.

This PR is not related to any Issue

Which Traffic Control components are affected by this PR?

Documentation
Traffic Ops (DB)

What is the best way to verify this PR?

Building CDN-in-a-Box seems appropriate. That's what I did. Also, try dumping an older schema and comparing it with this. The only (real) difference should be the new latitude/longitude constraints.

If this is a bug fix, what versions of Traffic Ops are affected?

This isn't a bug fix, but on the topic of TO versions: this is a breaking change to the database, meaning rollbacks to migrations that previously existed will no longer be possible.

The following criteria are ALL met by this PR

This PR includes tests OR I have explained why tests are unnecessary
This PR includes documentation OR I have explained why documentation is unnecessary
This PR includes an update to CHANGELOG.md OR such an update is not necessary
This PR includes any and all required license headers
This PR ensures that database migration sequence is correct OR this PR does not include a database migration
This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY

asfgit · 2019-04-25T19:00:16Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3586/
Test PASSed.

asfgit · 2019-04-25T19:05:30Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3587/
Test PASSed.

limited · 2019-04-26T17:17:13Z

What happens if someone is not on the latest minor version before migrating to this version?
I think it is an unreasonable expectation that a user install every version we have released in order as part of an upgrade.

docs/source/admin/traffic_ops/installation.rst

ocket8888 · 2019-04-26T19:03:50Z

You should only have to run an upgrade once to get from any arbitrary minor version to the latest. Unless you mean trying to allow people to go from e.g. v1.x -> v4.x skipping everything in between? That's never been possible afaik.

traffic_ops/app/db/seeds.sql

asfgit · 2019-04-29T14:33:12Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3595/
Test PASSed.

asfgit · 2019-04-29T16:40:24Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3601/
Test PASSed.

asfgit · 2019-04-29T17:26:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3603/
Test PASSed.

asfgit · 2019-05-08T14:38:47Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3637/
Test PASSed.

dangogh · 2019-05-08T16:04:43Z

just resolved the CHANGELOG.md conflict straight from GH

traffic_ops/app/db/migrations/20190424165437_valid_latitudes_and_longitudes.sql

dangogh

there are a number of things that changed in addition to collapsing the migrations. This PR should be limited to only creating the db as if it had gone through the migrations and nothing more.

Any other fixes to tables or data should be addressed in a separate PR. The extra constraint migration should be alone in a separate PR.

ocket8888 · 2019-05-09T13:44:54Z

Traffic Ops will fail to install without any migrations - any change that collapses migrations must also do one of:

Replace Goose with something that won't crash if it has nothing to do
Change admin.go to silently check for migrations before running them, and not attempt to do so if none are found
Modify postinstall to continue even if admin crashes (bad idea IMO)
Add at least one database migration.

ocket8888 · 2019-05-09T15:46:33Z

Instead of introducing a new migration, this now merely leaves in the migrations introduced since the release of 3.0.x as the only remaining migrations after collapse.

asfgit · 2019-05-09T16:02:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3642/
Test PASSed.

asfgit · 2019-05-09T16:03:28Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3643/
Test FAILed.

asfgit · 2019-05-09T16:08:11Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3644/
Test PASSed.

asfgit · 2019-05-20T18:15:34Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3693/
Test PASSed.

asfgit · 2019-05-20T20:47:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3695/
Test PASSed.

asfgit · 2019-05-21T11:27:54Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3698/
Test PASSed.

asfgit · 2019-05-21T12:15:15Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3699/
Test PASSed.

asfgit · 2019-05-28T18:08:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3736/
Test PASSed.

dangogh

Realized that I hadn't run the patch step on previous one, so some diffs I noted before are no longer there.

Couple of things still different that should be fixed.. Still want to run some more data thru it..

traffic_ops/app/db/create_tables.sql

dangogh

still have the last_edited_by error and conflict in CHANGELOG.md that need to be addressed.

…tudes

Ordinarilly this shouldn't matter - enums are meant to be opaque values, but we re-model these enums occasionally in client code.

…ly migration

Also add/removed some missing/incorrect NOT NULL etc. stuff

asfgit · 2019-05-31T15:01:02Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3769/
Test PASSed.

asfgit · 2019-05-31T18:25:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3774/
Test PASSed.

ocket8888 force-pushed the the-great-migration branch from a163858 to 537c744 Compare April 25, 2019 18:57

mitchell852 added Traffic Ops related to Traffic Ops improvement The functionality exists but it could be improved in some way. labels Apr 25, 2019

dangogh reviewed Apr 26, 2019

View reviewed changes

docs/source/admin/traffic_ops/installation.rst Outdated Show resolved Hide resolved

mitchell852 reviewed Apr 26, 2019

View reviewed changes

traffic_ops/app/db/seeds.sql Outdated Show resolved Hide resolved

mitchell852 assigned dangogh May 6, 2019

mitchell852 reviewed May 8, 2019

View reviewed changes

traffic_ops/app/db/migrations/20190424165437_valid_latitudes_and_longitudes.sql Outdated Show resolved Hide resolved

dangogh requested changes May 9, 2019

View reviewed changes

ocket8888 force-pushed the the-great-migration branch from ae8674d to d7c95a3 Compare May 9, 2019 15:42

ocket8888 force-pushed the the-great-migration branch from f91a8bd to 45398a9 Compare May 9, 2019 15:57

ocket8888 force-pushed the the-great-migration branch from 45398a9 to 9074696 Compare May 20, 2019 17:57

ocket8888 force-pushed the the-great-migration branch from 0eefb3a to 28ea824 Compare May 21, 2019 11:42

dangogh requested changes May 30, 2019

View reviewed changes

traffic_ops/app/db/create_tables.sql Outdated Show resolved Hide resolved

traffic_ops/app/db/create_tables.sql Outdated Show resolved Hide resolved

dangogh self-requested a review May 31, 2019 14:33

dangogh requested changes May 31, 2019

View reviewed changes

ocket8888 added 16 commits May 31, 2019 08:51

collapsed migrations

d3c0d8a

Added the first migration - constraints for valid latitudes and longi…

69a2038

…tudes

fixed a comment; more explicit upgrade tip

bd5b262

lat/long value constraints now apply on miss lat/long on DS table

f6c442e

removed portal role from default seeded data set

f0488fc

removed ill-advised default profile type

923b412

restored enums to original ordering

0ddc6e9

Ordinarilly this shouldn't matter - enums are meant to be opaque values, but we re-model these enums occasionally in client code.

Removed coordinate constraints - leaving max_origin_connections as on…

c95f450

…ly migration

Added migrations missing from 3.0.x release

3763666

Fixed syntax errors in create_tables.sql

68f3b04

fixed trailing commas for REAL this time

8ec9dfe

fixed migrations that weren't rolled back

f20c62e

Resolved constraint naming differences

0a697a2

Also add/removed some missing/incorrect NOT NULL etc. stuff

made some constraints implicit

42dfbbb

Fixed the last of the diffs with a master schema (I hope)

c0f42f6

fixed typo in fk constraint

24b1f8c

ocket8888 force-pushed the the-great-migration branch from e62662b to 24b1f8c Compare May 31, 2019 14:52

Fixed the last of the diffs

040e37b

dangogh approved these changes May 31, 2019

View reviewed changes

dangogh merged commit 41cc65a into apache:master May 31, 2019

ocket8888 deleted the the-great-migration branch June 1, 2019 02:35

dangogh mentioned this pull request Jun 3, 2019

fix seeds.sql #3656

Merged

7 tasks

mitchell852 added the database relating to setup/installation/structure of the Traffic Ops database label Jun 27, 2019

zrhoffman mentioned this pull request Jul 15, 2021

Database migrations should be collapsed for ATC 6 #6027

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collapse database migrations #3524

Collapse database migrations #3524

ocket8888 commented Apr 25, 2019 •

edited

Loading

asfgit commented Apr 25, 2019

asfgit commented Apr 25, 2019

limited commented Apr 26, 2019

ocket8888 commented Apr 26, 2019

asfgit commented Apr 29, 2019

asfgit commented Apr 29, 2019

asfgit commented Apr 29, 2019

asfgit commented May 8, 2019

dangogh commented May 8, 2019

dangogh left a comment

ocket8888 commented May 9, 2019

ocket8888 commented May 9, 2019 •

edited

Loading

asfgit commented May 9, 2019

asfgit commented May 9, 2019

asfgit commented May 9, 2019

asfgit commented May 20, 2019

asfgit commented May 20, 2019

asfgit commented May 21, 2019

asfgit commented May 21, 2019

asfgit commented May 28, 2019

dangogh left a comment

dangogh left a comment

asfgit commented May 31, 2019

asfgit commented May 31, 2019

Collapse database migrations #3524

Collapse database migrations #3524

Conversation

ocket8888 commented Apr 25, 2019 • edited Loading

What does this PR (Pull Request) do?

Which Traffic Control components are affected by this PR?

What is the best way to verify this PR?

If this is a bug fix, what versions of Traffic Ops are affected?

The following criteria are ALL met by this PR

asfgit commented Apr 25, 2019

asfgit commented Apr 25, 2019

limited commented Apr 26, 2019

ocket8888 commented Apr 26, 2019

asfgit commented Apr 29, 2019

asfgit commented Apr 29, 2019

asfgit commented Apr 29, 2019

asfgit commented May 8, 2019

dangogh commented May 8, 2019

dangogh left a comment

Choose a reason for hiding this comment

ocket8888 commented May 9, 2019

ocket8888 commented May 9, 2019 • edited Loading

asfgit commented May 9, 2019

asfgit commented May 9, 2019

asfgit commented May 9, 2019

asfgit commented May 20, 2019

asfgit commented May 20, 2019

asfgit commented May 21, 2019

asfgit commented May 21, 2019

asfgit commented May 28, 2019

dangogh left a comment

Choose a reason for hiding this comment

dangogh left a comment

Choose a reason for hiding this comment

asfgit commented May 31, 2019

asfgit commented May 31, 2019

ocket8888 commented Apr 25, 2019 •

edited

Loading

ocket8888 commented May 9, 2019 •

edited

Loading