Magma support for version rollback in case of failure in production deployment

[prudhviblinqnet]

We are planning to install and consume product version 1.4
lets say in future if we want to upgrade this to 1.5 to get benefit of the latest product features, what will be the upgrade and rollback process.
i can see in documentation it says for upgrade but there isn't any documented for rollback if something got failed over prod is there any quick turn around ?

[hcgatewood]

At this point in time, Orc8r does not explicitly support rolling back. If you want to support this on your own, you'll need to take a DB snapshot before the upgrade, perform the upgrade, then if it fails, reinstall the previous deployment and restore the DB snapshot.

The reason for this is most upgrades involve data migrations, and they are often non-reversible.

@karthiksubraveti or @tmdzk can speak to the AGW upgrade rollback

[prudhviblinqnet]

Thnq @hcgatewood @hcgatewood but what if if we are implementing this in Blue-Green deployment where we don't upgrade orc8r but we have different versions of orc8r like current and latest .how do we know if the DB schema and interface remain the same between two different ORC8R version. If suppose the schema got changed is there any data migration process or DB snapshot process that we can follow ?

[sudhikan]

AGW rollback is not trivial. You can do a dpkg --purge remove magma magma-sctpd and then install the previous version... but the dependencies don't get cleaned up quite easily.

When rolling back I usually wipe out:

• /usr/local/lib/python3.5/dist-packages/orc8r/*
• /usr/local/lib/python3.5/dist-packages/feg/*
• /usr/local/lib/python3.5/dist-packages/lte/*
• /etc/magma/* (this usually is nuked when we do the dpkg remove)

These steps do not do anything to the underlying ovs installation.... that stays as is (but I don't think we have upgraded / downgraded ovs; I think in 1.3 and 1.4 we've been using ovs-vsctl (Open vSwitch) 2.8.9).

cc: @tmdzk

[hcgatewood]

@prudhviblinqnet for Orc8r, we will announce on the relevant upgrade page (e.g. Upgrade Orc8r to v1.4) whether a DB migration is necessary when upgrading to that version from the immediately previous version

[jpad-freedomfi]

@berezovskyi-oleksandr ^^ FYI AGW rollback procedure used by Sudhi. This is more extensive than I tried when update to v1.4 failed.

[jpad-freedomfi]

@119Vik See note above about roll-back (and preparation for attempting roll-back i.e. DB snapshot) of orc8r if update to 1.4 goes poorly.