Formalise CSV transition strategy

[BClark09]

Following on from #299

The majority of systems can now be upgraded in the basic sense, but we need to define a CSV format to tell the upgrade scripts what to do in special cases. The proposal in #299 was to read through a simple list of instructions e.g:

MOVE,$OPENHAB_USERDATA/etc/some_karaf_file.profiteroles,$OPENHAB_RUNTIME/some_karaf_file.profiteroles
DELETE,$OPENHAB_USERDATA/etc/some_other_file.xml

We should discuss the exact stategy here.

CC all who discussed this previously: @kaikreuzer, @mhilbush, @BClark09, @xsnrg, @whopperg, @ThomDietrich, @bdleedy, @ghys

[BClark09]

I do think we'd need some versioning here. The update scripts already check which version is currently running, and what version is required to update. Say the system is on 2.1.0 the user wants to upgrade to 2.3.0. The CSV file in 2.3.0 should contain:

[2.1.0]
MOVE,***,***
DELETE,***

[2.2.0]
MOVE,$$$,$$$
DELETE,$$$
MOVE,$$$,$$$

[2.3.0]
MOVE,£££,£££

The script parses the CSV file and should perform:

MOVE,$$$,$$$
DELETE,$$$
MOVE,$$$,$$$
MOVE,£££,£££

[ThomDietrich]

Taken from #426 (comment) :

---

I can't think of any higher complexity (delete file x only if file y diesn't exist and z was created on christmas eve) demanding a more sophisticated format. Here are all examples I can think of:

• DELETE;relative/path/to/file
• DELETE;$OPENHAB_CONF/pseudo-relative/path/to/file
• MOVE;filepath_old;filepath_new
• MOVE;filepath_old;$OPENHAB_CONF/pseudo-relative/path/to/file
• TOUCH;some_file_or_folder (probably not needed, will create folders which might be of interest)

General remarks:

• All commands are executed in the order they are given in the file
• paths are relative and can start with one of the defined $OPENHAB variables
• path delimiter is the forward slash
• paths can contain wildcards * and ?

One additional idea: A second parameter could contain either of ERROR/WARN/INFO/NONE telling the upgrade script how to react on a problem, e.g. file to move doesn't exist, ...

[ThomDietrich]

Hey @BClark09, I agree we should include a versioning.

• CSV doesn't support sections. We should discuss if or if not staying with the CSV syntax is of interest
• Your example only covers Upgrading. While I'm not saying we should support downgrades per se, the part which is currently [2.2.0] should at least make clear that these are the upgrade steps from-to.

The list above doesn't contain an action to Modify/Update a file. @kaikreuzer suggested an APPEND action. My concern is, that this would only cover a small portion of intended use cases. A diff/patch file would be the better approach for general modifications but would also add unwanted complexity. Wdyt?

[bdleedy]

INI files? INI files have sections and there are usually great tools/extensions for using them. I know it's an odd use of the format but it might fit nicely.

[ThomDietrich]

INI creates the same issue. You have sections but only key=value pairs.

[BClark09]

When I was thinking csv, I was also thinking that whatever script could implement sections itself. In Pseudocode:

currentVersion=getCurrentVersion()
if (firstParam == "[*]") then
  if ( * < currentVersion ) then 
    skipToNextSection 
  end
end

While I'm not saying we should support downgrades per se

Wouldn't downgrades work exactly in reverse? MOVE for example:

MOVE,filepath_old,filepath_new

##upgrade
mv $1 $2

##downgrade
mv $2 $1

Fine until a DELETE action, because an UNDELETE would be impossible.

[ThomDietrich]

Well no doubt that is possible! It's just not following the standard idea of CSV - which would be okay with me...

[BClark09]

Ah, I see. Quite honestly I didn't know there was a CSV standard as such. Every day is a school day!

If we give it a file extension other than csv, this should be fine. (.trans, or .transition?)

[bdleedy]

@ThomDietrich

INI creates the same issue. You have sections but only key=value pairs.

I'm not seeing a situation in which this couldn't be used.

[2.0.0-2.1.0]
Move1="Here","There"
Move2="Here1","There1"
Delete1="This"

[2.0.0-2.2.0]
and so on....

Each possible scenario has it's own section. Keys are simply TypeNumber and some sort of delimitation if needed. Is that not what @BClark09 is describing in the 2nd comment but without the impossible nested Sections?

[BClark09]

Whilst that would be a fairly simple strategy, what would you do if 2.3.0 was introduced and has the same structure as 2.2.0? Add one for each version anyway?

In addition, The number of [From-To] you would need would be start becoming too large. For the number of versions openHAB may have (n), the number of sections would equal [(n^2)+n]/2

[bdleedy]

Yeah, it would be a vertically large file, whereas a .csv could be horizontally large. The two could be "combined" even more than I am suggesting like

Moves=here1,there1;here2,there2;here3,there3;...

Using the two delimiters could vertically simplify the file.

I also use an .ini with 'nesting' much like .inf drivers do.

[MAIN SECTION]
Key1=Value1
Key2=Value2
Key3=Value3

[Value1]
Key=Value

[Value2]
and so on

This could solve the same structure between versions. 2.2.0=2.2.0 and 2.3.0=2.2.0 as well. I'm not saying this is the best or better option. But I think it's a viable one to consider.

[BClark09]

Notes for what is necessary so far:

[2.2.0]
MOVE $OPENHAB_USERDATA/etc/org.ops4j.pax.logging.cfg $OPENHAB_USERDATA/etc/org.ops4j.pax.logging.cfg.bak

[2.3.0]
DELETE $OPENHAB_USERDATA/etc/overrides.properties
DELETE $OPENHAB_USERDATA/etc/org.openhab.addons.cfg

[BClark09]

If I might add another feature to this: Would it be worth placing 'breaking change' messages in this file? For example

[2.2.0]
ALERT Logging configuration has changed, org.ops4j.pax.logging.cfg has been backed up and restored to defaults

[2.3.0]
ALERT Z-Wave binding has changed, users must delete and re-add Z-Wave things.

This way, the respective repos can use these messages to display important change logs.

[kaikreuzer]

I like the idea!

[whopperg]

Me too! Should be very useful and could minimize the support threads and troubleshooting....

[BClark09]

Alright! I've got the basics set up: With the following update.trans file (for illustration only)...

[[PRE]]
[2.2.0]
MOVE;$OPENHAB_USERDATA/etc/org.ops4j.pax.logging.cfg;$OPENHAB_USERDATA/etc/org.ops4j.pax.logging.cfg.bak

[[POST]]
[2.2.0]
NOTE;Logging configuration has changed. 'org.ops4j.pax.logging.cfg' has been backed up and restored to defaults!
NOTE;*.rules files are now validated upon startup. Files with errors will from now on be logged and ignored by the runtime.

[2.3.0]
DELETE;$OPENHAB_USERDATA/etc/overrides.properties
DELETE;$OPENHAB_USERDATA/etc/org.openhab.addons.cfg
ALERT;Oceanic Binding: The 'softener' Thing Type no longer exists and is replaced by the 'serial' and 'ethernet' Thing Types

And the shellscript in this gist. I get the following output:

I realised that there should be a distinction between tasks that we want to execute before the main upgrade, and tasks we want to execute after the upgrade. Hence, the [[PRE]] and [[POST]] sections were necessary.

Otherwise the script will run any command that is underneath a higher version than the current. For example, if the current version is 2.2.1, it will skip 2.2.0 and execute anything under 2.3.0.

WDYT?

[wborn]

We've been using these automated upgrades using the update.lst and some scripts on Linux and Windows for a while now. 👍

Do you think we can now close this issue?

[kaikreuzer]

Sounds fair to me.