Easier installation and upgrade process

[JC5]

See also https://github.com/monicahq/monica

One command: firefly:install, that will call migrations, seeds, and data upgrades. Will also ask for the first registration and setting "allow other registrations yes/no". Will link to website/GitHub for common issues (Docker).

Perhaps this can also be done webbased.

When upgrading, an update of the code base should be enough. First visit to FF3 (must be admin) will trigger upgrade command. Verifies by comparing the version in config.php to a new FF3 configuration item ( "db_version"). Simply calls firefly:upgrade and various other commands, then updates config var and proceeds.

[Aquariu]

JC5,

+1 for a more automated upgrade process :) I"ve been stuck to 4.6.10 just because I can't find the time to test everything after upgrade.

From the other web tools I host on my personnal server, I've seen that most often, upgrading the codebase through a web updated requires the www-data user to have stronger than usual access rights to the webroot directory.

My other concern is about changes in configuration. One reason i'm staying on a 3 months-old version is that I know there are new and interresting features in a newer one that require new configurations for my automated import process. Particulary the .json files. It would be great if the changelog or the documentation contained what I need to change.

Regards,

Olivier

[JC5]

A quick reply from my phone. I agree. A code based full upgrade will never happen. At best, update the code yourself and FF3 will do migrations etc. In light of the development of Docker and similar tools this would be madness to build. Also insecure.

Als for your upgrade. No great backwards incompatible changes as far as I know. Running of rules and matching of bills need a config entry but without them nothing changes.

The only incompatible change I recall are possible migrations you should be able to run easily. Json config has one important change: “match_rules” became “match-rules”. Same for the bills related command.

Firefly III is growing up so a more professional and reliable change log is certainly necessary. 👍

[moay]

For my setup, I'd love some way to auto update my instance. It's currently kind of a huge deal. The easiest way I found was the github way - but thats something one could automate. Why not build a backend way to do just this? I'd use curl to get the latest release from github, then remove vendors and stuff like this, then use a downloaded composer.phar to execute the vendor update and execute the needed migrations and checks. The entire step could be done in a subfolder which will only replace the current release if everything went well. Also, a database backup should be performed before the update.

The entire process CAN be automated as long as there are enough checkups before, like:

• will I be able to use curl or wget
• will I be able to download from github?
• is there enough space for the download
• do the hashes match
• am I able execute php via exec or passthru

There should be some sort of update script in the repo which allows to throw in needed update steps before composer install, before migrations and after migrations.

Here is a possible workflow for the tool:

• before even showing the "update"-button do some checks if the tool is able to work.
• backup current database to a useful folder
• download current release from github to new release folder
• check hashes and extract if they match. Retry x times if download was corrupt.
• check from the update script if automatic update is possible from current release to new release
• copy over .env file
• copy over uploads
• execute composer install
• check in the update info if special operations are needed from the old version to the new one
• migrate databases
• update databases from included update script if needed for new version
• warmup cache (?)
• verify setup
• If everything is fine: Rename the old release to "backup", rename the new releases folder to the correct foldername, everything done.
• If not: Delete downloaded release, rebuild the old DB, inform the user.
• At any point: If an Exception is thrown: Rollback and stop.

The only problem I see there is to get a checksum for the downloaded release. But this could realized with some sort of release signing.

What do you think about this approach?

[moay]

Some thoughts on the checksum stuff: If @JC5 considers this updating process an option (there could be warnings like "be careful!" (or more drastic warning labels)), we'd need to develop a backend tool first, which will generate the checksum stuff. I'd suggest a heroku node which gets triggered by github with a new release. The tool should then clone the repo and download the additional github releases (zip and tarball). Unpack everything, check if the files do exactly match, create a checksum for both releases, add a signed version of both hashes (signed using openssl_sign()) and calculate the filesizes. Finally write everything into a "checksum.json" and append it to the release.

This way, there would be hashes which could be verified using a public key (Which could be hardcoded in the update tool), a validated way to ensure the downloaded version is intact and everyone might be happy. :-)

Of course don't update if

• there is no "checksum.json"
• there are no hashes for the release (yet)
• the hashes don't match
• the hash signature is not valid
• Anything else is strange

If you like the idea, I could build a tool for this, starting with the checksum generation tool.

[JC5]

Except for some checks isn't this easily solvable when running a Docker instance? I have to admit I feel this is a bit much for an automated upgrade.

[moay]

Sorry, I don't know much about Docker yet. Just began looking into it, but maybe it is. But apart from docker, I think that upgrades need to run in as much other environments as possible. I do currently run Firefly on a vserver, so it's kind of a shared hosting environment. I can't make a git pull there because git is not available via ssh.

A lot of people will be running Firefly on things outside of docker... Also, I guess that Firefly has great potential for non-techies who are looking for a good budgeting solution. They probably won't look into docker... ;-)

[JC5]

Haha yes you're right of course. Not everybody uses Docker. I don't even use Docker.

But I'm not going to go "full Wordpress" with this, if you know what I mean. A web-based upgrade is necessary because hosting services like Sandstorm won't give users access to the command line. And they'll want to upgrade as well.

But the "upgrade" will be limited to database migrations, and possibly some new seeds. If the user can get the code in the right place, FF3 should be able to do the rest.

[moay]

I'm not going to go "full Wordpress" with this

:D Yes, I know what you mean. I would love to find a way to have FF3 getting the code in the right place, but I see your concerns. My idea was to have one click and have all cli commands handled by PHP (passthru or exec). Additionally, it should be possible to have no cli commands at all and handle everything via javascript. The entire hashing thing would be invisible, automated and outside of FF3.

As I said, on my hosting a git pull is not possible. Also, downloading and copying the entire project there can be a bit daunting when I have to make sure that some folders don't get overwritten or something like that. Reinstalling via composer is also not an option for me, as this would need multiple commands or major file operations as well.

I may end up writing a bash script that downloads the latest release via wget, unzips it, replaces .env and the uploads and then deletes the old release. That's if I find a way to run a composer update after that... I'd still prefer the one click solution in the backend. ;-)

[pkoziol]

If you are interested, I've automated upgrading Firefly using Ansible.

Here is my playbook which I execute from my laptop with ansible-playbook firefly-upgrade.yml and after ~minute upgrade is done:

PLAY [vps] *******************************************************************************

TASK [Gathering Facts] *******************************************************************
ok: [vps]

TASK [Check if Firefly III 4.7.0 directory exists] ***************************************
ok: [vps]

TASK [Create Firefly III 4.7.0 project] **************************************************
changed: [vps]

TASK [Copy .env file] ********************************************************************
changed: [vps]

TASK [Copy storage/upload/ directory] ****************************************************
changed: [vps]

TASK [Copy storage/export/ directory] ****************************************************
changed: [vps]

TASK [Cleanup bootstrap/cache/ directory] ************************************************
changed: [vps] => (item=absent)
changed: [vps] => (item=directory)

TASK [Run artisan migrate] ***************************************************************
changed: [vps]

TASK [Run artisan cache:clear] ***********************************************************
changed: [vps]

TASK [Run artisan firefly:upgrade-database] **********************************************
changed: [vps]

TASK [Update symbolic link] **************************************************************
changed: [vps]

TASK [Update permissions of files in storage/ directory] *********************************
changed: [vps]

TASK [Update permissions of directories in storage/ directory] ***************************
changed: [vps]

TASK [Update owner of storage/ directory] ************************************************
 [WARNING]: Consider using file module with owner rather than running chown

changed: [vps]

PLAY RECAP *******************************************************************************
vps                        : ok=13   changed=1   unreachable=0    failed=0	

Drawbacks:

1. No database backup (I use another playbook, but plan to automate creating backups every day/week/month).
2. Previous versions are stored indefinitely and take disk space.
3. I wouldn't notice if JC5 changed upgrade instruction.
4. No checksum/signature verification (unless composer has some?).

I belive, it requires python 2 on the server (to run Ansible) and composer available in $PATH (but that probably could be avoided by using command module instead of composer module).

[moay]

Maybe this is worth a look... http://mothership.ec/blog/introducing-up-a-library-for-self-updating-web-applications

EDIT: Forget that. But I still am in favor of automating the download. ;-)

[Nik-vr]

But the "upgrade" will be limited to database migrations, and possibly some new seeds. If the user can get the code in the right place, FF3 should be able to do the rest.

I think that most users of virtual hosting (including me) will have enough of this update algorithm:

1. The user downloads the archive with the new version of FireFly with Github.
2. The user replaces all files of FireFly on their virtual hosting.
3. The user starts a certain php script that performs database migration, clears the cache and performs other necessary actions.
   In principle, this script can be started automatically if the first time a new version is launched, FireFly will find out that the database version does not match the version of the code.

Similarly, with the new FireFly installation for virtual hosting: copy the files, create the database (manually), and run the script that prepares the database and all that.

I think it will not be difficult for you to implement such an algorithm. And for many users it will be very useful. It's even possible that FireFly will become more popular :)

[Findus23]

At least for me updates are already pretty simple. I switched to using the git repository as it allows simply updating with a git pull on master.
For the rest of migrations (running rm -rf vendor/ and composer twice are not needed in most cases), I simply added them to a bash script.
That way my modification are also preserved.

[JC5]

For the next release, and the ones afterwards, Firefly III will verify if the database is up to date and upgrade as necessary. This is also the case for new installations. This means that once you set up the .env file correctly (which is automatically in Docker instances), you can surf to Firefly III's index page and the database migrations will fire automatically.

This applies to upgrades as well.

For as far as making the installation + upgrade simpler, this is about as simple as it gets.