firmware: scheduled updates

[StrikerTwo]

My first OPNsense update went smoothly... but still ;)

Sophos UTM has a cool feature that allows you to schedule updates for later. I would just select the available updates and tell it to update and restart itself at 01:00 am or whatever. Provided everything works okay (and you don't schedule all cluster members for the same time) this allows updates without noticable downtime and without the admin having to work late.

Under the hood it's probably nothing more complex than a simple cron job, but it would be nice to have that configurable in the GUI.

[fichtner]

:) We already have a cron job, but it goes straight for the update if available, also for the major ones if explicitly configured. I know what you would like to see, it shouldn't be too hard to do this, but keep in mind that updates are transient. If you schedule an update and there is a newer update it will take the latest one, not what you approved. If that's ok, we can add a nifty little feature. Cheers, Franco

…

On 31. Aug 2017, at 17:34, StrikerTwo ***@***.***> wrote: My first OPNsense update went smoothly... but still ;) Sophos UTM has a cool feature that allows you to schedule updates for later. I would just select the available updates and tell it to update and restart itself at 01:00 am or whatever. Provided everything works okay (and you don't schedule all cluster members for the same time) this allows updates without noticable downtime and without the admin having to work late. Under the hood it's probably nothing more complex than a simple cron job, but it would be nice to have that configurable in the GUI. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[StrikerTwo]

I think I could live with that. If there is a newer update in the (usually) short time between scheduling and actually updating, it's probably there for a good reason - like a quick hotfix for the last update because something was broken.

[fichtner]

Alright, so right now we already have a recurring type cron job.... but you still want a one-shot type upgrade via cron? Cron always runs, but only upgrades if the trigger was given and the trigger is deleted after upgrade?

[StrikerTwo]

I don't care that much about the implementation, you are the experts ;)
I would just like to have a button "schedule update" next to the existing "update now" button. Click it, select a date and time in the future, and at that time the latest update gets installed and the system reboots automatically.

[fichtner]

Alright, so that's something completely detached from the current cron job in design, at least from the user perspective. Not what I hoped for, but maybe we can reuse the cron job anyway.

Ideally we just need a date picker and time, write a timestamp to the disk and let a cron job pick it up when it expired, given that we poll this every few minutes... Maybe just by the hour plus a bit of fuzz to avoid annoying the update servers every full hour.

I assume everything within the particular hour would be ok?

Might take a while to materialise, but I'll keep it on the firmware TODO list.

Thanks,
Franco

[StrikerTwo]

I guess up to plus 30 minutes would be okay.
Announcing a downtime from 2am to 3am and having the update start on 2:59 could be annoying.

Or maybe you could download the files as soon as the user clicks the button, and schedule just the actual installation for later? Then you could do it at exactly the chosen time, plus the user would get exactly the update he saw when he clicked the button.

[fichtner]

Prefetch is indeed a nice idea. That would work, although it should be pointed out that if we prefetch updates are being executed on schedule or the next reboot (also a crash, worst case), as major upgrades are being done currently.

But I have what I need now, let's see what we can conjure up :)

Thank you,
Franco

[Woi]

I like to add my bits here. While StrikerTwo asks for a feature to manually trigger an update for later installation, I see a use case for automatically install updates some time after release.

Think about the following situation: multiple OPNsense boxes deployed on various locations. To minimize maintenance, all have automatic updates configured.
To be able to test the updates but also allow easy recovery in case of problems, it would be nice to install updates on the locale site immediately after release, but delay them on remote branches by release + t.
Locale box: automatic updates runs every night and installs if an update is available
Remote boxes: automatic updates runs every night and installs if an update is available since at least 2 days.

However an alternative would be to set up an own/personal repository and use this for the remote boxes. But it seems to be a lots of effort for just a handful of remote locations.

[Woi]

I like to add that it makes a lot of sense from usability to have some link or hint on ui/core/firmware/#updates to the cron jobs.

[fichtner]

We should probably hint to the cron job from the firmware page like we do for IDS and the Web Proxy, that's a good idea. :)

But I have to stress the point that "allow update in x days" is a bit problematic, because we do not track each upgrade step, so what ends up happening is that the latest available update in x days is installed, not the one that is approved.

We also have no way of explaining to the user what an update is going to be, it could be:

• a minor update for a security software update of the non-OPNsense components (no reboot)
• it could be a hotfix of OPNsense GUI component or plugins (no reboot)
• it could be a minor release of OPNsense (may reboot)
• (it could be a major release of OPNsense (must reboot))

The major case is separate due to its critical nature, but shown here for illustration.

It's true that we could prefetch in the precise moment and schedule an update, but that may not work for minor releases with kernel and base updates, because they are not part of the package ecosystem itself (they are "sets" like changelog and bogons updates for historic reasons in FreeBSD that does not support updating everything through its package manager yet).

But even when we prefetch there is a risk to this: a scheduled reboot or unscheduled reboot will trigger the updates, unless we want to add even more logic to prevent that, which I don't find very favourable. More code usually equals more problems (@Woi already found a bug in the firmware code so that's just going to keep happening with increasing complexity).

So in the spirit of progress, we'll keep this ticket for a while and start with a minimum viable solution:

• Cron job creation from the firmware page
• Single-shot approval of an update at a specified time, maybe with a mandatory reboot (not major upgrade)

The remaining items would be:

• Prefetch updates to be installed and schedule them for the next reboot a bit like the single-shot approval (plus scheduling that reboot)
• A background polling mechanism that allows the user to be informed of available updates (we value privacy so there is no automatic polling unless the cron job is enabled)
• Make sure the features don't get in the way of each other ;)

Thoughts, comments? :)

Cheers,
Franco

[AdSchellevis]

I would suggest to keep it simple.... different companies use different update/upgrade strategies, I don't think we should try to automate more then the basics which we can do with cron and are stateless.

If people want more then that... Chef or Puppet are likely more suitable and do support all kinds of logic to automate things.

[fichtner]

The simplest one from the work flow perspective that fits most would be to prefetch and reboot at the scheduled time. Least amount of surprises.

For the automated side, I am not sure. But the API is available if we do the former so it could be automated... check for update... prefetch and schedule reboot/apply.

[AdSchellevis]

I agree, make sure the workflow can be automated by an external tool, but don't try to be the automation tool..... It's just a different type of solution, and before you know you get the request of automating the rest of the world around the firewall (which is in my opinion what we should try to avoid).

[StrikerTwo]

For me it's more about scheduling the outage caused by the update. If the box crashes and reboots before the scheduled downtime, the outage is there anyway - if it applies the update then, all the better ;)

[Woi]

I wasn't aware about all the corner cases and complexity of my suggestion. And I totally agree with keeping thinks simple. Especially since configuration management or a own repository are better solutions anyway. Sorry for the confusion.

Do you guys like a own feature request for tracking the

link or hint on ui/core/firmware/#updates to the cron jobs.

feature?

[fichtner]

@StrikerTwo makes total sense, thanks for the perspective

@Woi we'll track this from here, shouldn't be a tough one to add

[fichtner]

I need to move this to 18.7, not enough time unless somebody lends a hand.

[Woi]

Hey guys,

for the "schedule update" button, at(1) might be handy. At least on Linux it even survives reboots.

I think about adding a reboot option for the auto update cron job. But I'm confused since my box rebooted after updating to 17.7.8 and 17.7.10 but not after updating to 17.7.9. Why is this? I thought the cron job is supposed to not reboot but for mayor upgrades? I'm using "ALLOW_RISKY_MAJOR_UPGRADE" should this matter.

[fichtner]

Hi Woi,

If the reboot is required it reboots like the GUI does but without the confirmation. We do this on FreeBSD updates, which need a kickstart for the new kernel...

ALLOW_RISKY_MAJOR_UPGRADE is for allowing major upgrades only. This one reboots multiple times.

Cheers,
Franco

[Woi]

So, if there is no reboot it's simply not necessary to restart?
May be I got something wrong or read about an old and long improved behaviour, but IIRC the cron job could update daemons without restarting them. The suggestion was to manually reboot, to make sure the changes take effect. Can you please confirm that it's not necessary to manually restart daemons or reboot the box?

[fichtner]

I cannot confirm this. If the box does not reboot, you are in charge of restarting services either by reboot or restarting the affected services from the GUI like it always was:

https://github.com/opnsense/changelog/blob/a4008b74e0315b1fd9fda7ef042ec7e297237a77/doc/15.1/15.1.9.1#L12-L13

The cron job does exactly what the console option 12 does. Not more, not less. :)

[BuhtigithuB]

It would maybe a bit simpler to have the notion of minor and major update... CRON JOB button for minor upgrade which trigger a permanent job that update/reboot opnsense on new minor version when ever available...

Just my 2 cents

[fichtner]

It’s in the cron commands. You can do minor or minor+major.

[fichtner]

I'm handing this over to #6011, likely an upcoming business feature.