Define Remote Node Patch Procedure

[paidforby]

I noticed there are a lot of bugs that are labelled "node patch pending". In particular, bugs #1 #5 and #17 are all resolved issues, but the associated patches are waiting to be pushed to active nodes. Now that bug #8 has been mostly resolved (pending the patching of #17), perhaps it is a good time to consider how we handle remote pushing (or pulling?) of updates on to active nodes. At the very least, we can determine a policy or protocol for collecting and pushing patches.

[jhpoelen]

Here's my suggestion for remote matching protocol:

1. select a bug in state "node patch pending"
2. create a patch by using git patch or writing a little script to make the changes along with instructions on how to apply the patch
3. apply the patch in a controlled environment using your own home node (or test node) and your own exit node (test exit node)
4. review and share results with at least on other person
5. convince a node whisperer with access to run the patch on active nodes
6. communicate via mesh mailing list and notify of remote patch

Something like that. Curious to hear your thoughts and let me know what you think would be a good place to put the patch instructions .

[paidforby]

Thanks for clarifying that procedure, it seems reasonable to me. I would add, in step 2, that you should have at least two test home nodes connected to your test exit node. I'll look into creating git patches or scripts to address the bugs i referenced.

Also, in regards to step 5, how do we determine when all active nodes have been patched, how do we define "active nodes"? Could we create an cron task on the exitnode to push out updates every so often to any newly connected nodes?

I would consider adding this procedure to the node-whisperer-program or the exitnode README or the node operator manual whichever seems more relevant. I feel all three of these need to be synced in some way.

[jhpoelen]

I'd vote for the whisperer program or https://github.com/sudomesh/bugs , because it covers both home nodes and exit nodes.

Also, a simple log file "patch.log" in the root of each home node could be introduced. The log would be appended after applying each patch and can be inspected by humans or their robot "friends" when needed. I imagine that openwrt / linux has a more elaborate way of doing this: if so, I am not aware of it yet.

[jhpoelen]

I think that patch automation would be nice after we figure out a somewhat controlled way to roll-out patches semi-automatically.

[paidforby]

Made a first pass at a standardized way of rolling out patches to nodes, https://github.com/sudomesh/makenode/tree/master/patches.

Also added instructions on building your own patches in the makenode README.

It seems to work well on my home node for patching #17 please try for yourself. I have yet to try running from a test exitnode (I just tested from my computer over the private ethernet interface)

Perhaps it would make more sense to store the patches and documentation for patching in the bugs repo, instead of in makenode or sudowrt-firmware? What do y'all think?

[bennlich]

@paidforby awesome BYO patch readme!

Perhaps it would make more sense to store the patches and documentation for patching in the bugs repo, instead of in makenode or sudowrt-firmware? What do y'all think?

There's currently no code in the bugs repo. I think it's probably better to keep it that way--let the repo continue to exist specifically for discussing bugs, and not for tracking changes to code.

To me it makes sense to keep the patches in the same repo as the files that they are patching. So makenode seems like a good spot to me, although it sort of conflicts with the goal of deleting makenode.

sudowrt-firmware also seems reasonable. I guess sudowrt-firmware already has openwrt_patches. I wonder when these patches get applied?

I wonder if we need guidelines for how to write patch.sh helper scripts. E.g., maybe they should be written such that it is always safe to run the same patch.sh twice?

Maybe every time you run patch.sh, it should append a timestamp to a patch log on the node.

[paidforby]

@bennlich thanks for the thoughts. Agreed on your opinion of keeping the bugs repo code-free. However, since patches could be needed on any given node (home, exit, or extender) for any given service (tunneldigger, babeld, admin dashboard), I'd rather not spread them across three of four repos, including one that we are actively trying to deprecate. Perhaps we could start a "patches" repo as a catch-all for any and all peoplesopen related patches? Could also be a good way of starting working on a patch/updates server/service that could be run on any given mesh node.

[paidforby]

created https://github.com/sudomesh/patches @bennlich would be cool if you (or someone else) could try patching one of the other "node patch pending" bugs using my instructions, bug #1 and #5 both need work.

[bennlich]

@paidforby sweet! I like the idea of not spreading patches all over the place too. patches repo sounds good to me. I think my main heuristic for deciding where to put stuff is, "if someone were looking for X, where would they look?" Looking for patches in a repo called "patches" makes a lot of sense to me :-)

I will try to write a patch for #5 following your readme.

[paidforby]

awesome thanks, looking at the diff from the related commit should help

[bennlich]

@paidforby do you know if there's a way to delete a line with an inexact match? e.g. if I want the patch to delete:

  option log_prefix <some-variable-hostname-here>

[bennlich]

@paidforby thinking about the patch procedure. How will we know which nodes are patched and with which patches?

Example scenario: you attempt to apply patch A today. 20 ips are connected to the exit node. You ssh into all 20 and successfully apply the patch.

A week later you want to apply patch B. This time, 25 ips are connected. You wonder which new ones might be missing patch A. You log into each one, check some kind of patch log, apply A to each node that didn't get A applied. Then you apply B. Complication ensues when it's time for patch C.

Wondering about a more wholesale instead of piecemeal strategy. Instead of "patching" specific files/bugs, what if the process is always to log into a node, check what commit sha (or version) its config is at, and if it's old, wholesale regenerate and overwrite all config files using the latest stable makenode/firmware commit.

An optimization for this approach would be to only replace files that have changed since the last version.

I suppose we could do something like this with the patches-for-specific-bugs approach. Would just need a tool that automatically logs in and applies all patches that haven't been applied. But newer nodes probably won't need old patches.

Am thinking specifically about patches to config files in this case. Haven't thought about what other kinds of patches we might need.

I think this might be related to ideas @gobengo brings up in #20.

[bennlich]

More thoughts, since it sounds like we might start applying patches soon.

My main goal is to avoid patch-hell, which is where we patch some things, forget what we've patched, start to forget the state of the world, etc.

Currently, the deployed software/firmware/files that live on a node are a function of:

1. sudowrt-firmware commit (this can be found in /opt/sudowrt/build_version)
2. makenode commit
3. user-specified info (hostname, admin/root passwords, up/down rate config, private SSID name/password)

Does anyone @paidforby @jhpoelen @Juul know if the makenode version used to configure a node is stored anywhere on the node? I was unable to find this.

Ideally, every patch would be associated with a (sudowrt-firmware commit sha, makenode commit sha). Then a patch tool can log into a node, determine if the node is missing the patch, apply it if it's missing, or skip the patch if the node doesn't need it (e.g. the patch has already been applied or the node was built with a newer firmware image/makenode).

Until we have some kind of ipk distribution mechanism, I think this will help us stay sane as we try to patch nodes in the wild.

[jhpoelen]

@paidforby @bennlich @gobengo suggest to move this discussion to in-person discussion or https://peoplesopen.net/chat rather than github issue. We have https://github.com/sudomesh/patches with some example patches. Please feel free to re-open if you disagree.