Refactoring of some components

[suorcd]

I tested this on a Ubuntu 22.04 VPS image.

Please test and let me know if changes are needed.

[Cameron-IPFSPodcasting]

It all looks good, but I'll test it to be sure. Thanks for the 2nd (or 3rd) set of eyes.

[CaffeinatedDNB]

Thank you for taking the time to review it.

Just took a look at the suggested modifications. I agree with the "service" to "systemctl" change and related adjustment needed to parameters as Debian based installs don't recognize it which I came across while testing on Debian 11. 👍

the "adduser ipfs" doesn't need a pre-check as the VPS running would be a new instance and thus wouldn't have any user accounts in existence besides root.

The entire section of the IPFS daemon file should be left as-is in the original as ExecStop won't work as intended. During testing, the system start then stop it right away. At times, it would run and when the "systemctl stop ipfs" command was issued, it would show that it was shutting down but never did. Please see the latest test below on a new instance I used to get the proof. The stop command had been issued 2+ minutes and it still hadn't shutdown. I let it be and checked again 8 minutes and 30 minutes later and it was still running able to "ipfs add" without issue.

"ipfs shutdown" worked just fine and was immediate:

ipfs@ubuntu:~$ systemctl status ipfs
● ipfs.service - IPFS daemon
Loaded: loaded (/etc/systemd/system/ipfs.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2022-09-12 03:22:27 UTC; 2min 12s ago
Process: 18669 ExecStart=/usr/local/bin/ipfs daemon (code=exited, status=0/SUCCESS)
Process: 18703 ExecStop=/usr/local/bin/ipfs shutdown (code=exited, status=0/SUCCESS)
Main PID: 18669 (code=exited, status=0/SUCCESS)

Sep 12 03:22:12 ubuntu ipfs[18669]: Swarm announcing /ip4/127.0.0.1/udp/4001/quic
Sep 12 03:22:12 ubuntu ipfs[18669]: Swarm announcing /ip6/::1/tcp/4001
Sep 12 03:22:12 ubuntu ipfs[18669]: Swarm announcing /ip6/::1/udp/4001/quic
Sep 12 03:22:12 ubuntu ipfs[18669]: API server listening on /ip4/127.0.0.1/tcp/5001
Sep 12 03:22:12 ubuntu ipfs[18669]: WebUI: http://127.0.0.1:5001/webui
Sep 12 03:22:12 ubuntu ipfs[18669]: Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Sep 12 03:22:12 ubuntu ipfs[18669]: Daemon is ready
Sep 12 03:22:27 ubuntu ipfs[18669]: Received interrupt signal, shutting down...
Sep 12 03:22:27 ubuntu ipfs[18669]: (Hit ctrl-c again to force-shutdown the daemon.)
ipfs@ubuntu:~$ echo Testing | ipfs add -
added QmQu6f8nDzxKHRk7rY8VQSyxjXdNrnu2UctqTSMEzjRrbU QmQu6f8nDzxKHRk7rY8VQSyxjXdNrnu2UctqTSMEzjRrbU
8 B / 8 B 100.00%

ipfs@ubuntu:~$ ipfs cat QmQu6f8nDzxKHRk7rY8VQSyxjXdNrnu2UctqTSMEzjRrbU QmQu6f8nDzxKHRk7rY8VQSyxjXdNrnu2UctqTSMEzjRrbU
Testing

I had considered including the "--enable-gc" parameter in the ExecStart but I did some research on https://docs.ipfs.tech/ and came across the part that automatic garbage collection runs once an hour and that's quite a disk-intensive task. In addition, @Cameron-IPFSPodcasting 's ipodcastingnode.py script runs garbage collection once a week so automatic garbage collection through the daemon is beyond the scope of the intent to get an IPFS Podcasting node up and running as smooth as possible. In addition, @Cameron-IPFSPodcasting is already working on his script to keep data storage to a certain percentage of available drive space to prevent disk full conditions. 😎

And to conclude comments on the IPFS service section, there's no need to invoke "systemctl daemon-reload" as that would only be required if the ipfs.service had been enabled and then modified. In the case of the installation script, the end of it calls for a system reboot so that's another point for not needing the daemon-reload command in this use case.

As for the "sysctl -w net.core.rmem_max=1200000" line, thank you! I had the echo line during early testing and was inadvertently omitted from the final script. Good catch!

And finally, the "-s /bin/bash" addiition to "su - ipfs" is usually not needed, at least for Ubuntu and Debian, which are the two platforms things were tested on, but it won't hurt. In any event, properly commented scripts tell the system which shell to use. 👍

[CaffeinatedDNB]

I didn't test it on Ubuntu 22.04 LTS as various underlying system changes have taken place that I don't particular care for plus it's too new. Ubuntu 20.04.5 LTS (as of this writing in Sep 2022) is solid and still has 3 more years of updates from Canonical (until 2025.)

I went ahead and checked out Ubuntu Server 22.04 on a new VPS instance and the installation process got interrupted by a kernel update screen requiring several steps to get through. That doesn't happen in 20.04 LTS. That's why I was specific as to which OS version it was tested under. 😎

[CaffeinatedDNB]

@Cameron-IPFSPodcasting

I've uploaded a newly saved version of the script incorporating some of the suggestions made by "suorcd" and his contribution was noted at each modification.

I also removed sudo in my original script that I found towards the top.
Example: sudo apt update -qq && sudo apt -y -qq upgrade (it didn't affect things either way as the user would be running as root in any event but it's better/cleaner to not have it unless absolutely necessary.

I added the following:
Adding Universe repository in the event it's not active
so fail2ban and related packages don't fail

add-apt-repository -y universe

Believe it or not, I've never used Github for any pull/merge requests so I'll rely on you to make those commits. 👍

Once again, thank you @suorcd

2022-09-12-IPFS-Node-Installation.txt

[suorcd]

@CaffeinatedDNB Adding a check shouldn't cause issues, especially if the script fails for some reason and the user tries to run again.

[suorcd]

@CaffeinatedDNB Using usermod -aG sudo ipfs is a bit more widely available, but adduser ipfs sudo would follow the previous use of adduser.

[suorcd]

@CaffeinatedDNB The '--enable-gc' was from a previous version that i had found, if the current implementation doesn't need no reason not to it remove it.

[suorcd]

@CaffeinatedDNB Seems like ExecStop might have a difference in behavior between 22.04 and older versions, I did not exeperience any issues when using it for testing; but there was at least one other user that said it caused some issues.

[suorcd]

@CaffeinatedDNB As for the slightly odd way to invoke the ipfs user, I think it should be a system user without sudo, password, or shell; and that is a good way to invoke a system user with a shell.
It would require more refactoring ipfspodcasting-install.sh and IPFS-Node-Installation.sh than I can do at this moment.

And if you are assuming running as root, the sudo would not be necessary.

[suorcd]

@CaffeinatedDNB What is the target audience, skill level, etc. for this script?

[suorcd]

I didn't test it on Ubuntu 22.04 LTS as various underlying system changes have taken place that I don't particular care for plus it's too new. Ubuntu 20.04.5 LTS (as of this writing in Sep 2022) is solid and still has 3 more years of updates from Canonical (until 2025.)

I went ahead and checked out Ubuntu Server 22.04 on a new VPS instance and the installation process got interrupted by a kernel update screen requiring several steps to get through. That doesn't happen in 20.04 LTS. That's why I was specific as to which OS version it was tested under. sunglasses

There are definitely some more jarring interface choices, but most of the new installs will most likely be on 22.04

[CaffeinatedDNB]

I'm a proponent of K.I.S.S. (Keep It Simple Stupid)

The goals and intended audience have been clearly spelled out. :-)

From the script with documentation I've provided:

To be done via Hosted VPS Web console or whatever
console you're using; VM or dedicated computer.

Generally, many will be running their IPFS nodes from a VPS provider
so read up on their documentation on configuring the firewal

---

From my initial post on:

#1

"Provide concise documentation for the would-be IPFS Node admin to have a new instance up and running in about 10-20 minutes time with limited knowledge and/or interaction. Code documentation provides insight as to what each step is doing."

---

I'm going to address your points individually:

"@CaffeinatedDNB Adding a check shouldn't cause issues, especially if the script fails for some reason and the user tries to run again."

Once again, there's no need to add a check as it's a brand new VPS instance. Even if the user runs the script again, the "adduser" command would simply state that the user exists and move along. A non-issue.

---

"@CaffeinatedDNB Using usermod -aG sudo ipfs is a bit more widely available, but adduser ipfs sudo would follow the previous use of adduser."

Either way would be fine. What counts is that the original command gets the job done and works on the tested versions of Ubuntu Server 20.04 / 22.04 and Debian 11.

---

"@CaffeinatedDNB The '--enable-gc' was from a previous version that i had found, if the current implementation doesn't need no reason not to remove it."

I addressed that point earlier. Using that flag causes garbage collection to happen once an hour and as the IPFS node grows in size would unnecessarily tax the system.

---

"@CaffeinatedDNB Seems like ExecStop might have a difference in behavior between 22.04 and older versions, I did not exeperience any issues when using it for testing; but there was at least one other user that said it caused some issues."

I came across it in testing as well which is why the final code I provided didn't include it. "ipfs shutdown" works just fine, is consistent and is available to the user IF the need arises.

---

"@CaffeinatedDNB As for the slightly odd way to invoke the ipfs user, I think it should be a system user without sudo, password, or shell; and that is a good way to invoke a system user with a shell.
It would require more refactoring ipfspodcasting-install.sh and IPFS-Node-Installation.sh than I can do at this moment.

And if you are assuming running as root, the sudo would not be necessary."

I'll respond to the last point first. I didn't use "sudo" to invoke the user change. That's a change that @cameron did to my code since he wasn't running from root on a VPS or VM instance. :-D All the "sudo" entries you came across weren't from me and won't be in the final revision.

The "ipfs" user with a password and sudo privileges is what's needed to run things properly from a non-root account and easy IPFS setup using the "ipfspodcasting-install.sh" which I do not modify one bit. It's used as is provided by @cameron. I went down this path of contributing because running IPFS daemon, in this case, is not a sound security practice from a system admin's point of view.

The proof that the installation wrapper I've contributed works, which makes use of "ipfspodcasting-install.sh", is that I've spun up numerous VPS instances and have a fully functioning IPFS node in about 10-15 minutes time.

In reality, once the setup is complete and verified to be online, @cameron's node management script handles the rest. There's no additional "refactoring" to do. :-) The 7 nodes I have are running smoothly.

---

suorcd commented
"There are definitely some more jarring interface choices, but most of the new installs will most likely be on 22.04"

I beg to differ. A VPS provider, which is the intended deployment platform, offer versions of Ubuntu all the way back to 16.04 LTS.

And lastly, people are free to choose their platform of choice to deploy IPFS. However, even @cameron's "RunNode" page (https://ipfspodcasting.com/RunNode/Hosted) clearly states Ubuntu and Debian. :-)

The show must go on! There are many more IPFS Podcasting nodes waiting to be spun properly.

Thank you for your contributions and discussion points!

[Cameron-IPFSPodcasting]

Thanks for the discussion. I'm still reading all the comments and will work on integrating everything (hopefully to everyone's satisfaction).

The added "sudo's" was my bad, and have been removed.

On garbage collection. It's not managed by the client script (yet). I'm planning to send GC commands in the request/response when I notice a node has reached it's usage limit (in the next client release). The client script will restart IPFS if it crashes (without --enable-gc). I don't think --enable-gc will be necessary on startup.

The only config change to IPFS is on the Umbrel where I add SwarmRelayClient in case the Umbrel is behind a firewall. Could add it to this client for consistency, but running your own server (more advanced) shouldn't need it.

[CaffeinatedDNB]

@Cameron-IPFSPodcasting

No problem. Sounds like a plan to me! 😎

As for the garbage collection, I thought you had mentioned in an earlier discussion that you would do garbage collection via "ipfs repo gc" once a week. Then again, I could be wrong. 😆 In any event, the "--enable-gc" function shouldn't be necessary at startup as you mentioned.

As for those running your config using Umbrel, I have zero knowledge of that framework so please feel free to fork my install wrapper and included documentation to tweak it as needed for setup using Umbrel.

Looking forward to the updated script going live! 👍

Thanks!

[Cameron-IPFSPodcasting]

Sorry, I didn't get to updating this today. Had other issues to take care of.

I had thought garbage collection ran weekly by default (if StorageMax was reached). I think I was looking at an old config. It looks like it never runs(!), which is more incentive to get that feature working in the next release.

Umbrel is a self-contained docker image that only needs to run ipfs init when the user enables the app. Everything else is already setup.

[CaffeinatedDNB]

No need to apologize. I know how that is all too well! We do what we can when we can. 👍

Thanks for the update.

Regarding weekly garbage collection, perhaps you can add cron job to that end?

If you prefer to put a finer point on it, run the GC function when the system is usually idle (after collecting some system data.)

Or, put it on the admin by adding a cron job and asking them to choose a preferred time and day to run in your documentation.

As for the Umbrel setup, that sounds good. The scripts I contributed can possibly assist, after some modification (e.g. like removing the addition of the ipfs.service file, etc,) to secure the system by installing fail2ban, etc.

[CaffeinatedDNB]

@Cameron-IPFSPodcasting

Keeping busy huh? Or is that an understatement? 🤔😆

When you have a moment, please upload the latest version of the installation scripts and documentation from my recent post last week as it includes some of the suggestions made by "suorcd".

That way anyone that downloads it has the latest and greatest.

Thanks!

[Cameron-IPFSPodcasting]

Updated to the new file from @CaffeinatedDNB which includes a few of the changes from @suorcd.

Closing this PR. Thanks for the updates.