Add cloud sync/backup scripts

[ridgekuhn]

Closes #37

"Sometime next weekend" came a week early and I decided to just hammer out a draft tonight. Please note that I wrote this entirely on my desktop machine and haven't tested it on my RG351P at all yet. (Making the unlikely assumption that I didn't make any mistakes, you should be able to just download the 12272020/update.sh script to your device and run it.) Here's what I ended up with:

Notes:

• Uses rclone
  Configuration file is stored at /roms/backup/rclone.conf (and symlinked to the default path, /home/ark/.config/rclone/rclone.conf)
• Exposes one new script to EmulationStation:
  • /opt/arklone/dialogs/settings.sh
  • /opt/system/Advanced/Backup Settings to Cloud.sh
• Creates directories:
  • /roms/retroarch/saves
  • /roms/retroarch/states
• Overrides:
  • /home/ark/.config/retroarch/retroarch.cfg
  • /home/ark/.config/retroarch32/retroarch.cfg
    (Both changed so savefiles and savestates are now stored in above directories instead of content directories)

Caveats:

• Existing ArkOS users will have to manually move their saves from the content dirs to the new paths. rclone uses the same syntax as rsync, and I've had a difficult time in the past while trying to sync saves in content dirs on my local network (ie, targeting specific file extensions in nested subdirectories). If we could figure that syntax out, we could present the user with an option to choose where their saves are stored and sync appropriately. (It would also mean not having to overwrite their retroarch.cfg files on this update!)
• The savefile script sends the local directories and then downloads the cloud copy, which is handy for people like myself who use RetroArch on multiple devices. However, this might cause the user to download a bunch of savefiles for games they don't have on their handheld. I don't think we can get around this, but maybe it's not an issue since saveRAM/memcards/etc are relatively small?
• Creating Dropbox API credentials is relatively easy for end-users, but Google Drive (and possibly others) seems pretty difficult bc you can't just generate an OAuth2 token from the Google API dashboard and plug it in. (You have to make a request to the HTTP REST API from rclone or other capable app to receive a usable token.)

Todos:

• I forgot to have the update.sh script grant executable permissions to the sync scripts
• The retroarch.cfg.bak files included in the update are the stock ArkOS files and should be generated from the user's existing retroarch.cfg files instead.
• Make "Sync Savefiles to Cloud.sh" parse retroarch/retroarch.cfg and retroarch32/retroarch.cfg for savefile/savestate dirs
• Present select list to user in "Cloud settings.sh" to choose cloud storage string from rclone.conf
• Present select list to user in "Backup Settings to Cloud.sh" to choose cloud storage from rclone.conf (Also, I just realized I forgot to implement the selection variable in this script)
• Create path units based on retroarch.cfg files
• Integrate ArkOS backup script
• Integrate changes to emulationstation.service & es_systems.cfg
• auto-generate additional path units if sort_savefiles/states_enable = "true" in retroarch.cfg
• generate-path-units.sh should automatically enable units if auto-syncing is on
• Fix ArkOS backup script crashing
• Convert to production update:
  • Rename update lock file to production date and regenerate update payload
  • Rename update folder to production date
  • Change repo URL in update.sh to production URL
  • Migrate update.sh to Update-RG351P.sh (or merge Compartmentalize update scripts (for easier collaboration) #101 😉)
• Write documentation for end-users (don't forget to mention paths in retroarch.cfg must be absolute)

@christianhaitian, I've seen msgbox and osk in your scripts. Is there a similar utility for presenting the user with a couple select-boxes?

[ridgekuhn]

@christianhaitian Here's the instructions for generating API tokens for Google Drive: https://rclone.org/drive/. If you follow the example and just name it "remote" in the first step, you can just swap the generated rclone.conf with /roms/backup/rclone.conf for testing

[christianhaitian]

I will review this when I can. Unfortunately there is not a similar utility for select boxes yet. Once I find a solution, I'd like to include the ability to download themes with such a utility.

[ridgekuhn]

@christianhaitian ok cool, as I was writing the PR notes, I realized there's a couple things I should fix up before you try it out. I'll probably get to them tomorrow evening and let u know! Bummer about the select boxes, I was hoping we could let the user set up several rclone configs and select from them, but it looks like we might have to just run with the default "remote" one and they'll have to change it manually.

[ridgekuhn]

Here's what your Dropbox app should look like. I don't think you need to set up a Redirect URI unless you use the rclone config tool. You should just be able to click on the "Generate" access token button and plug the key/secret/token into the example file at roms/backup/rclone.conf

[ridgekuhn]

Just occurred to me that since the user is staring at the TTY output while these scripts are running, we could probably just use bash's select built-in to make the kind of menus we want. I just need to figure out how to make use of the RG351P inputs.

[christianhaitian]

That would be great if possible. Maybe a modification to this is possible to allow this? https://github.com/lualiliu/RG351P_virtual-gamepad

[ridgekuhn]

Most of this is beyond my expertise, but it looks like the msgbox and osk binaries might listen for xbox gamepad input, which is why that executable is launched at the start of various scripts using it? If there's a way to map it to keyboard input, then we could use that to manipulate a selection variable, or possibly use (dialog)[https://linux.die.net/man/1/dialog] (that might actually be ideal). I'll try to investigate this week.

Also, I read up on rclone's filter flag today, and it looks like only syncing the save files from the content dirs won't be too tough after all, so I'm going to make the scripts default to that. If the menu thing works out, I'd still like to have the option for the user to choose between /roms/retroarch/saves and the content dirs

Edit: I'm so dumb, we can just have the script parse the retroarch/retroarch.cfg and retroarch32/retroarch.cfg files to get those paths and not bother the user. 🤦‍♂️

[ridgekuhn]

I think I may have the solution. Instead of dialog, we can use whiptail which already exists in ArkOS. The RetroPie project will come to the rescue, with their joy2key.py script which is used to control dialog in retropie_setup.sh, raspi-config, etc. Finally, we can use some functions from RetroPie's helpers.sh for setup/teardown. (Both files will need a few modifications, of course.) It looks like the whole RetroPie project is released under a GNU license, and since this is all on GitHub, etc, we should be good.

[ridgekuhn]

Today's bounty:
https://youtu.be/XDwlcrnJLCE

[christianhaitian]

How did you achieve this?

[christianhaitian]

I see it now. I was just starting to look at modifying joy2key from retropie myself. Nice!

[ridgekuhn]

@christianhaitian lol, well that worked out, then! It will live in /opt/joy2key/ in case you want to re-use it!

One problem I ran into is that if you run RetroPie's joy2keyStart and your script exits uncleanly, it won't have a chance to run joy2keyStop and the joypad will keep sending keypresses (which is especially troublesome if your analog sticks are "drifting").

To solve that, I've written a wrapper script for it. With the wrapper, you can pass your own script as an argument (plus optional keymap args), and the wrapper will handle setup/teardown/errors.

$ /opt/joy2key/listen.sh myCoolScript.sh [keyMapping1, keyMapping2, etc...]

[ridgekuhn]

@christianhaitian The more I think about it, I would be very annoyed if I had to manually sync every time I played a game, so the savefile/savestate part should be a daemon using inotify to automatically sync in the background if/when wlan0 is up. So instead of manually running Options -> Sync Saves to Cloud, that selection should take you to a whiptail menu which might look like:

1. Select cloud service (current: dropbox)
2. Manual sync savefiles/savestates to cloud
3. Enable/Disable automatic sync savefiles/savestates to cloud
4. Manual full ArkOS backup to cloud

I think rclone should handle partial transfers gracefully, so we wouldn't have to worry about waiting for a sync to finish if the user abruptly powered off. I'll work on the daemon and whiptail menu this week.

[christianhaitian]

Maybe make it optional? Let the user decide if they want that or not. Myself, I would prefer to control that. Sigh...I should read more carefully. I see you already include that option. nice.

[ridgekuhn]

lol, no worries, I also jump the gun a lot! 😂

[ridgekuhn]

joy2key should be final now. Also added the daemonized rclone script. I named it "Arklone", lol. Almost at the finish line!

[christianhaitian]

I noticed that when attempting to run a whiptail script from within emulationstation, it fails to load. However, if emulationstaton is stopped the script runs fine. Thoughts?

[ridgekuhn]

I'm by no means a Linux expert, so I'm mostly just thinking out loud and could be totally wrong! 😅

I tried hooking up the example I posted earlier to EmulationStation and it complained about having no TERM var set. It looks like EmulationStation is started by systemd and runs as a detached process, so I guess that makes sense:

ark@rg351p:~$ ps -aux | awk '/USER/ || /emulationstation/ {print}'
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ark        310  0.0  0.0   2048   584 ?        Ss   Jan01   0:00 /bin/sh /usr/bin/emulationstation/emulationstation.sh
ark        327 25.8 19.4 1063584 178856 ?      Sl   Jan01   3:09 /usr/bin/emulationstation/emulationstation

So, I think this line in es_systems.cfg has probably worked until now because nothing has ever cared about receiving stdin specifically from a TTY:

<command>sudo chmod 666 /dev/tty1; %ROM% > /dev/tty1; printf "\033c" >> /dev/tty1</command>

This probably isn't valid XML, but I think something like this may work.

<command>%ROM%  < /dev/tty1 > /dev/tty1</command>

Another option might be modifying the EmulationStation systemd unit. I came across settings for redirecting stdin/stdout and specifying a path to a tty device the other day when I was writing the rclone daemon. I'm not sure how this might affect EmulationStation. Possibly not at all if those values are currently undefined?

[ridgekuhn]

Just pulled the es_systems.cfg off my RetroPie and I think I'm on to something. It looks like they just use /dev/tty. I can't find where they launch EmulationStation from, but that implies that it's connected to a tty when it does. I'll do some digging on this after I finish the whiptail script.

<command>sudo /home/pi/RetroPie-Setup/retropie_packages.sh retropiemenu launch %ROM% &lt;/dev/tty &gt;/dev/tty</command>

[christianhaitian]

Getting close. The following allows the whiptail to show when launched from emulationstation but then the controls don't work.

sudo chmod 666 /dev/tty1; %ROM% 2>/dev/tty1; printf "\033c" >> /dev/tty1

[christianhaitian]

Also the terminal mode has to be set either in the listener or the actual script
export TERM=xterm-color
Then unset when the script is done.
unset TERM

[ridgekuhn]

It’s bc stdin isn’t getting redirected to the script, we’ll need to send all three streams to get user input, display output, and capture return vals. I don’t understand why `2>` makes it display though, are u swapping stdout/stderr inside your script before whiptail is called? I think other env vars get set in addition to TERM when u open a TTY. Might be worth looking into if whiptail uses those as well but imo, if we can just launch emulationstation at say, /dev/tty1 instead of running as a bg process, that would be ideal, I think!(?) My apologies, I thought I’d be working on this today but I haven’t had time to finish up the other stuff yet.

…

Sent from my iPhone

On Jan 2, 2021, at 7:59 PM, christianhaitian ***@***.***> wrote:  Also the terminal mode has to be set either in the listener or the actual script export TERM=xterm-color Then unset when the script is done. unset TERM — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[christianhaitian]

Yup. You can see it here: https://github.com/christianhaitian/oga_controls/blob/13da587f264b419488656479d88eddd9c8feafb8/main.c#L1892

[ridgekuhn]

Wonderful! Thanks!

[ridgekuhn]

Sorry, disregard the earlier post, jumped the gun and found some bugs.

[ridgekuhn]

Sorry about last night’s false alarm! 😅 I didn’t realize I actually had the modified versions of emulationstation.service and es_systems.cfg on my device already, and that they’re still needed with oga_controls.

@christianhaitian To recap bc we talked about it like a billion comments above:

• The modified emulationstation.service connects EmulationStation to tty1 instead of running detatched by systemd


• The modified es_systems.cfg changes the "Options" <command> to redirect all I/O streams to %ROM instead of only stdout


• This should be backwards-compatible with all existing scripts, since they’re already opening tty1 for reading/writing and redirecting I/O to it

• For new scripts, you shouldn’t have to write those sudo chmod 666 /dev/tty1 and redirection hacks anymore. Hopefully running third-party scripts should be easier now, and first-party ones more portable.


• The modded versions live directly in the install script (and the uninstall script reverts them).

I’ve nuked the entire pull request’s ref history down to one commit for sanity's sake when merging. I picked 10092021 as the update folder since it's recent and unused. I'll leave it up to you to integrate it into the individual device’s update scripts, so each device can be tested first. All you have to do is call the install script.

Anyone who wants to test it on your own device, run the install script manually. If you already have the previous version installed, you’ll have to do a semi-manual uninstall first:

#!/bin/bash

# Uninstall the old version of arklone
sudo /opt/arklone/uninstall.sh

# Remove joy2key
sudo rm -rf /opt/joy2key

# Remove the settings dialog from EmulationStation's Options menu
sudo rm "/opt/system/Cloud Settings.sh"

To install the new version:

#!/bin/bash

cd ~
wget https://raw.githubusercontent.com/ridgekuhn/arkos/cloudbackups/10092021/install.sh -O ./installArklone.sh
chmod u+x ./installArklone.sh
./installArklone.sh
rm ./installArklone.sh

To uninstall the new version:

#!/bin/bash

cd ~
wget https://raw.githubusercontent.com/ridgekuhn/arkos/cloudbackups/10092021/uninstall.sh -O ./uninstallArklone.sh
chmod u+x ./uninstallArklone.sh
./uninstallArklone.sh
rm ./uninstallArklone.sh

As far as testing and maintaining goes, I need all the help I can get. I only have an RG351P, so it’s difficult for me to keep up with new features. I might sell it and get an RG351MP soon-ish. There are probably ports, etc missing that will need to be integrated; existing ports may sync irrelevant files, etc.

I’ve tried to make this updated codebase easy to extend, and documented everything as much as I could. See the developer docs and inline comments in the code.

If you find any problems with arklone itself, please open an issue on the arklone repo.

#37

[ridgekuhn]

I've added some quality-of-life features like progress bars and being able to manually send/receive all directories at once. The input listener script also got moved to a new path, so the commit I just made addresses that for launching from EmulationStation

[ridgekuhn]

More bugfixes and new features. I moved the input listener again, hence this most recent commit. (This should be the last time, as I've already forked the project for a RetroPie version.) This PR should be good for all future updates, the user will just get the most recent release from the master branch whenever this PR gets merged in.

Video with ArkOS and RetroPie: https://youtu.be/-jmoS1xwVcI

[skykiller30]

This would be an awesome add (the ability to sync across devices via the cloud) I currently use this tool with retropie to do it all via OneDrive. https://github.com/Jandalf81/rclone_script
His script takes advantage of the runcommand.sh script inside of retropie and allows you to pass in game names and system names. Something like that would be nice. But even a simple option i could run that could grab the saves from the cloud... I would play for a while.... then run a second script that would sync all my new saves/states back into the cloud.

If you could get the files structure to have game saves and states in one folder and then subfolders based off the system directory names I could use my ArkOS devices along with my RetroPie.

..just a thought.

[Corey-Evans]

I ended up just writing a simple rclone script and adding it to the Tools system in Emulationstation.

[ridgekuhn]

@skykiller30

arklone lets u choose your own save settings via retroarch.cfg and supports all savefile/savestate directory setting combinations, see arklone docs and/or retroarch docs. if u already have an rclone.conf, that's the most obnoxious part of setting up arklone, so u can skip all that and drop yours right in and be good to go.

for arkos:
https://github.com/ridgekuhn/arklone-arkos/

for retropie:
https://github.com/ridgekuhn/arklone-retropie/

video of it working on both arkos and retropie:
https://youtu.be/-jmoS1xwVcI

[christianhaitian]

Has anyone tried this? https://syncthing.net/
It doesn't sync to a particular cloud but does some kind of sync among multiple devices. I haven't tried it yet but someone else told me about it and seems to work on here.

[ridgekuhn]

That’s interesting! I only perused the docs quickly but it seems like each instance is a client and server? Might be too heavy for low-RAM devices but it’s got me curious!

…

On Jan 16, 2022, at 8:12 AM, christianhaitian ***@***.***> wrote:  Has anyone tried this? https://syncthing.net/ It doesn't sync to a particular cloud but does some kind of sync among multiple devices. I haven't tried it yet but someone else told me about it and seems to work on here. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

[bphenriques]

Hello, this looks great! Great work!

I was in the middle of doing something of the sort until I noticed that there was already some work for this 😅 I may have useful dialog interfaces that may be useful to solve file conflicts - I can share once I have something more presentable 😃

Just quick questions:

• Would this work for any save-games under ports?
• Can I set directories so that /roms/arcade gets synced to dropbox:/roms/fbneo (reason: Emudeck uses fbneo folder)?

Edit: I think so to both using manual sync

[ridgekuhn]

@bphenriques Yes, it works with ports. I've included a few but I'm not familiar with the majority of them so it's up to u, all u need to do is create a systemd path unit file pointing to the proper directory and optionally, a filter file. See the developer docs for more info and examples.

I'm not familiar w EmuDeck but I would suggest just moving your ROMs from the arcade dir to the fbneo dir on ArkOS (and if an fbneo dir isn't supported by ArkOS, just add a new entry to /etc/emulationstation/es_systems.cfg for it)

[christianhaitian]

hey, I'm closing this as it's not something I will natively integrate at this point as I've lost interest in it and many seem to prefer using syncthings now. I also noticed you've created another git for it here:https://github.com/ridgekuhn/arklone-arkos that folks can refer to if they want to still use this tool. Thanks for your work on this. I wish you well.

[ridgekuhn]

lol, i thought u closed this like a year ago! 😂