[Feature]: Option to sync files (data) between mesh chat servers

[hickey]

Contact Details

No response

Enhancement Type

File sharing

What is your idea or what can be improved?

This feature request was originally stated in https://github.com/kn6plv/meshchat/issues/21.

[hickey]

Because of the amount of content in https://github.com/kn6plv/meshchat/issues/21, the following is a transcript of that issue to simplify having to go between 2 issues.

@csadams

Long term feature request...
It would be nice for redundancy and EmComm reliability if Mesh Chat servers could sync actual files between the servers. They currently synch the list of files on other servers, which is great most of the time, but that doesn't help if the other server goes down. This would require some options and configurability, so that server owners could pick what and how much is synced, to avoid overloading storage on servers with limited storage space... as well as avoiding other potential issues. Maybe this could be done with tagging files for syncing on one or both ends? Or having sync and non-sync folders? I'm not sure what the right solution to that is.

@hickey

This is an interesting thought. Let me ask a question or two to bat the idea around and flush out a few details.

• Do you have any thoughts or ideas on the transfer mechanism?
  • Yes, scp is there, but it may be better with something like rsync so that transfers can be restarted for unreliable links.
• Any ideas of how to identify server owners and/or how to select files to sync to a server?
• Should there be a size limitation on files that get transferred?

Again, this is to start a discussion to build a better framework of how this may get designed.

@csadams

Thanks for the questions.

• I don't know much about transfer mechanisms, so can't help with that, sorry.
• Server owner should be easy to identify in most cases... you can safely assume it's the callsign on the node that the server is hosted on or connected to. Is that what you meant? Oh... maybe you're asking about giving the server owner more control/options than other users, which I guess would require them to be able to log in somehow? I can see how that could add a lot of complexity. Hmm... not sure, but given how nothing is truly "secure" on the network, there are probably some relatively simple solutions?
• As for selecting files to sync... how about for files uploaded on the local server, you can toggle on/off (checkbox?) the ability for them to be synced out to others (maybe make it default on)? And for files seen on other servers, give the option to sync them to the local server (default off, only available if they're enabled for sharing on the server they're hosted on)? Just a first idea... would need to brainstorm on it a while. My thought process is that if I upload a really big file, I wouldn't want it to sync out automatically to others who have more limited storage... and if I'm managing a server, then I would want to control what comes in. One more step could be to allow some filtering like only sync files under a certain size... or do/don't sync certain file types/extensions. Anyway, lots of options, happy to discuss/brainstorm.
• As mentioned above, maybe provide some filter to the server owner to say that only files under a certain size get synced over? Or maybe make it so that under size X they auto-sync, between X and Y it's optional (owner needs to confirm each file to sync), and above Y never sync. Just an idea. Could also just set a max size (really big) to not pass over the network unless downloading?
• Lots of possibilities... start small with paths to ramp it up?

@csadams

Oh, one more thought on the importance of this idea of file sharing...
As background, earlier today I was reminded that Winlink has a maximum message size of 120kb, including attachment!! So you really can only pass tiny files via Winlink, which is very unfortunate for a Mesh Network enabled EmComm scenario. It makes a lot of sense to limit size in the packet and HF world, but would be interesting to see if Winlink could enable larger messages/attachments when going to a Telnet Post Office via Mesh. Of course you never know what other path(s) the message will end up taking, so would need some feedback or error reporting if it hits a size limited connection... but that's for Winlink to figure out.
So... what does this have to do with MeshChat? Well, if I can't send a large file via Winlink, maybe I can upload it somewhere on the mesh, and then send a link to the file via Winlink. And MeshChat is one of the few file sharing services on AREDN right now...there are others, but MeshChat is much more common (in my experience here in the Bay Area).
So long story short.... we need a way to share larger files (at least upto a few MB) on the mesh, and since Winlink isn't up to the task, maybe MeshChat can do it... with the reliability of redundant hosting of files!

@hickey

OK, multiple thoughts and comments related to the discussion and I will try to keep them straight as to not add confusion.

Concerning the transfer protocol, rsync is an available package for OpenWRT and weights in at just under 180KB. While this can address the actual transfer of files it will not address the coordination of moving the files around. That will take some thought and planning. On the other hand something like syncthing would handle much of these aspects and allow building of decentralized storage but it weighs in at 8.5MB. I think that is pushing the limits of the filesystem on nodes.

If we wanted to move the file sync feature away from nodes or maybe have a simpler file sync to nodes (but it would be very manual and maybe a bit clumsy) then syncthing is a more viable option. Having the syncing of files to Raspberry Pi / Linux host / VM would be quite doable with syncthing and things would just work. This would allow both one way and bi-lateral syncs between systems on the mesh network. In fact, a system would not need to really be a MeshChat server to have files synced to/from it. This opens some thoughts of having automation produce file content.

On the matter of system owner. This is not so simple but also moderately simple to solve. Not so simple in the sense that I may own the node (say a hAP in my house), but I do not have the technical skill to maintain the MeshChat installation on the node. So I have a fellow ham next door that maintains the installation. I would consider myself the system owner and the fellow ham the "service" owner. Ideally we could drop call signs into the config file for enabling the additional administrative functions. But then anyone could use anyone else's call sign to get to the admin functions. I really don't want to go down the path of "oh we need passwords on each callsign" and then building an infrastructure to distribute those passwords to all the MeshChat instances (that is not just a rat whole, but a rat condominum!). The nice thing about MeshChat is the simplicity that any ham can use it at any time without having to setup accounts and such (i.e. low bar for adoption).

Thinking about this after posting my initial questions it occurred to me that it is the same problem that I had with AAM Manager. In that case I store a password in the configuration file to enable the admin functions. I think that is probably the simplest solution here also. Then any user with the password could then "log in" and manage the syncing of files or other admin tasks.

The Winlink message size is something that is completely controllable on the server side. I remember seeing a reference that a Winlink post office on an AREDN network was updated to allow attachments in the multi MB range. But Winlink is not really a good file sharing application. But then MeshChat is only a little bit better than Winlink for file sharing.

Another problem that is going to be encountered moving forward is how to properly locate the file on the MeshChat network. To illustrate the problems I see popping up here, let's take a network of 3 MeshChat servers (A, B and C). Server A and server C share files and server B only sees the listings (much like it works today). User on server A clicks on a file and that file is downloaded from server A. The same happens with server C (i.e. download from server C). But when a user on server B clicks a file, where does it download from? Again let's assume that the file originated from server A and server B references the file on server A. Something in the network topology prevents the user on server B from connecting to server A, so the user is unable to get the file. But the file may be available to the user on server C. How does the user get the file from server C?

I don't think that there is any good (i.e. easy) why to automatically direct the user to server C. The only thought I have at the moment is that when the user clicks on a file to download they are presented with information where the file is located on the network and they can choose which one to download. That is fine when there is just a couple of servers. What happens when there is a large AREDN infrastructure with many MeshChat instances (such as the Tennessee SouthMesh or the Bay area network)? Is this idea scalable when there are 10, 20 or 30 MeshChat instances? I don't know yet.

Also thinking about how this is going to affect the way that the file sharing code communicates with the other MeshChat servers, I suspect that this will be a breaking change and would need to be marked for a v3.x milestone. I can not say for certain given that I have not even looked at any of the file sharing code, but I think that this is going to be a drastically different code base that may make being able to be interoperable with the older nodes difficult. As such I am going to make a v3.0 milestone and attach this issue to it.

@csadams

Thanks for all the thought on this!

Yeah, it's possible that someone manages the service attached to someone else's node, but I think that's going to be very uncommon, and we can assume that those 2 folks will have a reasonable level of communication between them. In the end, the service is run through a node, and I figure whoever has their callsign on it is responsible (at least from the FCC point of view?).

Definitely don't want to make this require password management infrastructure. Simple is good.

I'll have to look into the Winlink limits more. I found the 120k limit mentioned in their docs, but didn't dig deeper. And agreed, Winlink isn't great for file sharing, but being able to send photos, spreadsheets and forms is one of the things that it's touted to be super useful for. I'll try to do some more digging on that. All that makes me wonder if it would be better to build a dedicated file sharing service for AREDN, that integrates easily with MeshChat and Winlink, etc.

Regarding your 3 service illustration. Maybe have the file linked for direct download if it's local on the service, and if not local, then instead provide a link to the host service's Files page? Then again, what do you do when the same file is on A and C, but not on B.... does B provide a link back to A or to C?? Maybe the first one where it was uploaded or synced from? Though that might cause consistency issues if it's updated with a newer file of the same name on one of them?? Or yeah, like you suggested... provide a list of options for download links. Hmm... complexity.

Here on the Bay Area Mesh, last week we had only 3 mesh chat services on the default "MeshChat" zone (and a few others on custom zones for their groups). I helped one more set up from scratch, and assisted another in configuring the zone correctly, so now we're up to 5 servers in our default zone. That said, if it becomes more popular, you're right that we could end up with 20 or more servers, which would indeed cause some UI issues for download links, etc.

Yeah, probably good to think about this more for a v3.x... or as mentioned above, a separate file sharing service??

[hickey]

@csadams, Sorry I missed your last comment until now when I moved the transcript over to this issue.

I sort of like the idea of a separate file sharing service that is mesh aware. That would allow MeshChat to focus on just chat functions and defer any file references to the file sharing service.

There are a number of file sharing solutions out there and some of them are on AREDN networks, but none of them are mesh aware to where they just "work". I can see a new service being created out there called MeshShare :-)

[csadams]

Yeah, would be great to think about what a "MeshShare" (name TBD) service would looke like.
I currently have an FTP service on my RPi, so that's one way to do it... but with most common browsers having dropped FTP support, it's unfortunately not so attractive anymore. Hopefully it wouldn't be too hard to build a simple system that allows file upload & hosting, browsing & download, some sort of folder structure, and managed syncing with other servers. I'll try to think a bit more about what the UX could look like.

[nc8q]

Not knowing the available storage size of linked MeshChat servers,
I feel that syncing files with other MeshChat servers should be the
choice of the MeshChat owner of which file to receive.
Then again, any MeshChat user can download from one MeshChat
server and upload it to a different MeshChat server.
Thumbs down on the issue of automagically syncing MeshChat files.
73, Chuck