Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request: Allow federated storage to be stored locally [$5] #8459

Open
sunjam opened this issue Feb 20, 2018 · 15 comments

Comments

@sunjam
Copy link

commented Feb 20, 2018

Steps to reproduce

  1. Mount remote federated share
    2.take remote federated share offline
    3.attempt to access remote federated share data locally

Expected behaviour

Federated data from remote host will still be accessible from the web client while the remote host is offline. Federated share owner can allow those with remote access to keep local copies of the shared data.

Actual behaviour

All access to federated remote shares is lost until host goes back online.

Server

Nextcloud 13 and Nextcloud 12.0.5


There is a $5 open bounty on this issue. Add to the bounty at Bountysource.

@andreicociuba

This comment has been minimized.

Copy link

commented Feb 22, 2018

wouldnt this be a privacy/security concern?

i mean, federated content can be taken down for legitimate reasons. how do you propose to ensure that any caches are taken down when that happens?

@sunjam

This comment has been minimized.

Copy link
Author

commented Feb 22, 2018

What I’m requesting is persistent federated shares. The concern in this case is not security, but availability and uptime.

Example: most admins might very well have multiple federated instances of Nextcloud with no concern over internal content needing to be taken down. The issue in this case is when content needs to stay up for legitimate reasons, but having shares on a single instance becomes a point of failure by going offline.

@MorrisJobke

This comment has been minimized.

Copy link
Member

commented May 31, 2018

cc @nextcloud/sharing

The last state here was that we don't want to implement the whole syncing logic on the server as well and that it then would also work against the "your data is on your server" principle we have right now.

@e-alfred

This comment has been minimized.

Copy link

commented Jun 12, 2018

@MorrisJobke Well, this stance is a bit of an "security by obscurity" thing because if I share data to another Nextcloud they can copy this data anyway (for example if they download it with the desktop client or simply copy it in the web interface let alone automatic snapshotting on Windows or Linux or just making screenshots of documents/images, I think there are 100s of ways). If the data is handed out to a foreign entity, there is no control over what happens with it. Everything boils down to a technical solution for a social problem (trust in those that the data is shared with) which mostly won't work except maybe for superagressive DRM (which also doesn't work at the end of the day because people go against it with enough determination and it is contrary to FLOSS principles).

Aside from that, allowing sync data between federated servers could be a very useful and easy backup solution and take away the problem of the desktop client showing errors all the time if a federated share is offline even though ones own server is fully functional.

@MorrisJobke

This comment has been minimized.

Copy link
Member

commented Jun 12, 2018

@MorrisJobke Well, this stance is a bit of an "security by obscurity" thing because if I share data to another Nextcloud they can copy this data anyway (for example if they download it with the desktop client or simply copy it in the web interface let alone automatic snapshotting on Windows or Linux or just making screenshots of documents/images, I think there are 100s of ways). If the data is handed out to a foreign entity, there is no control over what happens with it. Everything boils down to a technical solution for a social problem (trust in those that the data is shared with) which mostly won't work except maybe for superagressive DRM (which also doesn't work at the end of the day because people go against it with enough determination and it is contrary to FLOSS principles).

True point. 👍

There is still the technical issue with keeping things in sync and how to deal with conflicts. It basically means we need to implement the sync engine of the desktop clients on the servers as well.

@nachoparker

This comment has been minimized.

Copy link
Member

commented Jun 12, 2018

This would indeed be very interesting! +1

@rullzer

This comment has been minimized.

Copy link
Member

commented Jun 12, 2018

The problem is also that we can't enforce a lot of things on the receiving end.

While it is true that once you have access you can just copy the files. But then of course there are more things at play. For example the data owner might have acces rules in place. Of have rules in palce that forbid the upload of certain files.

@sunjam

This comment has been minimized.

Copy link
Author

commented Jun 12, 2018

Hi, I'd like to mention a couple scenarios I've found. Thanks for all the comments!

  • The single point of failure that plagues self hosters running a small Nextcloud server.
  • The individual points of failure that occur when a series of federated Nextclouds attempt to federate a folder between them... I've realized how I lack the infrastructure of Dropbox or Google by using a Raspberry Pi, but being able to distribute data with systems I choose can address this problem.
  • You spin up multiple Nextcloud instances and actually do own + admin all of them. Mirroring my data between them is the first thing I want to do.
  • You share a photo publicly via url with other Nextcloud servers who could help host it, but your computer goes offline.
  • You would like a secondary instance to have a copy of the federated data, even if it is not regularly synced. It might not be fully up to date, but it certainly beats everything having gone offline.
  • Getting to interact with files that are stored locally is easier in Nextcloud. :)
@putt1ck

This comment has been minimized.

Copy link

commented Jun 16, 2018

Some conversation around a related issue was had sometime back:

https://help.nextcloud.com/t/hybrid-on-prem-and-cloud-replication-of-storage/2773/24

Biased, me, but I think NC level replication is a really robust approach that is ideal for orgs with multiple offices; covers offsite backups and availability when connections are down. Would need further thought on how to replicate DB tables for apps (a table that apps can chose to write "and me!" notifiers into?), but otherwise it's not a problem that hasn't already been solved.

@bpcurse

This comment has been minimized.

Copy link

commented Jun 16, 2018

If mirroring was implemented, a "do not cache on federated instance" flag, that all nextclouds respect, should be considered.
Meaning that either the sharing instances' admin can set this globally and/or the user as a simple checkbox "per share".
As mentioned above this should not be seen as a security measure but more as a "control where the data is physically stored and who has internal access" feature for GDPR compliance.

@nextcloud-bot nextcloud-bot added the stale label Jul 17, 2018
@jcklpe

This comment has been minimized.

Copy link

commented Oct 30, 2018

I'm very much a newbie here so maybe what I'm fixing to suggest is gibberish, but I'm curious if this feature could be implemented using some kind of p2p technology similar to bitorrent?

Instead of distinct servers sharing access to data through federation, each one only storing their own data, you could have the servers in the federated network acting as a cluster, each of them maintaining redundant copies etc, so if one of the servers goes down, even the server that "owned" the original data would be able to recover that data from copies on the other servers. You could have it so that instead of downloading that data from a single server, it could actually download it from all the other servers (similar to how bittorrenting allows you to connect to thousands of other people "seeding" parts of the download to you).

You could perhaps even use this as a sort of rudimentary CDN system? So if you and 5 other friends who all live in different geographical locations were to federate, you could store the large binary assets for a website (images, video, etc) in the federated shares, and then the website could source from whatever particular server was nearest to it in order to get the best performance possible.

Currently the only thing that keeps me from self hosting my own website is because of how easy my host makes it to use cloudflare CDN, and that's an absolute must to make sure my website feels nice and snappy. But I have friends who live all over the world, and I could see it being possible for us to create our own ad hoc CDN through something like this.

I also have read that it's possible to self host your own CDN, but I'm pretty sure that's "self host" in the "control of the machine down to the root level" and not self host in the "own the physical machine and have it in your bedroom closet" kind of way.

Again though, I'm not the best when it comes to backend stuff so maybe I'm just talking nonsense. It feels like a p2p powered CDN would be a HUGE deal though. It would give NC a huge extra value added proposition. People could form federated alliances that provided distributed server space on the level of Google or AWS cdn etc. That would be a huge deal.

@nextcloud-bot nextcloud-bot removed the stale label Oct 30, 2018
@sunjam

This comment has been minimized.

Copy link
Author

commented Oct 31, 2018

@jcklpe Hi, what you are requesting is different from this request. I'm requesting that the Nextcloud web app optionally store federated data locally. This is to keep access to the data or could possibly act as a basic mirror if that federated data goes offline.

What you are requesting is interesting, but should be posted as a separate thread. See this issue #11653 requesting Zot protocol support (account mirror, merge, remove across many servers, aka Nomadic Identity) in php and this issue requesting greater integration with Syncthing #8384

@sunjam

This comment has been minimized.

Copy link
Author

commented Jun 28, 2019

Bounty Added! Please consider donating to it.

@sunjam sunjam changed the title Request: Allow federated storage to be stored locally Request: Allow federated storage to be stored locally ($5) Jul 1, 2019
@QEDeD

This comment has been minimized.

Copy link

commented Sep 27, 2019

The bounty is now 60$ @sunjam

@sunjam

This comment has been minimized.

Copy link
Author

commented Oct 2, 2019

The bounty is now 60$ @sunjam

Looks like bounty was raised for Zot integration, which is awesome. Request here is basically to optionally allow admins to store copies of federated, remote files. Currently, they are only remotely accessible from the webui + literal copies are stored across desktop and mobiles devices.

@kesselb kesselb added the bounty label Oct 2, 2019
@kesselb kesselb changed the title Request: Allow federated storage to be stored locally ($5) Request: Allow federated storage to be stored locally [$5] Oct 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.