Repositories on extSDCard #29

Open
jonnojohnson opened this Issue Jun 5, 2014 · 119 comments

Projects

None yet
@jonnojohnson

SyncThing works great for repositories created on the internal SDCard. However if I try to create a rep on my external SDCard it doesn't complain. The other node attempts to sync & just shows sync progress set at 0%.
I'm running Android 4.4.2 on a GS4 & I know external SD card access for apps got restricted more since 4.4 but I think SyncThing is one of those apps where it makes complete sense to have ext SD access.
In the short term perhaps SyncThing for Android could complain if ext SD card access is not an option?

@Nutomic
Member
Nutomic commented Jun 6, 2014

I'll need a logcat for that. Unfortunately, I don't have an external sd card to test with myself.

@jonnojohnson

I'll try to do that when I get chance. I don't suppose you can point me to some instructions how to get the log? Not much of an Android expert.

@jonnojohnson

Incidentally I think this article pretty much explains the issue: http://www.androidcentral.com/kitkat-sdcard-changes
Essentially in 4.4 Android requires an app to have write permissions for the directory it wants to write to and external SD cards are FAT formatted which don't have permissions.
I think the app has to create a single folder on the ext SD card which it then has permission to write to. All repositories would need to be in there.

@Nutomic
Member
Nutomic commented Jun 6, 2014

I searched around, and it really seems there is no way to write to external sdcard without root. But I will add a directory chooser for repo creation in #5, and might add the external app dir to that.

@Nutomic Nutomic added the bug label Jun 8, 2014
@Nutomic
Member
Nutomic commented Jun 27, 2014

This should work with the new android.content.Context.getExternalMediaDirs() method in Android L.

@zombielinux

So is that new method already in the commits or is that coming down the pipes still?

@Nutomic
Member
Nutomic commented Jul 1, 2014

Nope, not implemented yet.

The above API only works on Android L (though it might come via support library), so I'll have to check which folders are writeable on external sdcard on older APIs.

@stilvoid
stilvoid commented Jul 2, 2014

It's worth noting that PowerTools is able to sync to folders in the emulated SD card. No idea how it does it but I've got it working just fine on my Nexus 4 (android 4.4)

@Nutomic
Member
Nutomic commented Jul 2, 2014

Okay I researched this a bit further:

  • 4.4 only: using Context.getExternalFilesDirs(), we can write to [device]/Android/data/com.nutomic.syncthingandroid
  • in 4.3 and lower, sdcard can be accessed directly, either by scanning /mnt/ or using eg Environment2 (which is commented in German, haven't seen that before for Android)
  • Using Storage Access Framework, we can access all locations (but folder support is only added in Android L afaik)

Of all these, option 3 seems to be the best, assuming it is accessible through support libraries. Otherwise, we might have to implement all three.

Edit: Looks like Context.ACTION_GET_CONTENT and Context.ACTION_OPEN_DOCUMENT would be even better (those would replace the folder chooser, but on 4.3 and lower, a seperate file browser app would be needed).

@ferongr
ferongr commented Jul 4, 2014

At least you can switch to the WebUI and enter the paths manually. Since the majority of devices are not running 4.4 they are unaffected by Google's boneheadedness it's be nice if the path picker could navigate anywhere around the filesystem.

@Nutomic
Member
Nutomic commented Jul 4, 2014

I plan to remove the internal file chooser and use external file browser using the intents in the edit above. The only disadvantage is that it won't work if the user has no file browser installed (on 4.3 and lower). So maybe the internal file chooser should be kept as a fallback for that. I'll look into that.

@ferongr
ferongr commented Jul 4, 2014

I don't know of any ROM that can't satisfy the file chooser intent, even
official manufacturer ones so you should be ok.

On 4 July 2014 18:42, Felix Ableitner notifications@github.com wrote:

I plan to remove the internal file chooser and use external file browser
using the intents in the edit above. The only disadvantage is that it won't
work if the user has no file browser installed (on 4.3 and lower). So maybe
the internal file chooser should be kept as a fallback for that. I'll look
into that.

โ€”
Reply to this email directly or view it on GitHub
https://github.com/Nutomic/syncthing-android/issues/29#issuecomment-48057160
.

@Nutomic
Member
Nutomic commented Jul 4, 2014

I'm pretty sure Android 4.3 and below doesn't have any file browser by default.

@ferongr
ferongr commented Jul 4, 2014

I don't know about AOSP but many manufacturer ROMs and CM do. But it needs
testing

On 4 July 2014 22:43, Felix Ableitner notifications@github.com wrote:

I'm pretty sure Android 4.3 and below doesn't have any file browser by
default.

โ€”
Reply to this email directly or view it on GitHub
https://github.com/Nutomic/syncthing-android/issues/29#issuecomment-48067366
.

@Nutomic
Member
Nutomic commented Jul 4, 2014

Yeah I'll do a test release first once I have it implemented.

@Nutomic
Member
Nutomic commented Jul 6, 2014

OK this won't work as well as I thought, because the DocumentsProvider returns an URI, and there's no general way to get file path from that.

@calmh Would it be possible to specify the repository folder as an URI?

@micahh2
micahh2 commented Jul 7, 2014

Can't you also use specific folders on on sd card? I'm not sure at this moment but something like this: com.exampleappname.subapp

@Nutomic
Member
Nutomic commented Jul 7, 2014

Right, but it would be better if we could also write to sdcard root (and I'm not quite sure how to get that path in Java).

@calmh
Member
calmh commented Jul 7, 2014

@Nutomic What would we then do with the URI? We need a path...

@Nutomic
Member
Nutomic commented Jul 7, 2014

At least Java can perform normal file operations on an URI using File. Maybe this is also somehow possible in Go.

Edit: The URI points to a local directory where we have full read/write permissions.

@calmh
Member
calmh commented Jul 9, 2014

Sounds like we could just strip any leading file:// and get a path then?

@Nutomic
Member
Nutomic commented Jul 9, 2014

That's what I tried basically, but Android uses its own URIs, they look like content://com.android.providers.media.documents/document/image:62 (for an image in this case, but similar for folders).

So we'd really have to access it through the URI.

@calmh
Member
calmh commented Jul 10, 2014

What we have available is a file system. If the URI can be mapped to a local directory as you say, we can write to that local directory. Accessing the URI as is would require going through some API, I guess, which sound like something that would need to happen Java-side or wherever.

@Nutomic
Member
Nutomic commented Jul 10, 2014

Ok so I guess I'll have to see myself how this could be done.

Or I might open an issue on golang for Android URI support (as part of general Android support).

@Nutomic Nutomic added a commit that referenced this issue Jul 27, 2014
@Nutomic Nutomic WIP Implemented folder picker using intents (ref #29).
This does not currently work, because golang does not support using URIs
as files, and we can't convert the URIs to paths.
7b23dc8
@sp00kie
sp00kie commented Aug 5, 2014

Would really appreciate it if you could find a workaround for this STUPID google decision. All of the stuff I want to sync is on my SD card! Who stores media in their main memory?

@crsaracco

Is it possible to let us type in the directory manually?

@Nutomic
Member
Nutomic commented Sep 8, 2014

You can always use the web GUI ;)

@crsaracco

Oooh huh, I hadn't thought of that. Awesome.

@martianrock

How do you edit path using web GUI? For me it is non-editable in GUIs on android and on Win.

@Nutomic
Member
Nutomic commented Sep 12, 2014

@martianrock Looks like syncthing doesn't support that, you'll have to recreate the repo (maybe this should also be changed in the native GUI?).

@calmh
Member
calmh commented Sep 12, 2014

Changing a repo's path can have surprising consequences and until that's sorted out (if ever) I recommend disallowing it.

@Nutomic
Member
Nutomic commented Sep 12, 2014

Alright I adjusted this in the native UI as well.

@TauPan
TauPan commented Sep 29, 2014

Would this help somehow? http://forum.xda-developers.com/showthread.php?p=44370296#post44370296

I've just tried this on my phone and it enables TitaniumBackup to make backups to my micro sd. However Syncthing isn't done scanning the folder yet. I just wanted to give a quick pointer before closing the laptop and starting the tv ;) (I prefer one screen at a time.)

@Nutomic
Member
Nutomic commented Sep 29, 2014

@TauPan No, because it requires root access. Also, it still doesn't give a proper API to get external device paths.

To get the paths, getExternalFilesDirs() might be a solution that I might try and implement. However, this only works on Kitkat+, and is somewhat limited.

@TauPan
TauPan commented Oct 7, 2014

@Nutomic Sorry, I thought it might give a hint regarding the changed permission names between android releases ('media_rw' as opposed to 'sdcard_rw'). But I have only very cursory knowledge about these things.

Nevermind. At least the link provides a workaround for rooted devices.

@Nutomic
Member
Nutomic commented Oct 13, 2014

Okay here's an updated apk that should allow paths on external storage:
ARM
x86

This will only work for Android Kitkat

However, that accounts for 70% of app installs, and there's no API for this on lower versions, so it's probably the best we'll get.

The apk will install next to your current install, so don't bother setting up syncs. Just tell me if it works, and hopefully it will go into the next version :)

@ppilk
ppilk commented Oct 16, 2014

Hello and sorry for my bad english
may be that i have some helpful information for this issue.

I test syncthing from version "syncthing-linux-amd64-v0.8.18"
to "syncthing-linux-amd64-v0.10.1+35-g598ce4b".
On android 4.2.2 and syncthing 0.9.18 a repository on external sd (/storage/extSdCard) work well.
Since syncthing 0.5.x on android it is possible to create a repository on external sd with the gui,
but if i restart the app syncthing 0.5.x stopped working.
If you need more information please give me some information how to debug the app on android.
Thank you for your great work an i wish you a lot of success.

@Nutomic
Member
Nutomic commented Oct 16, 2014

You should post a logcat (description here), and then open a new issue ;)

(because your specific problem seems to be a different one)

@Ferk
Ferk commented Oct 20, 2014

That's what I tried basically, but Android uses its own URIs, they look like content://com.android.providers.media.documents/document/image:62 (for an image in this case, but similar for folders).

So we'd really have to access it through the URI.

Doesn't uri.getPath() do the trick? Note that Android has its own implementation of the "Uri" class.

It would be ridiculous if it wasn't possible to do this conversion in the Android API, how else would anyone map it to a file.

@Nutomic
Member
Nutomic commented Oct 20, 2014

Unfortunately not, because that only returns the path section from the URI. So for content://media/external/audio/media/710 it would return media/external/audio/media/710 whereas the actual path on disk would be /sdcard/media/audio/ringtones/GetupGetOut.mp3.

@Ferk
Ferk commented Oct 21, 2014

That's pretty annoying.

I found some painful algorithm to do the conversion (here the file in github, apache 2.0 license from Open Intents), dunno if it's the cleanest solution.

No idea why Google wants to use these awkward uris.

I checked the ARM apk you posted above but my device is not kitkat (it's android 4.3) and I guess it falls back to the old behavior, no access to sd card. It gets paths from the internal storage fine, though.

@Nutomic
Member
Nutomic commented Oct 22, 2014

That solution seems to be pretty hackish, and will probably blow up rather often.

So I'll just commit the patch above knowing that it works at least on 4.3 and 4.4 (not perfect, but considering that 71% of users have Kitkat, it's pretty good).

@Nutomic Nutomic closed this in c82cdd4 Oct 22, 2014
@schuft69

Hello,
with the new folder selctor only /storage/emulated/0 is working (like in 0.5.5)
/storage/emulated/0/Android/data/com.numotic.syncthingandroid/files and
/storage/extSdCard/Android/data/com.numotic.syncthingandroid/files displaying "empty folder".
Tested on Samsung S4mini with Android 4.4.2

@Nutomic
Member
Nutomic commented Oct 26, 2014

Maybe that's because the folder is actually empty?

@schuft69

if

  • /storage/emulated/0/Android/data/com.numotic.syncthingandroid/files and /storage/emulated/0
  • /storage/extSdCard/Android/data/com.numotic.syncthingandroid/files and /storage/extSdCard

are the same then not - but maybe I misunderstood?

@Nutomic
Member
Nutomic commented Oct 26, 2014

No those are different folders.

The .../Android/data/com.numotic.syncthingandroid/files folders are app private folders that can only be accessed by that app (afaik).

@varod
varod commented Oct 27, 2014

i am having problem around this issue. i've created few weeks ago a repository on my sdcard using the web ui wihtout any problem. the goal was to bakcup my picture on my workstation.
today i am having "folder marker missing".
it is not related to android 4.4.2 of my S4 because it was working few updates ago.
my external folder is : /storage/extSdCard/DCIM
is there any way to make it works back ?
thanks

@Nutomic
Member
Nutomic commented Oct 27, 2014

It should be fixed automatically in 0.5.6, but you can also work around it by creating .stfolders in the repo root.

@cattledogit

First install of syncthing so have 0.5.6. Hope I'm posting in the right place.

Though I've made the required change to platform.xml per http://tinyurl.com/l3e8jod so that apps can access any folder on the ExtSD, and ES File Mgr. can edit all folders w/o granting it root access, syncthing can only access files in /storage/extSdCard/Android/data/com.numotic.syncthingandroid/files. This is useless though because Astro Player can't see media files in com.numotic.syncthingandroid, though it can see all non-data/com... folders.

Galaxy Note 2 with 4.4.2.

@Nutomic
Member
Nutomic commented Jan 29, 2015

@cattledogit At least /sdcard/ should be available for syncthing. You can also select the advanced folder picker in settings and select any folder.

Edit: And you should install the latest version ;)

@cattledogit

You can also select the advanced folder picker in settings and select any folder.

Thanks, fixed that problem for extSdCard.

@Nutomic Nutomic reopened this Apr 27, 2015
@GRMrGecko

I just got a Nexus 6P and had an issue where it couldn't write to anywhere in the "SD Card", I'm assuming maybe a folder specific for the app would be allowed but I wanted it on the "SD Card". I found out how to fix by going to Settings->Apps->Syncthing->Permissions->Storage. Thought I'd post this here for people who are also having this issue.

@Nutomic
Member
Nutomic commented Nov 11, 2015

@GRMrGecko That would be #483. There's a workaround in 0.7.1.

@taaem
taaem commented Nov 19, 2015

So no news on implementing the Storage Access API on Lollipop? Or is it still impossible to do because of the go language?

@wweich
Member
wweich commented Nov 19, 2015

@GRMrGecko your phone does not have an external sd card, so your problem (as Nutomic pointed out) has nothing to do with this issue here.

@taaem With Android 6 you can have full access to the external sd card like you have with the "internel sd card". So this issue will probably loose priority.

@necr0potenc3

This issue can be remediated (it's a hack, not a fix) by manually creating ".stfolder" file in external storage.

@Nutomic
Member
Nutomic commented Nov 29, 2015

@necr0potenc3 That only works for read-only folders though. Syncthing will not be able to write anything.

@tkedwards

It's really frustrating that Google put restrictions on using external SD cards like this, but the good news seems to be that the 'Adoptable Storage' feature in 6.0 should resolve this.
@Nutomic have you been able to test this yet on an Android 6 device using adoptable storage? Do you think it'll resolve this problem?

@wweich
Member
wweich commented Jan 6, 2016

Just tested this with my Moto X Style. After adding the card and setting it up as "internal storage" /storage/emulated/0/ is the SD card and the internal storage seems to be not accessible for storage access anymore.
So only Apps and data they place inside the "App partition" and not on the "storage partition" are located on the internal storage.

@NYPD
NYPD commented Jan 6, 2016

As others have pointed, creating a folder named "com.nutomic.syncthingandroid"
extSdCard/Android/data/com.nutomic.syncthingandroid

Does allow syncthing to do its thing. I'm able to sync a large music collection under
extSdCard/Android/data/com.nutomic.syncthingandroid/Phone Music

So at least this is an option for some perhaps.

edit: This is on Android 5.1

@wweich
Member
wweich commented Jan 6, 2016

Does syncthing have write permission inside that folder?
Do you know, if that is the default Android behaviour, or if perhaps your manufacturer changed Android to allow that?

@Kernald
Kernald commented Jan 6, 2016

Applications always had the ability to write on their external storage dedicated folder, at least up to 5.1 (I don't know for Android 6). However, I think read access to that folder is restricted from other applications.

@NYPD
NYPD commented Jan 6, 2016

The application I used to play the music from the extSdCard/Android/data/com.nutomic.syncthingandroid/Phone Music folder is GoneMad Music player. So perhaps some applications are not able to read data in there, but luckily for me my music player is able to.

@Nutomic
Member
Nutomic commented Jan 17, 2016

You don't have to create the Android/data/com.nutomic.syncthingandroid/ folder yourself, you can just select it in the app when creating a new folder ;)

One caveat about these folders is that they will be deleted when you uninstall Syncthing.

@alexforencich

This is a serious showstopper issue. However, this page appears to inicate that there is a way for apps to explicitly request write access on android 5.0: https://metactrl.com/docs/sdcard-on-lollipop/ . If it is possible to implement this in syncthing, then this could be an effective solution for this issue.

@bservet
bservet commented Jan 26, 2016

I thought that Syncthing couldn't write to an external sdcard because the underlying framework was missing a needed feature?

With my phone (5.01) Syncthing seems to refuse writing to the /storage/extsdcard/music folder, or should this be supported with the current version?

@Nutomic Nutomic added the help-wanted label Feb 25, 2016
@Nutomic Nutomic referenced this issue in syncthing/syncthing Mar 2, 2016
Open

Remote "cloud storage" support for folders #2647

@Kugeleis

I can confirm that using a folder as master (read only) it works copying a .stfolder file to this folder manually using ES file explorer. It starts syncing after a restart. I have a non-rooted Android 5.1.1.

@francescortiz

@Kugeleis hack works for me too with android 5.0.2

@vext01
vext01 commented Apr 26, 2016

The hack did not work on my LG-G4. Permission denied apparently.

@dper
dper commented Apr 26, 2016

I have a rooted 5.1. It seems like Syncthing can (at least sometimes) do the initial file transfer but after that will not write anything. The changes are visible in the web client on my desktop, and they /look/ like they're getting pushed, but in reality no. Then, if I try browsing to that folder using an Android file manager, the file manager freezes. In this case, I have a directory just under extSdCard, and am not using the Android subdirectory.

@Kugeleis

@vext01: Did you use ES-explorer? This tool handles the write protection correctly when copying files.

@vext01
vext01 commented Apr 27, 2016

I used the built-in file manager on my G4.

Regardless I think the bug is that not any arbitrary directory on the SD can be synced. Users (including myself) don't want to rely on obscure hacks to get things working.

@AudriusButkevicius
Member

Right, tell that to Google, who's own programming language is unable to access SD cards on their own OS.

@vext01
vext01 commented Apr 27, 2016

Agreed.

@vext01
vext01 commented Apr 27, 2016

Out of interest (I know very little about android internals), how is it the file manager app can write the SD card? It has some special credentials?

@Nutomic
Member
Nutomic commented Apr 27, 2016

It's written in Java, that's all. The problem is that Go doesn't support this.

@vext01
vext01 commented Apr 27, 2016

Ah. So, would it be possible (and I know it's awkward) to write a little Java proxy which could do file access on Syncthing's behalf?

I'm not sure how hard that is. Can you invoke Java code from go? Otherwise you would have to use some form of IPC I guess.

@dper
dper commented Apr 27, 2016

@vext01 Dealing with badness in Android 5.1 and external storage is not unique to this app, so your question is one that could be asked in a more general forum. And unfortunately (or fortunately?) Android 6 is different, or so I read. There's a question of how much time developers want to or should spend working around these regrettable decisions made by upstream Android developers, particularly if it's specific to Android 5.1. I would imagine it's not a particularly fun or interesting pursuit.

Sadly, this means some of us with Android phones running 5.1 have to deal with workarounds and fixes, such as they exist. But hey, it is what it is.

@vext01
vext01 commented Apr 27, 2016

Thanks. I'll hold out for a 6 update and see what happens then.

@wweich
Member
wweich commented Apr 27, 2016

With Android 6 you can choose to use the SD Card as main storage, instead of the internal storage.
If you do that, you will have no access to the internal storage anymore. File Managers will only see the SD Card. /storage/emulated/0/, /sdcard/ and /mnt/sdcard/ will all point to the SD Card, not the internal storage.
In some apps you will be able to set where to store the (offline) data (e.g. Google Play Music); some will use the SD Card (e.g. Spotify).

@Nutomic
Member
Nutomic commented Apr 27, 2016

@wweich Isn't the change in Android 6 that you can combine internal and external storage into a single storage? So you can still use both, but it's treated like a single partition.

@Kernald
Kernald commented Apr 27, 2016

@Nutomic yes, it's the feature named "adoptable storage". However, some device makers choose to not integrate it in their Marshmallow versions. I think it's the case on the Galaxy S7, for example.

@wweich
Member
wweich commented Apr 27, 2016

I have a Moto X Style (which AFAIK is one of the most stock like devices with SD Card), and with this device it is like I described.

My storage overview looks like this:
screenshot_20160427-143208

@wweich
Member
wweich commented Apr 27, 2016

According to the official doc, my Moto X is using "adoptable storage" and I also got the "erase & format" prompt as described in this Android Central article.
The doc says nothing about if the normal storage (internal) can still be used as before or if both storages are merged.

Apps can still store their data in either place (like the current version of Play Music can (the older one could only use internal)).

@Kernald
Kernald commented Apr 27, 2016

From what I see (on a Shield Tablet K1), you can move an application from the internal storage to the external one as you see fit. However, any application is able to write on the external storage as it was possible before Android 5 (or was it 4.4?). I just tested syncthing on it, and I was able to write files to my SD card.

@wweich
Member
wweich commented Apr 27, 2016

Yeah, moving apps between internal storage and SD Card is possible. But I never did that, as the internal is faster and has plenty of space left.
But can you access your internal storage with a file manager like Total Commander, ES Explorer or the like? And can you Sync files with Syncthing to internal storage?

@Kernald
Kernald commented Apr 27, 2016

I can't find anything related to the internal storage. I see a /data folder (I see it empty, but I simply don't have the permissions to see what's inside, I think), but anything in /mnt and the likes point to the same place. I think (but I'm not even sure) the files here are physically stored on the SD card. However, I guess that if the SD card gets full, the system will transparently begin to use the internal storage, without telling anything. That's, at least, how it's advertised - a single, merged space, using two physical supports behind the scenes.

@bminer
bminer commented May 19, 2016

Possible workaround is to install SDFix (assuming you have root access). Sometimes running Syncthing with root access is a bit buggy.

Links to SDFix are below:

@herdz
herdz commented May 23, 2016

Joining in with my 5.0 android device. Nothing works for extSDCard path.

@herdz
herdz commented May 23, 2016

OK, looks like only via the webui I am able to sync on the extSDCard. The toggle button in the application itself doesn't work.

@wweich
Member
wweich commented May 23, 2016

There is no toggle button for extSDCard. Are you talking about #647 ?

@herdz
herdz commented May 23, 2016

@wweich, yes I mean the folder setting's toggle button, if the folder is created by the device (as opposed to being shared). This doesn't work.

@skasync
skasync commented Jun 3, 2016

Looks like nobody mentioned it so far:

would the snippets of the second answer help to gain write access to the external SD card the traditional way?

http://stackoverflow.com/questions/26744842/how-to-use-the-new-sd-card-access-api-presented-for-lollipop

I mean, you use the current code to try to write to /sdcard/external/path_choosen, get denied, try ACTION_OPEN_DOCUMENT_TREE to gain permission for the whole SDcard (or just the subdir to be created / shared), and on success -> can access /sdcard/external/path_choosen with the code already in the app probably?

@Nutomic
Member
Nutomic commented Jun 3, 2016

@skasync No, because it doesn't work with the normal File APIs, but with special classes (which Go doesn't support).

@taaem
taaem commented Aug 6, 2016 edited

@Nutomic I had an idea but I don't know if its doable, what if one would create a virtual filesystem via libfuse and FUSE that would get mounted to /storage/emulated/0/... and everytime syncthing tries to create or read a file from there Syncthing gets that file via the Android File API.

@AudriusButkevicius
Member

It's probably easier just to implement android file api support in go...

@Nutomic Nutomic added the upstream label Sep 21, 2016
@jebeld17
jebeld17 commented Oct 2, 2016 edited

Just a little history from Google I/O 2014, where Google basically announced the fix for this whole problem:

Hey all, in KitKat we introduced APIs that let apps read/write file in app-specific directories on secondary storage devices, such as SD cards.
We heard loud and clear that developers wanted richer access beyond these directories, so in Lollipop we added the new ACTION_OPEN_DOCUMENT_TREE intent. Apps can launch this intent to pick and return a directory from any supported DocumentProvider, including any of the shared storage supported by the device. Apps can then create, update, and delete files and directories anywhere under the picked tree without any additional user interaction. Just like the other document intents, apps can persist this access across reboots.
This gives apps broad, powerful access to manage files while still involving the user in the initial selection process. Users may choose to give your app access to a narrow directory like "My Vacation Photos," or they could pick the top-level of an entire SD card; the choice is theirs.

  • Google @ Developer Conference 2014
@capi
Member
capi commented Oct 2, 2016

@jebeld17 Yes, we are aware of the DocumentProvider API, the problem being that the Syncthing native binary is not aware of it. There are now some fixes in recent Go versions, but it'll take some time until the native binary can/will support it. The issue at hand is, that the Syncthing for Android application is basically a wrapper around the native binary, which is developed in Go and is cross-platform. And this binary has its limitations regarding Android.

@adarazs
adarazs commented Oct 7, 2016

This is the 3rd time in 2 years I found this software, got super excited that I can replace Dropbox & DropSync and then ran into this issue. Of course everything that's worth syncing is on the SD card in my phone because the internal storage is small, so videos, pictures, music, etc. are on the external storage.

It blows my mind that this is not the number one problem of this project being worked on non-stop, instead being open since 2+ years. This looks like such an amazing project otherwise, but it's completely useless for me and probably a bunch of others without this fix. :(

@AudriusButkevicius
Member

Go complain about this to the go team, we have very little control around this.

@generalmanager
@adarazs
adarazs commented Oct 7, 2016 edited

Looks like this is a sensitive issue for the developers too. Sorry for bothering by adding my voice here as a potential but again disappointed user. I wanted to express that there are people who care about this problem strongly. Syncthing seems to be the only good open source solution for the syncing problem everyone is having and the Android app is a key part of it, and the external SD cards as well.

I saw the workarounds and fixes in the thread, they don't work for me. I don't have root and I can't convert my SD card into adoptable storage.

I understand the limitations are there because of go, however if a problem is literally impossible to fix with a tool used for a task it might mean not using the right tools. 3 years after the release of KitKat, 2 years after the issue being open, with $600 bounty going for the go fix and nobody even wanting to touch it (just noticed it was cancelled), new phones actually still coming out with SD card slots... it might be time to rethink the ways to solve this apart from saying it's impossible.

I'm not an Android or go developer, so this is just an idea: would it be possible to write the storage driver part outside the go app and use a storage abstraction to handle the file operations?

@capi
Member
capi commented Oct 7, 2016

It would technically be possible, if either Syncthing itself provided a way to have "external" storage (i.e. a means for external processes to provide list/read/write operations) or the whole Syncthing protocol (BEP) was re-implemented in Java/Android. The later would be a complete re-implementation of what Syncthing does and given that currently mostly only @Nutomic works on the Android part, it's not something that's likely going to happen.

At the moment it still looks more promising hoping that upstream Go will add proper support which is usable to us, so that we can continue to use the native Syncthing binary.

@frispete
frispete commented Oct 7, 2016

Although @Nutomic already did, here's the reference to the root issue again: golang/go#10588.

@Nutomic
Member
Nutomic commented Oct 29, 2016

If anyone wants to work on fixing this issue: It is now possible to call Android APIs from Go. With that, it should be possible to write a wrapper around the relevant Android APIs. This wrapper can then be used by Syncthing.

@capi
Member
capi commented Oct 30, 2016

But it would have to be added in the Syncthing binary, right? Nothing we can do on the Android side at the moment?

@AudriusButkevicius
Member
AudriusButkevicius commented Oct 30, 2016 edited

Correct, and it needs quite a big revamp (@calmh was working on it some while ago) to be able to support it on the syncthing side too.

@dionorgua

Hi,

There is a way to convert 'SD card' to adoptable storage on Samsung S7 (and probably all other Android 6 devices) without root. Just a few adb commands.

But could anybody confirm that converting SD card to adoptable storage will allow syncthing to write to such 'adoptable' storage?

@dper
dper commented Nov 15, 2016

@dionorgua It is my understanding that some manufacturers have removed adoptable storage from their (Android 6) builds for various devices.

@dionorgua

Samsung removed it only from UI. But you can convert it using adb. Don't want to advertise here, just google 'Samsung S7 adoptable storage'. Probably same for others.

@wweich
Member
wweich commented Nov 15, 2016

I've used adoptable storage on a near stock device (Moto X Style/Pure), so I don't know how it is on Galaxy S7.
But on that device, you cannot use both internal and SD card at the same time. In the UI, you have to migrate the data to the sd card. Then /storage/emulated/0/ will point to the sd card and no file explorer app (or Syncthing) can access the real internal storage anymore.

That way, Syncthing can write to sd card without problems.

As you seem to have to have an S7 and used that method, why don't you try it and confirm for us if it works.

@dionorgua

I've not used it (yet). Just discovered that it's possible to enable 'adoptable storage'. Since it'll erase everything on SD card I need to be ready for this.

In any case thanks for confirmation that syncthing works with adoptable SD card.

@dper
dper commented Nov 16, 2016

On the Samsung Galaxy Note 4, using adb I just enabled adoptable storage. As described above, now /storage/emulated/0 points to the SD card, and the real internal storage isn't used. It wastes storage space, but it successfully works around this bug.

@wweich
Member
wweich commented Nov 16, 2016

I used a 64GB card in my 32GB phone. But as the system and Apps are still on the phone memory, my usable space more than tripled.
Also, in some Apps (like Google Play Music or Spotify) you can choose where the offline data should be stored. That way, you have options to use both storage spaces.

@jult
jult commented Dec 1, 2016

What I find to be really strange is that I used to run syncthing silently in the background on my rooted Samsung Galaxy S3, it synced all my preferred external sdcard directories without a glitch for months, never an issue. Now android has 'upgraded' and syncthing doesn't do that anymore? What a sad sad way to go for android. Really, have they killed such basic functionality in favor of security theatre, or what's the reason for this change?

@dper
dper commented Dec 1, 2016

@jult That kind of thing is outside the scope of this issue. :-/ Here we're just trying to figure out how syncthing should handle the current situation, regardless of why it came about.

@licaon-kter
Contributor

@jult That's an old device, what Android upgrade did you make?

@frispete
frispete commented Dec 1, 2016

@jult Yes, security theatre obscure.

Isn't it quite interesting, that google phones doesn't come with an sd card slot, hence their users don't have this problem... This is further obscured, that the choice of implementing syncthing in go is the major blocker in this respect. Needless to say, that go is a - google innovation - too.

This is a perfect recipe for damaging the last remains of a good reputation.

@Nutomic Nutomic locked and limited conversation to collaborators Dec 2, 2016
@Nutomic Nutomic removed the help-wanted label Dec 30, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.