Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/mobile: Android Storage Access Framework support #10588

Closed
Nutomic opened this issue Apr 27, 2015 · 47 comments

Comments

Projects
None yet
@Nutomic
Copy link

commented Apr 27, 2015

On Android Kitkat and higher, the Storage Access Framework is the only way to read/write files on an external SD card (eg /storage/sdcard1/).

However, access to that is handled through URIs like this:

content://com.android.providers.media.documents/document/image:62

These URIs can't easily be converted to paths [1]. We need this for syncthing-android, and have discussed this already [2] [3].

Go Version: 1.4.2
OS: Android, any architecture

[1] http://stackoverflow.com/q/20067508/1837158
[2] syncthing/syncthing-android#29
[3] https://forum.syncthing.net/t/writing-to-external-sd-card-in-android-5-0-and-above/2492

@crawshaw crawshaw added this to the Unplanned milestone May 3, 2015

@crawshaw crawshaw changed the title Android: Support Storage Access Framework x/mobile: Android Storage Access Framework support May 3, 2015

@lfarkas

This comment has been minimized.

Copy link

commented Jun 4, 2015

is there any reason why it's unplanned? currently there is not any other way on android to use external storage!?

@crawshaw

This comment has been minimized.

Copy link
Contributor

commented Jun 8, 2015

Unplanned means we want to do it, but have not found the programmer hours to do it. We are not a large team.

@bassmaster187

This comment has been minimized.

Copy link

commented Jun 11, 2015

i think, syncthing android is useless, without access to sdcard.

bittorrent sync is able to sync the pictures of my phone (sony xperia z3 on lolipop) to all my windows clients and i really want to get rid of it.

@fcastilloec

This comment has been minimized.

Copy link

commented Mar 11, 2016

Any progress? It's been almost a year since this was reported. Access to the SD card is an extremely important feature.

@Eddy2909

This comment has been minimized.

Copy link

commented Apr 28, 2016

bump

Are there any news on that?

@fti7

This comment has been minimized.

Copy link

commented Jun 24, 2016

Guys, any new on that? Its an Important Feature :-(

@fti7

This comment has been minimized.

Copy link

commented Jul 17, 2016

@crawshaw do you accept bounties for that Issue? Maybe we can sponsor the "Programmer Hours" somehow?

@urza

This comment has been minimized.

Copy link

commented Jul 18, 2016

This is really a very important feature! I will also donate few dollars if needed..

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Aug 6, 2016

How could this be implemented though?
As far as I understand, it's impossible without tapping into the NDK, hence adding a mandatory cgo dependency?

@lkwg82

This comment has been minimized.

Copy link

commented Aug 7, 2016

For me I solved it with marshmallow release. I extend the space to the sdcard. So I have 70 sth gig on my S5.
But it does notsolve it for Lollipop releases.

@Cysioland

This comment has been minimized.

Copy link

commented Aug 7, 2016

@lkwg82 Not to mention, that not all Marshmallow phones allow Adoptable Storage
(vendors ruin all the fun, unfortunately).

@fti7

This comment has been minimized.

Copy link

commented Aug 8, 2016

Thats the problem... Adoptable Storage is disabled by some Vendors (I guess for Performance-Reasons) :-(

@Cysioland

This comment has been minimized.

Copy link

commented Aug 8, 2016

@frispete

This comment has been minimized.

Copy link

commented Aug 24, 2016

@lkwg82: you're paying a price for this: hope, you never uninstall syncthing, because all your synced data with go with it... Good luck!

I'm syncing about 100G with syncthing. Even using root, it creates major headaches to use arbitrary folders on an external SD card. Some are documented here:
syncthing/syncthing-android#719
Swallowing my sediments on this topic in order to not offend anybody.
The real causer don't see this anyway.

@leoluk

This comment has been minimized.

Copy link

commented Oct 3, 2016

The Syncthing guys have a $600 bounty over there:

https://forum.syncthing.net/t/600-bounty-for-writing-to-external-sd-card-in-android-5-0/2550

I'll contribute to a bounty to make this happen in Go.

@frispete

This comment has been minimized.

Copy link

commented Oct 7, 2016

Can't #16876 help to further advance on this issue?

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2016

So is the problem only path conversion?
Or is the problem that the writes have to go through some API that's only available in Java?
I know reads work via Go (as we have a workaround for read only access)

@Adonai

This comment has been minimized.

Copy link

commented Feb 11, 2017

Or is the problem that the writes have to go through some API that's only available in Java?

Yes, this is the problem. Android renders external SD card as Read-Only and only passes specific file descriptors with write-bit set if requested with ACTION_OPEN_DOCUMENT.

@LeonardoGentile

This comment has been minimized.

Copy link

commented Mar 14, 2017

Still unplanned after 2 years?

@Kuurusch

This comment has been minimized.

Copy link

commented Jun 11, 2017

Hey, any progress here?

@LeonardoGentile

This comment has been minimized.

Copy link

commented Jun 15, 2017

Could please someone from the team give an honest answer here?
Something unplanned for 2 year is very unlikely to be implemented.
Since the first answer from @crawshaw nobody else from the team bothered to answer to anybody, also ignoring a bounty and requests to further contribute to implement this.

This is blocking an essential feature for syncthing and apparently there is nothing the syncthing team can do about it.

If you don't have the time or you are not willing to do it just please tell us and we will stop to complain and look for alternatives.

Thanks 👍

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2017

So I am pretty sure nobody in the core team is interested to solve this, and I am fine with that as they have bigger fish to fry before Go becomes even mildly popular on Android. The thing I am not fine with however is lack of guidance. Even if I had the hours to do it, I haven't been provided any guidance on how this could be done even on a high level.

@griesemer

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2017

Unfortunately there's no one on the Go team actively supporting x/mobile at the moment. We are a small team and we are spread thin, and currently have to focus on different priorities. My apologies.

cc: @crawshaw in case he wants to add anything to this.

@dolanor

This comment has been minimized.

Copy link

commented Oct 5, 2017

cc: @eliasnaur do you have any ideas on what is the problem here, and what can be done about it?
(I'm pinging you because I know you're quite knowledgeable on the mobile part, sorry if I'm mistaken)

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 6, 2017

It's not clear to me what this issue is about. Is it enough to use

https://developer.android.com/reference/android/content/ContentProvider.html#openFile(android.net.Uri, java.lang.String)

to access the file with a specific URL and then use

https://developer.android.com/reference/android/os/ParcelFileDescriptor.html#getFd()

to extract a file descriptor to use with https://godoc.org/os#NewFile?

If not, please clarify.

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2017

I don't see a way to handle exceptions in case of reverse bindings?
I guess I am expected to write a Java wrapper that handles that gracefully?

@Adonai

This comment has been minimized.

Copy link

commented Oct 10, 2017

@eliasnaur , it's read-only fd that you can get this way. Getting rw fd is a bit more painful task involving showing activity to the user with access request, e.g. see SAF-wrapper here

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 10, 2017

Read FDs you can already get via os.Open...

@frispete

This comment has been minimized.

Copy link

commented Oct 10, 2017

The SAF-wrapper, that @Adonai points to seems exactly the part, that is missing so badly in syncthing...
Syncing is omnidirectional, hence RW file operations are required here.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

I still don't think this issue is specific enough. It seems to me, syncthing can create a Java wrapper for its file operations and expose that wrapper to Go through gomobile bind. If not, what specifically is blocking them?

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

Probably you are right, that should be sufficient. I need to workout how to do the android part, as the go part seems trivial.

@Nutomic

This comment has been minimized.

Copy link
Author

commented Oct 11, 2017

From what I understand, it should be possible to use gobind and call the Storage Access Framework APIs directly from Go. Main blocker for that is making Syncthing work with gomobile. So I guess this issue could be closed.

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

That's not a blocker, as we don't need to make syncthing work with storage access framework, it's the other way around, we need to make a library that uses Java reverse bindings, and then use that in syncthing.

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

Lets not close it until we work up a solution though.

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

If you create a Java wrapper, you don't need the reverse bindings. Creating and binding a Go interface and implementing it on the Java side should suffice. Note that gomobile bind doesn't copy []byte arguments, so read/writes across the language barrier should be efficient.

I'm going to close this as out of scope; I don't think direct support for Android SAF is something that belongs in gomobile so it's not clear how this issue can be resolved. A re-opens or a new issue has to specify (for example with small example programs) specific features wanted from gomobile bind that either doesn't work (a bug) or isn't yet done (a feature request) that blocks the writing of SAF support elsewhere.

@eliasnaur eliasnaur closed this Oct 11, 2017

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

Well you definately need a wrapper, because as it stands now, there is no way to handle JVM exceptions in Go, so you must have a exception free wrapper which you then reverse bind to.

@dolanor

This comment has been minimized.

Copy link

commented Oct 11, 2017

I don't know gomobile enough, but wouldn't a Java exception bubble up as a Go panic, and therefore can be recover'ed to make it safe?

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

I believe both bindings and reverse bindings support Java Exceptions. If not, please raise an issue about what doesn't work.

If you need examples, take a look in bind/testpkg.

@Adonai

This comment has been minimized.

Copy link

commented Oct 11, 2017

Hi, I see this is finally closed, how soon Syncthing will pick up the changes?

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

It's closed as it's not something Go developers are going to be addressing in the standard library, not that it has been resolved.

Either the docs are thin or I am bad at searching, but looking at #21033 it seemed to me that exception handling is not handled?

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

#21033 is about unexpected Java exceptions bubbling up to the Go side. It's the equivalent of a Go panic that isn't recovered. If you make sure to return error from your interface methods, any Java exception will be converted to a Go error for you.

#18210 is about a build failure. I can't reproduce it and I'm not sure syncthing even needs reverse bindings.

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

Sorry, but how does it access java APIs if we don't reverse bind? (Forward bind I assume is Java calls go, reverse is go calls java).

@eliasnaur

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2017

No, reverse bind is importing and accessing Java classes directly. It's a (very) experimental feature to explore not having to write Java code at all.

Regular bindings is automaticall generating Java wrapper classes and interfaces from Go exported structs, interfaces, functions, variables and constants.

With (regular) gomobile bind you can (for example) define in Go

package saf

type Writer interface {
     Write(b []byte) (int, error)
}

And then, from Java

import go.saf.Writer;

public class SAFWriter implements Writer {
     @Override public int write(byte[] b) throws Exception {
          ....
      }
}

And then pass instances of SAFWriter to a Go function or method expecting a Writer.

If you need more help, please write to golang-nuts where your questions are exposed a much larger audience.

@tomachinz

This comment has been minimized.

Copy link

commented Nov 7, 2017

I want this so I can remotely delete images, photos, audio from my phone. Fairly sure I got it to work once with Syncthing maybe on a different phone (not a Samsung). Definitely was able to get BTSync to sync the whole SD card such that I can even delete images from DCIM etc.

@TRSx80

This comment has been minimized.

Copy link

commented Jan 21, 2018

And this is why I don't want to use anything that touches Google in any way.

I have been hearing about Syncthing all over lately, finally had some time to dig into it, was starting to get very excited, everything sounds great, but now this.

Every time. So sick of it. For years now.

And this is why I will give no help, mind share, bug reports, free discussion/promotion of, etc. anything that has anything to do with Google. There are lots of other actually Free software projects that deserve our support and attention instead. In fact, whenever Syncthing is brought up in LUG, IRC, or anywhere else, I will be sure to let people know you can't even use it on your Android SD card.

Every. Freaking. Time. When will people learn?

EDIT: Purism Librem 5 can't get here soon enough. Once we can get actual GNU/Linux on a phone, Android's days will be numbered.

@AudriusButkevicius

This comment has been minimized.

Copy link
Contributor

commented Jan 25, 2018

So just to quantify, this is possible to solve, but essentially your go application has to become a java application calling out to Go, rather than the other way around.

For any vanilla Go application that is run as a simple binary has no way to get a handle on/initialize the JVM to call through to any Java API's. In syncthings case we have a java ui that pokes a go binary via a rest api. In this case we'd have to turn our binary into a library and load/link it in the java process, and then call Go API's from Java, which is a large effort.

@golang golang locked and limited conversation to collaborators Jan 25, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.