Upload from headless server

[ghost]

First, thank you for developing this incredibly useful project.

018/09/27 09:03:22 You will now be taken to your browser for authentication
2018/09/27 09:03:23 failed authenticating new client

Not sure how to go past this, is there a way?

[nmrshll]

Glad you appreciate it !

I didn't have this case in mind when developing it, but I can try to work on that when I have the time.
How would you see it work ? Would it be practical to have another tool that lets you simply connect to google from a computer that has a GUI interface and then configures the app remotely on the headless server ?

In the meantime for it to work you'd have to:

• create an oauth app on the google api console
• connect as a user to get a token (you can do that with postman)
• specify the same oauth app params in the config file of gphotos-uploader-cli on your headless server
• insert the token manually into your keyring at the right key on the headless server

so... not exactly straightforward

[sledge1977]

Also want to thank you for this tool, but also want to ask if you could implement a way to authenticate on a headless server. Would really like to use my VServer to automatically upload my photos to Google Photos.

[simonszu]

As an authentication workflow suggestion, check Grive. I don't specifically know their oauth workflow, but it asks the user to access a specific URL in their browser, where they authenticate and get a key in exchange, which has to be pasted to Grive in the command line.

+1 from my side for headless uploading, I can create a cron job which kind of syncs my photos folder to GPhotos, especially when adding new photos.

[vixing]

Any progress on headless to make it work like the python youtube upload script? It points me to an URL I could open on another PC.

nmrshell, you said "insert the token manually into your keyring at the right key on the headless server"
Where in Windows can I find the token, or how do I actually insert that token into the keyring on the server?
I tried to do it and installed the app on my Windows machine, just so I can copy the keyring back to linux but I'm stuck.

[nmrshll]

@vixing no progress at all yet as I'm in search of my next gig/job, which will steal a lot of time from me, but I intend to get back to it once that's done

Also I don't think any of this would work on windows yet (because of the config file paths and because I make use of the MacOS keychain / Linux keyring in the code). I don't have a windows machine to test this so I assume it's completely broken on windows.
What are the features that make you want to use this over the official backup & sync app ? Just asking to have a better clue what you guys actually expect of this tool (initially I set out to do just a replacement app for the GUI google photos uploader, but which works on linux)

[simonszu]

I don't have any Mac or windows machines at home and run only Linux, as well as my desktop as on my Homeserver. My preferred workflow would be to copy new photos from my camera to the Photos share I have set up on my Homeserver, which would then be automatically synced to Google Photos. Currently I still need to open Google Photos in the browser and upload them manually, so a few less steps in the workflow, and a "copy on server and forget" would be great.

[kurokh]

Hi, in your instruction for headerless server

• create an oauth app on the google api console
• connect as a user to get a token (you can do that with postman)
• specify the same oauth app params in the config file of gphotos-uploader-cli on your headless server
• insert the token manually into your keyring at the right key on the headless server

please explain in more detail last point exactly where i should insert token, sorry for english, I will be very grateful for the feedback!

[darryllee]

Hey @nmrshll I know linking to a spec and writing code are different things, but I remember an early version of Google Cloud Print Connector gave me a URL to copy-paste into a browser to authorize the app.

It looks like the new version uses Google Sign-In for TVs and Devices which seems like it would be a good fit.

Basically it generates a unique verification code that would be entered at https://www.google.com/device.

The code then takes you to the proper auth page for the app. Hum. Let me look at the code. Maybe it might not be that hard to add in myself.

[nhorvath]

I submitted a PR against your oauth project to add logs so the user can manually do the oauth steps on their main computer.

[nmrshll]

Oh cool, thanks, that sounds useful ! Don't have a computer right now or tomorrow but I should be able to have a look at this on tuesday. There would still need a way to re-inject the token into the headless uploader after the auth was done on your main computer, right ? I started turning gphotos-uplader into a proper cli tool with options for several use cases, this could be one.

…

On Sun, Nov 25, 2018, 15:07 Nick Horvath ***@***.***> wrote: I submitted a PR against your oauth project to add logs so the user can manually do the oauth steps on their main computer. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGw1NLMgw0UlG1atX7rxbJWL9ZSNUpOlks5uyrIUgaJpZM4W8NqU> .

[nhorvath]

Yes, the logs give instructions, saying you should curl the result url manually on the headless machine. They can be made more clear if you don't think they are. On Sun, Nov 25, 2018 at 2:27 PM Nicolas Marshall <notifications@github.com> wrote:

…

Oh cool, thanks, that sounds useful ! Don't have a computer right now or tomorrow but I should be able to have a look at this on tuesday. There would still need a way to re-inject the token into the headless uploader after the auth was done on your main computer, right ? I started turning gphotos-uplader into a proper cli tool with options for several use cases, this could be one. On Sun, Nov 25, 2018, 15:07 Nick Horvath ***@***.***> wrote: > I submitted a PR against your oauth project to add logs so the user can > manually do the oauth steps on their main computer. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #15 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AGw1NLMgw0UlG1atX7rxbJWL9ZSNUpOlks5uyrIUgaJpZM4W8NqU > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#15 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKE3PX_JyI0osU-FSnFUrd1kAqT7I_4ks5uyu8TgaJpZM4W8NqU> .

[lozbrown]

Any news on making this possible or instructions for doing this?

[nmrshll]

Not yet but I'm working actively on fixing bugs and other improvements today.

Though, I started with the most pressing stuff like the config init being broken and not uploading videos and gifs, hope that's all right

[rgusick]

Look at "bin/smregister" in https://github.com/marekrei/smuploader for a python headless way of configuring oauth

[gregbert42]

Adding a comment here so I know when headless server uploading is possible...

[rysi3k]

Hello
any progress with this?

Regards

[nhorvath]

My fork dumps a url you can manually put in a browser.
Not sure why @nmrshll hasn't merged nmrshll/oauth2-noserver#2 yet.
In the mean time you can use it by doing this:

cd ~/go/src/github.com/nmrshll
mv oauth2-noserver oauth2-noserver.bak
git clone https://github.com/nhorvath/oauth2-noserver.git
cd oauth2-noserver
git checkout manual-url

[optional] edit oauth2ns.go to set a local ip and device name to use for the callback (line 28,29) otherwise you will have to manually curl the url your browser winds up at after you do the oauth signin.

cd ../gphotos-uploader-cli/
go run cmd/gphotos-uploader-cli/main.go

[rysi3k]

Thanks,
but still I have problem with keyring, there is hard linux dependency of gnome-keyring, normally on headless server it is not available :(
Any thoughs ?

[nhorvath]

This project is used for the keystore https://github.com/zalando/go-keyring. Maybe take a look and see if you can find a way to provide a keyring it can use.

Otherwise... install gnome-keyring. I don't think it depends on a full gnome setup to work.

[tonymet]

@rysi3k I created a fork that let's you store the oauth cred in the main db. This way it can run on a non-desktop linux vm

See the branch, feature/headless

[chris-y]

I have a similar problem, to workaround I ran an X server on the remote desktop so I could see Chromium. However it tries to redirect to 127.0.0.1 after authentication and never connects. How do I get the resulting key?

[rfgamaral]

@tonymet Looked at your feature/headless branch but I couldn't find any code changes related to storing the oauth token in the main db. Did you forget to push a few commits or am I misreading something?

---

I'm working on a Docker Image for this tool and I believe that I have practically everything ready to build a working image. The only thing missing is a workaround to the keyring issue. Tried to install gnome-keyring and other required dependencies but eventually got to an error (Failed to execute program org.freedesktop.secrets: Operation not permitted) that I couldn't solve.

I don't think there's a workaround to this problem, the only solution is to use something else (like the main DB as @tonymet mentioned above) to store the authentication token.

Note: My Docker Image doesn't solve the issue with the manual authentication process, that's still required so that the user can given consent to Google Photos access at least once. I've used @nhorvath work to workaround this, kudos to him. See here fore more details (just changed one little thing 😄).

Hopefully we can workaround this keyring issue and have this amazing tool working on headless machines like a NAS (my own use case) or something.

[emmtte]

Need this nice feature too.
Hope will updated soon

[tonymet]

@rfgamaral Sorry about the confusion I forgot that there were 2 components needed.
Here is the other one:
https://github.com/tonymet/gphotos-uploader-cli/tree/feature/headless

See this commit: tonymet@8143d3b

To make the build easy...

go get github.com/nmrshll/gphotos-uploader-cli
cd $GOPATH/src/github.com/nmrshll/oauth2-noserver
git remote add tonymet	git@github.com:tonymet/oauth2-noserver
git checkout feature/headless
cd $GOPATH/src/github.com/nmrshll/gphotos-uploader-cli
git remote add tonymet	git@github.com:tonymet/gphotos-uploader-cli.git
git checkout feature/headless
make build

[tonymet]

@rfgamaral
There's one more change to run remotely on a vm. Change the redirect URL to use your VM's public hostname. It must also be added to your GCP Oauth whitelist

diff --git a/oauth2ns.go b/oauth2ns.go
index a35845a..27e786f 100644
--- a/oauth2ns.go
+++ b/oauth2ns.go
@@ -61,7 +61,7 @@ func AuthenticateUser(oauthConfig *oauth2.Config, options ...AuthenticateUserOpt

        // Redirect user to consent page to ask for permission
        // for the scopes specified above.
-       oauthConfig.RedirectURL = fmt.Sprintf("http://127.0.0.1:%s/oauth/callback", strconv.Itoa(PORT))
+       oauthConfig.RedirectURL = fmt.Sprintf("http://gphotos.mydomain.com:%s/oauth/callback", strconv.Itoa(PORT))

        // Some random string, random for each request
        oauthStateString := rndm.String(8)

[rfgamaral]

@tonymet Awesome work, I was finally to get everything up and running.

After reading this thread a couple of times and experimenting everything I'll sum up the things that IMO should be done to accommodate different scenarios for for gphotos-uploader-cli, especially the headless one:

1. Either by using @tonymet solution to store the token in the main DB or something else, the token should be stored in a safe and private location that is cross-platform (at least for Unix/Linux variants) and not tied to gnome-keyring.
2. Since OAuth requires a GUI for user consent, some sort of manual process is needed, I don't think there's any way around it. There are a couple of ways to achieve this:
   a) Go with @nhorvath solution and print a URL on the terminal that one should manually open on a browser, copy the resulting URL and curl that in the same machine we're authenticating.
   b) Go with @darryllee suggestion and use Google Sign-In for TVs and Devices approach.

For a), we could, perhaps, use ngrok to create a tunnel to a local server and allow "ngrok.io" in the OAuth consent page. This way we might be able to handle redirection automatically without requiring the user to curl the resulting address. I haven't tested this, I'm not sure if this is even possible. Either way, for this option, we still need to print the URL to manually open on a browser window if the machine where gphotos-uploader-cli is being used does not have a GUI.

If the "ngrok" solution doesn't work, I believe b) can be the most user friendly approach. The user starts the authentication process on the terminal, gets a simple URL to open and code to input on that page. On the server-side, gphotos-uploader-cli will be polling the token endpoint and automatically authenticate once the user inputs the code.

Hope this summary helps :)

[rfgamaral]

In the meantime, if you have Docker on the machine you want to use gphotos-uploader-cli, I have created a working Docker image with all the required code and instructions for the authentication process. Before advertising my project, I want to take a moment to thanks both @nhorvath and @tonymet for their work, this Docker image wouldn't have been possible without their contributions. I also want to thanks @nmrshll for this great CLI tool, of course, awesome job.

Anyway, if you want to try my Docker image, please follow the link below and look at the instructions carefully:

• https://hub.docker.com/r/rfgamaral/gphotos-uploader

Feel free to open any issues with bugs, suggestions or whatever on the GitHub repository. Only if related to the Docker image, everything else belongs to gphotos-uploader-cli's repository, of course.

[tonymet]

@rfgamaral It's good to see there's demand for this use-case. @nmrshll built a great app, and I think this featureset could help expand it's use on a VM or server.

For oauth, I like @darryllee's suggestion to use the Google login for TVs -- that way no public domain is needed.

I'll start by formalizing the headless work into options that can be integrated into master gracefully.

If someone wants to work on the google-login-for-tvs then we can meet in the middle and create a seamless "headless" feature out of it.

[rfgamaral]

For oauth, I like @darryllee's suggestion to use the Google login for TVs -- that way no public domain is needed.

I agree 😃

I'll start by formalizing the headless work into options that can be integrated into master gracefully.

Awesome 🎊

If someone wants to work on the google-login-for-tvs then we can meet in the middle and create a seamless "headless" feature out of it.

Would love to help but my Go skills are practically none... Hopefully someone else can pitch in and help get that done while @nmrshll focus on other things.

[tonymet]

with some basic reading i found that the scopes for "oauth for tv" may be limited...

https://developers.google.com/youtube/v3/guides/auth/devices#allowedscopes

But i'm still optimistic -- i've seen other docs say that certain scopes are limited when they are not in fact (perhaps the docs are out of date).

Still worth a try, just an fyi in case things don't work

[tonymet]

Would love to help but my Go skills are practically none... Hopefully someone else can pitch in and help get that done while @nmrshll focus on other things.

I can do the oauth stuff too it'd be a fun project...let's see if someone else chimes in otherwise I'm happy to work that component to.

[nmrshll]

For oauth, I like @darryllee's suggestion to use the Google login for TVs -- that way no public domain is needed.

I agree

I'll start by formalizing the headless work into options that can be integrated into master gracefully.

Awesome

If someone wants to work on the google-login-for-tvs then we can meet in the middle and create a seamless "headless" feature out of it.

Would love to help but my Go skills are practically none... Hopefully someone else can pitch in and help get that done while @nmrshll focus on other things.

@rfgamaral awesome work with the docker version ! Thanks for the help !

[tonymet]

@rfgamaral I think the oauth-for-devices won't work for this scope. i did testing with the scopes. here is an example testing just one and it fails with "invalid_scope"

# Control test : supported scope
➜ scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fanalytics
➜ echo $scope
https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fanalytics
➜  conditional-build curl -vd "client_id=$client_id&scope=${scope}" \
 https://accounts.google.com/o/oauth2/device/code
*{
  "device_code": "xxxxx",
  "user_code": "YYYY-YYYH",
  "expires_in": 1800,
  "interval": 5,
  "verification_url": "https://www.google.com/device"
}% 
# SUCCESS
# EXPERIMENT: photolibrary.readonly scope
➜ scope='https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly'
➜  curl -vd "client_id=$client_id&scope=${scope}" \
 https://accounts.google.com/o/oauth2/device/code
{
  "error": "invalid_scope"
* Connection #0 to host accounts.google.com left intact
}%
# FAIL

[rfgamaral]

Just tried it myself too, I guess the documentation is up to date 😆

The way I see it, there's a few alternatives to make the process as seamless as possible:

1. Use the ngrok.io approach I mentioned before. Again, not sure if the OAuth Consent thing accepts this for redirect URIs.
2. We could use Heroku and implement a similar approach to "oauth-for-devices" ourselves. I believe the free plan is sufficient for the time being. Also not sure if redirect URIs would work here or not.
3. Similar to 2, but buy ourselves a domain and host a server somewhere and again, implement a similar approach to "oauth-for-devices" ourselves.

Both option 2 and 3 are a lot more work though... And none of them avoid the step of manually opening some sort of URL in a different machine/device with a browser. This will always be required for "headless" servers.

EDIT: Forget about ngrok and Heroku, both ngrok.io and herokuapp.com are not allowed as "Authorized domains" in the OAuth consent screen 😢

[rfgamaral]

Another things guys, this is issue discussion is getting quite big and there's a few, different, but related, steps required to make a fully functional headless version of this tool:

1. Implement a mechanism for manual authentication if no GUI is available. Perhaps we could have a gphotos-uploader-cli oauth start command to do this manual authentication instead of messing with the code that does the GUI authentication on normal usages of the tool?
2. Store the access token in a cross-platform way or implement a mechanism that first tries the gnome-keyring thing and then falls back to the more cross-platform storage, like the main DB or something else more appropriate (if there is one).
3. One last thing that hasn't been mentioned here (but it was on oauth on each run #37) is that we need a mechanism to refresh the access token periodically and automatically so we don't have to bother with authentication again. Without thinking too much into this, I would suggest a command like gphotos-uploader-cli oauth refresh which could then be manually added to a cron job or something (I could easily automate this process in my Docker image). Another approach, perhaps, could be to always refresh the access token on every run of gphotos-uploader-cli, no cron job required and it's likely we'll always have a valid access token, unless the user didn't run the tool for a long time.

[rfgamaral]

Hey everyone,

I've been thinking about this issue, researching possible ideas, discussing it with @tonymet, etc... Here's the solution I propose:

1. Merge @tonymet's "headless" support for VMs (non-desktop linux instances) Issue #15 #41 without the RedirectURL option (more on this below) so that we can store the token without relying on dbus and gnome-keyring.
2. Improve on @nhorvath's Add logs to support manual auth steps nmrshll/oauth2-noserver#2 so that it's more automated:
   • Add a new configuration option to APIAppCredentials, something like HostIP and default it to 127.0.0.1.
     • Improve documentation to instruct the user to change the default HostIP to the local network IP of the headless machine he/she is setting up gphotos-uploader-cli. For instance, my home network is on 196.168.0.0/24 and my NAS (where I installed gphotos-uploader-cli) is 192.168.0.99; so that's the IP I would use for HostIP.
   • Try to open a browser with the default URL (using HostIP from above):
     • SUCCESS: Proceed as usual, let the user authenticate, give consent and have the token stored without further user interaction.
     • ERROR: Print the URL the user needs to manually open in a browser (possibly in green/yellow to make it standout from all other text) and wait until consent is given.
       • If HostIP is different from the default ( 127.0.0.1) and it's a local network IP, the OAuth process will only work if we pass along a device_id and device_name.
         • device_id: Perhaps something like machineid could be used. Go library, easy to use, secure and cross-platform.
         • device_name: Perhaps always default to the machine's hostname with os.Getenv("HOSTNAME")? We could have a configurable option for this but I honestly don't think it's worth it.
       • With the above configuration the user should be properly redirected and the token successfully stored without further user interaction. This, of course, assumes the user is in the same network where gphotos-uploader-cli is being installed, otherwise redirection won't work. But this is probably true for most scenarios so it shouldn't be a problem.

What do you think guys? What about you @nmrshll? If this sounds like a good solution maybe @tonymet and/or @nhorvath could work on clean branches for this? I'd do it myself but I'd rather leave this to someone who is more comfortable with Go than me.

[emmtte]

When?

[pacoorozco]

Most of the issues that are described here has been fixed in @rfgamaral docker fork.