add support for deployment on RStudio Cloud #278

mslynch · 2022-09-01T19:15:13Z

Description

Adds:

Support for adding RStudio Cloud credentials (currently token/secret pairs)
- to specify a credential is for Cloud, pass --server rstudio.cloud (or --server https://api.staging.rstudio.cloud for staging)
- if only the token/secret pair is provided, rsconnect-python will default to using shinyapps.io
Support for deployment to RStudio Cloud
- Outputs deployed to RStudio Cloud can optionally be associated with existing projects by providing that application's id through the LUCID_APPLICATION_ID variable (or --project-application-id).

Testing Notes / Validation Steps

We should validate the following cases:

Deploy to cloud using rsconnect deploy shiny
Deploy to cloud using rsconnect deploy manifest
Deploy to cloud using rsconnect deploy shiny and LUCID_APPLICATION_ID – the project should show the deployed output in its list of associated outputs
Deploy to cloud using rsconnect deploy manifest and LUCID_APPLICATION_ID – the project should show the deployed output in its list of associated outputs

rsconnect/main.py

rsconnect/actions.py

samperman · 2022-09-06T11:36:36Z

rsconnect/api.py

    def deploy_bundle(
        self,
        app_id: int = None,
+        project_application_id: str = None,


what is the difference between app_id and project_application_id?

There's sorta two separate workflows.

External to cloud you can write code locally and then publish you code to cloud as an output.

Internal to cloud you have a project with code - and you publish the code in that project to cloud as an output.

In cloud we create a link in the 2nd case between the project that has the code - and the output that ends up being the published. This supported through an environment variable LUCID_APPLICATION_ID which is present in every project - and consulted when rsconnect publishes an output.

So that's kinda the use case around this. I'm not sure that exposing this to an end user on the command line has much utility. It doesn't make a lot of sense for a person publishing locally to establish a link between the output that they are publishing - and some other random project in our system which doesn't have the code that they published. We also don't expose the projects application ID anywhere visible to the user...

I might not support this on the command line at all. If LUCID_APPLICATION_ID happens to be present - include it in the request. Otherwise don't try to make a link between a project and an output.

mbaynton

LGTM except comments already made by others

omar-rs

I think the comment about avoiding the name Lucid is important to fix, but otherwise looks good to me.

ssinnott

Nice! Looks good to me.

bcwu · 2022-09-09T15:30:46Z

QA note:

server store compatibility, i.e. credentials created does not affect regular on-prem usage, past credentials can be used by this version, etc...
downstream rsconnect-jupyter compatibility

bcwu · 2022-09-12T15:30:48Z

fixes #22123

kgartland-rstudio · 2022-09-13T12:43:14Z

I don't think I have an rstudio cloud account. Can someone give me access and the URL for testing?

kgartland-rstudio · 2022-09-13T15:34:23Z

Deploying a python shiny app fails.
app: https://github.com/rstudio/connect/tree/main/examples/top-5-income-share-shiny
server: https://api.staging.rstudio.cloud
account: pxb42b-kevin-gartland

> rsconnect deploy shiny ./ -n pxb42b-kevin-gartland -N -v -t new-one
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[OK]
Deploying bundle ...
Waiting for task: c3e3386d-150a-4ac1-8cf4-539865a2e7e2
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=build-output
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
  started - name=create-output-service
Task done: name=create-output-service
Application successfully deployed to https://3e4514a51468437b83e2ca666ef2bf08.app.staging.rstudio.cloud/
 	[OK]
Saving deployed information... 	[OK]

The link provided in the output returns a 404.
When I navigate to the app from https://staging.rstudio.cloud/content/, the browser console logs output a 404 as well with cross origin errors:

Failed to load resource: the server responded with a status of 404 ()
3e4514a51468437b83e2ca666ef2bf08.app.staging.rstudio.cloud/:1          
Failed to load resource: the server responded with a status of 404 ()
3e4514a51468437b83e2ca666ef2bf08.app.staging.rstudio.cloud/:1          
Failed to load resource: the server responded with a status of 404 ()
transport.js:23 Cross-Origin Read Blocking (CORB) blocked cross-origin response https://rum-http-intake.logs.datadoghq.com/v1/input/pub123fe85c7028cf974bac883aa1db50e6?ddsource=browser&ddtags=version:1.11.2&batch_time=1663083017778 with MIME type application/json. See https://www.chromestatus.com/feature/5629709824032768 for more details.
e.send @ transport.js:23
e.flush @ transport.js:64
(anonymous) @ transport.js:158
(anonymous) @ internalMonitoring.js:52
app.39d04f0ac55d89340b67.js:2 [Violation] 'setTimeout' handler took 50ms
transport.js:23 Cross-Origin Read Blocking (CORB) blocked cross-origin response https://rum-http-intake.logs.datadoghq.com/v1/input/pub123fe85c7028cf974bac883aa1db50e6?ddsource=browser&ddtags=version:1.11.2&batch_time=1663083066904 with MIME type application/json. See https://www.chromestatus.com/feature/5629709824032768 for more details.

mslynch · 2022-09-15T18:25:46Z

@kgartland-rstudio We've been having some instability in our staging outputs infrastructure, but deployments should be working now.

kgartland-rstudio · 2022-09-22T13:22:20Z

These errors are a little nebulous.

If you add credentials with a --secret length that has too few characters we throw this error:

-> rsconnect add --account pxb42b-kevin-gartland --name cloud --token {TOKEN} --secret {SECRET} --server https://api.staging.rstudio.cloud
Checking RStudio Cloud credential... Traceback (most recent call last):
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/actions.py", line 80, in cli_feedback
yield
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/main.py", line 278, in _test_rstudio_creds
test_rstudio_server(server)
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/actions.py", line 222, in test_rstudio_server
with api.RStudioClient(server) as client:
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/api.py", line 986, in init
self._key = base64.b64decode(rstudio_server.secret)
File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
[ERROR]
Internal error: Incorrect padding

With a --secret length that has too many characters:

-> rsconnect add --account pxb42b-kevin-gartland --name cloud --token 84E0E24166F1432FBE43D67746A6973A --secret vzo/Bj43ZGvojhHbTB5CIAkFkE0VBjZYc5O6lyIQf --server https://api.staging.rstudio.cloud
Checking RStudio Cloud credential... Traceback (most recent call last):
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/actions.py", line 80, in cli_feedback
yield
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/main.py", line 278, in _test_rstudio_creds
test_rstudio_server(server)
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/actions.py", line 222, in test_rstudio_server
with api.RStudioClient(server) as client:
File "/Users/kgartland/.pyenv/versions/3.8.2/envs/rsconnect-tag/lib/python3.8/site-packages/rsconnect/api.py", line 986, in init
self._key = base64.b64decode(rstudio_server.secret)
File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Invalid base64-encoded string: number of data characters (41) cannot be 1 more than a multiple of 4
[ERROR]
Internal error: Invalid base64-encoded string: number of data characters (41) cannot be 1 more than a multiple of 4

With a --secret length with the right number of chars but just incorrect:

-> rsconnect add --account pxb42b-kevin-gartland --name cloud --token 84E0E24166F1432FBE43D67746A6973A --secret vzo/Bj43ZGvojhHbTB5CIAkFkE0VBjZYc5O6lyI9 --server https://api.staging.rstudio.cloud
Checking RStudio Cloud credential... [ERROR]
Error: Failed to verify with RStudio Cloud (RStudio Cloud reported an error (calling /v1/users/me): bad signature).

Can all these errors instead just throw a consistent error like it does with an invalid --token? This error is thrown regardless of token length if it's incorrect.

-> rsconnect add --account pxb42b-kevin-gartland --name cloud --token k84E0E24166F1432FBE43D67746A6973A --secret vzo/Bj43ZGvojhHbTB5CIAkFkE0VBjZYc5O6lyIQ --server https://api.staging.rstudio.cloud
Checking RStudio Cloud credential... [ERROR]
Error: Failed to verify with RStudio Cloud (RStudio Cloud reported an error (calling /v1/users/me): invalid token).

kgartland-rstudio · 2022-09-22T15:09:21Z

should we update our --help menus? we currently only state RStudio Connect as the deploy target:

-> rsconnect deploy --help

Usage: rsconnect deploy [OPTIONS] COMMAND [ARGS]...

Deploy content to RStudio Connect.

Options:
--help Show this message and exit.

Commands:
api Deploy a Python API to RStudio Connect [v1.8.2+].
bokeh Deploy a Bokeh Application to RStudio Connect [v1.8.4+].
dash Deploy a Dash Application to RStudio Connect [v1.8.2+].
fastapi Deploy a Python FastAPI to RStudio Connect [v2021.08.0+].
html Deploy html content to RStudio Connect.
manifest Deploy content to RStudio Connect by manifest.
notebook Deploy Jupyter notebook to RStudio Connect [v1.7.0+].
other-content Describe deploying other content to RStudio Connect.
quarto Deploy Quarto content to RStudio Connect [v2021.08.0+].
shiny Deploy a Python Shiny Application to RStudio Connect
[v2022.07.0+].
streamlit Deploy a Streamlit Application to RStudio Connect [v1.8.4+].

-> rsconnect --help

Usage: rsconnect [OPTIONS] COMMAND [ARGS]...

This command line tool may be used to deploy Jupyter notebooks to RStudio
Connect. Support for deploying other content types is also provided.

The tool supports the notion of a simple nickname that represents the
information needed to interact with an RStudio Connect server instance. Use
the add, list and remove commands to manage these nicknames.

The information about an instance of RStudio Connect includes its URL, the
API key needed to authenticate against that instance, a flag that notes
whether TLS certificate/host verification should be disabled and a path to a
trusted CA certificate file to use for TLS. The last two items are only
relevant if the URL specifies the "https" protocol.

Options:
--help Show this message and exit.

Commands:
add Define a nickname for an RStudio Connect or shinyapps.io
server and credential.
content Interact with RStudio Connect's content API.
deploy Deploy content to RStudio Connect.
details Show details about an RStudio Connect server.
info Show saved information about the specified deployment.
list List the known RStudio Connect servers.
remove Remove the information about an RStudio Connect server.
version Show the version of the rsconnect-python package.
write-manifest Create a manifest.json file for later deployment.

-> rsconnect deploy shiny --help

Usage: rsconnect deploy shiny [OPTIONS] DIRECTORY [EXTRA_FILES]...

Deploy a Python Shiny Application module to RStudio Connect. The "directory"
argument must refer to an existing directory that contains the application
code.

Options:
-n, --name TEXT The nickname of the RStudio Connect server to deploy
to.
-s, --server TEXT The URL for the RStudio Connect server to deploy to.
-k, --api-key TEXT The API key to use to authenticate with RStudio
Connect.
-i, --insecure Disable TLS certification/host validation.
-c, --cacert FILENAME The path to trusted TLS CA certificates.
-v, --verbose Print detailed messages.
-N, --new Force a new deployment, even if there is saved
metadata from a previous deployment. Cannot be used
with --app-id.
-a, --app-id TEXT Existing app ID or GUID to replace. Cannot be used
with --new.
-t, --title TEXT Title of the content (default is the same as the
filename).
-E, --environment TEXT Set an environment variable. Specify a value with
NAME=VALUE, or just NAME to use the value from the
local environment. May be specified multiple times.
[v1.8.6+]
-A, --account TEXT The shinyapps.io/RStudio Cloud account name.
-T, --token TEXT The shinyapps.io/RStudio Cloud token.
-S, --secret TEXT The shinyapps.io/RStudio Cloud token secret.
-e, --entrypoint TEXT The module and executable object which serves as the
entry point for the Python Shiny Application
(defaults to app)
-x, --exclude TEXT Specify a glob pattern for ignoring files when
building the bundle. Note that your shell may try to
expand this which will not do what you expect.
Generally, it's safest to quote the pattern. This
option may be repeated.
-p, --python PATH Path to Python interpreter whose environment should
be used. The Python environment must have the
rsconnect package installed.
-g, --force-generate Force generating "requirements.txt", even if it
already exists.
-I, --image TEXT Target image to be used during content execution
(only applicable if the RStudio Connect server is
configured to use off-host execution)
--help Show this message and exit.

-> rsconnect add --help

Usage: rsconnect add [OPTIONS]

Associate a simple nickname with the information needed to interact with an
RStudio Connect server. Specifying an existing nickname will cause its
stored information to be replaced by what is given on the command line.

Options:
-n, --name TEXT The nickname of the RStudio Connect server to deploy
to. [required]
-s, --server TEXT The URL for the RStudio Connect server to deploy to.
-k, --api-key TEXT The API key to use to authenticate with RStudio
Connect.
-i, --insecure Disable TLS certification/host validation.
-c, --cacert FILENAME The path to trusted TLS CA certificates.
-A, --account TEXT The shinyapps.io/RStudio Cloud account name.
-T, --token TEXT The shinyapps.io/RStudio Cloud token.
-S, --secret TEXT The shinyapps.io/RStudio Cloud token secret.
-v, --verbose Print detailed messages.
--help Show this message and exit.

kgartland-rstudio · 2022-09-26T18:41:59Z

Everything else looks good, I think the secret length errors above should be fixed but all other regression testing is passing.

mslynch · 2022-09-26T20:11:05Z

@kgartland-rstudio

Can all these errors instead just throw a consistent error like it does with an invalid --token? This error is thrown regardless of token length if it's incorrect.

I updated the error messages for the Internal errors to say Invalid secret. since those are local validation, but I think it's important to still return the API's error when it's available.

kgartland-rstudio · 2022-09-26T20:39:39Z

Looks good, thanks. Now regardless if the secret is not the right length, I get:

-> rsconnect add --account pxb42b-kevin-gartland --name cloud --token {TOKEN} --secret {SECRET}--server https://api.staging.rstudio.cloud
Checking RStudio Cloud credential... [ERROR]
Error: Invalid secret.

omar-rs

Latest changes look good

Matthew Lynch added 14 commits September 1, 2022 14:15

add support for deployment on RStudio Cloud

6fced34

remove extraneous logging

8b5be54

remove extraneous logging

e84a533

rename test

e4b6573

fix type checking

178fcbf

fix typing

900ec6b

fix test_deploy_manifest_cloud failure

bf4030d

pin mock_connect flask version to 2.1.3 for compatibility

b2f0c58

rename text fixtures folder

6833acc

add test case for associating output with project

1b3d5d1

update error messaging for cloud

c218918

pytest fixes

019fbc5

remove duplicate methods

75b3962

pytest fixes

a85fb82

mslynch requested review from bcwu, mbaynton, mmarchetti, omar-rs, samperman and ssinnott September 2, 2022 15:02

mslynch marked this pull request as ready for review September 2, 2022 15:02

bcwu reviewed Sep 2, 2022

View reviewed changes

rsconnect/main.py Outdated Show resolved Hide resolved

samperman reviewed Sep 6, 2022

View reviewed changes

rsconnect/actions.py Outdated Show resolved Hide resolved

samperman reviewed Sep 6, 2022

View reviewed changes

mbaynton approved these changes Sep 6, 2022

View reviewed changes

omar-rs approved these changes Sep 8, 2022

View reviewed changes

ssinnott approved these changes Sep 8, 2022

View reviewed changes

Matthew Lynch added 2 commits September 8, 2022 09:42

rename lucid naming to rsconnect

9e33141

remove project application id from CLI, but keep environment variable

acdd58d

samperman approved these changes Sep 8, 2022

View reviewed changes

bcwu requested a review from kgartland-rstudio September 9, 2022 15:31

don't fail on list if servers don't have insecure or ca_cert

4b131b8

Matthew Lynch added 2 commits September 26, 2022 15:06

better error messaging for invalid secret

2116e4c

update help messages to be reflective of RStudio Cloud

1cd0583

Merge branch 'master' into cloud-deploys

dee0e6a

bcwu approved these changes Sep 29, 2022

View reviewed changes

Matthew Lynch added 3 commits September 29, 2022 15:10

tweak deploy help messages

86aad3b

handle deploy timeouts on rscloud/shinyapps.io with error

19509b5

deploy output to same space as project

942e672

omar-rs approved these changes Sep 30, 2022

View reviewed changes

Matthew Lynch added 3 commits October 3, 2022 13:31

open content url for cloud outputs

eb72990

Merge branch 'master' into cloud-deploys

eae6ff4

fix test

f3a9321

mslynch merged commit 1221c76 into master Oct 11, 2022

mslynch deleted the cloud-deploys branch October 11, 2022 17:20

add support for deployment on RStudio Cloud #278

add support for deployment on RStudio Cloud #278

Uh oh!

Conversation

mslynch commented Sep 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing Notes / Validation Steps

Uh oh!

Uh oh!

Uh oh!

samperman Sep 6, 2022

Choose a reason for hiding this comment

Uh oh!

ssinnott Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbaynton left a comment

Choose a reason for hiding this comment

Uh oh!

omar-rs left a comment

Choose a reason for hiding this comment

Uh oh!

ssinnott left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bcwu commented Sep 9, 2022

Uh oh!

bcwu commented Sep 12, 2022

Uh oh!

kgartland-rstudio commented Sep 13, 2022

Uh oh!

kgartland-rstudio commented Sep 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mslynch commented Sep 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kgartland-rstudio commented Sep 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kgartland-rstudio commented Sep 22, 2022

Uh oh!

kgartland-rstudio commented Sep 26, 2022

Uh oh!

mslynch commented Sep 26, 2022

Uh oh!

kgartland-rstudio commented Sep 26, 2022

Uh oh!

omar-rs left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mslynch commented Sep 1, 2022 •

edited

Loading

ssinnott Sep 6, 2022 •

edited

Loading

ssinnott left a comment •

edited

Loading

kgartland-rstudio commented Sep 13, 2022 •

edited

Loading

mslynch commented Sep 15, 2022 •

edited

Loading

kgartland-rstudio commented Sep 22, 2022 •

edited

Loading