Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose the fileuploads API endpoint #894

Merged
merged 2 commits into from Sep 23, 2021

Conversation

vogelsgesang
Copy link
Collaborator

We had at least two independent re-implementations [1, 2] of file uploads
within the last 4 months. And this was despite the fact that both projects
already used TSC which would offer this functionality.

Currently, the upload functionality in TSC is hard to discover as it is not
exposed like all other REST functions. Instead of server.fileuploads, one
has to first create an instance of the (undocumented) Fileuploads class.

The upload functionality was probably because it should be usually unnecessary:
The uploaded files are usually part of publishing a workbook/datasource/...
and the corresponding datasources.publish (and similar) already take care of
the upload internally.
However, TSC isn't always up-to-date with new REST APIs, and by exposing file
uploads directly we can make sure to offer the best possible experience to
users of TSC also in those transition periods.

This commit:

  • turns the Fileuploads class into a normal endpoint class which is not tied
    to one upload (So far, Fileuploads was not stateless. Now it is)
  • adds the endpoint to server, such that file uploads are available as
    server.fileuploads
  • adjusts all other users to use server.fileuploads instead of constructing
    an ad hoc instance of the Fileuploads class

Documentation will be added in a separate commit.

[1] https://github.com/jharris126/tableau-data-update-api-samples/blob/41f51ae4d220de55caf63e91fe9eff5694b9456a/basic/basic_incremental_load.py#L23
[2] https://github.com/tableau/hyper-api-samples/blob/382e66481ec8339407cf9cfa5d41fcdcf3f6a0fb/Community-Supported/clouddb-extractor/tableau_restapi_helpers.py#L165

@t8y8
Copy link
Collaborator

t8y8 commented Sep 20, 2021

@bcantoni it looks like the action changes broke all the CI tests? It fails on the slack message and then stops.

I wouldn't recommend merging this until the tests are running again

t8y8
t8y8 previously approved these changes Sep 20, 2021
Copy link
Collaborator

@t8y8 t8y8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 I don't know why we really need it exposed but I agree it makes more sense, is more consistent, and is flexible enough for the future.

I'm a little worried about how deep it goes though -- when the pipeline is fixed we want to verify the tests pass, and probably run a manual test and post the results.

@bcantoni
Copy link
Contributor

@t8y8 I'll take a look - we had some Slack workspace changes over the weekend and it's possible that's what broke. (Maybe the webhook URL has moved.)

@vogelsgesang Could you please rebase these PRs off the development branch? We do all the work there, then merge to master when releases are cut.

@vogelsgesang
Copy link
Collaborator Author

@bcantoni I rebased on development branch and updated the pull request accordingly

@t8y8 we don't necessarily need to expose it. If you prefer, we could also mount it under the server._fileuploads instead of server.fileuploads, thereby marking it as private... but I think we should unify it with the design of the other endpoints (i.e. stateless and part of the server) as not having it there was pretty surprising

We had at least two independent re-implementations [1, 2] of file uploads
within the last 4 months. And this was despite the fact that both projects
already used TSC which would offer this functionality.

Currently, the upload functionality in TSC is hard to discover as it is not
exposed like all other REST functions. Instead of `server.fileuploads`, one
has to first create an instance of the (undocumented) `Fileuploads` class.

The upload functionality was probably because it should be usually unnecessary:
The uploaded files are usually part of publishing a workbook/datasource/...
and the corresponding `datasources.publish` (and similar) already take care of
the upload internally.
However, TSC isn't always up-to-date with new REST APIs, and by exposing file
uploads directly we can make sure to offer the best possible experience to
users of TSC also in those transition periods.

This commit:
* turns the `Fileuploads` class into a normal endpoint class which is not tied
  to one upload (So far, `Fileuploads` was not stateless. Now it is)
* adds the endpoint to `server`, such that file uploads are available as
  `server.fileuploads`
* adjusts all other users to use `server.fileuploads` instead of constructing
  an ad hoc instance of the `Fileuploads` class

Documentation will be added in a separate commit.

[1] https://github.com/jharris126/tableau-data-update-api-samples/blob/41f51ae4d220de55caf63e91fe9eff5694b9456a/basic/basic_incremental_load.py#L23
[2] https://github.com/tableau/hyper-api-samples/blob/382e66481ec8339407cf9cfa5d41fcdcf3f6a0fb/Community-Supported/clouddb-extractor/tableau_restapi_helpers.py#L165
@vogelsgesang
Copy link
Collaborator Author

vogelsgesang commented Sep 23, 2021

For some reason, Github Actions on this repo still don't trigger for me. Despite adding additional commits/force-pushes...
I see no way to run the test pipeline here in this PR.

There was a successful test run in https://github.com/vogelsgesang/server-client-python/actions/runs/1265585205, though.

probably run a manual test and post the results.

I patched datasources_endpoint.py with FILESIZE_LIMIT = 1 so that it always uses a separate file upload and then ran

python samples/publish_datasource.py --server https://us-west-2a.online.tableau.com --token-name "tableauserverclient test" --site=alpodev --token-value ...redacted...
--filepath test/assets/World\ Indicators.hyper --project Hyper --logging-level debug

The upload succeeded with the log showing that we indeed used a separate file upload:

DEBUG:tableau.endpoint:request get, url: https://us-west-2a.online.tableau.com/api/2.4/serverInfo
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us-west-2a.online.tableau.com:443
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "GET /api/2.4/serverInfo HTTP/1.1" 200 265
DEBUG:tableau.endpoint:Server response from https://us-west-2a.online.tableau.com/api/2.4/serverInfo:
        <?xml version='1.0' encoding='UTF-8'?><tsResponse xmlns="http://tableau.com/api" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tableau.com/api https://help.tableau.com/samples/en-us/rest_api/ts-api_2_4.xsd"><serverInfo><productVersion build="near.21.0916.1424">near</productVersion><restApiVersion>3.14</restApiVersion></serverInfo></tsResponse>
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "POST /api/3.14/auth/signin HTTP/1.1" 200 386
INFO:tableau.endpoint.auth:Signed into https://us-west-2a.online.tableau.com as user with id e5f2eef4-6b85-4b2e-b307-760ff9bf424d
INFO:tableau.endpoint.projects:Querying all projects on site
DEBUG:tableau.endpoint:request get, url: https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/projects
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "GET /api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/projects?pageNumber=1&pageSize=100&filter=name%3Aeq%3AHyper HTTP/1.1" 200 525
DEBUG:tableau.endpoint:Server response from https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/projects:
        <?xml version='1.0' encoding='UTF-8'?><tsResponse xmlns="http://tableau.com/api" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tableau.com/api https://help.tableau.com/samples/en-us/rest_api/ts-api_3_14.xsd"><pagination pageNumber="1" pageSize="100" totalAvailable="1"/><projects><project id="621105f7-acc4-44b5-a662-a0eb00ba3562" name="Hyper" description="Visualizations used to track the Hyper team's progress&#xa;&#xa;!https://cdns.tblsft.com/sites/default/files/800x447_2.png!" parentProjectId="5ba10e8f-d276-4ef0-a111-b3f7c86a48f3" createdAt="2016-10-28T15:06:09Z" updatedAt="2019-03-25T17:08:03Z" contentPermissions="ManagedByOwner"><owner id="41e2b37f-c47b-4f6e-9b03-409d980cf5b7"/></project></projects></tsResponse>
INFO:tableau.endpoint.datasources:Publishing World Indicators.hyper to server with chunking method (datasource over 64MB)
DEBUG:tableau.endpoint:request post, url: https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/fileUploads
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "POST /api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/fileUploads HTTP/1.1" 201 341
DEBUG:tableau.endpoint:Server response from https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/fileUploads:
        <?xml version='1.0' encoding='UTF-8'?><tsResponse xmlns="http://tableau.com/api" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tableau.com/api https://help.tableau.com/samples/en-us/rest_api/ts-api_3_14.xsd"><fileUpload uploadSessionId="190:ca63d4b349584be68fde32605e3f96b4-0:0" fileSize="0"/></tsResponse>
INFO:tableau.endpoint.fileuploads:Initiated file upload session (ID: 190:ca63d4b349584be68fde32605e3f96b4-0:0)
DEBUG:tableau.endpoint:request put, url: https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/fileUploads/190:ca63d4b349584be68fde32605e3f96b4-0:0
DEBUG:tableau.endpoint:request content: b'--90eeef6bb3269619312dd3cb20000754\r\nContent-Disposition: form-data; name="request_payload"; filename=""\r\nContent-Type: text/xml\r\n\r\n\r\n--90eeef6bb3269619312dd3cb20000754\r\nContent-Disposition: form-data; name="tableau_file"; filename="file"\r\nContent-Type: application/octet-stream\r\n\r\nHyper\x08\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe4\x07\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\xe4\x07\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00P0\x00\x00\x00\x00\x00\x00\x800\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00Tg_\x9c\x81\xd9\xf2\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "PUT /api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/fileUploads/190:ca63d4b349584be68fde32605e3f96b4-0:0 HTTP/1.1" 200 265
DEBUG:tableau.endpoint:Server response from https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/fileUploads/190:ca63d4b349584be68fde32605e3f96b4-0:0:
        <?xml version='1.0' encoding='UTF-8'?><tsResponse xmlns="http://tableau.com/api" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tableau.com/api https://help.tableau.com/samples/en-us/rest_api/ts-api_3_14.xsd"><fileUpload uploadSessionId="190:ca63d4b349584be68fde32605e3f96b4-0:0" fileSize="1"/></tsResponse>
INFO:tableau.endpoint.fileuploads:Uploading a chunk to session (ID: 190:ca63d4b349584be68fde32605e3f96b4-0:0)
INFO:tableau.endpoint.fileuploads:      Published 1MB
INFO:tableau.endpoint.fileuploads:File upload finished (ID: 190:ca63d4b349584be68fde32605e3f96b4-0:0)
DEBUG:tableau.endpoint:request post, url: https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/datasources?datasourceType=hyper&overwrite=true&uploadSessionId=190:ca63d4b349584be68fde32605e3f96b4-0:0
DEBUG:tableau.endpoint:request content: b'--7e4814f0bed875d535e06e753abf6b9a\r\nContent-Disposition: form-data; name="request_payload"; filename=""\r\nContent-Type: text/xml\r\n\r\n<tsRequest><datasource name="World Indicators"><project id="621105f7-acc4-44b5-a662-a0eb00ba3562" /></datasource></tsRequest>\r\n--7e4814f0bed875d535e06e753abf6b9a--\r\n'
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "POST /api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/datasources?datasourceType=hyper&overwrite=true&uploadSessionId=190:ca63d4b349584be68fde32605e3f96b4-0:0 HTTP/1.1" 201 791
DEBUG:tableau.endpoint:Server response from https://us-west-2a.online.tableau.com/api/3.14/sites/d76ee955-ccc7-4333-ab3b-25ad2f8b1aa8/datasources?datasourceType=hyper&overwrite=true&uploadSessionId=190:ca63d4b349584be68fde32605e3f96b4-0:0:
        <?xml version='1.0' encoding='UTF-8'?><tsResponse xmlns="http://tableau.com/api" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tableau.com/api https://help.tableau.com/samples/en-us/rest_api/ts-api_3_14.xsd"><datasource certificationNote="" contentUrl="WorldIndicators_16319652913270" createdAt="2021-09-18T11:41:31Z" encryptExtracts="false" hasExtracts="false" id="5f137013-a4f3-42c1-8b09-cbd5e4b40d8c" isCertified="false" name="World Indicators" type="hyper" updatedAt="2021-09-23T11:28:28Z" useRemoteQueryAgent="false" webpageUrl="https://us-west-2a.online.tableau.com/#/site/alpodev/datasources/7175333"><project id="621105f7-acc4-44b5-a662-a0eb00ba3562" name="Hyper"/><owner id="e5f2eef4-6b85-4b2e-b307-760ff9bf424d"/><tags/></datasource></tsResponse>
INFO:tableau.endpoint.datasources:Published World Indicators.hyper (ID: 5f137013-a4f3-42c1-8b09-cbd5e4b40d8c)
Datasource published. Datasource ID: 5f137013-a4f3-42c1-8b09-cbd5e4b40d8c
DEBUG:tableau.endpoint:request post, url: https://us-west-2a.online.tableau.com/api/3.14/auth/signout
DEBUG:urllib3.connectionpool:https://us-west-2a.online.tableau.com:443 "POST /api/3.14/auth/signout HTTP/1.1" 204 0
INFO:tableau.endpoint.auth:Signed out

@t8y8 @bcantoni are we fine with merging this (also take into account the additional commit regarding exception safety which I added to this PR)

@t8y8 t8y8 self-requested a review September 23, 2021 17:39
Copy link
Collaborator

@t8y8 t8y8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@t8y8
Copy link
Collaborator

t8y8 commented Sep 23, 2021

Yeah, I'm fine with merging given the manual test runs (I think if you branch from master, even if you rebase the PR on development, that prevents the CI from running. Don't know why)

@vogelsgesang vogelsgesang merged commit ae56feb into tableau:development Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants