chore: move samples from python-docs-sample (#66)

* Add XMPP Sample * Add Dataproc Sample * Add more region tags * Minor dataproc fixes * Fix Dataproc e2e for Python 3 * Update reqs * updating requirements [(#358)](GoogleCloudPlatform/python-docs-samples#358) Change-Id: I6177a17fad021e26ed76679d9db34848c17b62a8 * Update Reqs * Wrong arg description * Auto-update dependencies. [(#456)](GoogleCloudPlatform/python-docs-samples#456) * Auto-update dependencies. [(#459)](GoogleCloudPlatform/python-docs-samples#459) * Fix import order lint errors Change-Id: Ieaf7237fc6f925daec46a07d2e81a452b841198a * bump Change-Id: I02e7767d13ba267ee9fc72c5b68a57013bb8b8d3 * Auto-update dependencies. [(#486)](GoogleCloudPlatform/python-docs-samples#486) * Auto-update dependencies. [(#540)](GoogleCloudPlatform/python-docs-samples#540) * Auto-update dependencies. [(#542)](GoogleCloudPlatform/python-docs-samples#542) * Move to google-cloud [(#544)](GoogleCloudPlatform/python-docs-samples#544) * Auto-update dependencies. [(#584)](GoogleCloudPlatform/python-docs-samples#584) * Auto-update dependencies. [(#629)](GoogleCloudPlatform/python-docs-samples#629) * Update samples to support latest Google Cloud Python [(#656)](GoogleCloudPlatform/python-docs-samples#656) * Update README.md [(#691)](GoogleCloudPlatform/python-docs-samples#691) * Auto-update dependencies. [(#715)](GoogleCloudPlatform/python-docs-samples#715) * Auto-update dependencies. [(#735)](GoogleCloudPlatform/python-docs-samples#735) * Auto-update dependencies. * Fix language OCR sample * Remove unused import * Auto-update dependencies. [(#790)](GoogleCloudPlatform/python-docs-samples#790) * Remove usage of GoogleCredentials [(#810)](GoogleCloudPlatform/python-docs-samples#810) * Fix a typo [(#813)](GoogleCloudPlatform/python-docs-samples#813) * Remove cloud config fixture [(#887)](GoogleCloudPlatform/python-docs-samples#887) * Remove cloud config fixture * Fix client secrets * Fix bigtable instance * Fix reference to our testing tools * Auto-update dependencies. [(#914)](GoogleCloudPlatform/python-docs-samples#914) * Auto-update dependencies. * xfail the error reporting test * Fix lint * Auto-update dependencies. [(#922)](GoogleCloudPlatform/python-docs-samples#922) * Auto-update dependencies. * Fix pubsub iam samples * Auto-update dependencies. [(#1005)](GoogleCloudPlatform/python-docs-samples#1005) * Auto-update dependencies. * Fix bigtable lint * Fix IOT iam interaction * Auto-update dependencies. [(#1011)](GoogleCloudPlatform/python-docs-samples#1011) * Properly forwarding the "region" parameter provided as an input argument. [(#1029)](GoogleCloudPlatform/python-docs-samples#1029) * Auto-update dependencies. [(#1055)](GoogleCloudPlatform/python-docs-samples#1055) * Auto-update dependencies. * Explicitly use latest bigtable client Change-Id: Id71e9e768f020730e4ca9514a0d7ebaa794e7d9e * Revert language update for now Change-Id: I8867f154e9a5aae00d0047c9caf880e5e8f50c53 * Remove pdb. smh Change-Id: I5ff905fadc026eebbcd45512d4e76e003e3b2b43 * Fix region handling and allow to use an existing cluster. [(#1053)](GoogleCloudPlatform/python-docs-samples#1053) * Auto-update dependencies. [(#1094)](GoogleCloudPlatform/python-docs-samples#1094) * Auto-update dependencies. * Relax assertions in the ocr_nl sample Change-Id: I6d37e5846a8d6dd52429cb30d501f448c52cbba1 * Drop unused logging apiary samples Change-Id: I545718283773cb729a5e0def8a76ebfa40829d51 * Auto-update dependencies. [(#1133)](GoogleCloudPlatform/python-docs-samples#1133) * Auto-update dependencies. * Fix missing http library Change-Id: I99faa600f2f3f1f50f57694fc9835d7f35bda250 * Auto-update dependencies. [(#1186)](GoogleCloudPlatform/python-docs-samples#1186) * Auto-update dependencies. [(#1199)](GoogleCloudPlatform/python-docs-samples#1199) * Auto-update dependencies. * Fix iot lint Change-Id: I6289e093bdb35e38f9e9bfc3fbc3df3660f9a67e * Fixed Failed Kokoro Test (Dataproc) [(#1203)](GoogleCloudPlatform/python-docs-samples#1203) * Fixed Failed Kokoro Test (Dataproc) * Fixed Lint Error * Update dataproc_e2e_test.py * Update dataproc_e2e_test.py * Fixing More Lint Errors * Fixed b/65407087 * Revert "Merge branch 'master' of https://github.com/michaelawyu/python-docs-samples" This reverts commit 1614c7d3ef33630a8ab095792b27fc25fd91f0ad, reversing changes made to cd1dbfd25997a154a8a85cc754cc2a85b18a63c4. * Revert "Fixed b/65407087" This reverts commit cd1dbfd25997a154a8a85cc754cc2a85b18a63c4. * Fixed Lint Error * Fixed Lint Error * Auto-update dependencies. [(#1208)](GoogleCloudPlatform/python-docs-samples#1208) * Dataproc GCS sample plus doc touchups [(#1151)](GoogleCloudPlatform/python-docs-samples#1151) * Auto-update dependencies. [(#1217)](GoogleCloudPlatform/python-docs-samples#1217) * Auto-update dependencies. [(#1239)](GoogleCloudPlatform/python-docs-samples#1239) * Added "Open in Cloud Shell" buttons to README files [(#1254)](GoogleCloudPlatform/python-docs-samples#1254) * Auto-update dependencies. [(#1282)](GoogleCloudPlatform/python-docs-samples#1282) * Auto-update dependencies. * Fix storage acl sample Change-Id: I413bea899fdde4c4859e4070a9da25845b81f7cf * Auto-update dependencies. [(#1309)](GoogleCloudPlatform/python-docs-samples#1309) * Auto-update dependencies. [(#1320)](GoogleCloudPlatform/python-docs-samples#1320) * Auto-update dependencies. [(#1355)](GoogleCloudPlatform/python-docs-samples#1355) * Auto-update dependencies. [(#1359)](GoogleCloudPlatform/python-docs-samples#1359) * Auto-update dependencies. * update Dataproc region tags to standard format [(#1826)](GoogleCloudPlatform/python-docs-samples#1826) * Update submit_job_to_cluster.py [(#1708)](GoogleCloudPlatform/python-docs-samples#1708) switch region to new 'global' region and remove unnecessary function. * Auto-update dependencies. [(#1846)](GoogleCloudPlatform/python-docs-samples#1846) ACK, merging. * Need separate install for google-cloud-storage [(#1863)](GoogleCloudPlatform/python-docs-samples#1863) * Revert "Update dataproc/submit_job_to_cluster.py" [(#1864)](GoogleCloudPlatform/python-docs-samples#1864) * Revert "Remove test configs for non-testing directories [(#1855)](GoogleCloudPlatform/python-docs-samples#1855)" This reverts commit 73a73321579337312e8ba85c34fe9c37b42b7f6e. * Revert "Auto-update dependencies. [(#1846)](GoogleCloudPlatform/python-docs-samples#1846)" This reverts commit 3adc94f4d0c14453153968c3851fae100e2c5e44. * Revert "Tweak slack sample [(#1847)](GoogleCloudPlatform/python-docs-samples#1847)" This reverts commit a48c010481c166968d9f1bd58106054c5d1c58f9. * Revert "Non-client library example of constructing a Signed URL [(#1837)](GoogleCloudPlatform/python-docs-samples#1837)" This reverts commit fc3284d995a8a35c473a207e80490fad265782af. * Revert "GCF samples: handle {empty JSON, GET} requests + remove commas [(#1832)](GoogleCloudPlatform/python-docs-samples#1832)" This reverts commit 6928491ed3d52b0bec694e6b30257f08caac5f2b. * Revert "Correct the maintenance event types [(#1830)](GoogleCloudPlatform/python-docs-samples#1830)" This reverts commit c22840fd23586349b7b665d851dea046a94ba7c7. * Revert "Fix GCF region tags [(#1827)](GoogleCloudPlatform/python-docs-samples#1827)" This reverts commit 0fbfef27d35cea23ad0e20fd2c9df3e8a4a046cb. * Revert "Updated to Flask 1.0 [(#1819)](GoogleCloudPlatform/python-docs-samples#1819)" This reverts commit d52ccf99503311bba2cec2881e8cb0f9b5a6f2bf. * Revert "Fix deprecation warning [(#1801)](GoogleCloudPlatform/python-docs-samples#1801)" This reverts commit 981737e85f60eca5cc337f172249deddca9b291b. * Revert "Update submit_job_to_cluster.py [(#1708)](GoogleCloudPlatform/python-docs-samples#1708)" This reverts commit df1f2b22547b7ca86bbdb791ad930003a815a677. * Create python-api-walkthrough.md [(#1966)](GoogleCloudPlatform/python-docs-samples#1966) * Create python-api-walkthrough.md This Google Cloud Shell walkthrough is linked to Cloud Dataproc documentation to be published at: https://cloud.google.com/dataproc/docs/tutorials/python-library-example * Update python-api-walkthrough.md * Update list_clusters.py [(#1887)](GoogleCloudPlatform/python-docs-samples#1887) * Auto-update dependencies. [(#1980)](GoogleCloudPlatform/python-docs-samples#1980) * Auto-update dependencies. * Update requirements.txt * Update requirements.txt * Update Dataproc samples. [(#2158)](GoogleCloudPlatform/python-docs-samples#2158) * Update requirements.txt * Update python-api-walkthrough.md * Update submit_job_to_cluster.py * Update list_clusters.py * Update python-api-walkthrough.md [(#2172)](GoogleCloudPlatform/python-docs-samples#2172) * Adds updates including compute [(#2436)](GoogleCloudPlatform/python-docs-samples#2436) * Adds updates including compute * Python 2 compat pytest * Fixing weird \r\n issue from GH merge * Put asset tests back in * Re-add pod operator test * Hack parameter for k8s pod operator * feat: adding samples for dataproc - create cluster [(#2536)](GoogleCloudPlatform/python-docs-samples#2536) * adding sample for cluster create * small fix * Add create cluster samples * Fixed copyright, added 'dataproc' to region tag and changed imports from 'dataproc' to 'dataproc_v1' * Fix copyright in create_cluster.py * Auto-update dependencies. [(#2005)](GoogleCloudPlatform/python-docs-samples#2005) * Auto-update dependencies. * Revert update of appengine/flexible/datastore. * revert update of appengine/flexible/scipy * revert update of bigquery/bqml * revert update of bigquery/cloud-client * revert update of bigquery/datalab-migration * revert update of bigtable/quickstart * revert update of compute/api * revert update of container_registry/container_analysis * revert update of dataflow/run_template * revert update of datastore/cloud-ndb * revert update of dialogflow/cloud-client * revert update of dlp * revert update of functions/imagemagick * revert update of functions/ocr/app * revert update of healthcare/api-client/fhir * revert update of iam/api-client * revert update of iot/api-client/gcs_file_to_device * revert update of iot/api-client/mqtt_example * revert update of language/automl * revert update of run/image-processing * revert update of vision/automl * revert update testing/requirements.txt * revert update of vision/cloud-client/detect * revert update of vision/cloud-client/product_search * revert update of jobs/v2/api_client * revert update of jobs/v3/api_client * revert update of opencensus * revert update of translate/cloud-client * revert update to speech/cloud-client Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Doug Mahugh <dmahugh@gmail.com> * feat: dataproc quickstart sample added and create_cluster updated [(#2629)](GoogleCloudPlatform/python-docs-samples#2629) * Adding quickstart sample * Added new quickstart sample and updated create_cluster sample * Fix to create_cluster.py * deleted dataproc quickstart files not under dataproc/quickstart/ * Added quickstart test * Linting and formatting fixes * Revert "Linting and formatting fixes" This reverts commit c5afcbcdf9deccbb7a21ddd82ae0fc305e79c008. * Added bucket cleanup to quickstart test * Changes to samples and tests * Linting fixes * Removed todos in favor of clearer docstring * Fixed lint error Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * Update Python Cloud Shell walkthrough script [(#2733)](GoogleCloudPlatform/python-docs-samples#2733) Cloud Shell walkthrough scripts no longer support enabling APIs. APIs must be enabled by linking to the console. Updated product name: "Cloud Dataproc" -> "Dataproc". * fix: added cli functionality to dataproc quickstart example [(#2734)](GoogleCloudPlatform/python-docs-samples#2734) * Added CLI functionality to quickstart * Fixed Dataproc quickstart test to properly clean up GCS bucket [(#3001)](GoogleCloudPlatform/python-docs-samples#3001) * splitting up #2651 part 1/3 - dataproc + endpoints [(#3025)](GoogleCloudPlatform/python-docs-samples#3025) * splitting up #2651 * fix typos * chore(deps): update dependency google-auth to v1.11.2 [(#2724)](GoogleCloudPlatform/python-docs-samples#2724) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-cloud-storage to v1.26.0 [(#3046)](GoogleCloudPlatform/python-docs-samples#3046) * chore(deps): update dependency google-cloud-storage to v1.26.0 * chore(deps): specify dependencies by python version * chore: up other deps to try to remove errors Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Leah Cole <coleleah@google.com> * chore(deps): update dependency google-cloud-dataproc to v0.7.0 [(#3083)](GoogleCloudPlatform/python-docs-samples#3083) * feat: added dataproc workflows samples [(#3056)](GoogleCloudPlatform/python-docs-samples#3056) * Added workflows sample * chore(deps): update dependency grpcio to v1.27.2 [(#3173)](GoogleCloudPlatform/python-docs-samples#3173) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [grpcio](https://grpc.io) | minor | `==1.25.0` -> `==1.27.2` | | [grpcio](https://grpc.io) | minor | `==1.23.0` -> `==1.27.2` | | [grpcio](https://grpc.io) | minor | `==1.26.0` -> `==1.27.2` | | [grpcio](https://grpc.io) | patch | `==1.27.1` -> `==1.27.2` | --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Simplify noxfile setup. [(#2806)](GoogleCloudPlatform/python-docs-samples#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot <bot@renovateapp.com> * fix: add mains to samples [(#3284)](GoogleCloudPlatform/python-docs-samples#3284) Added mains to two samples: create_cluster and instantiate_inline_workflow_templates. Fixed their associated tests to accommodate this. Removed subprocess from quickstart/quickstart_test.py to fix [2873](GoogleCloudPlatform/python-docs-samples#2873) fixes #2873 * Update dependency grpcio to v1.28.1 [(#3276)](GoogleCloudPlatform/python-docs-samples#3276) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * Update dependency google-auth to v1.14.0 [(#3148)](GoogleCloudPlatform/python-docs-samples#3148) Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * chore(deps): update dependency google-auth to v1.14.1 [(#3464)](GoogleCloudPlatform/python-docs-samples#3464) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.0` -> `==1.14.1` | | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | minor | `==1.11.2` -> `==1.14.1` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.14.1`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1141-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1140v1141-2020-04-21) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.0...v1.14.1) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-cloud-storage to v1.28.0 [(#3260)](GoogleCloudPlatform/python-docs-samples#3260) Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * chore(deps): update dependency google-auth to v1.14.2 [(#3724)](GoogleCloudPlatform/python-docs-samples#3724) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.1` -> `==1.14.2` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.14.2`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1142-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1141v1142-2020-05-07) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.1...v1.14.2) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore: some lint fixes [(#3743)](GoogleCloudPlatform/python-docs-samples#3743) * chore(deps): update dependency google-auth to v1.14.3 [(#3728)](GoogleCloudPlatform/python-docs-samples#3728) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.2` -> `==1.14.3` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.14.3`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1143-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1142v1143-2020-05-11) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.2...v1.14.3) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [x] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency grpcio to v1.29.0 [(#3786)](GoogleCloudPlatform/python-docs-samples#3786) * chore(deps): update dependency google-cloud-storage to v1.28.1 [(#3785)](GoogleCloudPlatform/python-docs-samples#3785) * chore(deps): update dependency google-cloud-storage to v1.28.1 * [asset] testing: use uuid instead of time Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * update google-auth to 1.15.0 part 3 [(#3816)](GoogleCloudPlatform/python-docs-samples#3816) * Update dependency google-cloud-dataproc to v0.8.0 [(#3837)](GoogleCloudPlatform/python-docs-samples#3837) * chore(deps): update dependency google-auth to v1.16.0 [(#3903)](GoogleCloudPlatform/python-docs-samples#3903) * update google-auth part 3 [(#3963)](GoogleCloudPlatform/python-docs-samples#3963) * chore(deps): update dependency google-cloud-dataproc to v0.8.1 [(#4015)](GoogleCloudPlatform/python-docs-samples#4015) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | patch | `==0.8.0` -> `==0.8.1` | --- ### Release Notes <details> <summary>googleapis/python-dataproc</summary> ### [`v0.8.1`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#081-httpswwwgithubcomgoogleapispython-dataproccomparev080v081-2020-06-05) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v0.8.0...v0.8.1) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](GoogleCloudPlatform/python-docs-samples#4022) * Update dependency google-auth to v1.17.0 [(#4058)](GoogleCloudPlatform/python-docs-samples#4058) * chore(deps): update dependency google-auth to v1.17.1 [(#4073)](GoogleCloudPlatform/python-docs-samples#4073) * Update dependency google-auth to v1.17.2 [(#4083)](GoogleCloudPlatform/python-docs-samples#4083) * Update dependency google-auth to v1.18.0 [(#4125)](GoogleCloudPlatform/python-docs-samples#4125) * Update dependency google-cloud-dataproc to v1 [(#4109)](GoogleCloudPlatform/python-docs-samples#4109) Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * chore(deps): update dependency google-cloud-storage to v1.29.0 [(#4040)](GoogleCloudPlatform/python-docs-samples#4040) * chore(deps): update dependency grpcio to v1.30.0 [(#4143)](GoogleCloudPlatform/python-docs-samples#4143) Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * Update dependency google-auth-httplib2 to v0.0.4 [(#4255)](GoogleCloudPlatform/python-docs-samples#4255) Co-authored-by: Takashi Matsuo <tmatsuo@google.com> * chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloudPlatform/python-docs-samples#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <coleleah@google.com> * chore(deps): update dependency google-auth to v1.19.0 [(#4293)](GoogleCloudPlatform/python-docs-samples#4293) * chore(deps): update dependency google-cloud-dataproc to v1.0.1 [(#4309)](GoogleCloudPlatform/python-docs-samples#4309) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-dataproc](https://togithub.com/googleapis/python-dataproc) | patch | `==1.0.0` -> `==1.0.1` | --- ### Release Notes <details> <summary>googleapis/python-dataproc</summary> ### [`v1.0.1`](https://togithub.com/googleapis/python-dataproc/blob/master/CHANGELOG.md#101-httpswwwgithubcomgoogleapispython-dataproccomparev100v101-2020-07-16) [Compare Source](https://togithub.com/googleapis/python-dataproc/compare/v1.0.0...v1.0.1) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * chore(deps): update dependency google-auth to v1.19.1 [(#4304)](GoogleCloudPlatform/python-docs-samples#4304) * chore(deps): update dependency google-auth to v1.19.2 [(#4321)](GoogleCloudPlatform/python-docs-samples#4321) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.19.1` -> `==1.19.2` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.19.2`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1192-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1191v1192-2020-07-17) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.19.1...v1.19.2) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Update dependency google-auth to v1.20.0 [(#4387)](GoogleCloudPlatform/python-docs-samples#4387) * Update dependency pytest to v6 [(#4390)](GoogleCloudPlatform/python-docs-samples#4390) * Update dependency grpcio to v1.31.0 [(#4438)](GoogleCloudPlatform/python-docs-samples#4438) * chore(deps): update dependency google-auth to v1.20.1 [(#4452)](GoogleCloudPlatform/python-docs-samples#4452) * chore: update templates Co-authored-by: Bill Prin <waprin@google.com> Co-authored-by: Bill Prin <waprin@gmail.com> Co-authored-by: Jon Wayne Parrott <jonwayne@google.com> Co-authored-by: Eran Kampf <eran@ekampf.com> Co-authored-by: DPE bot <dpebot@google.com> Co-authored-by: aman-ebay <amancuso@google.com> Co-authored-by: Martial Hue <martial.hue@gmail.com> Co-authored-by: Gioia Ballin <gioia.ballin@gmail.com> Co-authored-by: michaelawyu <chenyumic@google.com> Co-authored-by: michaelawyu <michael.a.w.yu@hotmail.com> Co-authored-by: Alix Hamilton <ajhamilton@google.com> Co-authored-by: James Winegar <jameswinegar@users.noreply.github.com> Co-authored-by: Charles Engelke <github@engelke.com> Co-authored-by: Gus Class <gguuss@gmail.com> Co-authored-by: Brad Miro <bmiro@google.com> Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com> Co-authored-by: Doug Mahugh <dmahugh@gmail.com> Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: WhiteSource Renovate <bot@renovateapp.com> Co-authored-by: Leah Cole <coleleah@google.com> Co-authored-by: Takashi Matsuo <tmatsuo@google.com>
googleapis · Aug 8, 2020 · d3dcbbc · d3dcbbc
1 parent f9fb2d2
commit d3dcbbc
Show file tree

Hide file tree

Showing 33 changed files with 2,446 additions and 27 deletions.
diff --git a/packages/google-cloud-dataproc/.github/CODEOWNERS b/packages/google-cloud-dataproc/.github/CODEOWNERS
@@ -0,0 +1,8 @@
+# Code owners file.
+# This file controls who is tagged for review for any given pull request.
+#
+# For syntax help see:
+# https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners#codeowners-syntax
+
+
+/samples/**/*.py                           @bradmiro @googleapis/python-samples-owners
diff --git a/packages/google-cloud-dataproc/.gitignore b/packages/google-cloud-dataproc/.gitignore
@@ -46,6 +46,7 @@ pip-log.txt
 # Built documentation
 docs/_build
 bigquery/docs/generated
+docs.metadata
 
 # Virtual environment
 env/
@@ -57,4 +58,4 @@ system_tests/local_test_setup
 
 # Make sure a generated file isn't accidentally committed.
 pylintrc
-pylintrc.test
+pylintrc.test
diff --git a/packages/google-cloud-dataproc/.kokoro/build.sh b/packages/google-cloud-dataproc/.kokoro/build.sh
@@ -36,4 +36,10 @@ python3.6 -m pip uninstall --yes --quiet nox-automation
 python3.6 -m pip install --upgrade --quiet nox
 python3.6 -m nox --version
 
-python3.6 -m nox
+# If NOX_SESSION is set, it only runs the specified session,
+# otherwise run all the sessions.
+if [[ -n "${NOX_SESSION:-}" ]]; then
+    python3.6 -m nox -s "${NOX_SESSION:-}"
+else
+    python3.6 -m nox
+fi
diff --git a/packages/google-cloud-dataproc/.kokoro/docker/docs/Dockerfile b/packages/google-cloud-dataproc/.kokoro/docker/docs/Dockerfile
@@ -0,0 +1,98 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ubuntu:20.04
+
+ENV DEBIAN_FRONTEND noninteractive
+
+# Ensure local Python is preferred over distribution Python.
+ENV PATH /usr/local/bin:$PATH
+
+# Install dependencies.
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends \
+    apt-transport-https \
+    build-essential \
+    ca-certificates \
+    curl \
+    dirmngr \
+    git \
+    gpg-agent \
+    graphviz \
+    libbz2-dev \
+    libdb5.3-dev \
+    libexpat1-dev \
+    libffi-dev \
+    liblzma-dev \
+    libreadline-dev \
+    libsnappy-dev \
+    libssl-dev \
+    libsqlite3-dev \
+    portaudio19-dev \
+    redis-server \
+    software-properties-common \
+    ssh \
+    sudo \
+    tcl \
+    tcl-dev \
+    tk \
+    tk-dev \
+    uuid-dev \
+    wget \
+    zlib1g-dev \
+  && add-apt-repository universe \
+  && apt-get update \
+  && apt-get -y install jq \
+  && apt-get clean autoclean \
+  && apt-get autoremove -y \
+  && rm -rf /var/lib/apt/lists/* \
+  && rm -f /var/cache/apt/archives/*.deb
+
+
+COPY fetch_gpg_keys.sh /tmp
+# Install the desired versions of Python.
+RUN set -ex \
+    && export GNUPGHOME="$(mktemp -d)" \
+    && echo "disable-ipv6" >> "${GNUPGHOME}/dirmngr.conf" \
+    && /tmp/fetch_gpg_keys.sh \
+    && for PYTHON_VERSION in 3.7.8 3.8.5; do \
+        wget --no-check-certificate -O python-${PYTHON_VERSION}.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \
+        && wget --no-check-certificate -O python-${PYTHON_VERSION}.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \
+        && gpg --batch --verify python-${PYTHON_VERSION}.tar.xz.asc python-${PYTHON_VERSION}.tar.xz \
+        && rm -r python-${PYTHON_VERSION}.tar.xz.asc \
+        && mkdir -p /usr/src/python-${PYTHON_VERSION} \
+        && tar -xJC /usr/src/python-${PYTHON_VERSION} --strip-components=1 -f python-${PYTHON_VERSION}.tar.xz \
+        && rm python-${PYTHON_VERSION}.tar.xz \
+        && cd /usr/src/python-${PYTHON_VERSION} \
+        && ./configure \
+            --enable-shared \
+            # This works only on Python 2.7 and throws a warning on every other
+            # version, but seems otherwise harmless.
+            --enable-unicode=ucs4 \
+            --with-system-ffi \
+            --without-ensurepip \
+        && make -j$(nproc) \
+        && make install \
+        && ldconfig \
+  ; done \
+  && rm -rf "${GNUPGHOME}" \
+  && rm -rf /usr/src/python* \
+  && rm -rf ~/.cache/
+
+RUN wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \
+  && python3.7 /tmp/get-pip.py \
+  && python3.8 /tmp/get-pip.py \
+  && rm /tmp/get-pip.py
+
+CMD ["python3.7"]
diff --git a/packages/google-cloud-dataproc/.kokoro/docker/docs/fetch_gpg_keys.sh b/packages/google-cloud-dataproc/.kokoro/docker/docs/fetch_gpg_keys.sh
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# A script to fetch gpg keys with retry.
+# Avoid jinja parsing the file.
+# 
+
+function retry {
+    if [[ "${#}" -le 1 ]]; then
+	echo "Usage: ${0} retry_count commands.."
+	exit 1
+    fi
+    local retries=${1}
+    local command="${@:2}"
+    until [[ "${retries}" -le 0 ]]; do
+	$command && return 0
+	if [[ $? -ne 0 ]]; then
+	    echo "command failed, retrying"
+	    ((retries--))
+	fi
+    done
+    return 1
+}
+
+# 3.6.9, 3.7.5 (Ned Deily)
+retry 3 gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \
+      0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D
+
+# 3.8.0 (Łukasz Langa)
+retry 3 gpg --keyserver ha.pool.sks-keyservers.net --recv-keys \
+      E3FF2839C048B25C084DEBE9B26995E310250568
+
+# 
diff --git a/packages/google-cloud-dataproc/.kokoro/docs/common.cfg b/packages/google-cloud-dataproc/.kokoro/docs/common.cfg
@@ -11,12 +11,12 @@ action {
 gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
 
 # Use the trampoline script to run in docker.
-build_file: "python-dataproc/.kokoro/trampoline.sh"
+build_file: "python-dataproc/.kokoro/trampoline_v2.sh"
 
 # Configure the docker image for kokoro-trampoline.
 env_vars: {
     key: "TRAMPOLINE_IMAGE"
-    value: "gcr.io/cloud-devrel-kokoro-resources/python-multi"
+    value: "gcr.io/cloud-devrel-kokoro-resources/python-lib-docs"
 }
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
@@ -28,6 +28,23 @@ env_vars: {
     value: "docs-staging"
 }
 
+env_vars: {
+    key: "V2_STAGING_BUCKET"
+    value: "docs-staging-v2-staging"
+}
+
+# It will upload the docker image after successful builds.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE_UPLOAD"
+    value: "true"
+}
+
+# It will always build the docker image.
+env_vars: {
+    key: "TRAMPOLINE_DOCKERFILE"
+    value: ".kokoro/docker/docs/Dockerfile"
+}
+
 # Fetch the token needed for reporting release status to GitHub
 before_action {
   fetch_keystore {

diff --git a/packages/google-cloud-dataproc/.kokoro/docs/docs-presubmit.cfg b/packages/google-cloud-dataproc/.kokoro/docs/docs-presubmit.cfg
@@ -0,0 +1,17 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "STAGING_BUCKET"
+    value: "gcloud-python-test"
+}
+
+env_vars: {
+    key: "V2_STAGING_BUCKET"
+    value: "gcloud-python-test"
+}
+
+# We only upload the image in the main `docs` build.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE_UPLOAD"
+    value: "false"
+}
diff --git a/packages/google-cloud-dataproc/.kokoro/publish-docs.sh b/packages/google-cloud-dataproc/.kokoro/publish-docs.sh
@@ -18,26 +18,16 @@ set -eo pipefail
 # Disable buffering, so that the logs stream through.
 export PYTHONUNBUFFERED=1
 
-cd github/python-dataproc
-
-# Remove old nox
-python3.6 -m pip uninstall --yes --quiet nox-automation
+export PATH="${HOME}/.local/bin:${PATH}"
 
 # Install nox
-python3.6 -m pip install --upgrade --quiet nox
-python3.6 -m nox --version
+python3 -m pip install --user --upgrade --quiet nox
+python3 -m nox --version
 
 # build docs
 nox -s docs
 
-python3 -m pip install gcp-docuploader
-
-# install a json parser
-sudo apt-get update
-sudo apt-get -y install software-properties-common
-sudo add-apt-repository universe
-sudo apt-get update
-sudo apt-get -y install jq
+python3 -m pip install --user gcp-docuploader
 
 # create metadata
 python3 -m docuploader create-metadata \
@@ -52,4 +42,23 @@ python3 -m docuploader create-metadata \
 cat docs.metadata
 
 # upload docs
-python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket docs-staging
+python3 -m docuploader upload docs/_build/html --metadata-file docs.metadata --staging-bucket "${STAGING_BUCKET}"
+
+
+# docfx yaml files
+nox -s docfx
+
+# create metadata.
+python3 -m docuploader create-metadata \
+  --name=$(jq --raw-output '.name // empty' .repo-metadata.json) \
+  --version=$(python3 setup.py --version) \
+  --language=$(jq --raw-output '.language // empty' .repo-metadata.json) \
+  --distribution-name=$(python3 setup.py --name) \
+  --product-page=$(jq --raw-output '.product_documentation // empty' .repo-metadata.json) \
+  --github-repository=$(jq --raw-output '.repo // empty' .repo-metadata.json) \
+  --issue-tracker=$(jq --raw-output '.issue_tracker // empty' .repo-metadata.json)
+
+cat docs.metadata
+
+# upload docs
+python3 -m docuploader upload docs/_build/html/docfx_yaml --metadata-file docs.metadata --destination-prefix docfx --staging-bucket "${V2_STAGING_BUCKET}"