Skip to content

Commit

Permalink
Merge branch 'apache:master' into master
Browse files Browse the repository at this point in the history
  • Loading branch information
celeste-zeng committed Jul 28, 2023
2 parents b105fb4 + 7bfead0 commit e3d321b
Show file tree
Hide file tree
Showing 40 changed files with 3,051 additions and 48 deletions.
21 changes: 16 additions & 5 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,16 +83,22 @@ To test new workflows, we recommend the following pattern:

1) Fork the Beam repo
2) Add your proposed workflow to the main branch of your fork.
3) Run the workflow in the [Actions tab](https://github.com/apache/beam/actions) of your fork using the `Run workflow` button
3) Run the workflow in the [Actions tab](https://github.com/apache/beam/actions) of your fork using the `Run workflow` button.

Note: most workflows use [self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners)
with the main and ubuntu labels to execute ([example](https://github.com/apache/beam/blob/5a54ee6ddd8cb8444c41802929a364fe2561001e/.github/workflows/beam_PostCommit_Go_Dataflow_ARM.yml#L41)).
If you are testing on a fork, you likely will not have self-hosted runners set up.
To work around this, you can start using hosted runners and then switch over when you're ready to create a PR.
You can do this by changing `runs-on: [self-hosted, ubuntu-20.04, main]` (self-hosted, use in your PR) to `runs-on: ubuntu-20.04` (GitHub hosted, use for local testing).

## Testing Workflow Updates

If you need to make more changes to the workflow yaml file after it has been added to the repo, you can develop normally on a branch (using your fork or the main Beam repo if you are a committer).
To do so:
If you need to make more changes to the workflow yaml file after it has been added to the repo, you can develop normally on a branch (using your fork or the main Beam repo if you are a committer). If you are a non-committer, this flow has several caveats mentioned below.
To do this:

1) Make your change on a development branch.
3) Navigate to your workflow in the [Actions tab](https://github.com/apache/beam/actions). If your changes are on a fork, navigate to the fork's Actions tab instead. If you don't see the correct action, make sure that your fork's main branch is up to date with Beam's master branch.
4) Click run workflow. Before clicking submit, update to run on your branch.
2) Navigate to your workflow in the [Actions tab](https://github.com/apache/beam/actions). If your changes are on a fork, navigate to the fork's Actions tab instead. If you don't see the correct action, make sure that your fork's main branch is up to date with Beam's master branch.
3) Click run workflow. Before clicking submit, update to run on your branch.

Note: If you run a workflow from your fork of Beam, it will not have access to secrets stored in the Beam repository.
This will cause some things like authenticating to external services to fail, which may cause your workflow to fail.
Expand All @@ -101,6 +107,10 @@ If you run into this issue, you can either:
2) Upload secrets to your repo mirroring the secrets used in the main Beam repo.
3) Wait until the changes are merged into Beam to test (this should only be done rarely).

Additionally, as mentioned above your fork likely will not have self-hosted runners set up.
To work around this, you can start using hosted runners and then switch over when you're ready to create a PR.
You can do this by changing runs-on: [self-hosted, ubuntu-20.04, main] (self-hosted, use in your PR) to runs-on: ubuntu-20.04 (GitHub hosted, use for local testing).

# Workflows
Please note that jobs with matrix need to have matrix element in the comment. Example:
```Run Python PreCommit (3.8)```
Expand All @@ -121,3 +131,4 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| [ PreCommit Python Integration](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Python_Integration.yml) | ['3.8','3.11'] | `Run Python_Integration PreCommit (matrix_element)` | [![.github/workflows/beam_PreCommit_Python_Integration.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Python_Integration.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Python_Integration.yml) |
| [ RAT PreCommit ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_RAT.yml) | N/A | `Run RAT PreCommit` | [![.github/workflows/beam_PreCommit_RAT.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_RAT.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_RAT.yml) |
| [ PreCommit SQL Java11 ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_SQL_Java11.yml) | N/A |`Run SQL_Java11 PreCommit`| [![PreCommit SQL Java11](https://github.com/apache/beam/actions/workflows/beam_PreCommit_SQL_Java11.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_SQL_Java11.yml) |
| [ PreCommit Website Stage GCS ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) | N/A |`Run Website_Stage_GCS PreCommit`| [![PreCommit Website Stage GCS](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) |
116 changes: 116 additions & 0 deletions .github/workflows/beam_PreCommit_Website_Stage_GCS.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: PreCommit Website Stage GCS

on:
push:
tags: ['v*']
branches: ['master', 'release-*']
paths: ['website/**','.github/workflows/beam_PreCommit_Website_Stage_GCS.yml']
pull_request_target:
branches: ['master', 'release-*']
paths: ['website/**']
issue_comment:
types: [created]
schedule:
- cron: '0 */6 * * *'
workflow_dispatch:

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
cancel-in-progress: true

#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
permissions:
actions: write
pull-requests: read
checks: read
contents: read
deployments: read
id-token: none
issues: read
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

env:
gcsbucket: apache-beam-website-pull-requests
ghprbPullId:

jobs:
beam_PreCommit_Website_Stage_GCS:
timeout-minutes: 120
if: |
github.event_name == 'push' ||
github.event_name == 'pull_request_target' ||
github.event_name == 'schedule' ||
github.event.comment.body == 'Run Website_Stage_GCS PreCommit'
runs-on: [self-hosted, ubuntu-20.04, main]
name: beam_PreCommit_Website_Stage_GCS (Run Website_Stage_GCS PreCommit)
steps:
- name: Git checkout
uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Rerun on comment
if: github.event.comment.body == 'Run Website_Stage_GCS PreCommit'
uses: ./.github/actions/rerun-job-action
with:
pull_request_url: ${{ github.event.issue.pull_request.url }}
github_repository: ${{ github.repository }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: ${{ github.job }}
github_current_run_id: ${{ github.run_id }}
- name: Find Current Pull Request
uses: jwalton/gh-find-current-pr@v1
id: findPr
with:
state: open
- name: Echo PR number
run: echo "ghprbPullId=${{ steps.findPr.outputs.pr || github.event.issue.number }}" >> $GITHUB_ENV
- name: Setup self-hosted
uses: ./.github/actions/setup-self-hosted-action
with:
requires-py-39: false
requires-go: false
- name: Setup Gradle
uses: gradle/gradle-build-action@v2
with:
cache-read-only: false
- name: Authenticate on GCP
uses: google-github-actions/setup-gcloud@v0
with:
service_account_email: ${{ secrets.GCP_SA_EMAIL }}
service_account_key: ${{ secrets.GCP_SA_KEY }}
project_id: ${{ secrets.GCP_PROJECT_ID }}
export_default_credentials: true
- name: Run website_stageWebsite script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :website:stageWebsite
arguments: -PwebsiteBucket=$gcsbucket
- name: Add website link to the summary if pr
if: env.ghprbPullId != null
run: echo "Website published to http://$gcsbucket.storage.googleapis.com/$ghprbPullId/index.html" >> $GITHUB_STEP_SUMMARY
- name: Add website link to the summary
if: env.ghprbPullId == null
run: |
USERNAME="$(whoami)"
echo "Website published to http://$gcsbucket.storage.googleapis.com/$USERNAME-${GITHUB_REF##*/}/index.html" >> $GITHUB_STEP_SUMMARY
Original file line number Diff line number Diff line change
Expand Up @@ -658,7 +658,7 @@ class BeamModulePlugin implements Plugin<Project> {
google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20220924-$google_clients_version",
google_api_services_cloudresourcemanager : "com.google.apis:google-api-services-cloudresourcemanager:v1-rev20220828-$google_clients_version",
google_api_services_dataflow : "com.google.apis:google-api-services-dataflow:v1b3-rev20220920-$google_clients_version",
google_api_services_healthcare : "com.google.apis:google-api-services-healthcare:v1-rev20220818-$google_clients_version",
google_api_services_healthcare : "com.google.apis:google-api-services-healthcare:v1-rev20230510-$google_clients_version",
google_api_services_pubsub : "com.google.apis:google-api-services-pubsub:v1-rev20220904-$google_clients_version",
google_api_services_storage : "com.google.apis:google-api-services-storage:v1-rev20230301-$google_clients_version",
google_auth_library_credentials : "com.google.auth:google-auth-library-credentials", // google_cloud_platform_libraries_bom sets version
Expand Down Expand Up @@ -815,9 +815,12 @@ class BeamModulePlugin implements Plugin<Project> {
testcontainers_clickhouse : "org.testcontainers:clickhouse:$testcontainers_version",
testcontainers_elasticsearch : "org.testcontainers:elasticsearch:$testcontainers_version",
testcontainers_gcloud : "org.testcontainers:gcloud:$testcontainers_version",
testcontainers_jdbc : "org.testcontainers:jdbc:$testcontainers_version",
testcontainers_kafka : "org.testcontainers:kafka:$testcontainers_version",
testcontainers_localstack : "org.testcontainers:localstack:$testcontainers_version",
testcontainers_mssqlserver : "org.testcontainers:mssqlserver:$testcontainers_version",
testcontainers_mysql : "org.testcontainers:mysql:$testcontainers_version",
testcontainers_oracle : "org.testcontainers:oracle-xe:$testcontainers_version",
testcontainers_postgresql : "org.testcontainers:postgresql:$testcontainers_version",
testcontainers_rabbitmq : "org.testcontainers:rabbitmq:$testcontainers_version",
truth : "com.google.truth:truth:1.0.1",
Expand Down
40 changes: 40 additions & 0 deletions it/elasticsearch/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* License); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

plugins { id 'org.apache.beam.module' }
applyJavaNature(
automaticModuleName: 'org.apache.beam.it.elasticsearch',
)

description = "Apache Beam :: IT :: Elasticsearch"
ext.summary = "Integration test utilities for Elasticsearch."

def elastic_search_version = "7.9.2"

dependencies {
implementation project(path: ":it:testcontainers", configuration: "shadow")
implementation library.java.testcontainers_elasticsearch
implementation "org.elasticsearch.client:elasticsearch-rest-high-level-client:$elastic_search_version"
implementation "org.elasticsearch.client:elasticsearch-rest-client:$elastic_search_version"
implementation "org.elasticsearch:elasticsearch:$elastic_search_version"

testImplementation library.java.truth
testImplementation library.java.mockito_inline
testImplementation library.java.commons_lang3
testRuntimeOnly library.java.slf4j_simple
}
Loading

0 comments on commit e3d321b

Please sign in to comment.