Skip to content
Permalink
Browse files
Initial commit (#1)
  • Loading branch information
davidcavazos committed Jun 15, 2022
1 parent ef3c703 commit 9e9a97d573378b0a36a2545e8def087392bee907
Show file tree
Hide file tree
Showing 15 changed files with 403 additions and 1 deletion.
@@ -0,0 +1,12 @@
**Please** add a meaningful description for your change here

------------------------

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

- [ ] I agree that my contributions are licensed with both [Apache ASL2](../../LICENSE-APACHE) and [MIT](../../LICENSE-MIT).
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

See the [`CONTRIBUTING.md`](../../CONTRIBUTING.md) guide for more information.
@@ -0,0 +1,30 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

name: Test

on: [push]

jobs:
tests:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.7", "3.8", "3.9"]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install local package
run: |
pip install -U pip
pip install -e .
- name: Run tests
run: python -m unittest -v
@@ -0,0 +1,40 @@
# IDEs
.vscode/
.idea/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# Virtual environments.
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
@@ -0,0 +1,69 @@
# Contributing

🎉🎊 Thanks for taking the time to contribute! 🎉🎊

There are many ways to contribute, here are some.

## Filing an issue

If there's any issue you encounter or anything that needs to be fixed, feel free to [create a GitHub issue](https://github.com/apache/beam-starter-python/issues).

## Contributing to this starter project

If this is your first time contributing to a GitHub repo,
we recommmend going through the
[GitHub quickstart](https://docs.github.com/en/get-started/quickstart/hello-world).

It's a good idea to discuss your plans with the Beam community through the dev@beam.apache.org mailing list before doing any changes.

Here's a small overview of the process.

1. [Fork the repo](https://docs.github.com/en/get-started/quickstart/fork-a-repo).

1. Clone the repo.

```sh
export GITHUB_USERNAME="my-github-username"
git clone git@github.com:$GITHUB_USERNAME/beam-starter-python.git
```

1. Set the [upstream remote branch](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/configuring-a-remote-for-a-fork).

```sh
cd beam-starter-python
git remote add upstream git@github.com:apache/beam-starter-python.git
```

1. Create and change to a new branch.

```sh
git checkout -B my-branch-name
```

1. Modify the code! 😱

1. Run the tests. For steps on how to run them see the [`README.md`](README.md).

1. Commit and push your changes to your branch in `origin`.

```sh
git commit -m "one line description of your changes"
git push
```

1. [Create a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).

1. Add reviewers, and [address review comments](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests).

1. Once it's approved, we can merge the Pull Request.

For more information about proposing changes to a GitHub repository, see the
[Propose changes](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-branches)
page in the GitHub docs.

## Contributing to Apache Beam

For information on how to contribute to
[Apache Beam](https://github.com/apache/beam), see the
[Contribution guide](https://beam.apache.org/contribute/).
37 LICENSE
@@ -1,3 +1,38 @@
This software is distributed under the terms of both the MIT license and the
Apache License (Version 2.0).


MIT license

Copyright 2022 Google LLC

Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.


Apache 2 license

Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
@@ -198,4 +233,4 @@
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
limitations under the License.
@@ -0,0 +1 @@
include requirements.txt
5 NOTICE
@@ -0,0 +1,5 @@
Apache Beam
Copyright [2022-] The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
@@ -0,0 +1,78 @@
# Apache Beam starter for Python

If you want to clone this repository to start your own project,
you can choose the license you prefer and feel free to delete anything related to the license you are dropping.

## Before you begin

Make sure you have a [Python 3](https://www.python.org/) development environment ready.
If you don't, you can download and install it from the
[Python downloads page](https://www.python.org/downloads/).

We recommend using a virtual environment to isolate your project's dependencies.

```sh
# Create a new Python virtual environment.
python -m venv env
# Activate the virtual environment.
source env/bin/activate
```

While activated, your `python` and `pip` commands will point to the virtual environment,
so any changes or install dependencies are self-contained.

As a one time setup, let's install the project's dependencies from the [`requirements.txt`](requirements.txt) file.

```py
# It's always a good idea to update pip before installing dependencies.
pip install -U pip
# Install the project as a local package, this installs all the dependencies as well.
pip install -e .
```

> ℹ️ Once you are done, you can run the `deactivate` command to go back to your global Python installation.
### Running the pipeline

Running your pipeline in Python is as easy as running the script file directly.

```sh
# You can run the script file directly.
python main.py
# To run passing command line arguments.
python main.py --input-text="🎉"
# To run the tests.
python -m unittest -v
```

## GitHub Actions automated testing

This project already comes with automated testing via [GitHub Actions](https://github.com/features/actions).

To configure it, look at the [`.github/workflows/test.yaml`](.github/workflows/test.yaml) file.

## Using other runners

To keep this template small, it only includes the [Direct Runner](https://beam.apache.org/documentation/runners/direct/).

For a comparison of what each runner currently supports, look at the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).

To add a new runner, visit the runner's page for instructions on how to include it.

## Contributing

Thank you for your interest in contributing!
All contributions are welcome! 🎉🎊

Please refer to the [`CONTRIBUTING.md`](CONTRIBUTING.md) file for more information.

# License

This software is distributed under the terms of both the MIT license and the
Apache License (Version 2.0).

See [LICENSE](LICENSE) for details.
32 main.py
@@ -0,0 +1,32 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

from apache_beam.options.pipeline_options import PipelineOptions

from my_app import app


if __name__ == "__main__":
import argparse
import logging

logging.getLogger().setLevel(logging.INFO)

parser = argparse.ArgumentParser()
parser.add_argument(
"--input-text",
default="Default input text",
help="Input text to print.",
)
args, beam_args = parser.parse_known_args()

beam_options = PipelineOptions(save_main_session=True, setup_file="./setup.py")
app.run(
input_text=args.input_text,
beam_options=beam_options,
)
@@ -0,0 +1,9 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

from .app import *
@@ -0,0 +1,27 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

from typing import Callable, Optional
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions


def run(
input_text: str,
beam_options: Optional[PipelineOptions] = None,
test: Callable[[beam.PCollection], None] = lambda _: None,
) -> None:
with beam.Pipeline(options=beam_options) as pipeline:
elements = (
pipeline
| "Create elements" >> beam.Create(["Hello", "World!", input_text])
| "Print elements" >> beam.Map(print)
)

# Used for testing only.
test(elements)
@@ -0,0 +1 @@
apache-beam==2.39.0
@@ -0,0 +1,22 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

from setuptools import setup, find_packages

with open("requirements.txt") as f:
requirements = f.readlines()

setup(
name="My app",
version="1.0",
description="Python Apache Beam pipeline.",
author="My name",
author_email="my@email.com",
packages=find_packages(),
install_requires=requirements,
)
@@ -0,0 +1,7 @@
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.

0 comments on commit 9e9a97d

Please sign in to comment.