Skip to content
Apache Beam example
Python Makefile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
beam First commit. Jan 30, 2018
config First commit. Jan 30, 2018
utils First commit. Jan 30, 2018
.gitignore First commit. Jan 30, 2018
Makefile First commit. Jan 30, 2018
README.md First commit. Jan 30, 2018
__init__p.y First commit. Jan 30, 2018
app.yaml First commit. Jan 30, 2018
countries.json First commit. Jan 30, 2018
dev_requirements.txt First commit. Jan 30, 2018
factory.py First commit. Jan 30, 2018
generate.py First commit. Jan 30, 2018
input.txt First commit. Jan 30, 2018
main.py First commit. Jan 30, 2018
names.txt First commit. Jan 30, 2018
pylintrc First commit. Jan 30, 2018
requirements.txt First commit. Jan 30, 2018
setup.py First commit. Jan 30, 2018
web.py First commit. Jan 30, 2018

README.md

Beam example

This article will show a basic usage of Apache Beam library.

Introduction

The application is made for Google Cloud Platform, specifically Flexible Environment. Please read the google docs here () to get more information about that

Installation

First, create a virtualenv

virtualenv --no-site-packages beam
source beam/binactivate

After this, install all the dependencies:

pip install -r requirements.txt

Input files generation

In order to create a valid input file, use the generate.py script:

python generate.py

Execution

last, but not least, start the instance with:

make run

The endpoint to hit is http://localhost:8090/dataflow/from_text

Once you see the page showing ok, you can go on the filesystem and look at the output.txt-00000-of-00001 file.

Note

Pay attention that some python version might throw you an error:

TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
make: *** [run] Error 1

This happened to me when i tried to use a different Python version (still 2, but a different build). Sadly you have to address any of these errors contenxtually.

You can’t perform that action at this time.