Redis Queue) for asynchronous polling of a Cloud pull-subscriptionFlask server/client application with RQ (
This repository contains source code for a Flask application that may be used to poll and consume messages, published with Google's Cloud Pub/Sub. To create a working example messages published from Cloud Build service have been used, but this is only for the proof-of-concept, as similar procedure could be used for other Pub/Sub topics and subscriptions. Heroku dynos have been used as a remote-staging environment, utilizing the free-tier, with one web dyno and one worker dyno, allowing hosting of the application together with a single
psq, which is a Cloud Pub/Sub Task Queue for Python, developed by Google for this purpose. This is not an official product and is still in development, so RQ was chosen instead of
The project has been developed in a Python
virtualenv created and managed using
pipenv. It also adds/removes packages from your
Pipfile as you install/uninstall packages and generates the Pipfile.lock, which is used to produce deterministic builds. You will need
pipenv to be able to quickly and reliably reproduce a working environment to run the app locally. If you are unfamiliar with
pipenv have a look at a few quick tips to get you started.
Clone the repo and then
cd in the project's root directory (e.g.
gcpubsub) then create
virtualenv and install dependencies:
git clone https://github.com/chrisliatas/pubsub-monitor.git gcpubsub cd gcpubsub pipenv install
This will install all project dependencies. Please note that the project is built using Python 3.7. When installing of all requirements is finished you will have a new
virtualenv for the application that you can work-on by running:
You will need to have your new
virtualenv activated. You then have to:
export FLASK_APP=gcpubsub.py export GOOGLE_APPLICATION_CREDENTIALS=path_to_key_file.json export GCP_PROJECT_ID=your_GCP_project_name export PUBSUB_TOPIC=your_pubsub_topic_name export SUBSCRIPTION_NAME=your_subscription_name
Start a separate terminal in the same directory, activate your
pipenv shell) and run:
rq worker gcpubsub-tasks # You will get an output similar to: 09:16:13 RQ worker 'rq:worker:host-01.30454' started, version 0.12.0 09:16:13 *** Listening on gcpubsub-tasks... 09:16:13 Cleaning registries for queue: gcpubsub-tasks
On your first terminal run:
flask deploy flask run
You should be able now, to view the application at
http://localhost:5000. The application has a default user
admin with password
cannotguess, created by
flask deploy command, that you can use for your initial login. You only need to run
flask deploy for your first run. The default password can be changed after you login, from within the web interface.
Start polling will initiate the background task that listens for messages in your subscription. If you have any messages, these will be loaded and displayed as soon as they are received. You can use the
publish.py under the directory
helpers if you need to create and publish sample messages to test the app.
Would you like to learn more?
If you would like to know more about the app, usage considerations, Heroku deployment and alternatives, have a look at the post:
This app has been developed utilizing the following open source projects and libraries:
|github.com/metafizzy/infinite-scroll||GNU GPL license v3|
Using of the "Infinite Scroll" plugin ( released under GNU GPL license v3) requires this project to be released under the same license, as according to GPLv3 software, merely linking to the software it is considered by the GPLv3 authors to create a derivative work.
Even though my intention was to release it under the Apache License, Version 2.0, I have to respect the GPLv3, so this work is released under GNU GPL license v3.