Scripts to pull data from various sources for SUMO Dashboards and upload it to GCP.
Uses Google Analytics Reporting API v4 to pull dimensions and metrics for the Google Analytics SUMO report.
https://developers.google.com/analytics/devguides/reporting/core/v4/rest/
Make sure Analytics Reporting API is enabled in the GCP running the code. A valid service account should be permissioned to pull data from the SUMO report from the Google Analytics side. GoogleAnalytics/create_ga_tables.py creates Google Analytics BiqQuery tables with schema definition. GoogleAnalytics/get_ga_data.py pulls data for a given range. The data is written to local csv files in /tmp folder, and pushed to google storage gs:///googleanalytics/. The google storage files are uploaded to BigQuery dataset sumo table ga_*. After upload, the files are moved to the /processed subfolder. Some of the data pulls hit daily data limits so it is recommend to run data pulls in one month chunks. The Google Analytics API has a processing latency of 24-48 hours, https://support.google.com/analytics/answer/1070983?hl=en To prevent volatile numbers in the last 48 hours, the daily Google Analytics job retrieves data with a 48 hour lag.
Sentiment analysis on Twitter tweets and Kitsune questions. Pull data from GTrends for SUMO. The package uses an unofficial library, PyTrends, to pull data from GTrends.
The scripts are intended to be run on a Google Cloud Project with necessary account permissions.
Assumes Google storage folder structure:
gs:// <sumo-bucket>
/ googleanalytics => where google analytics data files are initially placed
/ googleanalytics / processed => where processed google analytics data files are placed after being uploaded to BigQuery
/ googleplaystore => where google data files are initially placed [deprecated]
/ googleplaystore / processed => where processed google analytics data files are placed after being uploaded to BigQuery [deprecated]
/ tmp => model param files, aggregation files in subfolder by model pararm
gs:// <data-bucket> => location of parquet input data files
packagemanager install awesome-project
awesome-project start
awesome-project "Do something!" # prints "Nah."
Here you should say what actually happens when you execute the code above.
Some projects require initial configuration (e.g. access tokens or keys, npm i
).
This is the section where you would document those requirements.
Here's a brief intro about what a developer must do in order to start developing the project further:
git clone https://github.com/your/awesome-project.git
cd awesome-project/
packagemanager install
And state what happens step-by-step.
If your project needs some additional steps for the developer to build the project after some code changes, state them here:
./configure
make
make install
python setup.py test
Sigh, maybe someday.
Define GCP storage bucket where files should go.
packagemanager deploy awesome-project -s server.com -u username -p password
And again you'd need to tell what the previous code actually does.
What's all the bells and whistles this project can perform?
- What's the main functionality
- You can also do another thing
- If you get really randy, you can even do this
Here you should write what are all of the configurations a user can enter when using the project.
Type: String
Default: 'default value'
State what an argument does and how you can use it. If needed, you can provide an example below.
Example:
awesome-project "Some other value" # Prints "You're nailing this readme!"
Type: Number|Boolean
Default: 100
Copy-paste as many of these as you need.
When you publish something open source, one of the greatest motivations is that anyone can just jump in and start contributing to your project.
These paragraphs are meant to welcome those kind souls to feel that they are needed. You should state something like:
"If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome."
If there's anything else the developer needs to know (e.g. the code style
guide), you should link it here. If there's a lot of things to take into
consideration, it is common to separate this section to its own file called
CONTRIBUTING.md
(or similar). If so, you should say that it exists here.
Even though this information can be found inside the project on machine-readable format like in a .json file, it's good to include a summary of most useful links to humans using your project. You can include links like:
- Project homepage: https://your.github.com/awesome-project/
- Repository: https://github.com/your/awesome-project/
- Issue tracker: https://github.com/your/awesome-project/issues
- In case of sensitive bugs like security vulnerabilities, please contact my@email.com directly instead of using issue tracker. We value your effort to improve the security and privacy of this project!
- Related projects:
- Your other project: https://github.com/your/other-project/
- Someone else's project: https://github.com/someones/awesome-project/
Licensed under ... For details, see the LICENSE file.