A complete Python application to automatize the SQL Mass Deployment process making it easy and simple
Meteor uses cutting-edge technologies like Parallel Executions which notably reduces the deployment duration in a logarithmic scale. Also allows to perform Cross-Region Executions avoiding transport delay.
Meteor is built leading to an end-to-end validation process to guarantee that no queries will harm any environment.
Tested in a real Production Environment using:
- MySQL (5.6 / 5.7)
- Amazon RDS for MySQL (5.6 / 5.7)
- Amazon Aurora MySQL (5.6 / 5.7)
- Amazon Aurora Serverless
1.1. Clone the Meteor Repository
$ git clone https://github.com/polius/Meteor.git
1.2. Install the Requirements
$ cd Meteor/app/ $ pip install -r requirements.txt --user
Before executing the Meteor, there are two files that have to be filled:
Meteor has built-in three different modes:
Starts the Validation Process (Credentials, Queries, Regions).
$ python meteor.py --environment "environment_name" --validate [ credentials | queries | regions | all ]
3.2. Test Execution
Performs the Test Execution executing only SELECT queries.
$ python meteor.py --environment "environment_name" --test
3.3. Deployment Execution
Performs the Deployment executing ALL queries.
$ python meteor.py --environment "environment_name" --deploy
Emergency Stop Button
To stop the Execution while is in process, use: Ctrl+C
Open the meteor/web/index.html with your web browser and import the generated execution file named meteor.js to see the execution results.
Here is a brief explanation about the tasks that Meteor performs during a deployment execution "--deploy".
Before performing any action, a validation process is initiated reviewing:
- Credentials. Validates the "credentials.json" file.
- Queries. Validates the user-defined queries located in "query_execution.py".
- Regions. Validates all the SSH/SQL Connections.
If all of the three steps are perfectly validated, Meteor continues its execution to the next phase.
In this phase the deployment process is started executing all queries.
When the Deployment Execution is finished, Meteor retrieves all the logs created in all servers and starts a compilation process generating one single file named meteor.js which contains the result of all executions.
Meteor analyzes all the execution data and builds a Summary that is shown in the terminal.
5) Post Execution Tasks
To finish the process, 5 remaining tasks are executed:
- AWS S3 Upload (Optional). Uploads the execution logs to the AWS S3.
- Clean Up. Cleans all the temporary files generated during the Execution both Remote and Local Environments.
- Slack (Optional). Sends the Summary to an Slack Channel.
- Execution Time. Shows the Execution Time Summary.
- Output. Shows the generated Logs Path and the Logs Url.
Under the Hood
The following graph shows the Meteor behaviour during a Deployment Process using the 3-Level Parallel Execution.
This project is licensed under the MIT license. See the LICENSE file for more info.