presto-gateway

A load balancer / proxy / gateway for presto compute engine.

How to setup a dev environment

Step 1: setup mysql. Install docker and run the below command when setting up first time:

docker run -d -p 3306:3306  --name mysqldb -e MYSQL_ROOT_PASSWORD=root123 -e MYSQL_DATABASE=prestogateway -d mysql:5.7

Next time onwards, run the following commands to start mysqldb

docker start mysqldb

Now open mysql console and install the presto-gateway tables:

mysql -uroot -proot123 -h127.0.0.1 -Dprestogateway

Once logged in to mysql console, please run gateway-ha-persistence.sql to populate the tables.

Step 2: Edit the configuration gateway-ha-config.yml

Step 3: Add below program argument to class HaGatewayLauncher and debug in IDE

server /path/to/gateway-ha/src/test/resources/config-template.yml

Build and run

run mvn clean install to build presto-gateway

Edit the config file and update the mysql db information.

cd gateway-ha/target/
java -jar gateway-ha-{{VERSION}}-jar-with-dependencies.jar server ../gateway-ha-config.yml

Now you can access load balanced presto at localhost:8080 port. We will refer to this as prestogateway.lyft.com

Gateway API

Add or update a backend

curl -X POST http://localhost:8080/entity?entityType=GATEWAY_BACKEND \
 -d '{  "name": "presto1", \
        "proxyTo": "http://presto1.lyft.com",\
        "active": true, \
        "routingGroup": "adhoc" \
    }'

curl -X POST http://localhost:8080/entity?entityType=GATEWAY_BACKEND \
 -d '{  "name": "presto2", \
        "proxyTo": "http://presto2.lyft.com",\
        "active": true, \
        "routingGroup": "adhoc" \
    }'

If the backend URL is different from the proxyTo URL (for example if they are internal vs. external hostnames). You can use the optional externalUrl field to override the link in the Active Backends page.

curl -X POST http://localhost:8080/entity?entityType=GATEWAY_BACKEND \
 -d '{  "name": "presto1", \ 
        "proxyTo": "http://presto1.lyft.com",\
        "active": true, \
        "routingGroup": "adhoc" \
        "externalUrl": "http://presto1-external.lyft.com",\
    }'

curl -X POST http://localhost:8080/entity?entityType=GATEWAY_BACKEND \
 -d '{  "name": "presto2", \ 
        "proxyTo": "http://presto2.lyft.com",\
        "active": true, \
        "routingGroup": "adhoc" \
        "externalUrl": "http://presto2-external.lyft.com",\
    }'

Get all backends behind the gateway

curl -X GET http://localhost:8080/entity/GATEWAY_BACKEND
[
    {
        "active": true,
        "name": "presto1",
        "proxyTo": "http://presto1.lyft.com",
        "routingGroup": "adhoc"
    },
    {
        "active": true,
        "name": "presto2",
        "proxyTo": "http://presto2.lyft.com",
        "routingGroup": "adhoc"
    }
]

Delete a backend from the gateway

curl -X POST -d "presto3" http://localhost:8080/gateway/backend/modify/delete

Deactivate a backend

curl -X POST http://localhost:8080/gateway/backend/deactivate/presto2

Get all active backend behind the Gateway

curl -X GET http://localhost:8080/gateway/backend/active | python -m json.tool

    [{
        "active": true,
        "name": "presto1",
        "proxyTo": "http://presto1.lyft.com",
        "routingGroup": "adhoc"
    }]

Activate a backend

curl -X POST http://localhost:8080/gateway/backend/activate/presto2

Query History UI - check query plans etc.

PrestoGateway records history of recent queries and displays links to check query details page in respective presto cluster.

Gateway Admin UI - add and modify backend information

The Gateway admin page is used to configure the gateway to multiple backends. Existing backend information can also be modified using the same.

Resource Groups API

Add a resource group

To add a single resource group, specify all relevant fields in the body.

curl -X POST http://localhost:8080/presto/resourcegroup/create \
 -d '{  "resourceGroupId": 1, \
        "name": "resourcegroup1", \
        "softMemoryLimit": "100%", \
        "maxQueued": 100, \
        "softConcurrencyLimit": 100, \
        "hardConcurrencyLimit": 100, \
        "environment": "test", \
        "schedulingPolicy": null, \
        "schedulingWeight": null, \
        "jmxExport": null, \
        "softCpuLimit": null, \
        "hardCpuLimit": null, \
        "parent": null, \
        "environment": "test" \
    }'

Get existing resource group(s)

If no resourceGroupId (type long) is specified, then all existing resource groups are fetched.

curl -X GET http://localhost:8080/presto/resourcegroup/read/{INSERT_ID_HERE}

Update a resource group

Specify all columns in the body, which will overwrite properties for the resource group with that specific resourceGroupId.

curl -X POST http://localhost:8080/presto/resourcegroup/update \
 -d '{  "resourceGroupId": 1, \
        "name": "resourcegroup_updated", \
        "softMemoryLimit": "80%", \
        "maxQueued": 50, \
        "softConcurrencyLimit": 40, \
        "hardConcurrencyLimit": 60, \
        "environment": "test", \
        "schedulingPolicy": null, \
        "schedulingWeight": null, \
        "jmxExport": null, \
        "softCpuLimit": null, \
        "hardCpuLimit": null, \
        "parent": null, \
        "environment": "test" \
    }'

Delete a resource group

To delete a resource group, specify the corresponding resourceGroupId (type long).

curl -X POST http://localhost:8080/presto/resourcegroup/delete/{INSERT_ID_HERE}

Add a selector

To add a single selector, specify all relevant fields in the body.

curl -X POST http://localhost:8080/presto/selector/create \
 -d '{  "resourceGroupId": 1, \
        "priority": 1, \
        "userRegex": "selector1", \
        "sourceRegex": "resourcegroup1", \
        "queryType": "insert" \
     }'

Get existing selectors(s)

If no resourceGroupId (type long) is specified, then all existing selectors are fetched.

curl -X GET http://localhost:8080/presto/selector/read/{INSERT_ID_HERE}

Update a selector

To update a selector, the existing selector must be specified with all relevant fields under "current". The updated version of that selector is specified under "update", with all relevant fields included. If the selector under "current" does not exist, a new selector will be created with the details under "update". Both "current" and "update" must be included to update a selector.

curl -X POST http://localhost:8080/presto/selector/update \
 -d '{  "current": {
            "resourceGroupId": 1, \
            "priority": 1, \
            "userRegex": "selector1", \
            "sourceRegex": "resourcegroup1", \
            "queryType": "insert" \
        },
        "update":  {
            "resourceGroupId": 1, \
            "priority": 2, \
            "userRegex": "selector1_updated", \
            "sourceRegex": "resourcegroup1", \
            "queryType": null \
        }
}'

Delete a selector

To delete a selector, specify all relevant fields in the body.

curl -X POST http://localhost:8080/presto/selector/delete \
 -d '{  "resourceGroupId": 1, \
        "priority": 2, \
        "userRegex": "selector1_updated", \
        "sourceRegex": "resourcegroup1", \
        "queryType": null \
     }'

Add a global property

To add a single global property, specify all relevant fields in the body.

curl -X POST http://localhost:8080/presto/globalproperty/create \
 -d '{
        "name": "cpu_quota_period", \
        "value": "1h" \
     }'

Get existing global properties

If no name (type String) is specified, then all existing global properties are fetched.

curl -X GET http://localhost:8080/presto/globalproperty/read/{INSERT_NAME_HERE}

Update a global property

Specify all columns in the body, which will overwrite properties for the global property with that specific name.

curl -X POST http://localhost:8080/presto/globalproperty/update \
 -d '{
        "name": "cpu_quota_period", \
        "value": "2h" \
     }'

Delete a global property

To delete a global property, specify the corresponding name (type String).

curl -X POST http://localhost:8080/presto/globalproperty/delete/{INSERT_NAME_HERE}

Contributing

Want to help build Presto Gateway? Check out our contributing documentation

References ✨

Scaling Presto Infra with gateway at Lyft

Presto-gateway at Pinterest