Squall Cluster Configs

We will explain the content of a config file on INSTALL_DIR/dip/SQLtoQueryPlanPlugin/confs/1G_hyracks_parallel:

DIP_DISTRIBUTED true
DIP_QUERY_NAME hyracks

DIP_TOPOLOGY_NAME_PREFIX teamX
DIP_DATA_ROOT /export/home/avitorovic/queries/tpch/
DIP_SQL_ROOT ../dip/SQLtoQueryPlanPlugin/SQLqueries/

# DIP_DB_SIZE is in GBs
DIP_DB_SIZE 1
DIP_MAX_SRC_PAR 4

#below are unlikely to change
DIP_EXTENSION .tbl
DIP_READ_SPLIT_DELIMITER \|
DIP_GLOBAL_ADD_DELIMITER |
DIP_GLOBAL_SPLIT_DELIMITER \|

DIP_ACK_EVERY_TUPLE true
DIP_KILL_AT_THE_END true
#used only in distributed mode
DIP_NIMBUS_HOST icdatasrv2
DIP_STORM_ZOOKEEPER_SERVERS icdatasrv2

Config file 1G_hyracks_parallel is the same as in 0.1G_hyracks_serial, except:

1. DIP_DISTRIBUTED is set to true.

2. DIP_TOPOLOGY_NAME_PREFIX is an optional parameter. It is used for distinguishing different users possibly running the same query at the same time on the cluster.

3. DIP_DATA_ROOT refers to a location on the cluster.

4. DIP_MAX_SRC_PAR is set to 4 to exploit parallelism in the cluster.

Thus, in order to change database size, only the DIP_DB_SIZE has to be changed, and for changing the query, we have to modify DIP_QUERY_NAME. You can find more examples of config files in INSTALL_DIR/dip/SQLtoQueryPlanPlugin/confs, or you can write new ones from scratch.

Keep in mind that for in each config file you need to set DIP_DATA_ROOT. In addition, DIP_QUERY_NAME must correspond to a query from INSTALL_DIR/dip/SQLtoQueryPlanPlugin/SQLqueries/.

You can run Squall with a desired config file as follows:

cd $INSTALL_DIR/bin
./pluginSQLClusterRun.sh $CONFIG_FILE

where CONFIG_FILE is a file from INSTALL_DIR/dip/SQLtoQueryPlanPlugin/confs which ends up with _parallel.

Due to the constrained main memory, you cannot run arbitrary large database with small component parallelism. For information on detecting this behavior, please consult Squall query plans vs Storm topologies, section How to know we run out of memory?. The way you control it is through MAX_SRC_PAR parameter - the larger the parameter is, bigger database can be processed.