Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a remote/server/cluster feature #48

Closed
neilotoole opened this issue May 11, 2017 · 1 comment
Closed

Add a remote/server/cluster feature #48

neilotoole opened this issue May 11, 2017 · 1 comment

Comments

@neilotoole
Copy link
Owner

You should be able to have the sq query execute remotely on a server/cluster. For example:

> sq remote create aws myaws1 ...
Created cluster and added remote "myaws1"
> sq --remote=myaws1 '@mysql1.tbluser, @pg1.tbladdress | join(.uid)'
# Add a pre-existing cluster, e.g. one set up by an admin
> sq remote add myazure1 http://my_sq_lb.azure.com --username=ETC
> sq --remote=myazure1 '@mysql1.tbluser, @pg1.tbladdress | join(.uid)'

There's two pieces here: the first is creating (or adding) the remote cluster, the second is executing the query remotely.

The service consists of a load balancer that directs requests to a bunch of sq server instances (just web servers wrapping libsq deployed in Docker containers). The sq --remote=myaws1 QUERY command posts the query and the local config to the server. If the query refers to a local data source, the server will send a response with upload URLs for the local files (e.g. http://myaws1.aws.amazon.com/upload/GUID). sq will upload those files, and the server will run the query, occasionally sending status updates to the client, while the client waits for the job to complete.

@neilotoole
Copy link
Owner Author

Out of scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant