Skip to content
Dremio Flight connector. Access Dremio using Arrow flight
Java
Branch: master
Clone or download
dremioJonny and rymurr Correct errors in Python Script in README.md (#1)
* Update README.md

* Update README.md

* Update README.md
Latest commit 7c7f4f4 Aug 15, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
command move to own version Jul 29, 2019
plugin move to own version Jul 29, 2019
.editorconfig moved off dremio parent Jul 4, 2019
.gitignore initial commit Jul 3, 2019
.travis.yml Update .travis.yml Aug 6, 2019
CONTRIBUTING.md initial commit Jul 3, 2019
LICENSE initial commit Jul 3, 2019
NOTICE initial commit Jul 3, 2019
README.md Correct errors in Python Script in README.md (#1) Aug 14, 2019
mvnw initial commit Jul 3, 2019
mvnw.cmd initial commit Jul 3, 2019
pom.xml update to most recent oss version of Dremio Aug 2, 2019

README.md

Dremio Flight Connector

Build Status

Building and Installation

  1. In root directory with the pom.xml file run mvn clean install
  2. Take the resulting .jar file in the target folder and put it in the \dremio\jars folder in Dremio
  3. Restart Dremio

Accessing Dremio via flight in python

The Flight endpoint is exposed on port 47470. The most recent release of pyarrow (0.14.1) has the flight client built in. To access Dremio via Flight first install pyarrow (conda install pyarrow -c conda-forge or pip install pyarrow). You must also generate the python class for the Command proto protoc -I=dremio-flight-connector/command/src/main/protobuf --python_out=. dremio-flight-connector/command/src/main/protobuf/command.proto. Then:

from pyarrow import flight
import pyarrow as pa
from .command_pb2 import Command

class HttpDremioClientAuthHandler(flight.ClientAuthHandler): 
  
  def __init__(self, username, password): 
    super().__init__() 
    self.username = username
    self.password = password
    self.token = None

  def authenticate(self, outgoing, incoming): 
    outgoing.write(base64.b64encode(self.username + b':' + self.password))
    self.token = incoming.read()

  def get_token(self): 
    return self.token

username = b'<USERNAME>'
password = b'<PASSWORD>'
sql = '''<SQL_QUERY>'''

client = flight.FlightClient.connect('grpc+tcp://<DREMIO_COORDINATOR_HOST>:47470')
client.authenticate(HttpDremioClientAuthHandler(username, password)) 
cmd = Command(query=sql, parallel=False, coalesce=False, ticket=b'')
info = client.get_flight_info(flight.FlightDescriptor.for_command(cmd.SerializeToString()))
reader = client.do_get(info.endpoints[0].ticket)
batches = []
while True:
    try:
        batch, metadata = reader.read_chunk()
        batches.append(batch)
    except StopIteration:
        break
data = pa.Table.from_batches(batches)
df = data.to_pandas()
You can’t perform that action at this time.