New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Result Set Streaming #24

Closed
jeffm13 opened this Issue Feb 18, 2015 · 3 comments

Comments

Projects
None yet
3 participants
@jeffm13
Copy link

jeffm13 commented Feb 18, 2015

In #6, @cjbj asked for preferences on a result set streaming interface. I thought I'd start a new issue to begin the conversation.

One of my goals is to be able to use node-oracledb to drive ETL-style activities. I used node-oracle in the past to pull about 25 million rows from an Oracle database to an in-memory Redis no-sql database. I expect I'll need to be able to take on that type of activity even more frequently in the future for various reasons. To do it efficiently, streaming is a reasonable pattern.

Node-oracle implemented a reader concept that went in a good direction. But I'd like to see node-oracledb fully implement a Node.js readable stream, similar to how it was implemented with node-mysql or node-pt-query-stream with support for data, end, and error events. There might be a couple types of stream: maybe a 'raw' object stream that would return data in an array format, a 'formatted' object stream that would return a row in object form with column labels. I'd also need access to metadata for the query that originated the stream, to help me transform the results if I need to.

I'd like to be able to pipe a node-oracledb stream through transform streams and into writable streams, so conformance with the Node.js stream interface is important.

I think it might also be a good idea to think about streams for LOBs (maybe similar to how streams work in JDBC), but that's a separate issue.

@llambda

This comment has been minimized.

Copy link

llambda commented Feb 18, 2015

I wrote a lib https://github.com/llambda/oracle-query-stream is this useful?

@cjbj

This comment has been minimized.

Copy link
Member

cjbj commented Feb 18, 2015

@jeff thanks for starting the thread. @llambda thanks for sharing. Let's see what other people think.,

@cjbj

This comment has been minimized.

Copy link
Member

cjbj commented Mar 16, 2015

I'm going to close this and we can continue discussion in #13

@cjbj cjbj closed this Mar 16, 2015

@oracle oracle locked and limited conversation to collaborators Aug 13, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.