Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

database/sql: add support for PutData and GetData #30561

Open
akhilravuri1 opened this issue Mar 4, 2019 · 13 comments
Open

database/sql: add support for PutData and GetData #30561

akhilravuri1 opened this issue Mar 4, 2019 · 13 comments

Comments

@akhilravuri1
Copy link

@akhilravuri1 akhilravuri1 commented Mar 4, 2019

Is there any idea to add support for Putdata() and Getdata() in the SQL package. Users store large data in the database and they want to send them to the database by chunks, where Putdata() is used. In the same way Getdata() is used to get chunks of data from the database.

Thanks.

@kevinburke

This comment has been minimized.

Copy link
Contributor

@kevinburke kevinburke commented Mar 4, 2019

Which database or ORM framework uses these function calls?

You can probably do what you are hoping to with the existing framework - no reason Query or QueryRow can't return you tens or hundreds of megabytes of data.

@akhilravuri1

This comment has been minimized.

Copy link
Author

@akhilravuri1 akhilravuri1 commented Mar 5, 2019

I think every Database has this kind of function calls. As I work on Db2(IBM database), it has SQLPutData() and SQLGetData().

The above API's take Statement handle as one of the parameters and variables in statement object (st,err := db.Prepare(query) ) are unexported and I can't use it even if I extend the sql.Stmt.

@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Mar 5, 2019

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Mar 5, 2019

Can you point me to the docs on what you are trying to do? Are you trying to upload large amounts or rows like a bulk row upload? Maybe an example that demonstrates how you would use this?

@akhilravuri1

This comment has been minimized.

Copy link
Author

@akhilravuri1 akhilravuri1 commented Mar 5, 2019

package main

import (
    "database/sql"
    "fmt"
    _"github.com/ibmdb/go_ibm_db"
)

func main() {
    con := "HOSTNAME=localhost;PORT=50000;DATABASE=dbname;UID=uname;PWD=password;"
    db, err := sql.Open("go_ibm_db", con)
    if err != nil {
        fmt.Println(err)
    }
	defer db.Close()
    st,err:=db.Prepare("Insert into putdata(a,b) values(?,?)")
	if err != nil{
	fmt.Println(err)
	}
	defer st.Close()
	
	// If user wants to send the data at execute time.
	// He will send the some empty string or empty byte array or something to notify the execute api
	// that data will be send later.
	st.Query("","")
	// From Here Its My vision of functions how to send the data.
	st.ParamData()
	//Here I am reading the normal data. He may read chunks of Data from file 
	st.PutData("Akhil")
	st.PutData(" Ravuri")
	//This paramData function will set the next parameter to be inserted.
	//If there are no Parameter's the it will execute the query.
	st.ParamData()
	st.PutData("Sai")
	st.PutData(" Ram")
	st.ParamData()
}

In the example I have taken the string, suppose if the user wants to store lob data in the database which is of 2GB (As of I know a buffer can't handle 2GB of Data at once) , then he wants to send them in chunks of data then he will call multiple Putdata's and then store the data.
In the same way, while getting the Data from the Database GetData is used.

When Binding the columns if we use this SQL_DATA_AT_EXEC (In Db2) in column value , then execute api will be waiting for the data and and when we send the data to the column, then it will execute the query.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Mar 5, 2019

Ah, yes, that makes much more sense. Thanks for the context. This is a usecase I'd like to enable with database/sql/v2, but the current API doesn't really allow for this at the moment. You'll need to use the direct DB2 API I think. I'll look into that driver.

@akhilravuri1

This comment has been minimized.

Copy link
Author

@akhilravuri1 akhilravuri1 commented Mar 5, 2019

Thanks for the quick turn around, As many users are store lob data in the database it will be useful and when Go is planning to release database/sql/v2.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Mar 5, 2019

If I understand your usecase correctly, after thinking about it for awhile, you can probably implement this in your driver with the current interface.

The key is to implement the https://godoc.org/database/sql/driver#NamedValueChecker Named Value Checker, then allow the user to pass in some type of reader, possibly an io.Reader or other chunk based reader. Then the driver can upload parameters as it reads it off in chunks.

@akhilravuri1

This comment has been minimized.

Copy link
Author

@akhilravuri1 akhilravuri1 commented Mar 5, 2019

NamedValueChecked is just declared in the driver file and no sign of it in the sql file. How to call that function. Is there any example on how to use it.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Mar 6, 2019

Fair point. I don't remember if we have examples for that driver member. Here is an example for a SQL Server driver, which supports a number of non-stdlib types:

https://github.com/denisenkom/go-mssqldb/blob/master/mssql_go19.go#L66

@akhilravuri1

This comment has been minimized.

Copy link
Author

@akhilravuri1 akhilravuri1 commented Mar 6, 2019

Thanks for the information. But that did not help me. Correct me If I am wrong, As of what I know If a user writes an application using the driver the flow is

Application(Database program) --->database/sql/sql.go -----> database/sql/driver/driver.go(where the function are declared) -----> imported Driver.

So if the user wants to use it(NamedValueChecked) at the application side it should be present in the database/sql/sql.go or else it will produce an error that function is not defined in the sql.go

So, what is the equivalent function in the in sql.go for NamedValueChecked?
I will go through the docs to check whether it is helpful for my case or not.

@akhilravuri1

This comment has been minimized.

Copy link
Author

@akhilravuri1 akhilravuri1 commented Mar 6, 2019

If I extend the interface and add a function to the Stmt handle it will not work because the PutData and get Data functions Require SQLHSTMT as one of the parameters and which is stored in the driver.Stmt. Since driver.Stmt is not exported in the Stmt I cannot access SQLHSTMT.

If there are Read, write specifiers like export and unexport then it will be too easy.

@andybons andybons changed the title Adding support for PutData and GetData in database/sql database/sql: add support for PutData and GetData Mar 9, 2019
@andybons andybons added this to the Unplanned milestone Mar 9, 2019
@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Jan 24, 2020

@akhilravuri1 The Db.Conn.Raw method may help you interact with a driver's native capability. However, I do not expect to add this type of functionality into the current database/sql interface.

I suggest we decline/close this request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.