Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk Insert get errors in inserting data into HANA #56

Closed
ghost opened this issue Sep 3, 2020 · 3 comments
Closed

Bulk Insert get errors in inserting data into HANA #56

ghost opened this issue Sep 3, 2020 · 3 comments
Labels

Comments

@ghost
Copy link

ghost commented Sep 3, 2020

{"_func":"HanaChaos/Executor.runTest","_level":"info","_loc":"sqljob.go:58","_time":"2020-09-01T03:21:05.517832857-07:00","msg":"Run Test Case:INSERT INTO DDL5 VALUES($2) "}
{"_func":"HanaChaos/Executor.runTest","_level":"info","_loc":"sqljob.go:60","_time":"2020-09-01T03:21:05.518816453-07:00","msg":"Run Test Case:INSERT INTO DDL5 VALUES($2) "}
{"_func":"HanaChaos/Executor.bulkInsert","_level":"info","_loc":"sqljob.go:96","_time":"2020-09-01T03:21:05.519362595-07:00","msg":"BULK INSERT INTO DDL5 VALUES(?, ?)"}
{"_func":"HanaChaos/Executor.bulkInsert","_level":"fatal","_loc":"sqljob.go:106","_time":"2020-09-01T03:21:05.679429537-07:00","_trace":"goroutine 15 [running]:\nruntime/debug.Stack(0x7b4789, 0x5, 0xc000759a50)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\nHanaChaos/Logging.preLogFatal(0xc000f345b0)\n\t/home/lzhang/go/src/HanaChaos/Logging/logger.go:66 +0x17a\nHanaChaos/Logging.Fatal(0xc000759c98, 0x1, 0x1)\n\t/home/lzhang/go/src/HanaChaos/Logging/logger.go:126 +0x26\nHanaChaos/Executor.bulkInsert(0xc0002e8300, 0xc000246120, 0x1b)\n\t/home/lzhang/go/src/HanaChaos/Executor/sqljob.go:106 +0x35b\nHanaChaos/Executor.runTest(0xc0002222a0, 0xc000222270, 0x0, 0x0)\n\t/home/lzhang/go/src/HanaChaos/Executor/sqljob.go:65 +0x1ce\nHanaChaos/Executor.(*SqlJob).Run(0xc000228080, 0xc00022e080, 0x2, 0x0)\n\t/home/lzhang/go/src/HanaChaos/Executor/sqljob.go:21 +0x44\nHanaChaos/Worker.Worker.Start.func1(0xc00007a420, 0xc000092900, 0xc000092960)\n\t/home/lzhang/go/src/HanaChaos/Worker/job.go:26 +0x108\ncreated by HanaChaos/Worker.Worker.Start\n\t/home/lzhang/go/src/HanaChaos/Worker/job.go:18 +0x53\n","msg":"sql: converting argument $2 type: unsupported lobCESU8Type conversion: string 0u5jwx7z9z"}
{"_func":"HanaChaos/Executor.bulkInsert","_level":"fatal","_loc":"sqljob.go:106","_time":"2020-09-01T03:21:05.680060927-07:00","_trace":"goroutine 15 [running]:\nruntime/debug.Stack(0x7b4789, 0x5, 0xc000759a50)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\nHanaChaos/Logging.preLogFatal(0xc000f348e0)\n\t/home/lzhang/go/src/HanaChaos/Logging/logger.go:66 +0x17a\nHanaChaos/Logging.Fatal(0xc000759c98, 0x1, 0x1)\n\t/home/lzhang/go/src/HanaChaos/Logging/logger.go:126 +0x26\nHanaChaos/Executor.bulkInsert(0xc0002e8300, 0xc000246120, 0x1b)\n\t/home/lzhang/go/src/HanaChaos/Executor/sqljob.go:106 +0x35b\nHanaChaos/Executor.runTest(0xc0002222a0, 0xc000222270, 0x0, 0x0)\n\t/home/lzhang/go/src/HanaChaos/Executor/sqljob.go:65 +0x1ce\nHanaChaos/Executor.(*SqlJob).Run(0xc000228080, 0xc00022e080, 0x2, 0x0)\n\t/home/lzhang/go/src/HanaChaos/Executor/sqljob.go:21 +0x44\nHanaChaos/Worker.Worker.Start.func1(0xc00007a420, 0xc000092900, 0xc000092960)\n\t/home/lzhang/go/src/HanaChaos/Worker/job.go:26 +0x108\ncreated by HanaChaos/Worker.Worker.Start\n\t/home/lzhang/go/src/HanaChaos/Worker/job.go:18 +0x53\n","msg":"sql: converting argument $2 type: unsupported lobCESU8Type conversion: string e1itpn70fg"}

@ghost ghost changed the title Bulk Insert get errors in to insert data into HANA Bulk Insert get errors in inserting data into HANA Sep 3, 2020
@stfnmllr
Copy link
Contributor

stfnmllr commented Sep 3, 2020

Please provide a proper issue description with an code example to reproduce the error.
Anyway, referring to the issue headline and if the log output is correct
>>> unsupported lobCESU8Type conversion: string e1itpn70fg
it looks like an attempt to insert a DB LOB type via a Go string parameter which is not supported. Please refer to the documentation and the LOB examples provided in go-hdb documentation.

@ghost
Copy link
Author

ghost commented Sep 4, 2020

We create one table and convert the type to lob as below. and insert the data as the bulk insert.
create column table T2 (a int, b varchar(10);
alter table T2 alter (B clob page loadable) online;
testSql := "InsertData: INSERT INTO T2 VALUES($2)"
func bulkInsert(db *sql.DB, testSql string) {
bulkInsertString := testSql
bulkInsertString = strings.Replace(bulkInsertString, "$2", "?, ?", -1)
bulkInsertString = strings.Replace(bulkInsertString, "INSERT", "BULK INSERT", -1)

log.Info(bulkInsertString)
stmt, err := db.Prepare(bulkInsertString) // Prepare bulk query.
if err != nil {
	log.Fatal(err)
}
defer stmt.Close()

for i := 0; i < 10000; i++ {
	randValue := CreateRandomString(10)
	if _, err := stmt.Exec(i, randValue); err != nil {
		log.Fatal(err)
	}
}
// Call final stmt.Exec().
if _, err := stmt.Exec(); err != nil {
	log.Fatal(err)
}

}

func randomResult(typeList []string) string {
colTypeIndex := rand.Intn(len(typeList))
return typeList[colTypeIndex]
}

@stfnmllr
Copy link
Contributor

stfnmllr commented Sep 4, 2020

Thanks for the explanation! Like presumed you are using a string variable (randValue) as argument for a CLOB field but you need to use a driver.Lob field instead (please see https://godoc.org/github.com/SAP/go-hdb/driver#Lob and the examples for reference).
To use 'bulk inserts' with Lob arguments please take care to use the latest go-hdb version (>= 0.100.13).
Tip: do not use LOB fields if not needed as you definitely get an negative impact on performance (at least the double amount of client - server roundtrips) compared to standard fields like a varchar.

@stfnmllr stfnmllr closed this as completed Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant