Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDBC support for Object array #5620

Closed
ramach opened this issue May 24, 2017 · 22 comments
Closed

JDBC support for Object array #5620

ramach opened this issue May 24, 2017 · 22 comments

Comments

@ramach
Copy link

ramach commented May 24, 2017

CrateDB version:
0.57.8

JVM version:
1.8.0_74
OS version / environment description:
MAC-OS X 10.11.6
Problem description:
Not able to insert a column (using JDBC preparedStatement) of Type -> Array of Object

[2017-05-23 16:22:15,738][WARN ][io.crate.analyze.SetStatementAnalyzer] SET STATEMENT WITH SESSION OR LOCAL WILL BE IGNORED: SetStatement{scope=SESSION, assignments=[Assignment{column="extra_float_digits", expressions=[3]}], settingType=TRANSIENT}
[2017-05-23 16:22:16,078][WARN ][io.crate.protocols.postgres.types.PGType] decodeUTF8Text failed. input= ..... type=_json

Steps to reproduce:
A simple code shown below will show this error.

List<PGobject> entities = new ArrayList<>();
    PGobject entityObject = new PGobject();
    entityObject.setType("json");
    entityObject.setValue(gson.toJson(customObject.getEntity().get(0)));
    entities.add(entityObject);
    statement.setArray(1, connection.createArrayOf("object", entities.toArray()));

relevant part of the schema

entity array(object as (
   id string,
   type string,
   name string INDEX using fulltext with (type = 'standard'),
   attr1 string,
   info string,
   count int
)),

Use case:

Feature description:

@joemoe
Copy link
Contributor

joemoe commented May 24, 2017

thanks @ramach for reporting this. we will have a look at it.

@joemoe
Copy link
Contributor

joemoe commented May 24, 2017

@ramach
Copy link
Author

ramach commented May 24, 2017 via email

@ramach
Copy link
Author

ramach commented May 25, 2017

Your example still internally uses String [] in value field.
When I use a custom object I get this error

[2017-05-24 17:56:02,018][WARN ][io.crate.protocols.postgres.types.PGType] decodeUTF8Text failed. input={"{\"entity\":{\"type\":\"json\",\"value\":\".......type=_json

@ramach
Copy link
Author

ramach commented May 25, 2017

this happens during statement.executeUpdate()

my set block is

statement.setArray(2,
        connection.createArrayOf("object", new Object[] {new HashMap<String, Object>() {
          private static final long serialVersionUID = 1L;
          {
            put("entity", entityObject);
          }}, new HashMap<String, Object>() {{
            put("entity2", entityObject2);
          }}
        }));

@mikethebeer
Copy link

@ramach I'm not sure if I get you right, but it seems that you want to insert an "already" jsonified value into a field of type object_array. This is actually not the common way to do that with crate-jdbc since we have the parsing from/to json already in place internally and it would result in a parsing issue if you work with a json parser like gson beforehand.

This is what this error is indicating. The value seems to be a json string and will be incorrectly decoded:

[2017-05-24 17:56:02,018][WARN ][io.crate.protocols.postgres.types.PGType] decodeUTF8Text failed. input={"{\"entity\":{\"type\":\"json\",\"value\":\".......type=_json

You should instead work with the CrateDB Data Types directly.

new HashMap<String, Object>() {{
	put("id", "foo");
	put("count", 1L);
	...
}}, new HashMap<String, Object>() {{
	put("id", "bar");
	put("count", 2L);
	...
}}

@joemoe
Copy link
Contributor

joemoe commented May 30, 2017

hi @ramach can you provide an answer to @mikethebeer?

@ramach
Copy link
Author

ramach commented May 30, 2017

Now am seeing this error. Looks like auto table create during test initialization does not happen(though I do exactly the way you have in your integration test) - with @BeforeClass

code snippet for array creation is shown below

            List<String> values = (List<String>) activity.get(field.pos());
            logger.debug("value is {}", values.get(0));
            //String[] values = (String[]) ((List<String>) activity.get(field.pos())).toArray();
            logger.debug("value is {}, {}", values.toArray()[0], field.name());
            try {
              preparedStatement.setArray(field.pos() + 1, connection.createArrayOf(field.name(), values.toArray()));
            } catch (Exception e) {
              logger.error("error {}", e);
              preparedStatement.setArray(field.pos() + 1, null);
            }

2017-05-30 07:36:13,220 ERROR [main] jdbc.PreparedStatementBinder (PreparedStatementBinder.java:bindRecord(94)) - error {}
org.postgresql.util.PSQLException: ERROR: TableUnknownException: Table 'pg_catalog.pg_namespace' unknown
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2458)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2158)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:291)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:433)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:359)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
at org.postgresql.jdbc.TypeInfoCache.getPGType(TypeInfoCache.java:371)
at org.postgresql.jdbc.TypeInfoCache.getPGArrayType(TypeInfoCache.java:447)
at org.postgresql.jdbc.PgConnection.createArrayOf(PgConnection.java:1300)
at com.shn.activity.loader.connect.jdbc.PreparedStatementBinder.bindRecord(PreparedStatementBinder.java:92)
at com.shn.activity.loader.connect.jdbc.ActivityJDBCIntegrationTest.activityInsertTest(ActivityJDBCIntegrationTest.java:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

@mikethebeer
Copy link

mikethebeer commented May 30, 2017

@ramach are you using the official crate-jdbc driver? The driver you are using wants to query a meta table pg_namespace that does not exist in CrateDB. This seems to me that you are using the official Postgres driver instead.

You will find the repo here: https://github.com/crate/crate-jdbc
Bintray (Maven/Gradle): https://bintray.com/crate/crate/crate-jdbc

@ramach
Copy link
Author

ramach commented May 30, 2017

I am positive am using only Crate JDBC driver. I include this

            <dependency>
                    <groupId>io.crate</groupId>
                    <artifactId>crate-jdbc</artifactId>
                    <version>${crate-jdbc.version}</version>
            </dependency>

In my build file (pom) that is only reference to crate. No explicit call to postgres. postgres library is only while creating dynamic Objects using json and PGObject as shown in your example

import org.postgresql.util.PGobject;

            <crate-jdbc.version>2.0.1</crate-jdbc.version> is what I am using

@ramach
Copy link
Author

ramach commented May 30, 2017

for connection I use
connection = DriverManager
.getConnection(String.format(Locale.ENGLISH, "jdbc:crate://%s:%d/", host, psqlPort));
(defaults to localhost and 5432

@mikethebeer
Copy link

@ramach pls consider to use the latest stable release of crate-jdbc which is 2.1.7 atm --> https://github.com/crate/crate-jdbc/releases/tag/2.1.7

It can be that you run into this issue that has been fixed since 2.1.0.

@ramach
Copy link
Author

ramach commented May 30, 2017

Thanks - that error is resolved with 2.1.7

Now I see this

io.crate.shade.org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of org.postgresql.util.PGobject. Use setObject() with an explicit Types value to specify the type to use.
at io.crate.shade.org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1007) ~[tp-activity-loader.jar:na]
at com.shn.activity.loader.connect.jdbc.PreparedStatementBinder.setCustomType(PreparedStatementBinder.java:154) ~[tp-activity-loader.jar:na]
at com.shn.activity.loader.connect.jdbc.PreparedStatementBinder.bindRecord(PreparedStatementBinder.java:105) ~[tp-activity-loader.jar:na]
at com.shn.activity.loader.connect.jdbc.BufferedEvents.testInsert(BufferedEvents.java:310) [tp-activity-loader.jar:na]
at com.shn.activity.loader.connect.jdbc.BufferedEvents.main(BufferedEvents.java:297) [tp-activity-loader.jar:na]

the relevant code is

private void setCustomType(int position, Gson gson, List values) throws SQLException {
PGobject customRecord = new PGobject();
customRecord.setType("json");
customRecord.setValue(gson.toJson(values.get(0)));
preparedStatement.setObject(position, customRecord);
}

@ramach
Copy link
Author

ramach commented May 30, 2017

the copy paste did not show the correct
it is actually
private void setCustomType(int position, Gson gson, List values) throws SQLException {

@ramach
Copy link
Author

ramach commented May 30, 2017

List is a generic array - I am not able to copy paste with <

@ramach
Copy link
Author

ramach commented May 30, 2017

List

@ramach
Copy link
Author

ramach commented May 30, 2017

The way am using to setObject is identical to what you have in your crate-sample-app (Guestbook)

@ramach
Copy link
Author

ramach commented May 31, 2017

At least with 2.0.2 setObject worked (though array had error)
With 2.1.7 am not able to set a custom object

the sql looks like

entity object as (
external_id string,
entity_type string,
file_name string INDEX using fulltext with (type = 'standard'),
file_folder_path string,
file_last_modified_time timestamp,
file_owner string,
file_sharing_enabled string,
file_size long,
file_type string INDEX using fulltext with (type = 'standard'),
report_info string,
report_object_count int
),

I can create a map and put all attributes inside - I prefer not doing that using Gson() is more compact as we have several fields in some objects

@ramach
Copy link
Author

ramach commented May 31, 2017

I see another problem. In latest version 2.1.7 f preparedStatement.executeBatch() causes an error though executeUpdate() works fine

@mikethebeer
Copy link

mikethebeer commented May 31, 2017

@ramach I have an example app here that uses the field type object_array for ingest and query. Please check and use this as a base.

package io.crate.sampleapp;

import java.sql.*;
import java.util.HashMap;
import java.util.Map;

public class ObjectArrayApplication {

    public static void main(String[] args) {

        // CREATE TABLE t1 (
        //   obj_col OBJECT AS (
        //     col_int INTEGER,
        //     col_str STRING
        //   ),
        //   obj_arr_col ARRAY(OBJECT AS (
        //     col_int INTEGER,
        //     col_str STRING
        //   ))
        // );

        try (Connection conn = DriverManager.getConnection("crate://localhost:5432/doc")) {
            System.out.println("INSERT");
            PreparedStatement insertStatement = conn.prepareStatement("INSERT INTO t1 (obj_col, obj_arr_col) VALUES (?, ?)");
            insertStatement.setObject(1, new HashMap<String, Object>(){{
                put("col_int", 1);
                put("col_str", "some text");
            }});
            insertStatement.setArray(2, conn.createArrayOf("object", new Map[]{
                    new HashMap<String, Object>() {{
                        put("col_int", 2);
                        put("col_str", "some other text");
                    }},
                    new HashMap<String, Object>() {{
                        put("col_int", 3);
                        put("col_str", "another text");
                    }}
            }));
            insertStatement.execute();

            System.out.println("REFRESH");
            Statement refreshStatement = conn.createStatement();
            refreshStatement.execute("REFRESH TABLE t1");

            System.out.println("SELECT");
            Statement selectStatement = conn.createStatement();
            ResultSet rs = selectStatement.executeQuery("SELECT obj_col, obj_arr_col FROM t1");
            while (rs.next()) {
                Map<String, Object> obj_col = (Map<String, Object>) rs.getObject("obj_col");
                System.out.println(obj_col);
                Array obj_arr_col = rs.getArray("obj_arr_col");
                System.out.println(obj_arr_col);
            }
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }
}

@ramach
Copy link
Author

ramach commented May 31, 2017

thanks this was very helpful. Looks like your sample application (guestbook) has not been updated - setObject still uses json & PGobject

@mikethebeer
Copy link

@ramach yes you are right. It depends on an older version where this was still necessary. https://github.com/crate/crate-sample-apps/blob/master/java/pom.xml#L19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants