An error is reported after repeated reading and writing IO Error: Cannot open file "C:\test\duckdb\test.db": wanted=262144 read=0

[ethink]

Hello

I created the library table of duckdb through the following method, and then query it. It is OK at first, but an error will be reported after trying several times.
The creation code is as follows:

public static void createTableFromFile() {
try {
Class.forName("org.duckdb.DuckDBDriver");Properties ro_prop = new Properties();
ro_prop.setProperty("duckdb.read_only", "false");
Connection conn = DriverManager.getConnection("jdbc:duckdb:C:\test\duckdb\test743.db",ro_prop);

Statement stmt = conn.createStatement();

stmt.execute("CREATE TABLE test AS SELECT * FROM parquet_scan('E:\\test\\test.parquet');");


stmt.close();
conn.close();
} catch (Exception e) {
	// TODO Auto-generated catch block
	e.printStackTrace();
}

}

The query code is as follows:

public static void main(String[] args) {
System.out.println("++++++++++++++");
String sql1="SELECT * FROM test";
queryTest(sql1);

String sql2="SELECT s_id,sum(s_value) FROM test group by s_id ";
queryTest(sql2);
}
public static int queryTest(String sql) {
int count=0;
System.out.println("Start query statement:"+sql);
try
{
Properties ro_prop = new Properties();
ro_prop.setProperty("duckdb.read_only", "false");
DuckDBConnection conn = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:C:\test\duckdb\test.db");
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql);
rs.close();
stmt.close();
conn.close();
System.out.println("query was successful!");
}catch(Exception e) {
e.printStackTrace();
}

return count;
}

The error log is as follows:

++++++++++++++
Start query statement:SELECT * FROM test
query was successful!
Start query statement:SELECT s_id,sum(s_value) FROM test group by s_id
java.sql.SQLException: IO Error: Cannot open file "C:\test\duckdb\test.db": wanted=262144 read=0
at org.duckdb.DuckDBNative.duckdb_jdbc_startup(Native Method)
at org.duckdb.DuckDBDatabase.(DuckDBDatabase.java:22)
at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:35)
at java.sql.DriverManager.getConnection(Unknown Source)
Error after multiple queries

[Alex-Monahan]

Hello!
Which version of DuckDB are you using?

[Alex-Monahan]

Could you please try with the latest version? We are all the way up to 0.3.4 now and there are some great improvements!

[hannes]

Yes this is a known bug please update

[ethink]

Thank you. Now, after changing to a new version, this problem is solved. However, if the amount of data is very large, such as more than 500 million pieces of data, the following errors will be reported:
java.sql.SQLException: Catalog Error: Table with name test does not exist!
LINE 1: SELECT * FROM testlimit 1000
^
at org.duckdb.DuckDBNative.duckdb_jdbc_prepare(Native Method)
at org.duckdb.DuckDBPreparedStatement.prepare(DuckDBPreparedStatement.java:75)
at org.duckdb.DuckDBPreparedStatement.executeQuery(DuckDBPreparedStatement.java:129)

What caused this problem?

[Alex-Monahan]

Hello!

Could you please provide some more details?

Which specific code caused this issue? What script was used to build the database and fill it with content?

[ethink]

The code is as follows：
public static void addDataToTable() {
try
{

		    	 Properties ro_prop = new Properties();
					ro_prop.setProperty("duckdb.read_only", "False");
				DuckDBConnection conn2=null;
				  try {
						conn2 = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:C:\\test\\duckdb\\test.db");
					} catch (SQLException e) {
						// TODO Auto-generated catch block
						e.printStackTrace();
					}
				Statement stmt = conn2.createStatement();
				
				stmt.execute("CREATE TABLE data2 (n_id VARCHAR, vendor_id VARCHAR, pickup_at VARCHAR,dropoff_at VARCHAR,passenger_count VARCHAR,trip_distance VARCHAR,rate_code_id VARCHAR,store_and_fwd_flag VARCHAR,pickup_location_id VARCHAR,dropoff_location_id VARCHAR,payment_type VARCHAR,fare_amount VARCHAR,extra VARCHAR,mta_tax VARCHAR,tip_amount VARCHAR,tolls_amount VARCHAR,improvement_surcharge VARCHAR,total_amount FLOAT,congestion_surcharge FLOAT)");
				DuckDBAppender appender = conn2.createAppender("main", "test");
			
				conn2.setAutoCommit(false);
				
				stmt.execute("START TRANSACTION");
		     
		    	  for(int rowIndex=0;rowIndex<500000000;rowIndex++) {
		    		  
			    	  appender.beginRow();
			    	  appender.append("n"+rowIndex);
			    	  for(int i=1;i<=16;i++) {
			    		 
			    			  appender.append("str_"+rowIndex);
			    		  
			    	  }
			    	  appender.append(1F);
			    	  appender.append(rowIndex);
			    	  appender.endRow();
			    	  
			    	  if(rowIndex%(1024000*2)==0) {
			    		  stmt.execute("COMMIT");
			    		  stmt.execute("START TRANSACTION");
			    		 
			    	  }
			      
		    	  }
		      
		    	  appender.close();
			      stmt.execute("COMMIT");
		      stmt.close();
		      conn2.close();
		    }catch(Exception e) {
		    	e.printStackTrace();
		    }

	}

There is no problem when this code generates 200 million pieces of data, but there is a problem when it increases to 500 million

java.sql.SQLException: Catalog Error: Table with name test does not exist!
LINE 1: SELECT * FROM testlimit 1000
^
at org.duckdb.DuckDBNative.duckdb_jdbc_prepare(Native Method)
at org.duckdb.DuckDBPreparedStatement.prepare(DuckDBPreparedStatement.java:75)
at org.duckdb.DuckDBPreparedStatement.executeQuery(DuckDBPreparedStatement.java:129)

What caused this problem?