Feature 163- DropBox tweaks to EasyImportManager

[rtdog]

Added thread interruption logic to allow for quickly turning off, on, and changing folders in EasyImportManager. Also added some critical sections to prevent timing related errors. For best results this PR requires an updated java7-fs-dropbox library which is not included in this PR, but it is functional with the current version.

[wolfgang-ch]

Member Sorting

Currently you have not applied the settings for the member sorting which is defined in http://mytourbook.sourceforge.net/mytourbook/index.php/development/dev-ide/code-style

-> Code is resorted

Field Formatting

Eclipse has the evil behavior that sometimes (not always) it moves blank lines between member fields, this happened in your code. The only workaround is to manually check the code before it is commited or improve Eclipse which is propably very complex.

[rtdog]

@wolfgang-ch , thank you for the correction on this. I had wondered about this before and actually had a short discussion on the topic recently https://sourceforge.net/p/mytourbook/discussion/622811/thread/a8eace28ec/ , but I incorrectly misread my obligation to modify the settings for the sort order. This also explains why I saw such large diffs when I did the checkins. I'll get it fixed up in the future and I appreciate you fixing up these files. Thanks!

[wolfgang-ch]

I've not fix the formatting issues, I just discovered and logged it here.

For me, to verify code changes with the github browser tools is useless as it shows also other changes and then to find only your changes it just wasting of time. I use the Eclipse compare tools to verify the code changes but to use it, I have to merge the PR.

[wolfgang-ch]

@rtdog I've cherry picked your commit without the formatting commit into the new branch https://github.com/wolfgang-ch/mytourbook/tree/main-with-cherry-pick-from-rtdoc

My tests with local and dropbox devices were successfull, can you check if it is also working for you because I had to solve some formatting conflicts.

[wolfgang-ch]

@rtdog
When starting MT, the import view is opened and dropbox is selected, then this exception occures. Saving a tour later on is working.

Can you fix this issue for 20.8?

Application started in : 12089ms
Jul 30, 2020 5:38:52 PM com.mchange.v2.log.MLog <clinit>
INFO: MLog clients using java 1.4+ standard logging.
Jul 30, 2020 5:38:52 PM com.mchange.v2.c3p0.C3P0Registry banner
INFO: Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10]
Jul 30, 2020 5:38:53 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager
INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hgf02hab18hyb9tixk9nm|28796d0e, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> org.apache.derby.jdbc.EmbeddedDriver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hgf02hab18hyb9tixk9nm|28796d0e, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:derby:tourbook, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements -> 100, maxStatementsPerConnection -> 20, minPoolSize -> 3, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
SQLException

SQLState: 08000
ErrorCode: 40000
Message: Connection closed by unknown interrupt.

java.sql.SQLNonTransientConnectionException: Connection closed by unknown interrupt.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
	at com.mchange.v2.c3p0.impl.NewProxyResultSet.next(NewProxyResultSet.java:2859)
	at net.tourbook.importdata.EasyImportManager.getDbFileNames(EasyImportManager.java:305)
	at net.tourbook.importdata.EasyImportManager.getImportFiles(EasyImportManager.java:415)
	at net.tourbook.importdata.EasyImportManager.checkImportedFiles(EasyImportManager.java:212)
	at net.tourbook.ui.views.rawData.RawDataView.thread_UpdateDeviceState(RawDataView.java:5302)
	at net.tourbook.ui.views.rawData.RawDataView.access$48(RawDataView.java:5296)
	at net.tourbook.ui.views.rawData.RawDataView$51.run(RawDataView.java:5496)
	at java.lang.Thread.run(Thread.java:748)
Caused by: ERROR 08000: Connection closed by unknown interrupt.
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.iapi.util.InterruptStatus.setInterrupted(Unknown Source)
	at org.apache.derby.iapi.util.InterruptStatus.throwIf(Unknown Source)
	at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.checkCancellationFlag(Unknown Source)
	at org.apache.derby.impl.sql.execute.MultiProbeTableScanResultSet.getNextRowCore(Unknown Source)
	at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown Source)
	... 10 more

!ENTRY net.tourbook.common 4 0 2020-07-30 17:39:12.570
!MESSAGE 
!STACK 0
java.lang.NullPointerException
	at net.tourbook.common.util.SQL.showException(SQL.java:144)
	at net.tourbook.importdata.EasyImportManager.getDbFileNames(EasyImportManager.java:313)
	at net.tourbook.importdata.EasyImportManager.getImportFiles(EasyImportManager.java:415)
	at net.tourbook.importdata.EasyImportManager.checkImportedFiles(EasyImportManager.java:212)
	at net.tourbook.ui.views.rawData.RawDataView.thread_UpdateDeviceState(RawDataView.java:5302)
	at net.tourbook.ui.views.rawData.RawDataView.access$48(RawDataView.java:5296)
	at net.tourbook.ui.views.rawData.RawDataView$51.run(RawDataView.java:5496)
	at java.lang.Thread.run(Thread.java:748)

[rtdog]

@wolfgang-ch Yes I can take a look at this, thanks for finding it. Is there a hardening or code freeze period prior to the 2020.08 release and how does that work with the various branches? Thanks!

[wolfgang-ch]

The product is built from the main branch.

An official code freeze period do not exist but at the time, when I send the translators a mail to do the translations, which was on July 20, 2020 for 20.8, then I also create a test version that they can test their translations. At this time I also run my production with this test version, then there must be no database structure changes or other major changes only code fixes.

The translators have 2 weeks to do the translation (they could extend the time when requested) so the release date will be August 4, 2020 but this date is not fixed it also depends if I've updaate the website.

The built process itself takes only 3 minutes and with my 1GBit internet access it's published very fast, in the early days it took 2 hours to upload the product files.

[rtdog]

@wolfgang-ch , I have been unable to reproduce the issue you reported above. Furthermore, I'm not sure we're looking at the same code:

java.lang.NullPointerException
at net.tourbook.common.util.SQL.showException(SQL.java:144)
at net.tourbook.importdata.EasyImportManager.getDbFileNames(EasyImportManager.java:313)
at net.tourbook.importdata.EasyImportManager.getImportFiles(EasyImportManager.java:415)
at net.tourbook.importdata.EasyImportManager.checkImportedFiles(EasyImportManager.java:212)
at net.tourbook.ui.views.rawData.RawDataView.thread_UpdateDeviceState(RawDataView.java:5302)
at net.tourbook.ui.views.rawData.RawDataView.access$48(RawDataView.java:5296)
at net.tourbook.ui.views.rawData.RawDataView$51.run(RawDataView.java:5496)
at java.lang.Thread.run(Thread.java:748)

Notice this stack dump you provided shows "getImportFiles" on line 415 of EasyImportManager. My git history shows that command now on line 429 at the head of branch "main-with-cherry-pick-from-rtdoc". Is it possible that you were testing on a different commit? I admit that I have only used git for a few weeks so I'm still learning here. Thanks.

[wolfgang-ch]

I thought to have done the test with your commit but it was not. When doing the test with main-with-cherry-pick-from-rtdoc branch then it looks good, the expection still occures when doing the test with the current main branch.

Sorry for this mistake

I just tested with local and dropbox device and the import works smoothly. Now I will merge your commit into main.

Thanks for fixing the (for me) complex thread locking/interruption issue, this is something where I can still improve my skills :-)

[FJBDev]

@rtdog Thanks for improving the EasyImportManager :-)

My understanding is that the Dropbox integration revealed some issues that were never found with a local device as the performance of the latter is way faster than the performance of Dropbox