Settings/database migration from 0.2.231 fails #10

whispy · 2018-01-06T16:24:27Z

Steps:

Install Docker version of NZBHydra2 and start it — it starts with no issues.
Navigate to Docker version of NZBHydra 1 and copy the settings.cfg and nzbhydra.db into a sub-folder of NZBHydra2's /data volume.
Open NZBHydra2's web UI and navigate to System > Control > Migrate from NZBHydra 1.
Select NZBHydra 1 is running on another machine or in a container.
Enter locations of the files (see screenshot: https://vgy.me/1BebGq.png) and click Start Migration.
Migration runs.

Results:

Indexer and settings import shows as successful in the log and UI, but there are no indexers listed when I go to Config > Indexers. Other settings also seem to have failed to import.
Database migration fails with java.sql.SQLException: no such column: 'author' (more info in log section below).

Note: I have not yet tried running with --xmx 512M, as Synology's Docker UI makes this difficult and my log files make it seem like this is not the issue. However, if you request, I will figure out how to run it with this argument.

Log (debug output enabled and indexer names redacted):

2018-01-06 16:14:44.910  WARN --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Deleting all indexers, indexer searches, searches, downloads and API accesses from database
2018-01-06 16:14:45.407  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Starting database migration
2018-01-06 16:14:45.665  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating 13 indexers from old database
2018-01-06 16:14:45.700 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.515 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.517 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.520 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.522 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.524 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.527 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.529 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.530 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.532 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.535 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.537 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.538 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.540  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Successfully migrated indexers from old database
2018-01-06 16:14:46.657  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating 215 searches from old database
2018-01-06 16:14:46.818 ERROR --- [0.0-5076-exec-5] o.n.migration.FromPythonMigration        : [IP: 172.17.0.1] Error while migrating database

java.sql.SQLException: no such column: 'author'
	at org.sqlite.jdbc3.JDBC3ResultSet.findColumn(JDBC3ResultSet.java:48)
	at org.sqlite.jdbc3.JDBC3ResultSet.getString(JDBC3ResultSet.java:443)
	at org.nzbhydra.migration.SqliteMigration.migrateSearches(SqliteMigration.java:278)
	at org.nzbhydra.migration.SqliteMigration.migrate(SqliteMigration.java:110)
	at org.nzbhydra.migration.SqliteMigration.migrate(SqliteMigration.java:100)
	at org.nzbhydra.migration.SqliteMigration$$FastClassBySpringCGLIB$$72e2874a.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at org.nzbhydra.migration.SqliteMigration$$EnhancerBySpringCGLIB$$7dae4580.migrate(<generated>)
	at org.nzbhydra.migration.FromPythonMigration.startMigration(FromPythonMigration.java:103)
	at org.nzbhydra.migration.FromPythonMigration.migrateFromFiles(FromPythonMigration.java:65)
	at org.nzbhydra.migration.FromPythonMigration$$FastClassBySpringCGLIB$$25126370.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at org.nzbhydra.migration.FromPythonMigration$$EnhancerBySpringCGLIB$$7c1da1b2.migrateFromFiles(<generated>)
	at org.nzbhydra.migration.MigrationWeb.migrateFromFiles(MigrationWeb.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.nzbhydra.auth.HydraAnonymousAuthenticationFilter.doFilter(HydraAnonymousAuthenticationFilter.java:98)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

Please let me know if I can provide any more info or testing. Thanks for putting all this effort into the rewrite!

The text was updated successfully, but these errors were encountered:

theotherp · 2018-01-06T16:27:19Z

Thanks for the report. I'm getting feedback left and right, so it might take a bit until I can take a look.

whispy · 2018-01-06T16:30:02Z

I'm running NZBHydra 1, version 0.2.231 (see: https://vgy.me/pEfIX2.png).

theotherp · 2018-01-06T16:38:25Z

Yeah sorry, I didn't see that in the headline ;-)

theotherp · 2018-01-06T16:42:26Z

There's a column missing in the database that was added with 0.2.151. I can't really explain why it's not there. That should not be possible. That could mean that there are any number of possible database states out there. Oh my god.

I'll have to release a new update for v1 which makes sure everything is expected but that will take time. Considering how many reports I'm getting for v2 you're better off waiting a bit ;-)

whispy · 2018-01-06T16:47:31Z

Okay, thanks. Is that also why the settings.cfg import fails? I can do without my old database, but manually recreating all the settings and indexers in the UI would be a bit of a pain (but doable)

EDIT: I can send you my DB file too, if you'd like.

theotherp · 2018-01-06T16:50:37Z

Please post your debug infos. I didn't really expect so many (or any issues) with the migration.

The settings should be migrated even if the database migration fails.

theotherp · 2018-01-06T16:51:22Z

Please post your debug infos from v2 (http://127.0.0.1:5076/system/bugreport)

whispy · 2018-01-06T16:51:41Z

Debuginfos: https://dl.dropboxusercontent.com/s/sm8ckzmtvys6ya3/nzbhydra-debuginfos-2018-01-06-16-26.zip

theotherp · 2018-01-06T16:53:32Z

I'm seeing

Error while reading old settings file
java.io.IOException: Is a directory

Make sure that the settings file you provided is correct.

whispy · 2018-01-06T16:55:07Z

That was from a previous attempt where I just entered /data/nzbhydra-old-migration into the UI to see if it needed the filename or not. Other attempts in the debuginfos should have the full file paths.

EDIT: And I was doing the manual migration, as my V1 was running in a Docker container.

theotherp · 2018-01-06T16:57:55Z

Hm shit, the migration will fail when the database migration doesn't work.

As a workaround you could shut down your v1 instance, move the database files and start v1 again. Then try to migrate again. The v1 database should be reinitialized and match the expectations of the migration process.

theotherp · 2018-01-06T17:03:32Z

I really want to know what your database looks like and why it looks like that. The missing columns were added 2016 and Hydra shouldn't even work without them...

whispy · 2018-01-06T17:08:45Z

I figured it out, and it was me being silly — apologies for wasting your time.

I noticed when moving the database files, as you wrote earlier, that I was mistakenly using an older DB file from the end of 2016 (probably left over from some previous troubleshooting) that was in the top-level of my /hydra folder, and not the most up-to-date database. I corrected this and used the most recent DB file, and the migration worked fine.

Sorry for any potential stress! Keep up the awesome work!

whispy · 2018-01-06T17:15:07Z

Sent you a small donation to apologize :P

theotherp · 2018-01-06T17:18:52Z

Aaargh :-D

Well, I'm really glad that you figured it out quickly. I'll add code that checks the database version in the database file directly.

Thanks for the donation.

whispy closed this as completed Jan 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Settings/database migration from 0.2.231 fails #10

Settings/database migration from 0.2.231 fails #10

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018

theotherp commented Jan 6, 2018

theotherp commented Jan 6, 2018 •

edited

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018 •

edited

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018

Settings/database migration from 0.2.231 fails #10

Settings/database migration from 0.2.231 fails #10

Comments

whispy commented Jan 6, 2018 • edited

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018

theotherp commented Jan 6, 2018

theotherp commented Jan 6, 2018 • edited

whispy commented Jan 6, 2018 • edited

theotherp commented Jan 6, 2018 • edited

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018 • edited

theotherp commented Jan 6, 2018 • edited

whispy commented Jan 6, 2018 • edited

theotherp commented Jan 6, 2018

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018

whispy commented Jan 6, 2018 • edited

theotherp commented Jan 6, 2018

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018 •

edited

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018 •

edited

whispy commented Jan 6, 2018 •

edited

theotherp commented Jan 6, 2018 •

edited

whispy commented Jan 6, 2018 •

edited

whispy commented Jan 6, 2018 •

edited