Dataverse 5.3 upgrade process: Dataverses and datasets missing

[ipcraiurosario]

Hi,

I've upgraded my dataverse from 4.9.4 version to 5.3 version on my development server however, it seems that some of my data is missing when I compare the 2 versions of dataverse that I have:

this is the datasets/dataverses count on my production 4.9.4 dataverse:

and this is my datasets/dataverses count on my 5.3 development dataverse:

This is what the harvesting server of my 4.9.4 dataverse shows:

and this is what the 5.3 shows:

The logs are just telling me that dataverse could not find the storage driver:

[2021-04-05T12:35:03.393+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103393] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/GZ2DRY]]

[2021-04-05T12:35:03.394+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103394] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/DAXR9U]]

[2021-04-05T12:35:03.394+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103394] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/RPQIYX]]

[2021-04-05T12:35:03.394+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103394] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/WIILZY]]

[2021-04-05T12:35:03.395+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103395] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/XHKWLL]]

[2021-04-05T12:35:03.399+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103399] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/8TMRLC]]

[2021-04-05T12:35:03.400+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103400] [levelValue: 900] [[
  Could not find storage driver for: s3://urdataverse:176b52da3fc-5e0777bb2072]]

[2021-04-05T12:35:03.404+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103404] [levelValue: 900] [[
  Could not find storage driver for: s3://10.34848/FK2/RD5NIK]]

[2021-04-05T12:35:03.404+0000] [Payara 5.2020.6] [WARNING] [] [edu.harvard.iq.dataverse.dataaccess.DataAccess] [tid: _ThreadID=66 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1617626103404] [levelValue: 900] [[
  Could not find storage driver for: s3://urdataverse:17294cec892-42684d75ef82]]

This could be related with an error that I'm getting when I click one published dataset:

[2021-04-05T12:47:42.507+0000] [Payara 5.2020.6] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=64 _ThreadName=http-thread-pool::http-listener-1(1)] [timeMillis: 1617626862507] [levelValue: 900] [[ StandardWrapperValve[Faces Servlet]: Servlet.service() for servlet Faces Servlet threw exception java.lang.NullPointerException at edu.harvard.iq.dataverse.dataset.DatasetUtil.getDownloadSize(DatasetUtil.java:473) at edu.harvard.iq.dataverse.DatasetPage.getSizeOfDatasetOrig(DatasetPage.java:2959) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.el.BeanELResolver.getValue(BeanELResolver.java:299) at com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:156) at com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:184) at com.sun.el.parser.AstValue.getValue(AstValue.java:114) at com.sun.el.parser.AstValue.getValue(AstValue.java:177) at com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:183) at org.jboss.weld.module.web.el.WeldValueExpression.getValue(WeldValueExpression.java:50) at com.sun.faces.facelets.el.TagValueExpression.getValue(TagValueExpression.java:73) at javax.faces.component.ComponentStateHelper.eval(ComponentStateHelper.java:170) at javax.faces.component.ComponentStateHelper.eval(ComponentStateHelper.java:157) at javax.faces.component.UIParameter.getValue(UIParameter.java:113) at com.sun.faces.renderkit.html_basic.OutputMessageRenderer.encodeEnd(OutputMessageRenderer.java:79) at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:595) at org.primefaces.renderkit.CoreRenderer.renderChild(CoreRenderer.java:88) at org.primefaces.renderkit.CoreRenderer.renderChildren(CoreRenderer.java:70) at org.primefaces.component.commandlink.CommandLinkRenderer.encodeEnd(CommandLinkRenderer.java:142) at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:595) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:286) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:80) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:278) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:283) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:80) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:278) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:80) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:278) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:80) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1647) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1650) at javax.faces.render.Renderer.encodeChildren(Renderer.java:152) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:566) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1647) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1650) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1650) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:468) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:170) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:196) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:132) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:132) at org.omnifaces.viewhandler.OmniViewHandler.renderView(OmniViewHandler.java:121) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:102) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:76) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:199) at javax.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:708) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:451) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1636) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:331) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:282) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) at java.lang.Thread.run(Thread.java:748) ]]
This issue does not happens when I click a dataset unpublished

I think that this could be fixed by making a copy of the storage from the production dataverse and setting this new storage on the domain.xml.. am I right?

Hope you guys can help!

best regards,

Juan

[djbrooke]

Thanks @ipcraiurosario, this is sounding like it may be a duplicate of #7663, but I'm not sure.

Can you try to run this API:

https://guides.dataverse.org/en/latest/api/native-api.html#datafile-integrity

... and let me know if that resolves the issue? Apologies if we've already suggested this step, as it's come up a bit recently.

[ipcraiurosario]

Hi @djbrooke I ran the api and it keeps showing the same information... however if I run this:
curl http://localhost:8080/api/info/metrics/datasets

I get the in my development server shows the 19 published datasets that should be shown on the platform view:
Here's the curl result:

and here's what is shown if I'm not logged in as the dataverse admin:

It should say 19 instead of 13.

[qqmyers]

The display may not update until you reindex after you fixed the earlier issue.

[ipcraiurosario]

@qqmyers after a full reindex (https://guides.dataverse.org/en/latest/admin/solr-search-index.html#full-reindex) the problem seems to be solved!

Thanks to @qqmyers and @djbrooke for your help!

[ipcraiurosario]

Just to close this issue ... the second part that I asked about can be fixed with my guess?

[qqmyers]

Not sure I fully understand the second part but - if the problem is that the new server can't access the s3 bucket, then setting up a store using the same bucket/credentials as before should work. That still involves jvm options stored in domain.xml (and also settable with the bin/asadmin create-jvm-options command). However, as of 4.20, Dataverse now supports multiple file and s3 stores so the jvm options used are different than they were in 4.9.4. If you check the guides for the s3 section and/or the 4.20 release notes, you'll see which jvm options are needed now.

[ipcraiurosario]

Ok, thanks for the reply!

I'll close this issue and try what you said.

Thanks again.