RData file not showing up as RData file, but as GZIP archive; doesn't ingest

[bseeger]

What steps does it take to reproduce the issue?

Note, this issue was originally about RDS files, but that turned out to be inaccurate -- the files in question are actually all RData files, they just mistakenly have RDS as the file extension

We have 4 files that are RDS RData files. 3 convert successfully, the 4th does not and remains as a gzip archive file.

• When does this issue occur?
  Upon upload

• Which page(s) does it occurs on?
  File upload page

• What happens?
  4 files that are supposed to be RDS RData files (compressed) are uploaded, but only three are ingested. The 4th file remains as a zip archive. I'm not sure why - if it's the file format or something going on in dataverse.

In the the log an exception is thrown:

[2026-07-23T18:16:23.359+0000] [Payara 7.2026.2] [SEVERE] [] [] [tid: _ThreadID=103 _ThreadName=http-thread-pool::jk-connector(2)] [timeMillis: 1784830583359] [levelValue: 1000] [[
  java.io.EOFException: Unexpected end of ZLIB input stream
        at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:266)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:175)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:149)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:123)
        at edu.harvard.iq.dataverse.ingest.IngestableDataChecker.testRDAformat(IngestableDataChecker.java:593)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at edu.harvard.iq.dataverse.ingest.IngestableDataChecker.detectTabularDataFormat(IngestableDataChecker.java:669)
        at edu.harvard.iq.dataverse.ingest.IngestableDataChecker.detectTabularDataFormat(IngestableDataChecker.java:639)
        at edu.harvard.iq.dataverse.util.FileUtil.determineFileType(FileUtil.java:458)
        at edu.harvard.iq.dataverse.engine.command.impl.CreateNewDataFilesCommand.execute(CreateNewDataFilesCommand.java:218)
        at edu.harvard.iq.dataverse.engine.command.impl.CreateNewDataFilesCommand.execute(CreateNewDataFilesCommand.java:69)
        at edu.harvard.iq.dataverse.EjbDataverseEngineInner.submit(EjbDataverseEngineInner.java:36)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.glassfish.exousia.AuthorizationService.lambda$invokeBeanMethod$2(AuthorizationService.java:498)
        at org.glassfish.exousia.AuthorizationService.runInScope(AuthorizationService.java:582)
        at org.glassfish.exousia.AuthorizationService.invokeBeanMethod(AuthorizationService.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:272)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4716)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:654)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:832)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:604)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:868)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:831)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:604)
        at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:78)
        at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:868)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:831)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:375)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4689)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4678)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at jdk.proxy69/jdk.proxy69.$Proxy353.submit(Unknown Source)
        at edu.harvard.iq.dataverse.__EJB31_Generated__EjbDataverseEngineInner__Intf____Bean__.submit(Unknown Source)
        at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:300)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.glassfish.exousia.AuthorizationService.lambda$invokeBeanMethod$2(AuthorizationService.java:498)
        at org.glassfish.exousia.AuthorizationService.runInScope(AuthorizationService.java:582)
        at org.glassfish.exousia.AuthorizationService.invokeBeanMethod(AuthorizationService.java:498)
        at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:272)
        at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4716)
        at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:654)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:832)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:604)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
        at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:868)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:831)
        at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:604)
        at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:78)
        at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:868)
        at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:831)
        at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:375)
        at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4689)
        at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4678)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
        at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
        at jdk.proxy69/jdk.proxy69.$Proxy351.submit(Unknown Source)
        at edu.harvard.iq.dataverse.__EJB31_Generated__EjbDataverseEngine__Intf____Bean__.submit(Unknown Source)
        at edu.harvard.iq.dataverse.EditDatafilesPage.handleFileUpload(EditDatafilesPage.java:2041)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.glassfish.expressly.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:186)
        at org.glassfish.expressly.parser.AstValue.invoke(AstValue.java:253)
        at org.glassfish.expressly.MethodExpressionImpl.invoke(MethodExpressionImpl.java:248)
        at org.jboss.weld.module.web.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
        at org.jboss.weld.module.web.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:70)
        at org.primefaces.component.fileupload.FileUpload.broadcast(FileUpload.java:74)
        at jakarta.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:834)
        at jakarta.faces.component.UIViewRoot.processValidators(UIViewRoot.java:1271)
        at com.sun.faces.lifecycle.ProcessValidationsPhase.execute(ProcessValidationsPhase.java:49)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:72)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:131)
        at jakarta.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:689)
        at jakarta.faces.webapp.FacesServlet.service(FacesServlet.java:449)
        at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1554)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:331)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
        at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:83)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
        at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
        at edu.harvard.iq.dataverse.filter.CorsFilter.doFilter(CorsFilter.java:89)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)
        at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:366)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
        at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
        at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
        at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
        at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
        at java.base/java.lang.Thread.run(Thread.java:1583)
]]

Code line it failed on:

dataverse/src/main/java/edu/harvard/iq/dataverse/ingest/IngestableDataChecker.java

Line 593 in b485249

sb.append(String.format("%02X", gzin.read()));

This makes me think the file format may not have been complete? Or should this be relaxed? RStudio and R can read this file just fine.

Zulip Discussion: #troubleshooting > RDS looks like gzip file

File:

College_Debt_Data.zip

• To whom does it occur (all users, curators, superusers)?
  Only these four RDS files in this record: https://doi.org/10.60600/YU/R7ILPI

• What did you expect to happen?
  I expected all 4 files to get ingested to tab files.

Which version of Dataverse are you using?
6.10.1

Any related open or closed issues to this bug report?

Not that I'm aware of.

Screenshots:

The file that failed ingest:

The three that succeeded in ingest:

[landreev]

We have 4 files that are RDS files. 3 convert successfully, the 4th does not and remains as a gzip archive file.

Per the discussion on Zulip, and having looked at the dataset - the first 3 files were not RDS, but RData. An important distinction, since these are different, distinct formats and only RData has been supported as an "ingestable" type.

[bseeger]

Hello, @landreev, thank you for looking into this. When you say the first three are RData, how did you determine that? When I did a hex dump on them I see they are compressed, but beyond that I don't know enough about R to tell if something is RDS or RData and I would like to learn. (I have a programming background, so sharing R code is fine, if that's how you determined it).

So it appears that Dataverse could ingest (convert) (compressed) Rdata files fine, but maybe not (actual compressed) RDS files?

[landreev]

Hello, @landreev, thank you for looking into this. When you say the first three are RData, how did you determine that? When I did a hex dump on them I see they are compressed, but beyond that I don't know enough about R to tell if something is RDS or RData and I would like to learn. (I have a programming background, so sharing R code is fine, if that's how you determined it).

Hi,

How do we know that those 3 files were RData:

Usually RData files have the .RData filename extension; I noticed just now that yours had the extension .RDS, adding to the confusion. (Dataverse may rely on the file extension to identify some file types/in some situations; however, it does not for R files);

Internally, RData can be identified by the signature header. The simplest test is to uncompress and look at the first 5 bytes:

gzip -d < Coronavirus_Data.RDS | head -c5
RDX3

"RDX3" is not the only valid signature. The pattern is "RD[ABX][123]", with the last 2 bytes encoding the format type and version; (always followed by the 5th null byte).
For example, "RDA1" would also be a valid signature.
This is ALL that Dataverse does internally - if the first 5 bytes of a gz-compressed file match this pattern, Dataverse assigns it the mime type application/x-rlang-transport. Then it may or may not attempt to ingest it, depending on how it is configured.

RDS files similarly have their own, different signatures. In the case of the file SuppFig2_CandidateGenes.RDS from your dataset: hex 58 0A 00 00 00 03 ("X" followed by one new line, 3 null bytes, and \003 - with the sixth byte indicating version 3 of R format; I learned that just now, from Google 😄). [EDIT: Sorry, the file SuppFig2_CandidateGenes.RDS is NOT from your dataset - it was from a TDL Dataverse, and was linked by another user in the Zulip thread!]

For some historical reasons, only the RData format has been supported in Dataverse; both in terms of format identification, and ingest.

I will add some extra bits of info tomorrow, and summarize what (I think) we should do.

[landreev]

Re: how to tell an RData file from an RDS using R itself: You probably figured it out already, but it really sounds like the easiest way is simply load() vs readRDS().
The former will work for RData, but will bomb for RDS:
my_data <- load("Coronavirus_Data.RDS") (works quietly)

my_data <- load("SuppFig2_CandidateGenes.RDS") bombs:

Error in load("SuppFig2_CandidateGenes.RDS") : 
  bad restore file magic number (file may be corrupted) -- no data loaded
...

The latter, the other way around:
my_data <- readRDS("SuppFig2_CandidateGenes.RDS") (works)

my_data <- readRDS("Coronavirus_Data.RDS") bombs:

Error in readRDS("Coronavirus_Data.RDS") : unknown input format

To be clear, I am not an R expert at all; and haven't used it much at all in recent years.
But I happen to know/remember a bit about the old code in Dataverse that handles R files.

[landreev]

OK, mystery (and/or confusion) thickens; please see my edit in the comment from yesterday; I just realized that the RDS file, SuppFig2_CandidateGenes.RDS I was looking at (that I also used in my examples just now) was NOT from your dataset - it was linked, as an example by a different user in the Zulip thread.

I finally downloaded, and looked at that 4th ".RDS" file from your dataset - College_Debt_Data.RDS... it does in fact have the proper "magic number" signature of an RData file, just like the first 3:

gzip -d < College_Debt_Data.RDS | head -c5       
RDX3

In my defense, "RDS" was prominently mentioned all over the Zulip thread as well as this issue; I assumed of course that it was about support for the actual RDS format in Dataverse, or lack thereof... As it was, all 4 of these files were RData, and it was for some entirely different reason that Dataverse failed to recognize this particular one as such. That gzip-related exception you posted early on must be the key to it somehow. I'll take another look next week.

Everything I have said so far about RDS - that we do want to recognize and properly label it in Dataverse, etc. - still stands. But no, it was not relevant to your use case at all. 😅

[bseeger]

Thank you, @landreev - all the above info is very useful. The RDS mis-labeling of the file is definitely confusing and I relayed to the curator of that collection that the file extension is incorrect.

[bseeger]

I updated the title and description to note how the files have the incorrect extension.

[landreev]

@bseeger Great, thank you. I will add a final summary to the top of the issue, describing where it's breaking and what we should do in practical terms. (when I have a sec)