Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major Update to Metadata blocks #1114

Closed
posixeleni opened this issue Nov 14, 2014 · 16 comments
Closed

Major Update to Metadata blocks #1114

posixeleni opened this issue Nov 14, 2014 · 16 comments
Assignees
Labels

Comments

@posixeleni
Copy link
Contributor

In order to make sure we are DDI-Lite compliant I am working on modifying the metadata blocks in Dataverse prior to 4.0.

Under Citation Block

  • Added Subtitle
  • Added Alternative Title (DANS request)
  • Created a Dataset Contact Compound field: for email now use: datasetContactEmail which is now a child of the compound datasetContact (this will affect how the email contact button works)
  • made Description compound field: added Description Date and allow multiples. @sekmiller this may affect displaying the description on the results page
  • made Keyword compound field: term, Vocabulary, URL
  • Moved Topic Classification compound from Social Science to Citation Block
  • Made Related Publication (Citation, ID Type and ID Number) Advance Searchable (help w/ DASH integration)
  • Moved Series compound from Social to Citation
  • Moved Software compound from Social to Citation
  • Controlled Vocabulary: Added "Data Curator" to contributorType list
  • Controlled Vocab: Add LCNA to Author Identifier
  • Added to both Distributor and Producer fields: URL and Logo URL
  • Made sure "Dataset" was capitalized in all tooltips (where applicable)

Under Social Science Block

  • moved fields: Series, Software, Topic Class to Citation (see above)
  • changed: Universe allowmultiples column to TRUE (DDI-Lite compliant)
  • changed Unit of Analysis allowmultiples column to TRUE (DDI-Lite compliant)
  • made Target Sample compound field (prev. no children)
  • removed sample frame

Under Geospatial Block

  • changed: Geographic Unit allowmultiples column to TRUE (DDI-Lite compliant)
  • capitalized all tooltips w/ Dataset in them
@posixeleni posixeleni added this to the Beta 10 - Dataverse 4.0 milestone Nov 14, 2014
@posixeleni posixeleni self-assigned this Nov 14, 2014
@posixeleni posixeleni changed the title Major Update to Citation metadata block Major Update to Citation & Social metadata blocks Nov 19, 2014
@posixeleni posixeleni changed the title Major Update to Citation & Social metadata blocks Major Update to Citation & Social Science metadata blocks Dec 8, 2014
@posixeleni posixeleni changed the title Major Update to Citation & Social Science metadata blocks Major Update to Metadata blocks Dec 8, 2014
posixeleni added a commit that referenced this issue Dec 8, 2014
See ticket #1114 this will need a schema.xml change so assigning to
@pdurbin
@posixeleni posixeleni assigned pdurbin and unassigned posixeleni Dec 8, 2014
@posixeleni
Copy link
Contributor Author

@pdurbin this will need a schema.xml change. I had to commit this in now to help with @ekraffmiller import work. Please let me know if anything is unclear.

Thanks!

posixeleni added a commit that referenced this issue Dec 8, 2014
pdurbin added a commit that referenced this issue Dec 9, 2014
scripts/search/tests/data/dataset-finch1.json needs an update
@pdurbin
Copy link
Member

pdurbin commented Dec 9, 2014

Ok, I updated the Solr schema.xml in 9265f8e

@posixeleni remember when you used to push your changes in a branch? And test in Vagrant? That was awesome because it meant I wouldn't have to immediately update the Solr schema. The minute you update the tsv files it affects dvn-build and developers the next time they pull. My search tests are currently broken.

Unfortunately, I can no longer create a dataset using the JSON at https://github.com/IQSS/dataverse/blob/master/scripts/search/tests/data/dataset-finch1.json (I'm getting the exception below).

@posixeleni do you want to see if you can adjust that JSON to make it work? I'm giving this ticket back to you.

[2014-12-09T08:47:39.661-0500] [glassfish 4.0] [WARNING] [] [edu.harvard.iq.dataverse.api.Dataverses] [tid: _ThreadID=22 _ThreadName=http-listener-1(5)] [timeMillis: 1418132859661] [levelValue: 900] [[
  Error parsing dataset version from Json
java.lang.ClassCastException: org.glassfish.json.JsonStringImpl cannot be cast to javax.json.JsonObject
    at edu.harvard.iq.dataverse.util.json.JsonParser.parseCompoundValue(JsonParser.java:131)
    at edu.harvard.iq.dataverse.util.json.JsonParser.parseField(JsonParser.java:101)
    at edu.harvard.iq.dataverse.util.json.JsonParser.parseMetadataBlocks(JsonParser.java:81)
    at edu.harvard.iq.dataverse.util.json.JsonParser.parseDatasetVersion(JsonParser.java:62)
    at edu.harvard.iq.dataverse.api.Dataverses.createDataset(Dataverses.java:147)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4695)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:630)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:582)
    at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:46)
    at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:582)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
    at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4667)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4655)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
    at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
    at com.sun.proxy.$Proxy397.createDataset(Unknown Source)
    at edu.harvard.iq.dataverse.api.__EJB31_Generated__Dataverses__Intf____Bean__.createDataset(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:125)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:253)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160)
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673)
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174)
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:357)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:260)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:188)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:191)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:168)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:189)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
    at java.lang.Thread.run(Thread.java:745)
]]

@pdurbin pdurbin assigned posixeleni and unassigned pdurbin Dec 9, 2014
@posixeleni
Copy link
Contributor Author

@pdurbin I don't understand what the JSON is outputting above. @scolapasta is there any chance you can help me figure out what is causing this issue? I had to make several major changes to support DDI-Lite and this includes changing how the Dataset Contact is mapped (see my first comment at the top). Could this be the cause? Thanks!

@posixeleni
Copy link
Contributor Author

Please note that these changes would also require a database drop.

@ekraffmiller
Copy link
Contributor

@pdurbin & @posixeleni - looks like the change is that one of the fields we have change to allowMultiples= true, is in one of the Json data test files. When allowMultiples= true, then the Json data has to store the value in an array, rather than a simple string. Phil, I can look at the data with you and find where the problem is.

@raprasad
Copy link
Contributor

raprasad commented Dec 9, 2014

Not sure if this is connected but JSON that had been parsing fine for months "broke" on me yesterday.

  • usual JSON: { "token_value" : "--some-string 64 chars long--" }
  • usual retrieved value: --some-string 64 chars long--
    • (no quotes and 64 chars long)
  • code that worked before yesterday: pulled value as an Object and converted to a String
Object worldmapTokenObject = jsonTokenInfo.get(WorldMapToken.GEOCONNECT_TOKEN_KEY);
if (worldmapTokenObject==null){
      logger.warning("Token is null found. Permission denied.");
       return null;
}
String worldmapTokenParam = worldmapTokenObject.toString(); 
  • yesterday retrieved value: "--some-string 64 chars long--"
    • (with quotes and 66 chars long, causing error)
  • updated code, uses getString instead of getObject
String worldmapTokenParam = jsonTokenInfo.getString(WorldMapToken.GEOCONNECT_TOKEN_KEY);       
if (worldmapTokenParam==null){      // shouldn't happen
     logger.warning("worldmapTokenParam is null when .toString() called.  Permission denied.");
     return null;
 }

\cc @pdurbin

@sekmiller
Copy link
Contributor

The update to the Dataset Contact (making it a compound field) broke the create dataset page. I just checked in a fix that will allow you create a dataset via the interface.

NOTE: I did a dbdrop/ re-run setup-all.sh to get this to work….

On Dec 9, 2014, at 10:23 AM, Raman Prasad <notifications@github.commailto:notifications@github.com> wrote:

Not sure if this is connected but JSON that had been parsing fine for months "broke" on me yesterday.

  • usual JSON: { "token_value" : "--some-string 64 chars long--" }
  • usual retrieved value: --some-string 64 chars long--
  • (no quotes and 64 chars long)
  • code that worked before yesterday: pulled value as an Object and converted to a String

Object worldmapTokenObject = jsonTokenInfo.get(WorldMapToken.GEOCONNECT_TOKEN_KEY);
if (worldmapTokenObject==null){
logger.warning("Token is null found. Permission denied.");
return null;
}
String worldmapTokenParam = worldmapTokenObject.toString();

  • yesterday retrieved value: "--some-string 64 chars long--"
  • (with quotes and 66 chars long, causing error)
  • updated code, uses getString instead of getObject

String worldmapTokenParam = jsonTokenInfo.getString(WorldMapToken.GEOCONNECT_TOKEN_KEY);
if (worldmapTokenParam==null){ // shouldn't happen
logger.warning("worldmapTokenParam is null when .toString() called. Permission denied.");
return null;
}

\cc @pdurbinhttps://github.com/pdurbin


Reply to this email directly or view it on GitHubhttps://github.com//issues/1114#issuecomment-66299069.

@posixeleni posixeleni assigned ekraffmiller and unassigned posixeleni Dec 9, 2014
@posixeleni
Copy link
Contributor Author

@ekraffmiller please reassign to @pdurbin or me if the JSON is sorted out

Thanks so much for your help!

@ekraffmiller
Copy link
Contributor

Ok, json updated.

@ekraffmiller ekraffmiller assigned pdurbin and unassigned ekraffmiller Dec 9, 2014
posixeleni pushed a commit that referenced this issue Dec 9, 2014
Per #1114 needed to remove sample frame which @ekraffmiller brought to
my attention that this was not done yet. @pdurbin this may need a
schema.xml change and another db drop whenever you're ready since I had
to drop a pre-existing field.
@pdurbin
Copy link
Member

pdurbin commented Dec 10, 2014

Created a Dataset Contact Compound field: for email now use: datasetContactEmail which is now a child of the compound datasetContact (this will affect how the email contact button works)

In #921 (comment) we said...

We don't have a way to express "Contact E-mail" or "Subject" in dcterms so we now populate the former with the email address of the parent dataverse and the later with "Other".

... and now we need to adjust the addDatasetContact method in https://github.com/IQSS/dataverse/blob/master/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordServiceBean.java to properly enter the value. I get so lost in updating dataset fields (which is why I opened #761) that I'm hoping @scolapasta or @sekmiller can fix that method. Passing this ticket to @scolapasta

(There's a new related ticket about how datasets with no contact email address shouldn't have been allowed to be created in the first place #1237 .)

@scolapasta
Copy link
Contributor

This is fixed now,

@scolapasta scolapasta removed their assignment Dec 10, 2014
pdurbin added a commit that referenced this issue Dec 10, 2014
other changes are a no-op, an artifact of sorting on Mac vs. Linux?
@pdurbin
Copy link
Member

pdurbin commented Dec 10, 2014

I saw in b9ef7b8 that @posixeleni slipped another tsv change it and I just updated the Solr schema in 5c786bd but since a field was removed, not added, people don't need to update the Solr schema. I'm sure it will be updated soon enough.

@pdurbin
Copy link
Member

pdurbin commented Dec 10, 2014

Hmm, I should note that @posixeleni mentions the possibility of a db drop at b9ef7b8 but I don't know if this is required.

@kcondon
Copy link
Contributor

kcondon commented Dec 14, 2014

@posixeleni I saw the following issues, can you comment?
-Sample Frame was not removed from the UI in the Soc Sci block.
-Related publication does not show multiples in the metadata summary section, subject does. Probably an existing issue.
-Added to Distributor field: URL and Logo but could not find a distributor field.

Otherwise tested all changes and verified changed fields are usable and searchable.

@kcondon
Copy link
Contributor

kcondon commented Dec 15, 2014

Closing unless I hear otherwise.

@kcondon kcondon closed this as completed Dec 15, 2014
@posixeleni
Copy link
Contributor Author

@kcondon to answer your questions (let me know if anything is unclear):

  • Sample Frame: as you saw from mine and Phil's comments above. I had forgotten to add the changes to remove "Sample Frame" from the Social Sci block but added them in after. Not sure if a db drop is required for the changes to show on this one.
  • Related publication on the Summary page will always only show the first publication and the rest will be shown when you click on the "Metadata" tab. This is expected UI behavior
  • Distributor Field will show up in the Citation Block when you select "Edit Dataset Metadata" (unless there's some bug where it sometimes doesnt show? here's a screenshot from dvn-build)
    image

Please let me know if I can assist with anything else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants