[DS-3755] integration test for submission configuration endpoint #1929

lap82 · 2018-01-24T18:45:43Z

This PR ( built on top of #1889 ) contains:

base integration test
implementation for missed endpoint /api/config/submissiondefinitions/<:definition-name>/collections

…3740

…c6902

…ocessing step;

…e (getData)

…nto SelectCollectionStep

…erimental code to works without authorization)

…n this phase)

…D4CRIS-337

…retrieved with findrel

lap82 · 2018-01-26T14:04:59Z

Ok, now the merge with the master branch is completed. I discovered another issue in the code base of the master and I fixed it. I have just to perform the changes requested by @tdonohue in the related PR #1889 to move this PR ready to merge.

tdonohue

Overall, this is looking good. But, after giving this a quick scan, I believe all my questions/suggestions from the previous PR (#1889) have not yet been implemented. I found the same code in this new PR.

That said, none of my comments should necessarily block this PR. But, I do feel there are minor refactors and additional comments that should be made. I also still don't much like the new class name DirectlyAddressableRestModel, but maybe that's just me.

tdonohue · 2018-01-26T14:22:53Z

dspace-api/src/main/java/org/dspace/app/itemupdate/AddBitstreamsAction.java

@@ -210,7 +210,7 @@ protected String addBitstream(Context context, ItemArchive itarch, Item item, Fi
 				if (group != null)
 				{
                    authorizeService.removeAllPolicies(context, bs);  // remove the default policy
-                    authorizeService.createResourcePolicy(context, bs, group, null, ce.permissionsActionId, null);
+                    authorizeService.createResourcePolicy(context, bs, group, null, ce.permissionsActionId, null, null, null, null, null);


I'm still seeing a large number of these createResourcePolicy() calls that now end in 5 null parameters. While it's not necessarily bad code, it looks very odd / unclear to me (and is extremely duplicative). So, as I mentioned in #1889, I wonder if we should have multiple createResourcePolicy() methods, with different sets of attributes passed in, rather than continually passing in nulls for the majority of the params.

So, for example, retain the existing authorizeService.createResourcePolicy(context, bs, group, null, ce.permissionsActionId, null) call in this line, and have its method code now call another createResourcePolicy() method with all 10 params, and the last 4 being set to null. This would cut down on a number of changes in the PR which simply append 4 more null attributes on every createResourcePolicy call.

Does this make sense? Or am I missing something in why we'd rather do it this way?

Yes, I agree. I left the method createResourcePolicy() after refactoring the method to highlight the changes, no other reason.

tdonohue · 2018-01-26T14:28:52Z

dspace-api/src/main/java/org/dspace/submit/step/AccessStep.java

-    }
+	@Override
+	public void doProcessing(Context context, Request req, InProgressSubmission wsi) {
+		// TODO Auto-generated method stub


I'm still seeing quite a few of these TODO Auto-generated method stub comments in what looks to be empty classes. Should we remove these classes too?

I suggest to do not remove these classes. The unused classes will be remove on the next round (I left the classes on to simplify also diff with the history commit).

tdonohue · 2018-01-26T14:31:41Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

 *
 * @author Andrea Bollini (andrea.bollini at 4science.it)
 * @author Tom Desair (tom dot desair at atmire dot com)
 * @author Frederic Van Reet (frederic dot vanreet at atmire dot com)
 *
 */
 @RestController
-@RequestMapping("/api/"+BitstreamRest.CATEGORY +"/"+ BitstreamRest.PLURAL_NAME + "/{uuid:[0-9a-fxA-FX]{8}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{12}}/content")
+@RequestMapping("/api/" + BitstreamRest.CATEGORY + "/" + BitstreamRest.PLURAL_NAME
+		+ "/{uuid:[0-9a-fxA-FX]{8}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{12}}/content")


As noted in #1889, I'd like to see a comment that explains these very large regex. It could be an example of paths that should match here, or note that it's trying to match valid UUIDs (if that's what this is doing). It's not a very straightforward regex, so it needs comments.

yes please add a comment as suggested by @tdonohue
The requestMapping value matches valid string representation of an UUID

tdonohue · 2018-01-26T14:32:09Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/RestResourceController.java

 			@PathVariable Integer id, @RequestParam(required = false) String projection) {
 		return findOneInternal(apiCategory, model, id, projection);
 	}

-	@RequestMapping(method = RequestMethod.GET, value = "/{id:[A-z0-9]+}")
+	@RequestMapping(method = RequestMethod.GET, value = "/{id:^(?!^\\d+$)(?!^[0-9a-fxA-FX]{8}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{12}$)[\\w+\\-]+$+}")


Another very long regex with no comments/examples. (There's also several more in this same class)

I agree that the regexes do not improve the readability of the code. In addition, I think it's the responsability of the RestRepository to parse a given String into the correct "ID format" (see the discussion in https://jira.duraspace.org/browse/DS-3785?focusedCommentId=57801&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-57801)

for now I recommend to add a comment in the javadoc to explain a bit
"The requestMapping value matches any alphanumeric string except than string composed by only digit or uuid string representation"

tdonohue · 2018-01-26T14:34:02Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/RestResourceController.java


 	private <ID extends Serializable> ResourceSupport findRelEntryInternal(HttpServletRequest request, String apiCategory, String model,
 			String id, String rel, String relid, Pageable page, PagedResourcesAssembler assembler, String projection) {
 		checkModelPluralForm(apiCategory, model);
-		DSpaceRestRepository<RestModel, ID> repository = utils.getResourceRepository(apiCategory, model);
-		Class<RestModel> domainClass = repository.getDomainClass();
+		DSpaceRestRepository<DirectlyAddressableRestModel, ID> repository = utils.getResourceRepository(apiCategory, model);


As noted in #1889, I'm still not a fan of calling this class DirectlyAddressableRestModel, as the DirectlyAddressable portion has very little meaning to me. I'm not sure of a better name yet.

tdonohue · 2018-01-26T14:34:25Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/RestResourceController.java

+		return deleteInternal(apiCategory, model, id);
+	}
+
+	@RequestMapping(method = RequestMethod.DELETE, value = "/{uuid:[0-9a-fxA-FX]{8}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{12}}")


Another long regex with no comment.

Same as above... maybe we can add define a constant UUID_REGEX, NO_UUID_OR_DIGIT_REGEX and put the comment on top of these constants

tdonohue · 2018-01-26T14:37:44Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/model/DirectlyAddressableRestModel.java

+ * @author Andrea Bollini (andrea.bollini at 4science.it)
+ *
+ */
+public interface DirectlyAddressableRestModel extends RestModel {


Here's the actual DirectlyAddressableRestModel class. I'm not sure I understand the terminology here, and why it is now "directly addressable" because it has a Category and Controller? Maybe I'm the only one though who feels the name here is obscure/unclear?

Maybe RestAddressableModel is the name appropriate for this interface.

tdonohue · 2018-01-26T14:40:47Z

dspace/config/dspace.cfg

-# The user interface you will be using for DSpace. Common usage is either xmlui or jspui
-dspace.ui = xmlui
+# The user interface you will be using for DSpace.
+dspace.ui = dspace-spring-rest


Again, as noted in #1889, "dspace-spring-rest" isn't a "ui". We might want to consider renaming this configuration (either in this PR or a future one).

I agree. Now I remove the dspace.ui configuration.

lap82 · 2018-01-26T14:53:43Z

Thanks @tdonohue to report the requested changes from #1889. In my previous comment what I would say is "I am still applying the changes proposed by Tim and then I will do the PR". Sorry for that!

tdonohue · 2018-01-26T14:55:42Z

Oh, sorry, @lap82. I had misunderstood your previous statement :) Now I understand that you still plan to apply my suggestions in the coming days. Thanks!

DS-3755 add links for the resource returned by the POST and PATCH method;

…pace.path or a dspace.rest.url could be introduced)

…bleRestModel) DS-3755 add missing interface declaration (RestAddressableModel) for the search rest class

tomdesair

Overall this PR is OK. Nice job on the PATCH request implementation @lap82 👍!

I've posted a few remarks on which I would like some feedback.

I did not yet built and deployed this PR to give it a test run. I'll try to do that later today or tomorrow.

tomdesair · 2018-01-30T10:52:45Z

dspace-api/src/main/java/org/dspace/app/util/Util.java

@@ -543,4 +546,25 @@ public static String getSourceVersion()
 		}
        return toReturn;
    }
+
+	public static List<String> diff(Collection fromCollection, Collection toCollection) throws DCInputsReaderException {


Since this method belongs to a general Util class, I think we should give this method a better name. "diff" can mean a lot of things:

Difference in metadata?

Difference in items?

If I understood the method correctly, we could name this method differenceInSubmissionFields.

tomdesair · 2018-01-30T11:02:12Z

dspace-api/src/main/java/org/dspace/content/DSpaceObjectServiceImpl.java

+				addMetadata(context, dso, schema, element, qualifier, rr.getLanguage(), rr.getValue(),
+						rr.getAuthority(), rr.getConfidence());
+			}
+			if (idx == to && to>from) {


I think that if-tests if (idx == to && to<from) and if (idx == to && to>from) can be collapsed to if (idx == to) because the code inside the if-test is the same.

No, because the operation move the resource, the order of the add metadata is important to save it to the right place.

Suppose idx == to is true, then

if to < from, we'll execute addMetadata(context, dso, schema, element, qualifier, moved.getLanguage(), moved.getValue(), moved.getAuthority(), moved.getConfidence())

if to > from, we'll execute addMetadata(context, dso, schema, element, qualifier, moved.getLanguage(), moved.getValue(), moved.getAuthority(), moved.getConfidence());

So it looks like that in both cases, we're doing the same thing. So I don't understand why the distinction is really necessary. But if you are really sure, you can leave this.

The clue is that we have to move an element in the right position related to the element of the iteration. If idx == to then we have to insert the element. So the question is the "from" element goes before or after the current element of the iteration? Depends on the "to" index related to the "from" index.

tomdesair · 2018-01-30T11:35:07Z

dspace-api/src/main/java/org/dspace/submit/model/UploadConfigurationService.java

+
+	private Map<String, UploadConfiguration> map;
+
+	public Map<String, UploadConfiguration> getMap() {


It might be useful to document what this map contains. What are the keys?

tomdesair · 2018-01-30T11:50:58Z

dspace-api/src/test/data/dspaceFolder/config/submission-forms.xml

+         <label>Title</label>
+         <input-type>onebox</input-type>
+         <hint>Enter the name of the file.</hint>
+         <required>You must enter a main title for this item.</required>


I suppose this is about a file. So maybe we should rephrase this to "You must enter a name for this file."

tomdesair · 2018-01-30T11:58:00Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/RestResourceController.java

 			@PathVariable Integer id, @RequestParam(required = false) String projection) {
 		return findOneInternal(apiCategory, model, id, projection);
 	}

-	@RequestMapping(method = RequestMethod.GET, value = "/{id:[A-z0-9]+}")
+	@RequestMapping(method = RequestMethod.GET, value = "/{id:^(?!^\\d+$)(?!^[0-9a-fxA-FX]{8}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{4}-[0-9a-fxA-FX]{12}$)[\\w+\\-]+$+}")


I agree that the regexes do not improve the readability of the code. In addition, I think it's the responsability of the RestRepository to parse a given String into the correct "ID format" (see the discussion in https://jira.duraspace.org/browse/DS-3785?focusedCommentId=57801&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-57801)

tomdesair · 2018-01-30T13:24:01Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/submit/SubmissionService.java

+	public UploadBitstreamRest buildUploadBitstream(ConfigurationService configurationService, Bitstream source) throws SQLException {
+		UploadBitstreamRest data = new UploadBitstreamRest();
+
+		for (MetadataValue md : source.getMetadata()) {


This feels like code that should go to a UploadBitstreamRestConverter class similar to how all other REST models are built in a converter class.

I agreed with @tomdesair but investigated in depth I left this code outside a "converter" because at moment there are code that it is not initialized as spring bean (see org.dspace.app.rest.submit.step.UploadStep). The converter will be available after the submission reader enhancement (i.e. the element into item-submission.xml will defined as spring bean)

tomdesair · 2018-01-30T13:34:40Z

...spring-rest/src/main/java/org/dspace/app/rest/submit/step/validation/MetadataValidation.java

+ * @author Luigi Andrea Pascarelli (luigiandrea.pascarelli at 4science.it)
+ *
+ */
+public class MetadataValidation extends AbstractValidation {


Would it be possible to move this class to dspace-api? It looks like "metadata validation" is business logic and is not directly related to the REST API.

At moment this class contains logic to transport error message to the client. Usually metadata validation in DSpace is a prerogative of the UI and the batch for example assume that the inserted values are pre validated.

tomdesair · 2018-01-30T13:38:47Z

dspace-spring-rest/src/main/resources/spring/spring-dspace-addon-validation-services.xml

+	xsi:schemaLocation="http://www.springframework.org/schema/beans
+           http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
+
+	<bean name="metadataValidation" class="org.dspace.app.rest.submit.step.validation.MetadataValidation" scope="prototype">


Can we move these beans (or the entire XML) to dspace/config/spring/rest? Then they end up in the config folder of the dspace install directory which makes them easier to modify afterwards.

So, I suggest to postpone it. In the next round I try to move the xml because it depends by the classpath for launch the batch script in DSpace and dependency for the other webapps to the new REST API (the dspace-rest jar need to compare in the lib of dspace installation and in the other webapp classpath (oai, sword, etc)

tomdesair · 2018-01-30T13:39:27Z

dspace-spring-rest/src/main/resources/spring/spring-dspace-core-services.xml

+	xsi:schemaLocation="http://www.springframework.org/schema/beans
+           http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
+
+	<bean id="patchConfigurationService" class="org.dspace.app.rest.submit.PatchConfigurationService">


Can we move these beans (or the entire XML) to dspace/config/spring/rest? Then they end up in the config folder of the dspace install directory which makes them easier to modify afterwards.

tomdesair · 2018-01-30T13:41:43Z

dspace/config/submission-forms.xml

+         <label>Title</label>
+         <input-type>onebox</input-type>
+         <hint>Enter the name of the file.</hint>
+         <required>You must enter a main title for this item.</required>


I think this should be updated to "You must enter a name for this file".

abollini

I suggest to immediately apply the small improvements suggested by all the reviewers to make the code more readable and create issues for the thing that need to be postponed.

lap82 · 2018-01-31T11:36:35Z

I just ended with the suggestions. I hope that this go well. I tried to do a simple javadoc, also because I have some limitation with the english :)
Need to create other ticket to manage Exception as reported by @abollini, please @tomdesair open a separated ticket and feel free to assign me. This PR is ready to merge!

tdonohue

👍 This now looks good to me (and all my previous concerns/questions have been addressed). Thanks for the hard work, @lap82! I think this is ready to merge, provided that someone creates the various follow-up tickets mentioned in comments above.

tomdesair

It seems that my concerns will be addressed in new PRs and tickets. So for now, I'm OK with this. Thanks @lap82 (and @abollini) for the great effort!

tomdesair · 2018-02-01T15:06:18Z

dspace-api/src/main/java/org/dspace/content/DSpaceObjectServiceImpl.java

+				addMetadata(context, dso, schema, element, qualifier, rr.getLanguage(), rr.getValue(),
+						rr.getAuthority(), rr.getConfidence());
+			}
+			if (idx == to && to>from) {


Suppose idx == to is true, then

if to < from, we'll execute addMetadata(context, dso, schema, element, qualifier, moved.getLanguage(), moved.getValue(), moved.getAuthority(), moved.getConfidence())

if to > from, we'll execute addMetadata(context, dso, schema, element, qualifier, moved.getLanguage(), moved.getValue(), moved.getAuthority(), moved.getConfidence());

So it looks like that in both cases, we're doing the same thing. So I don't understand why the distinction is really necessary. But if you are really sure, you can leave this.

abollini · 2018-02-01T19:02:17Z

Created JIRA issues for the postponed activities:
https://jira.duraspace.org/browse/DS-3825
https://jira.duraspace.org/browse/DS-3826

@tomdesair @tdonohue I hope to haven't forget anything, I will appreciate a double check here

abollini and others added 30 commits November 2, 2017 12:28

DS-3740 initial draft of the workspaceitems endpoint

e71c036

Merge branch 'DS-3740' of https://github.com/4Science/DSpace into DS-…

03007a8

…3740

format code

dcacb03

stub: introduce PATCH try to manage as https://tools.ietf.org/html/rf…

eb9b2b6

…c6902

DS-3740 introduce findBySubmitter

d41ccfa

DS-3740 finalize fromModel converter method; start refactoring for pr…

6bed6ac

…ocessing step;

Merge branch 'DS-3699' into DS-3740

6523b2f

DS-3740 stub refactoring for processing step;

8af7a74

Merge branch 'DS-3699' into DS-3740

4716128

DS-3740 remove old DTO class for submission process

910dfcc

DS-3740 comment in patch method; add rest step to define data exposur…

b2b035c

…e (getData)

DS-3740 implement POST method, add create workspaceitem by default

f521593

DS-3740 use standardize utility method

d4cc435

DS-3740 design preprocessing step method; implemented preprocessing i…

16108d2

…nto SelectCollectionStep

DS-3740 introduced dry run with a fake user (NOTE this code is an exp…

363320a

…erimental code to works without authorization)

DS-3740 minor comment

f3d09ee

DS-3740 remove method doPreProcessing (step have todo no operations i…

0539c92

…n this phase)

DS-3740 the init of workspaceitem is demanded on external service

1b278ec

DS-3740 refactoring use of the fake user move into rest

6eab911

DS-3740 add missed set submission definition

9139a84

avoid ambigous mapping when the id is digit-only with alphanumeric id

db33e62

Merge branch 'D4CRIS-337' of https://github.com/4Science/DSpace into …

fcf57ed

…D4CRIS-337

DS-3740 add missed commit to save the fake user

8dcfeba

DS-3740 move fake turnoff to create fake user

f789561

fix mapping for entities with string ID

ddb545e

DS-3740 add commit transaction on create method

06d4c72

DS-3740 stub implementation of update

624258d

DS-3743 introduce SectionData

595c044

avoid ambigous mapping when the id is digit-only with alphanumeric id

7f84c4b

fix mapping for entities with string ID

0f4ccf2

lap82 added 2 commits January 26, 2018 13:28

DS-3755 add test to check links on embeddedpage when the resource is …

3a09cd5

…retrieved with findrel

DS-3755 add missed link support for embedded object findrelation

4a615ec

tdonohue reviewed Jan 26, 2018

View reviewed changes

lap82 added 4 commits January 26, 2018 18:08

DS-3755 change response content for upload (now return the full object);

8630f12

DS-3755 add links for the resource returned by the POST and PATCH method;

DS-3755 split method to cut down on a number of changes in the PR

d690e67

DS-3755 remove the dspace.ui configuration (maybe in a future PR a ds…

feb2709

…pace.path or a dspace.rest.url could be introduced)

DS-3755 renamed class into RestAddressableModel (was DirectlyAddressa…

13a3660

…bleRestModel) DS-3755 add missing interface declaration (RestAddressableModel) for the search rest class

tomdesair suggested changes Jan 30, 2018

View reviewed changes

lap82 added 5 commits January 30, 2018 17:09

DS-3755 add javadoc

4a38057

DS-3755 minor comment

5f1e670

DS-3755 rename method in differenceInSubmissionFields (was "diff")

3426935

DS-3755 remove api class; add javadoc;

51823d1

DS-3755 minor changes

6bf5019

abollini reviewed Jan 30, 2018

View reviewed changes

lap82 added 3 commits January 31, 2018 11:09

DS-3755 fix message

77dab9c

DS-3755 remove try/catch to avoid unexpected NullPointerException

9e3e653

DS-3755 introduce constant to define request mapping regex

ddd2fff

lap82 removed the work in progress PR is still being worked on & is not currently ready for review label Jan 31, 2018

tdonohue approved these changes Jan 31, 2018

View reviewed changes

tomdesair approved these changes Feb 1, 2018

View reviewed changes

lap82 merged commit e2fdc75 into DSpace:master Feb 1, 2018

abollini deleted the DS-3755 branch February 1, 2018 18:12

tdonohue modified the milestones: 7.0, 7.0preview Jan 26, 2021

dspace-bot mentioned this pull request Aug 31, 2021

[DS-3826] Use an HandlerMethodArgumentResolver to create the Patch object #7173

Open


		private Map<String, UploadConfiguration> map;

		public Map<String, UploadConfiguration> getMap() {

[DS-3755] integration test for submission configuration endpoint #1929

[DS-3755] integration test for submission configuration endpoint #1929

Conversation

lap82 commented Jan 24, 2018 • edited Loading

lap82 commented Jan 26, 2018

tdonohue left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lap82 commented Jan 26, 2018 • edited Loading

tdonohue commented Jan 26, 2018

tomdesair left a comment

Choose a reason for hiding this comment

tomdesair Jan 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abollini left a comment

Choose a reason for hiding this comment

lap82 commented Jan 31, 2018

tdonohue left a comment

Choose a reason for hiding this comment

tomdesair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abollini commented Feb 1, 2018

lap82 commented Jan 24, 2018 •

edited

Loading

lap82 commented Jan 26, 2018 •

edited

Loading

tomdesair Jan 30, 2018 •

edited

Loading