DS-3651: Range header support #1884

tomdesair · 2017-11-10T22:10:24Z

@Frederic-Atmire and I implemented a solution for https://jira.duraspace.org/browse/DS-3651 and https://jira.duraspace.org/browse/DS-3527 (for bitstreams).

Frederic created a SAF archive you can import and use to test this PR:

https://www.dropbox.com/s/jsaoe9bd5o5biq2/DS-3651-SAF.zip?dl=0 (Note: this file will only remain available while the PR is open).
Make sure the bitstream format registry is up to date: bin/dspace registry-loader -bitstream config/registries/bitstream-formats.xml
bin/dspace import -a -s ~/Downloads/ -z DS-3651-SAF.zip -c 123465789/2 -m mapfile -e admin@mail.com

The SAF archive contains 3 items: one with a PDF, one with a MP4 file and one with a MP3 file. Each modern browser should open the files without problems. The MP4 and MP3 files should be downloaded by parts. You should also be able to jump within the media file to undownloaded sections.

Alternatively, you can also test using curl as illustrated in the integration tests or with a Download mananger that can pause and resume downloads.

coveralls · 2017-11-10T22:24:12Z

Coverage increased (+0.5%) to 21.305% when pulling b84c2183283511c1e8a252bb647aeb7cc12aeba1 on atmire:DS-3651_Range-Header-support into 716912f on DSpace:master.

coveralls · 2017-11-12T21:42:26Z

Coverage increased (+0.5%) to 21.341% when pulling 4afa0f240893de74e4a26aa7bcd2a738e5f26426 on atmire:DS-3651_Range-Header-support into 716912f on DSpace:master.

abollini · 2017-11-16T11:02:45Z

dspace-api/src/test/data/dspaceFolder/config/spring/api/solr-services.xml

+    <!--<bean class="org.dspace.discovery.SolrServiceIndexOutputPlugin" id="solrServiceIndexOutputPlugin"/>-->
+
+    <!-- Statistics services are both lazy loaded (by name), as you are likely just using ONE of them and not both -->
+    <bean id="elasticSearchLoggerService" class="org.dspace.statistics.ElasticSearchLoggerServiceImpl" lazy-init="true"/>


is this really required? the plan is to withdrawn ES support in DSpace7

I moved these services from here to this new file so that we can override them in the integration tests.

I suggest we remove the elasticSearchLoggerService bean once we remove the complete Elastic Search implementation. I don't want to partly cleanup Elastic Search now with the risk of forgetting stuff.

ok, I agree

abollini · 2017-11-16T11:04:04Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

 import org.springframework.web.bind.annotation.PathVariable;
 import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.RequestMethod;
 import org.springframework.web.bind.annotation.RestController;

 /**
 * This is a specialized controller to provide access to the bitstream binary content
- * 
- * @author Andrea Bollini (andrea.bollini at 4science.it)


please add the name of the main authors of the class

abollini · 2017-11-16T11:09:20Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

+            try {
+                authorizeService.authorizeAction(context, bit, Constants.READ);
+            } catch (AuthorizeException e) {
+                response.sendError(HttpServletResponse.SC_UNAUTHORIZED);


as said I'm unsure about adding the auth check here. In any case we should not deal with the response directly but throw a new REST runtime AuthorizeException that can be annotated see https://github.com/DSpace/DSpace/blob/master/dspace-spring-rest/src/main/java/org/dspace/app/rest/exception/RepositoryNotFoundException.java

We can also implement a @ControllerAdvice class that has a handler method for AuthorizeExceptions. That way we don't have to manually convert this exception to a REST runtime exception. What do you think?

abollini · 2017-11-16T11:15:54Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

+			}
+
+		} catch(IOException ex) {
+			log.debug("Client aborted the request before the download was completed. Client is probably switching to a Range request.", ex);


are you sure about that? an IOException can occur if we have issue to read from the underline storage. If the client abort the request the exception should occur on the Servlet container stack as writing to the response is in some way "buffered" by tomcat, etc. (I admit I haven't check)

We'll make the check more specific on the "Client aborted" use case.

abollini · 2017-11-16T11:23:30Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

+
+		} catch(IOException ex) {
+			log.debug("Client aborted the request before the download was completed. Client is probably switching to a Range request.", ex);
+		} catch (Exception e) {


No please, avoid catch all block. If we have other not runtime exceptions they should be encapsulated in a REST Exception to have single trasversal code to deal with logging and response code. We should move to the SpringMVC exception handling way https://spring.io/blog/2013/11/01/exception-handling-in-spring-mvc

abollini · 2017-11-16T11:24:57Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

+		}
+	}
+
+	private boolean isNotAnErrorResponse(HttpServletResponse response) {


once you switch to the mvc exception handing this should be not required anymore

Our Range header utility class also sets some error codes. So we cannot remove this method.

abollini · 2017-11-16T11:29:56Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/converter/EPersonConverter.java

 import org.dspace.app.rest.model.EPersonRest;
+import org.dspace.app.rest.model.GroupRest;


I know that we have an open conversation about setting code style to avoid such kind of issue... but can you exclude from the PR the file that only have fix to the import? this should be done in a single dedicated pr when we have set our codestyle. This PR doesn't really need to touch more than 5-10 files I guess

abollini · 2017-11-16T11:42:36Z

dspace/config/dspace.cfg

@@ -1972,6 +1963,9 @@ mail.helpdesk = ${mail.admin}
 # Should all Request Copy emails go to the helpdesk instead of the item submitter?
 request.item.helpdesk.override = false

+# 2 MB


can you add more details about the intend of the parameter? it is a limit for the range requests? it is discoverable in some way?

This is the size of the byte buffer used for copying data from the bitstream InputStream to the request OutputStream. We can also omit the configuration property and give it a static value.

abollini · 2017-11-16T11:48:17Z

dspace/config/spring/rest/README.md

@@ -0,0 +1 @@
+You should only add Spring XML definition files here if there is really no way to load them through automatic component scanning.


A good use case to place xml here is to allow override of the provided configuration such as the listners enabled for usage events that you have included

I hope that in the future we also find an easy way (using XML?) to override the default bean implementations we now have in dspace-spring-rest. For example, what if I want to use a custom ItemConverter bean in my own custom repository.

abollini · 2017-11-16T11:53:25Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

+
+		Context context = ContextUtil.obtainContext(request);
+
+        Bitstream bit = getBitstreamIfAuthorized(context, uuid, response);


We should throw the authorize exception and use the spring mvc excpetion handling here

Is it OK if we take the global exception handling approach using @ControllerAdvice?

tdonohue · 2017-11-28T17:43:06Z

dspace-spring-rest/src/main/java/org/dspace/app/rest/BitstreamContentRestController.java

 * @author Andrea Bollini (andrea.bollini at 4science.it)
+ * @author Atmire NV (info at atmire dot com)


While we haven't really "policed" @author tags closely in recent years, I feel we should likely keep them pointing at individuals (who want credit for creating/refactoring specific classes) instead of organizations. I'm a bit worried that labeling classes with organizational contact info is a bit misleading, as it no longer refers to the specific creator of the class, and also could become an advertising gimmick/battle within our codebase (over which company gets more credit). The only company that "owns" the code is DuraSpace, and we just own the copyright in order to ensure the code remains open source, etc.

So, personally, I feel we should avoid this sort of change in our codebase. Either leave off the @author altogether, or reference the individual(s) who created / refactored the class.

However, if others disagree strongly here, I'll gladly bring this to discussion within the Committers group and/or Steering Group, so that we can formalize a policy on @author tags

I only noted this in the first file I found with this company @author tag. There are a few others

While Frederic and I did most of the development, there are also other people at Atmire who provided input and ideas for this (and other) pull-requests. And it is because other colleagues are working full-time on other projects, that Frederic and I have time allocated to work on DSpace 7. So it doesn't feel fair to only put my and Frederic's name as this PR would not have been possible without the support of Atmire.

And while I understand your concerns on the "advertising gimmick/battle", our codebase already contains a lot of company names/websites and contacts as people tend to use their professional e-mail address. There already are other examples of general company authors.

@tomdesair : Thanks for pointing out the Lyncode @author tags. I suspect those slipped in during the massive OAI refactor (i.e. creation of XOAI) many years back. It's simply hard to police this sort of thing. Had I noticed these at the time, I also would have questioned them. And, I'd rather remove them then add in more.

FWIW, other major projects (Apache Foundation and Gradle are two, see this blog post for example) do not allow any @author tags whatsoever. Their policy seem to be they quickly get out of date or are misleading, and simply cause clutter. While I understand your reasoning, I think individuals should be attributed via commits or via name, and not via a blanket generic statement. As-is, I think "Atmire NV (info at atmire dot com)" gives us no information about who actually worked on this code at Atmire, so there's no way to attribute those individuals in our Contributor list, or even (eventually) consider those individuals for Committership

coveralls · 2017-11-29T17:46:21Z

Coverage increased (+0.5%) to 21.564% when pulling 41a59ce on atmire:DS-3651_Range-Header-support into b578b03 on DSpace:master.

coveralls · 2017-11-29T17:53:27Z

Coverage increased (+0.5%) to 21.564% when pulling 41a59ce on atmire:DS-3651_Range-Header-support into b578b03 on DSpace:master.

coveralls · 2017-11-30T08:48:35Z

Coverage increased (+0.5%) to 21.565% when pulling ad0187f on atmire:DS-3651_Range-Header-support into b578b03 on DSpace:master.

tomdesair · 2017-11-30T13:12:39Z

@tdonohue I've corrected the author tags to unblock this PR. Once there is a decision on the usage of @author I'll update them accordingly in a separate PR.

coveralls · 2017-11-30T13:25:30Z

Coverage increased (+0.5%) to 21.565% when pulling 71b0bc4 on atmire:DS-3651_Range-Header-support into b578b03 on DSpace:master.

abollini · 2017-12-03T03:52:57Z

I have solved a minor conflict in the dspace-spring-rest/src/test/java/org/dspace/app/rest/test/AbstractControllerIntegrationTest.java
and fixed dspace-spring-rest/src/test/java/org/dspace/app/rest/AuthenticationRestControllerIT.java#L120 (to switch to the static way to create builder for test as the constructor is now protected)

artlowel added the interface: REST API v7+ REST API for v7 and later (dspace-server-webapp module) label Nov 13, 2017

abollini requested changes Nov 16, 2017

View reviewed changes

tdonohue requested changes Nov 28, 2017

View reviewed changes

frederic and others added 21 commits November 29, 2017 13:33

DS-3651: MultipartFileSender with inputstreams

4cedffc

DS-3651: Fixes and refactoring

eb65a3f

DS-3651: several improvements

db54bc1

DS-3651 configurable buffer

34e6676

DS-3651: Small refactoring

a58e650

DS-3651: Small improvements

91ae2e5

DS-3651: Call the event service on bitstream download

71f0dc5

DS-3651 MultiPartFileSender testclass

a02bda7

DS-3651 Multirange fix

40e6e95

DS-3651 Fix If-Match implementation

4c747a2

DS-3651 Unit tests for MultipartFileSender

39d202c

DS-3651: Log event when downloading bitstream via new REST API

fd4dde3

DS-3651: Update unit test

26dfe8f

DS-3651: Adding integration tests

5165ab0

DS-3651: Finished integration tests

df950ea

DS-3651: Fix imports

f9236d7

DS-3651: Check bitstream authorizatoins + tests

8fe17da

DS-3651: Fixing imports and authors

14e9bce

DS-3651 Exceptions with @ControllerAdvice

22ced22

DS-3651 Throw exceptions instead of catching

9b65d72

DS-3651: Correct ControllerAdvice exception handling + tests

b0bb8eb

tomdesair force-pushed the DS-3651_Range-Header-support branch from 44b4b2f to b0bb8eb Compare November 29, 2017 16:51

DS-3651: Remove unnecessary catch

41a59ce

DS-3651: Close database connection during download/streaming

ad0187f

DS-3651: Correct Atmire @author tag

71b0bc4

tdonohue approved these changes Nov 30, 2017

View reviewed changes

abollini approved these changes Dec 3, 2017

View reviewed changes

abollini merged commit 71b0bc4 into DSpace:master Dec 3, 2017

tomdesair deleted the DS-3651_Range-Header-support branch December 4, 2017 10:49

tdonohue modified the milestones: 7.0, 7.0preview Jan 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DS-3651: Range header support #1884

DS-3651: Range header support #1884

tomdesair commented Nov 10, 2017 •

edited

Loading

coveralls commented Nov 10, 2017

coveralls commented Nov 12, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017

abollini Nov 21, 2017

abollini Nov 16, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017

abollini Nov 16, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017 •

edited

Loading

abollini Nov 16, 2017

tomdesair Nov 20, 2017

abollini Nov 16, 2017

tomdesair Nov 20, 2017

tdonohue Nov 28, 2017

tdonohue Nov 28, 2017

tomdesair Nov 29, 2017

tdonohue Nov 29, 2017

coveralls commented Nov 29, 2017

coveralls commented Nov 29, 2017

coveralls commented Nov 30, 2017

tomdesair commented Nov 30, 2017

coveralls commented Nov 30, 2017

abollini commented Dec 3, 2017

		import org.dspace.app.rest.model.EPersonRest;
		import org.dspace.app.rest.model.GroupRest;

		@@ -0,0 +1 @@
		You should only add Spring XML definition files here if there is really no way to load them through automatic component scanning.


		Context context = ContextUtil.obtainContext(request);

		Bitstream bit = getBitstreamIfAuthorized(context, uuid, response);

		* @author Andrea Bollini (andrea.bollini at 4science.it)
		* @author Atmire NV (info at atmire dot com)

DS-3651: Range header support #1884

DS-3651: Range header support #1884

Conversation

tomdesair commented Nov 10, 2017 • edited Loading

coveralls commented Nov 10, 2017

coveralls commented Nov 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomdesair Nov 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Nov 29, 2017

coveralls commented Nov 29, 2017

coveralls commented Nov 30, 2017

tomdesair commented Nov 30, 2017

coveralls commented Nov 30, 2017

abollini commented Dec 3, 2017

tomdesair commented Nov 10, 2017 •

edited

Loading

tomdesair Nov 20, 2017 •

edited

Loading