Stream video with GridStoreAdapter (implements byte-range requests) #2437

Bragegs · 2016-08-01T22:51:30Z

Apple :
"HTTP servers hosting media files for iOS must support byte-range requests, which iOS uses to perform random access in media playback. (Byte-range support is also known as content-range or partial-range support.) Most, but not all, HTTP 1.1 servers already support byte-range requests."

Allows users of GridStoreAdapter to stream videos to IOS devices. Not tested on Android device.
(First PR)

codecov-io · 2016-08-01T23:08:11Z

Current coverage is 91.76% (diff: 40.81%)

Merging #2437 into master will decrease coverage by 0.71%

@@             master      #2437   diff @@
==========================================
  Files            93         93          
  Lines         10602      10718   +116   
  Methods        1309       1320    +11   
  Messages          0          0          
  Branches       1721       1743    +22   
==========================================
+ Hits           9804       9835    +31   
- Misses          798        883    +85   
  Partials          0          0

Powered by Codecov. Last update cbcd44b...a77fa21

flovilmart · 2016-08-03T14:30:34Z

src/Adapters/Files/GridStoreAdapter.js

+
+    gridFile.seek(start, function () {
+      // get gridFile stream
+              var stream = gridFile.stream(true);


fix indent.

flovilmart · 2016-08-03T14:41:21Z

Hi @Bragegs, Thanks for your PR, that seems to be pretty useful!

Can you address the different comments? And fix the indents to 2 spaces instead of 4?

flovilmart · 2016-08-03T14:42:35Z

src/Adapters/Files/GridStoreAdapter.js

+  handleVideoStream(filename, range, res, contentType) {
+    return this._connect().then(database => {
+      return GridStore.exist(database, filename)
+        .then(() => {


please put on the same line as previous one

Bragegs · 2016-08-04T15:27:38Z

Hi @flovilmart! Will address the typos Monday when returning from holiday, sry for the inconvenience.

"shouldn't we infer the streaming based upon the request headers like Range.":
That sounds more logical, indeed! I will test if the "Range" request header is sent by the clients when requesting video. According to IETF: "The "Range" header field on a GET request modifies the method semantics to request transfer of only one or more subranges of the selected representation data, rather than the entire selected representation data." (point 3.1). Maybe change the method name from "handleVideoStream" to "handleRangeRequest", as it may occur on other files than videos and optimizes bandwidth usage?

"Also the header seem to be Content-Range not Range.":
Of what I can understand it seems as the "Content-Range" header is only implemented in the response from the server and indicates the bytes present in the response body. Sample range request session

flovilmart · 2016-08-04T21:20:58Z

src/Controllers/FilesController.js

+    * If not; handle the request as usual with "getFileData".
+    */
+  handleVideoStream(filename, range, res, contentType) {
+      if (this.adapter.constructor.name == 'GridStoreAdapter') {


Can you expose just check if the function exists on the adapter for forward compatibility?

flovilmart · 2016-08-04T21:29:19Z

Another major nit, would it be possible to return a promise with the stream (and other things like length etc...) so the adapter don't need to know about setting headers and don't get the express response expose to keep the separation of concerns?

ghost · 2016-08-09T01:56:47Z

@Bragegs updated the pull request.

facebook-github-bot · 2016-08-09T02:09:42Z

@Bragegs updated the pull request.

…with_gridstoreadapter # Conflicts: # src/Routers/FilesRouter.js

facebook-github-bot · 2016-08-09T02:33:43Z

@Bragegs updated the pull request.

flovilmart · 2016-08-09T11:11:21Z

src/Routers/FilesRouter.js

-      res.end('File not found.');
-    });
+    const contentType = mime.lookup(filename);
+    if (req.get['Range']) {


Can we just check that typeof this.adapter.getFileStream === 'function'

This way we'll fall through regular response for non supporting adapters and it will be forward compatible?

…illed.

ghost · 2016-08-09T21:36:52Z

@Bragegs updated the pull request.

Bragegs · 2016-08-12T20:02:20Z

@flovilmart Thx for guiding me through! You doin a great job

Miladinho · 2017-02-05T23:26:45Z

Is this still in the latest version? I am using version 2.2.25 and I do not see the code changes mentioned in @Bragegs document https://notehub.org/8ncql

flovilmart · 2017-02-06T02:47:41Z

This should be available since a while.

Miladinho · 2017-02-12T03:31:32Z

@flovilmart I am looking at the changes to the files mentioned int he above link form @Bragegs and I do not see them in my version, nor do I see them in the current version of this repo. For example, I do a ctrl-f for "video" in parse-server-example\node_modules\parse-server\lib\Routers\FilesRouter and I only get 1 hit from a comment, there should be for example a 'var video' according to the link https://notehub.org/8ncql

Update:
Previously I was using version 2.2.18 not 2.2.25
Now I have upgraded to 2.3.3
Issue still persists
Although I have the S3 adaptor in my package.json, I have not instantiated it in my server code
So presumably I am still using grid store. There is some confusion out there among people who do not know parse well that files are stored in the mongodb, but from experiments I know that they are saved on my was ec2 instance (as long as it is not restarted or shutdown)

Miladinho · 2017-03-01T02:09:34Z

@flovilmart any updates on my last post?

flovilmart · 2017-03-04T15:27:20Z

If you made changes directly in the lib folder, this won't be deployed as npm will always fetch the build package from npm.

Miladinho · 2017-03-04T17:39:19Z

@flovilmart Maybe you did not understand what I am saying. For starters let me say that I am using parse-server-example. What I did is I made the code changes as described above in the parse-server folder of parse-server-example's node_modules folder. When deployed to AWS everything works fine except video streaming, I get an error regarding the code changes I made "handleVideoStream is not a function". So technically, I believe you did not understand my question because clearly the code update is not affected when pm runs because it sees that I already have the latest version so it will not update (probably nam looks at some file somewhere to verify that).

Look, I beg for you to really look into this. NONE of the code changes described here show up in the files...maybe someone removed them in future versions of parse-server. Furthermore, try building the parse-server-example project and look at the parse-server files int he node_modules after you run "nam install", you will notice that there is no "src" folder, just a "lib" folder. Now I may be a noob so forgive me if my understanding is bad, but why is it that when you clone parse-server by itself, all the files get sync as is seen int he github, but within parse-server-example it is not?

flovilmart · 2017-03-04T17:49:10Z

The src folder is built into the lib folder through babel upon publishing the module.

The lib folder is not committed to git as it would be inconvenient.
The src folder is not available in the npm module either, as it's useless.

NONE of the code changes described here show up in the files

There is no reference of a 'var video' whatsoever in that pull request, I'm not sure what you're referring to.

Any changes made to parse-server should be done in the src folder, run through npm run build.

If you wish to run your own fork of parse-server with your changes, you need to change multiple things. @acinader had an example at one point how to achieve that.

Miladinho · 2017-03-04T18:10:57Z

As I said in my first comment, I am referencing this https://notehub.org/8ncql when I speak about code changes, these changes seem to have fixed most people's issues with the video streaming. Either way, I cannot for the life of me figure out why I still cannot stream a video that I posted. It works through web when I download my videos in Chrome but not in mobile iOS or Safari, which makes sense given the comment above

"HTTP servers hosting media files for iOS must support byte-range requests, which iOS uses to perform random access in media playback. (Byte-range support is also known as content-range or partial-range support.) Most, but not all, HTTP 1.1 servers already support byte-range requests."

I am looking at the pull requests and I see that some of the code is not in the newest version
878718d

flovilmart · 2017-03-04T18:13:21Z

I am looking at the pull requests and I see that some of the code is not in the newest version

What do you mean by that???? The code is here... You're looking to a particular commit, where the naming was handleVideoStream later renamed in a77fa21

Miladinho · 2017-03-04T18:58:29Z

@flovilmart

Ok, thanks for the clarification. So everything is ok with the code then. But the problem persists. I cannot stream videos. I have deployed to parse-server-example with AWS Elastic Bean Stalk, backed with a t2.small ec2 instance. I have updated nginx so that I can upload files of size 20mb, so I can successfully post videos. I can;t stream videos and there are no errors in my logs, so I see all the GET requests coming in for mp4 files and no issues are reported after the request

flovilmart · 2017-03-04T19:04:31Z

@Bragegs may be more likely to answer that problem, I'm not sure if it's related to the headers not sent correctly, iOS behaving weirdly, or anything else. Feel free to open an issue with filling the issue template completely with all relevant logs, server AND client.

Miladinho · 2017-03-04T19:15:33Z

@flovilmart Thank you. One last question. Since with the old parse server (pre-open source) when parse was not shut down everything was working fine, some of the issues we are having now must be related to the way in which people are deploying their instances correct?

For example, if I were to run my parse-server with an https server I do not think I would be getting this issue with the video streaming. Also, I remember reading that old parse-server (pre-open source) was using AWS S3 to store content, is that true? Because if it is, it would suggest that i may be able to fix the issue doing that (which I will test anyways). This would also validate @dpaid comment on this thread where he/she claims that the S3 adapter fixed the streaming issue (even before @Bragegs fix). Am I thinking along the right lines here?

acinader · 2017-03-04T23:47:54Z

@Miladinho if you're deploying on elastic beanstalk, do you have an Elastic Load Balancer in front of your instances? Do you know if they are classic or app? If they are classic, I don't think that you are going to be able to stream using ws protocol.....sorry if i am way of topic :)

Miladinho · 2017-03-05T00:18:27Z

@acinader Hi,

Thank you for contributing your help. I actually just added the load balancer so I guess it is now an app instance. But it is not working. I added it because I thought of two solutions, 1 was to run HTTPS for which I needed to add the load balancer, 2 was to try saving files to S3. Right now it is running S# with load balancer and I still can't get the stream to work.

acinader · 2017-03-05T00:37:07Z

@Miladinho well, there are two different types of load balancers. classic and application. how do you know that you configured an application load balancer? for me, i am doing all of the config from the command line, so I don't know how to tell you to check using the console.

here's the docs on how to configure the application load balancer.

FYI, I haven't set up web streaming yet, but will need to pretty soon. until then, my ability to help you is very limited.....

Miladinho · 2017-06-01T23:52:04Z

Hi @acinader
Sorry if this is late, but man did you help me out with that hint! Not only did you help me solve my immediate issue and get my app running, but it helped me learn more about AWS and also web sockets. Eternally grateful for your help man. Let me know if there is ever anything I can help out with on one of your projects.

Closes: #5834 Similar to #2437 I ran into this issue while trying to view a mov file in safari from the dashboard.

#6028) * Stream video with GridFSBucketAdapter (implements byte-range requests) Closes: #5834 Similar to #2437 I ran into this issue while trying to view a mov file in safari from the dashboard. * Rename getFileStream to handleFileStream

parse-community#6028) * Stream video with GridFSBucketAdapter (implements byte-range requests) Closes: parse-community#5834 Similar to parse-community#2437 I ran into this issue while trying to view a mov file in safari from the dashboard. * Rename getFileStream to handleFileStream

Stream video with GridStoreAdapter

878718d

ghost added the GH Review: review-needed label Aug 1, 2016

flovilmart reviewed Aug 3, 2016
View reviewed changes

flovilmart added GH Review: needs-revision and removed GH Review: review-needed labels Aug 3, 2016

flovilmart reviewed Aug 3, 2016
View reviewed changes

flovilmart reviewed Aug 4, 2016
View reviewed changes

fixing nits. Removing test(Range not accepted as header)

0a94b9b

ghost added GH Review: review-needed and removed GH Review: needs-revision labels Aug 9, 2016

nit

b639383

Merge remote-tracking branch 'ParsePlatform/master' into streamVideo_…

5f8bad1

…with_gridstoreadapter # Conflicts: # src/Routers/FilesRouter.js

flovilmart reviewed Aug 9, 2016
View reviewed changes

flovilmart added this to the 2.2.18 milestone Aug 9, 2016

Changed names. Added function to check if stream-requirements is fulf…

a77fa21

…illed.

flovilmart merged commit e690b73 into parse-community:master Aug 12, 2016

wooknight mentioned this pull request Mar 22, 2017

Playing video with PFFile.url not working #1440

Closed

3 tasks

wooknight mentioned this pull request Apr 5, 2017

Unable to stream Video using PFFile #3690

Closed

postgetme mentioned this pull request Jul 25, 2019

hope GridFSBucketAdapter support getFileStream interface for client request range content #5834

Closed

dplewis added a commit that referenced this pull request Sep 5, 2019

Stream video with GridFSBucketAdapter (implements byte-range requests)

a71efb9

Closes: #5834 Similar to #2437 I ran into this issue while trying to view a mov file in safari from the dashboard.

dplewis mentioned this pull request Sep 5, 2019

Stream video with GridFSBucketAdapter (implements byte-range requests) #6028

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream video with GridStoreAdapter (implements byte-range requests) #2437

Stream video with GridStoreAdapter (implements byte-range requests) #2437

Bragegs commented Aug 1, 2016

codecov-io commented Aug 1, 2016 •

edited

Loading

flovilmart Aug 3, 2016

flovilmart commented Aug 3, 2016 •

edited

Loading

flovilmart Aug 3, 2016

Bragegs commented Aug 4, 2016

flovilmart Aug 4, 2016

flovilmart commented Aug 4, 2016

ghost commented Aug 9, 2016

facebook-github-bot commented Aug 9, 2016

facebook-github-bot commented Aug 9, 2016

flovilmart Aug 9, 2016

ghost commented Aug 9, 2016

Bragegs commented Aug 12, 2016

Miladinho commented Feb 5, 2017

flovilmart commented Feb 6, 2017

Miladinho commented Feb 12, 2017 •

edited

Loading

Miladinho commented Mar 1, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017 •

edited

Loading

acinader commented Mar 4, 2017

Miladinho commented Mar 5, 2017

acinader commented Mar 5, 2017

Miladinho commented Jun 1, 2017

Stream video with GridStoreAdapter (implements byte-range requests) #2437

Stream video with GridStoreAdapter (implements byte-range requests) #2437

Conversation

Bragegs commented Aug 1, 2016

codecov-io commented Aug 1, 2016 • edited Loading

Current coverage is 91.76% (diff: 40.81%)

flovilmart Aug 3, 2016

Choose a reason for hiding this comment

flovilmart commented Aug 3, 2016 • edited Loading

flovilmart Aug 3, 2016

Choose a reason for hiding this comment

Bragegs commented Aug 4, 2016

flovilmart Aug 4, 2016

Choose a reason for hiding this comment

flovilmart commented Aug 4, 2016

ghost commented Aug 9, 2016

facebook-github-bot commented Aug 9, 2016

facebook-github-bot commented Aug 9, 2016

flovilmart Aug 9, 2016

Choose a reason for hiding this comment

ghost commented Aug 9, 2016

Bragegs commented Aug 12, 2016

Miladinho commented Feb 5, 2017

flovilmart commented Feb 6, 2017

Miladinho commented Feb 12, 2017 • edited Loading

Miladinho commented Mar 1, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017

flovilmart commented Mar 4, 2017

Miladinho commented Mar 4, 2017 • edited Loading

acinader commented Mar 4, 2017

Miladinho commented Mar 5, 2017

acinader commented Mar 5, 2017

Miladinho commented Jun 1, 2017

codecov-io commented Aug 1, 2016 •

edited

Loading

flovilmart commented Aug 3, 2016 •

edited

Loading

Miladinho commented Feb 12, 2017 •

edited

Loading

Miladinho commented Mar 4, 2017 •

edited

Loading