Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download media object #296

Merged
merged 15 commits into from
Aug 14, 2024
Merged

Download media object #296

merged 15 commits into from
Aug 14, 2024

Conversation

elv-preethi
Copy link
Collaborator

@elv-preethi elv-preethi commented Jul 25, 2024

The script OfferingDownloadMedia performs the following:

  • Retrieves streams metadata from either transcodes or offerings for each provided streamKey.
  • Then downloads the parts, concatenates them, and trims the media file for each streamKey.

Steps:

  • export PRIVATE_KEY="xxx"
  • set demov3 network in TestConfiguration.json or using --config-url
  • Run the command:
node ./testScripts/OfferingDownloadMedia.js 
--startTime 10 
--endTime 50 
--objectId=iq__bSS84FakdSgxHSiW3kJcA1obH6P 
--out ./out_dir 
--config-url "https://demov3.net955210.contentfabric.io/config"
--streamKey "video,polish_5_1,english_stereo"
  • For each content with a startTime and endTime, a new folder will be created under the output directory, as shown below:
./out_dir
└── iq__bSS84FakdSgxHSiW3kJcA1obH6P_00-00-40.500_00-01-10.000
    ├── english_stereo
    │   ├── 0001.hqpeBJhFb8U8gedQRc1gPT4uKd6fxawWmQBUjRcQuiRT2TKp1hVZ.mp4
    │   └── 0002.hqpe4oNGvobdiDeX9r9RnTAqtESNjKqtxryUAx51JhMxuhh4fUFq.mp4
    ├── english_stereo.mp4
    ├── parts_english_stereo.txt
    ├── parts_polish_5_1.txt
    ├── parts_video.txt
    ├── polish_5_1
    │   ├── 0001.hqpeNjNmLQvLQxfx3hWvXZzJ4nv2hF41DdrzSefRxxBDVRLyCsZ7.mp4
    │   └── 0002.hqpe6iHZFyZ7qroTSfz6xiVrhtDRyYktLveofuE27NB8EaXGtfCh.mp4
    ├── polish_5_1.mp4
    ├── trimmed
    │   ├── english_stereo_trimmed.mp4
    │   ├── polish_5_1_trimmed.mp4
    │   └── video_trimmed.mp4
    ├── video
    │   ├── 0001.hqpedi3ftDawHChtviNtzCbcVsnVukpjbqm6bmM1TaPJRutgz8x3Q.mp4
    │   └── 0002.hqpe2tXuAapeuTdPFNcwqnJFKyV1RxbVqz4Jc3ejJtn5iYURQDjeWS.mp4
    └── video.mp4

5 directories, 15 files

For instance, for english_stereo streamKey:
english_stereo folder => contains all the parts that fall within the start and end times.
english_stereo.mp4 => the concatenated file
english_stereo_trimeed.mp4 => this file has been both concatenated and trimmed to the specified time range.

Command usage:

 node ./testScripts/OfferingDownloadMedia --help
Options:
  --help                         Show help                                                                                                                                                       [boolean]
  --debug                        Print debug logging for API calls                                                                                                                               [boolean]
  --configUrl, --config-url      URL pointing to the Fabric configuration, enclosed in quotes. e.g. for Eluvio demo network: --configUrl "https://demov3.net955210.contentfabric.io/config"       [string]
  --elvGeo, --elv-geo            Geographic region for the fabric nodes.                       [string] [choices: "as-east", "au-east", "eu-east", "eu-west", "na-east", "na-west-north", "na-west-south"]
  --objectId, --object-id        Object ID (should start with 'iq__')                                                                                                                             [string]
  --versionHash, --version-hash  Object Version Hash (should start with 'hq__')                                                                                                                   [string]
  --offeringKey                  offering key                                                                                                                                [string] [default: "default"]
  --streamKey                    comma separated list of offerings stream key                                                                                                  [string] [default: "video"]
  --startTime                    start time to retrieve parts in seconds                                                                                                               [number] [required]
  --endTime                      end time to retrieve parts in seconds                                                                                                                 [number] [required]
  --out                          output directory                                   

@eponymous301
Copy link
Collaborator

For consistency with other scripts I recommend renaming objectHash to versionHash and offeringType to offeringKey

Note also that rather than using TestConfiguration.json to set demov3 network you should be able to use --configUrl https://demov3.net955210.contentfabric.io/config in command line, or alternate option is to set env var FABRIC_CONFIG_URL=https://demov3.net955210.contentfabric.io/config

Copy link
Collaborator

@eponymous301 eponymous301 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, I made some suggestions, the only ones that I feel strongly about are:

  • add a --streamKey argument (optionally defaulting to "video")
  • change --objectVersion to --versionHash and --offeringType to --offeringKey for consistency with other scripts
  • use default: ... in option() declarations instead of setting the value via || (this will make the default values show up automatically in help output when user is missing an option or specifies --help)

I also suggest renaming ObjectDownloadMedia.js to OfferingDownloadMedia.js to reflect the fact that it targets a single mez offering.

throw new Error(`start or end time provided are not within range: ${totalDuration}s`);
}

const sourcesTimeInfo = sourcesMetadata.map((item) => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recommend renaming item variable to part or partInfo for greater clarity

if(startTime < 0 || endTime > totalDuration) {
throw new Error(`start or end time provided are not within range: ${totalDuration}s`);
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional check?

  • if startTime >= endTime throw...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly more checks,

  • totalDuration < startTime < endTime, throw error
  • startTime < totalDuration < endTime, throw warning but set endTime to the total Duration

})
.option("startTime", {
describe: "start time to retrieve parts",
type: "number"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add
demandOption: true,
or
default: 0,
?

describe: "start time to retrieve parts",
type: "number"
})
.option("endTime", {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add
demandOption: true,
?


let objectId = this.args.objectId;
const versionHash = this.args.objectHash;
const offeringType = this.args.offeringType || "default";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can declare with default: "default", in .option(...)

const offeringType = this.args.offeringType || "default";
const startTime = this.args.startTime;
const endTime = this.args.endTime;
const out = this.args.out || "./out";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can declare with default: "./out", in .option(...)

objectId,
partHash
});
let filePath = path.join(dirPath, partHash);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to prepend an index so that e.g. instead of creating files:

hqpeKzJ5bBqKr38dEdNsX16epCwnqmwm5acnRcL13PvQkZhCnR9si
hqpe2NC2ojcbkPLrEbEKfFFSqroeST8Y2wgrJWRWtP61Lg7TfZ7nRT

the files wind up being

0001.hqpeKzJ5bBqKr38dEdNsX16epCwnqmwm5acnRcL13PvQkZhCnR9si
0002.hqpe2NC2ojcbkPLrEbEKfFFSqroeST8Y2wgrJWRWtP61Lg7TfZ7nRT

so that they are sorted in directory listings and user does not have to save part list from output to preserve ordering info.

Alternately you might be able to open a single file handle and write each part to same file handle in append mode, resulting in just one concatenated file being output

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prepended index to file name.

}


const sourcesJsonPath = "offerings." + offeringType + ".media_struct.streams.video.sources[*]";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to add an option --streamKey that defaults to "video", and then statement would be "offerings." + offeringType + ".media_struct.streams." + streamKey + ".sources[*]";

Video stream is not guaranteed to be stored under the key "video", and (maybe?) a future user might want to download an audio track

@elv-preethi
Copy link
Collaborator Author

Looks good overall, I made some suggestions, the only ones that I feel strongly about are:

  • add a --streamKey argument (optionally defaulting to "video")
  • change --objectVersion to --versionHash and --offeringType to --offeringKey for consistency with other scripts
  • use default: ... in option() declarations instead of setting the value via || (this will make the default values show up automatically in help output when user is missing an option or specifies --help)

I also suggest renaming ObjectDownloadMedia.js to OfferingDownloadMedia.js to reflect the fact that it targets a single mez offering.

Thanks for reviewing. Made the changes as per comment.
Also, changed the code to append all parts to single file and runs the ffmpeg at the end to concat all the parts and generate single mp4 file.

out_dir
├── parts_video.txt
├── video
│   ├── hqpe2NC2ojcbkPLrEbEKfFFSqroeST8Y2wgrJWRWtP61Lg7TfZ7nRT.mp4
│   └── hqpeKzJ5bBqKr38dEdNsX16epCwnqmwm5acnRcL13PvQkZhCnR9si.mp4
└── video.mp4

@elv-preethi
Copy link
Collaborator Author

elv-preethi commented Jul 26, 2024

@elv-serban When attempting to trim a video, I have observed that if the duration is 19.7s, it is rounded down to 19s on my system, instead of rounding to 20s. However, the ffmpeg @eponymous301 is using rounds the duration up to 20s. I am not sure how to fix this. @elv-haoyu, could you please try the script when you have a moment?

The ./out/video.mp4 has the untrimmed version comprising of all parts within duration and ./out/video_hh-mm-ss.ms_hh-mm-ss.ms.mp4 has the trimmed version.

@elv-preethi
Copy link
Collaborator Author

elv-preethi commented Aug 5, 2024

As per @elv-haoyu requests, Made following changes:

  • For each content with a startTime and endTime, a new folder will be created under the output directory, as shown below:
  1. streamKey == "both" :
├── iq__2BbhsB7V3DBVoSYmhszCeUBBVzpY_00-00-10.000_00-00-43.000
│   ├── audio
│   │   ├── 0001.hqpeDgq3121TLyBFf2NHiuv4kSvPbGiANyG1vGeN2cBxoo1J7oAu.mp4
│   │   └── 0002.hqpe4Yt7TBcceK1dau35nVWkbwDXBm8WNxXPwczKtXkRJ2xSP6Fx.mp4
│   ├── audio.mp4
│   ├── out.mp4
│   ├── out_00-00-10.000_00-00-43.000.mp4
│   ├── parts_audio.txt
│   ├── parts_video.txt
│   ├── video
│   │   ├── 0001.hqpeKzJ5bBqKr38dEdNsX16epCwnqmwm5acnRcL13PvQkZhCnR9si.mp4
│   │   └── 0002.hqpe2NC2ojcbkPLrEbEKfFFSqroeST8Y2wgrJWRWtP61Lg7TfZ7nRT.mp4
│   └── video.mp4

  1. streamKey == "audio":
└── iq__2BbhsB7V3DBVoSYmhszCeUBBVzpY_00-00-10.500_00-00-43.500
    ├── audio
    │   ├── 0001.hqpeDgq3121TLyBFf2NHiuv4kSvPbGiANyG1vGeN2cBxoo1J7oAu.mp4
    │   └── 0002.hqpe4Yt7TBcceK1dau35nVWkbwDXBm8WNxXPwczKtXkRJ2xSP6Fx.mp4
    ├── audio.mp4
    ├── out_00-00-10.500_00-00-43.500.mp4
    └── parts_audio.txt

  1. streamKey == "video":
└── iq__2BbhsB7V3DBVoSYmhszCeUBBVzpY_00-00-10.500_00-00-43.500
    ├── out_00-00-10.500_00-00-43.500.mp4
    ├── parts_video.txt
    ├── video
    │   ├── 0001.hqpeKzJ5bBqKr38dEdNsX16epCwnqmwm5acnRcL13PvQkZhCnR9si.mp4
    │   └── 0002.hqpe2NC2ojcbkPLrEbEKfFFSqroeST8Y2wgrJWRWtP61Lg7TfZ7nRT.mp4
    └── video.mp4

  • By default, if steamKey is not provided. Both audio and video files will be downloaded, concatenated and trimmed.
  • Final trimmed output is at out_start-time_end-time.mp4.

}
};

if(streamKey === "both") {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function works well. A few action items to generalize the method:

  1. Support for Multiple Audio Streams
  • Content object could have multiple audio streams, for instance english_5_1, english_stereo, french_parisian_5_1, french_parisian_stereo.
  • Allow users to specify multiple stream names in the --streamKey argument, separated by spaces (e.g., --streamKey "video english_stereo french_parisian_stereo").
  • Check if the specified streams exist in the content metadata before downloading.
  1. Handling Duplicate Downloads
  • When downloading the same content with the same start and end time but different streams, get an error Directory already exists at /Users/jenniezhang/elvProjects/elv-client-js Error: Directory already exists at /Users/jenniezhang/elvProjects/elv-client-js/iq__3QhxBgWHZDkcN87irAYqf5hSGdya_00-01-09.820_00-02-04.820, the method should:
    • Avoid errors due to existing directories.
    • Allow downloading multiple streams for the same content without conflicts.

Example commands:

Download video: node ./testScripts/OfferingDownloadMedia.js  --startTime 69.82 --endTime 124.82 --objectId=iq__3QhxBgWHZDkcN87irAYqf5hSGdya  --config-url "https://main.net955305.contentfabric.io/config" --out . --streamKey video
Download audio: node ./testScripts/OfferingDownloadMedia.js  --startTime 69.82 --endTime 124.82 --objectId=iq__3QhxBgWHZDkcN87irAYqf5hSGdya  --config-url "https://main.net955305.contentfabric.io/config" --out . --streamKey "audio english_stereo french_parisian_stereo"

@elv-preethi
Copy link
Collaborator Author

elv-preethi commented Aug 13, 2024

@elv-haoyu @eponymous301 @elv-serban made the changes as discussed. The details are here.
Also, @elv-haoyu all of the trimmed files will be under iq_XXX_start-time_end-time/trimmed folder.

Copy link

@elv-haoyu elv-haoyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test and looks good.

@elv-preethi elv-preethi merged commit 35ff381 into develop Aug 14, 2024
@elv-preethi elv-preethi deleted the download-media-object branch August 14, 2024 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a download method to get video or audio in mp4 files for a given time period
3 participants