feat(server): transcode bitrate and thread settings

[mertalev]

This PR allows users to set the max bitrate and thread count of transcoded videos through the admin transcoding page. This addresses two pain points with transcoding: high CPU usage and large file sizes.

Max Bitrate
This option is used for constrained quality (CQ) mode in FFmpeg. This mode respects the chosen CRF so long as it doesn't go beyond this bitrate, in which case it lowers quality dynamically. Setting an upper bound allows for predictable file sizes and prevents the issue of transcoded videos being much larger than the original.

Threads
Transcoding is naturally very CPU-intensive, and this can have an effect on other tasks and apps running on a device. Setting a thread count ensures that transcoding can only occupy a certain level of CPU resources.

However, the exact effect of this depends on the exact library used. While libvpx-vp9 respects FFmpeg's -threads flag, x264 and x265 create their own thread pools internally with heuristics set to max utilization. Disabling thread pooling led to a much closer relationship between -threads and actual utilization, though there seems to be something else that isn't being affected by this option.

Setting Descriptions
With the growth of the transcoding page over time, I think it's helpful to give some pointers on what these settings mean and what they can be useful for.

Extra
I also added support for two-pass transcoding, which does a fast transcode to generate statistics and uses those statistics to produce a better encoded video. CRF is ignored in this setting, and the actual quality is derived from the max bitrate set (following these recommendations). This setting is particularly good for VP9, as libvpx is optimized to take advantage of it for both quality and efficiency. The downside, of course, is that it takes more time to transcode in two passes than one.

It can be enabled by setting the environmental variable ENABLE_TWO_PASS to true.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
immich	⬜️ Ignored (Inspect)	Visit Preview		May 22, 2023 5:32pm

[alextran1502]

Tagging @zackpollard since this is his baby

[alextran1502]

It can be enabled by setting the environmental variable ENABLE_TWO_PASS to true.

Can we move this option to the setting page as well? and have it defaulted to false?

[mertalev]

It can be enabled by setting the environmental variable ENABLE_TWO_PASS to true.

Can we move this option to the setting page as well? and have it defaulted to false?

Yup, I added it as a slider in the admin page.

[vercel]

@mertalev is attempting to deploy a commit to the immich Team on Vercel.

A member of the Team first needs to authorize it.

[brighteyed]

Could you try to fix switch position?

[Sherlock1979]

Hi,

I am a complete noob when it comes to transcoding. But as a user, I need to tweak ffmpeg settings as the defaults just did not work for me. I need smaller files so that there is no buffering in low bandwidth scenarios while finding a balance with acceptable quality.

Hence, I started reading about vp9 recommendations and I realised that in Immich I cannot apply some settings that are recommended by many articles, including ffmpeg website and Google. I found this article particularly useful: https://streaminglearningcenter.com/codecs/encoding-vp9-in-ffmpeg-an-update.html#The_Recommended_Command_String

I am very pleased to see this PR and thank you for solving some of these challenges.

However, I feel that a couple of other options (or hardcoded defaults? for vp9) would make sense. E.g. "-row-mt 1" is highly recommended.

I can also imagine that hundreds of other users may come up with hundreds of other needs. I have seen in some applications that they solve such problems by letting the users edit any ffmpeg options as they will. Maybe as an 'advanced option' on the UI?

Thanks for considering my use case and ideas.

Note, as per the linked article and Google's m vp9 guide (https://developers.google.com/media/vp9/settings/vod) I think that your below statement may not be fully correct because they use CRF with 2-pass encoding. But again, I am a noob, so it is very likely that I am wrong:

I also added support for two-pass transcoding, which does a fast transcode to generate statistics and uses those statistics to produce a better encoded video. CRF is ignored in this setting, and the actual quality is derived from the max bitrate set (following these recommendations).

Thanks again! 🙏

[mertalev]

Could you try to fix switch position?

CSS is not my strong suit but it should be fixed now

[mertalev]

However, I feel that a couple of other options (or hardcoded defaults? for vp9) would make sense. E.g. "-row-mt 1" is highly recommended.

Yup, this PR adds that flag for all VP9 transcodes.

Note, as per the linked article and Google's m vp9 guide (https://developers.google.com/media/vp9/settings/vod) I think that your below statement may not be fully correct because they use CRF with 2-pass encoding.

You're right, VP9 allows for two-pass transcoding with constant quality based on CRF. The original PR didn't support this since the other two codecs (h264 and hevc) want bitrate ranges for two-pass encoding.

That said, I later added support for VP9's CRF-based two-pass mode. The way two-pass mode works now is that if the max bitrate is 0, VP9 will use the provided CRF, else it will use the max bitrate specified. (For h264 and h265, setting the max bitrate to 0 will still disable two-pass mode even if it's enabled.)

[Sherlock1979]

Thanks for the clarifications. Music to my ears. :) I am very excited to see this PR getting accepted and released.

That said, I later added support for VP9's CRF-based two-pass mode. The way two-pass mode works now is that if the max bitrate is 0, VP9 will use the provided CRF, else it will use the max bitrate specified. (For h264 and h265, setting the max bitrate to 0 will still disable two-pass mode even if it's enabled.)

Hmmm, I might have read that VP9 can do max bitrate AND CRF combined AND two-pass. Or have I misinterpreted something?

[mertalev]

Hmmm, I might have read that VP9 can do max bitrate AND CRF combined AND two-pass. Or have I misinterpreted something?

According to the FFmpeg wiki, two-pass mode can be done with either bitrates or CRF, but it doesn't mention using both. Some of the commands from the Google page actually do have both bitrate ranges and CRF, but the article you shared suggests that the CRF in them was ignored (in this section). It's possible that just setting a max rate would use both, but the resources I've seen only mention that combination in the context of one-pass encoding.

[Sherlock1979]

Thanks for the clarification. You are right, I think the Google page confused me. Sorry for the trouble.

[brighteyed]

I have tested all possible combinations of constrainMaximumBitrate and ffmpeg.twoPass:

constrainMaximumBitrate	ffmpeg.twoPass	h264/h265	vp9
true	true	✔️	✔️
false	true	⛔ for h264 second pass fails, ✔️ h265 is fine	✔️
true	false	✔️	✔️
false	false	✔️	✔️

After some googling it looks like h264 doesn't support 2-pass encoding with crf option specified. I believe we should ignore ffmpeg.twoPass in this case. What do you think?

[mertalev]

I have tested all possible combinations of constrainMaximumBitrate and ffmpeg.twoPass:

constrainMaximumBitrate ffmpeg.twoPass h264/h265 vp9
true true ✔️ ✔️
false true ⛔ for h264 second pass fails, ✔️ h265 is fine ✔️
true false ✔️ ✔️
false false ✔️ ✔️
After some googling it looks like h264 doesn't support 2-pass encoding with crf option specified. I believe we should ignore ffmpeg.twoPass in this case. What do you think?

Ah, my mistake. I handled that case in getFfmpegOptions by ignoring two-pass mode and falling back to CRF, but I didn't actually disable the ffmpeg.twoPass flag. It should be fixed now.

[brighteyed]

Ah, my mistake. I handled that case in getFfmpegOptions by ignoring two-pass mode and falling back to CRF, but I didn't actually disable the ffmpeg.twoPass flag. It should be fixed now.

All encoding options are fine now. It is definitely a notable step for video transcoding. Thank you for the PR!
Btw, great descriptions for options!

[mertalev]

Thanks for the detailed reviews! @brighteyed @jrasm91 @michelheusschen

[Sherlock1979]

Hi @mertalev,

Once again, thank you for this change. The improvements are fantastic.

I am doing vp9/opus transcoding and I have also noticed that only about 30-40% of my CPU capacity is utilised and meanwhile transcoding feels to take significantly slower than before. So I am trying to investigate the situation and better understand what is going on behind the scenes.

My ffmpeg Immich UI settings:

- CFR: 34
- PRESET: faster
- AUDIO CODEC: opus
- VIDEO CODEC: vp9
- TARGET RESOLUTION: 720p
- MAX BITRATE: 0
- THREADS: 0
- TRANSCODE: Videos higher than target resolution or not in the desired format
- TWO-PASS ENCODING: yes

Actual ffmeg paramteres while running:
ffmpeg -i upload/library/admin/[original file name].mp4 -y -vcodec vp9 -acodec opus -strict experimental -movflags faststart -vf scale=-2:720 -cpu-used 5 -row-mt 1 -crf 34 -b:v 0 -pass 2 -passlogfile upload/encoded-video/[some-uuid] upload/encoded-video/[some-uuid2]/[some-uuid3]

So, I have noticed that there is a "-cpu-used 5" parameter. Is this hardcoded? Or is this a representation of the preset that I set in the Immich UI?

• If yes then what is the relationship / mapping between the two?
• If no then is it possible that the preset UI setting is actually not having any effect? (as I don't see any other related parameters while ffpmeg is running)

Also, can you please point me in the right direction so that I can better understand the logic behind presets vs quality/deadline vs cpu-used/speed? I did some googling but could not find a source that explains it well and is trustworthy.

Thanks.

[mertalev]

@Sherlock1979 Great questions!

I have also noticed that only about 30-40% of my CPU capacity is utilised

If you set threads to 0, the threading is actually the same as the old behavior. The difference comes from a recent change in the queue to transcode videos one at a time instead of two at a time. If you want higher utilization, you can try setting it to a number yourself since FFmpeg doesn't always choose the most optimal number.

So, I have noticed that there is a "-cpu-used 5" parameter. Is this hardcoded? Or is this a representation of the preset that I set in the Immich UI?

The latter. The preset options map from 0-5 with veryslow being 0 and faster being 5. This was added with the PR so the presets were meaningful for all three supported codecs. Before this, the ffmpeg default would have been 1.

Another change was to mirror H.264 and HEVC in making constant quality the default when only using CRF. Omitting b:v 0 still constrains bitrate if I understand the wiki correctly. But I don't know whether this affects performance, or if it may only affect it in two-pass mode.

Speaking of two-pass mode, I think the slower speed you noticed comes primarily from that. It just fundamentally adds a good chunk of work for each video.

[Sherlock1979]

@mertalev

If you set threads to 0, the threading is actually the same as the old behavior. The difference comes from a recent change in the queue to transcode videos one at a time instead of two at a time. If you want higher utilization, you can try setting it to a number yourself since FFmpeg doesn't always choose the most optimal number.

I tried with 0, 5, and 10 (I have 12 cores). Overall utilisation was below 50% in all cases.

Speaking of two-pass mode, I think the slower speed you noticed comes primarily from that. It just fundamentally adds a good chunk of work for each video.

Actually, first pass seems to be very fast so it should not have a significant impact on my overall experience. Typically first pass is done in between 1-2 minutes.

[mertalev]

I did some testing with FFmpeg and it seems it's a limitation of libvpx-vp9. 720p only uses up to 40% of my CPU, 1080p can reach 60%, and 4K can fully saturate it. Upon doing some research, it seems libvpx doesn't have frame-level parallelism for encoding (only decoding), so the level of parallelism you can get is limited by resolution (i.e. how many tiles the frame can be divided into).

[mertalev]

Hmm, given that, it might be good to make the job queue concurrency user-configurable. Encoding more than one video in parallel seems to be the only way to scale vp9 further at lower resolutions.

[Sherlock1979]

Hmm, given that, it might be good to make the job queue concurrency user-configurable. Encoding more than one video in parallel seems to be the only way to scale vp9 further at lower resolutions.

Thanks for looking into it.

Yes, I would love to be able to change concurrency.