Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle bluray folders #1588

Open
breezytm opened this issue Mar 29, 2019 · 65 comments
Open

Handle bluray folders #1588

breezytm opened this issue Mar 29, 2019 · 65 comments

Comments

@breezytm
Copy link

Hello Clinton,

Is there a way for nzbToMedia to transcode the bluray to mkv?

@clinton-hall
Copy link
Owner

I thought it did...

is this just the folder structure, or an iso?

@clinton-hall
Copy link
Owner

I have added the basics for this... but don't have anything to test it on here at the moment.
If you want to test, switch to bluray-1 branch and test it out.

@breezytm
Copy link
Author

I will go head and switch my branch to nightly and test the changes you made. It is interesting that I had one iso that worked fine and two others failed. perhaps it could be folder structure difference. Please let me know if I should send you the folder structure of one of the failed one.

@clinton-hall
Copy link
Owner

Yes, any detaiks as to folder structure would help.

Please use bluray-1 branch, not nightly...

@breezytm
Copy link
Author

breezytm commented Apr 3, 2019

I have not been able to switch to the latest branch but here's a log of one of the issues. It's not always the same.
https://ufile.io/kq4ez

:/opt/nzbget/downloads/completed/movie/A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel$ ls
A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel.iso
:/opt/nzbget/downloads/completed/movie/A.Star.is.Born.2018.1080p.CEE.Blu-ray.AVC.TrueHD.Atmos.7.1-wezjhOurBits-Rakuvfinhel$

The file is still an iso. It wasn't transcoded into an MKV

@clinton-hall
Copy link
Owner

clinton-hall commented Apr 3, 2019

Yes, the issue here is that there was a bug if you have "Concat" enabled and no .vob files are found.

This is fixed in bluray-1 branch (along with checking for bluray structure in addaition to DVD .vob structures.

You will need to switch to bluray-1 branch to test this. I can guarantee that nightly branch will not currently trasncode any bluray images.

@breezytm
Copy link
Author

breezytm commented Apr 3, 2019

I am about to test this right now. I'll report back with my findings.

@breezytm
Copy link
Author

breezytm commented Apr 3, 2019

What am I doing wrong?

admin@CYBERDYNE:/opt/nzbget/scripts/nzbToMedia$ sudo git checkout bluray-1
Branch bluray-1 set up to track remote branch bluray-1 from origin.
admin@CYBERDYNE:/opt/nzbget/scripts/nzbToMedia$ sudo git pull
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), done.
From https://github.com/clinton-hall/nzbToMedia
82d1ba8..0a3b2cd bluray-1 -> origin/bluray-1
cb3f61f..825b48a nightly -> origin/nightly
Updating 82d1ba8..0a3b2cd
Fast-forward
core/transcoder.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

https://ufile.io/5xze2

@breezytm
Copy link
Author

breezytm commented Apr 3, 2019

Also can you merge the HVEC change to this branch as well. I could test them both in this use case.

@clinton-hall
Copy link
Owner

ok, I have merged h265 changes into bluray-1
You have correctly switched branch
probably worth doing another sudo git pull and then all should be ready to test

@breezytm
Copy link
Author

breezytm commented Apr 4, 2019

What I was saying is I switched branch but it still failed. log file is attached at the bottom of my previous comment.

@clinton-hall
Copy link
Owner

ok, I see the error now. At least it isn't crashing, and now just says it cent find matching files to process.

What I need to see is just what the file structure is in that .iso....

I'll see tonight if I can download the same iso and check this to find out what it isn't matching....
are you able to email the .nzb file to fock_wulf@hotmail.com?

@breezytm
Copy link
Author

breezytm commented Apr 4, 2019

Sure. Emailing right now

@clinton-hall
Copy link
Owner

thanks.... in about 7 hours I should be able to start testing!

Hopefully I'll get some time over the weekend to debug the bluray parsing and get this working for you.

@clinton-hall
Copy link
Owner

Does that image actually work to be played by bluray players (e.g. in kodi)?

When I use 7zip to view the contents, there is a com directory, a META-INF, and a kudu directory along with a whole stack of *.class files

There is not a single .mts file, there is no BDMV or SOURCE directory... so as it stands nzbToMedia can't do anything with this.

All searching I have done indicates that Bluray folder structure should have a BDMV and a CERTIFICATE directory at the root level.

https://en.wikipedia.org/wiki/Blu-ray#Directory_and_file_structure

So there could be a couple of things going on here:

    1. This .iso is not a proper bluray image.
    1. The bluray .iso is somehow encrypted and needs to be decrypted before the actual folder structure can be properly parsed.
    1. 7zip can't read bluray .iso files correctly.

@clinton-hall
Copy link
Owner

ok... this is a limitation with 7zip.... (And I updated to v16.02)
When I mount the .iso I can view the BDMV and STREAM directories fine.

So I had another discussion somewhere in here recently about doing away with 7zip and using a python library that supports iso parsing and extraction... looks like I have some work ahead of me.

@clinton-hall
Copy link
Owner

Ok... I just tried using https://github.com/barneygale/isoparser

isoparser/source.py", line 93, in unpack_volume_descriptor
    raise SourceError("Wrong volume descriptor identifier")
isoparser.source.SourceError: Wrong volume descriptor identifier

I am wondering if we need to fall-back to mounting the iso and then parsing the contents.

@breezytm
Copy link
Author

breezytm commented Apr 4, 2019

ok... this is a limitation with 7zip.... (And I updated to v16.02)
When I mount the .iso I can view the BDMV and STREAM directories fine.

So I had another discussion somewhere in here recently about doing away with 7zip and using a python library that supports iso parsing and extraction... looks like I have some work ahead of me.

That was me :)
#1302

@clinton-hall
Copy link
Owner

LOL... well, aren't you giving me a headache :P ?
Thanks for linking that issue back in. I knew I had looked at these...

So I can now confirm that isoparser does NOT support bluray iso files.
I have not had a chance to test https://github.com/clalancette/pycdlib yet

@breezytm
Copy link
Author

breezytm commented Apr 4, 2019

LOL this is me helping you helping me helping everyone if that makes sense.

@breezytm
Copy link
Author

breezytm commented Apr 4, 2019

In all fairness I don't believe that particular iso is a valid one. MakeMKV can read it though.

@clinton-hall
Copy link
Owner

What I will like do is keep the 7zip parsing (if it works, it is efficient) but then if that doesn't work, try a mount and parse (as a fall-back).

The only other things I'll need to check is if I need to try and make this smarter and actually read the various play-list files.... current methodology is to transcode all .mts files into a single video file. but that may very well see all additional materials embedded before.after the main feature movie...

@clinton-hall
Copy link
Owner

Ok. I have added the fall-back to mounting .iso files to parse (only for linux at this time).
This is very theoretical, as I am not able to specifically test this functionality (working remotely).

I would be fascinated to see the output of logging from the script against this same download (after updating to latest bluray-1 branch).

If it transcodes, the big question will be what does the video file look like? i.e. is the movie continuous with extra features added at the end, or is it all jumbled and out of order?

If it does not transcode, what does the logging show us to reveal what I have missed?

@clinton-hall
Copy link
Owner

as of this moment, using bluray-1 I am transcoding that very .iso :)
So that part is working.... once it has transcoded I'll then need to verify what the movie itself looks like

@clinton-hall
Copy link
Owner

wow... this has been transcoding for over 11 hours! The output video is only up to 3 GB!

So it appears to be working... but testing on a NAS is slow going.
I still need to verify just how the transcoded video looks and if it it continuous.

@breezytm
Copy link
Author

breezytm commented Apr 5, 2019

Something is not right. MakeMKV does all of that in under 30 minutes for me but then again I am using a computer with decent specs. Let me use my server to test. I should post process much quicker. I am also wondering if it is transcoding the video as well since it is HVEC. I know you were supposed to merge the changes. Maybe it isn't working correctly. That could explain the longer transcoding time.

@breezytm
Copy link
Author

breezytm commented Apr 5, 2019

Before I even redownload the file, can you please confirm that when I select MKV as the outputDefaultk, it would copy the video instead of transcoding it as requested in this thread #1592

@breezytm
Copy link
Author

breezytm commented Apr 6, 2019

And yes I did do a pull and I did select mkv-bluray

remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 26 (delta 11), reused 14 (delta 8), pack-reused 0
Unpacking objects: 100% (26/26), done.
From https://github.com/clinton-hall/nzbToMedia
066e93e..13523ea bluray-1 -> origin/bluray-1
bbcef52..0ba4b9d nightly -> origin/nightly
Updating 066e93e..13523ea
Fast-forward
autoProcessMedia.cfg.spec | 2 +-
core/init.py | 10 +++++++++-
core/transcoder.py | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------
nzbToCouchPotato.py | 2 +-
nzbToLidarr.py | 2 +-
nzbToMedia.py | 2 +-
nzbToNzbDrone.py | 2 +-
nzbToRadarr.py | 2 +-
nzbToSickBeard.py | 2 +-
9 files changed, 84 insertions(+), 27 deletions(-)

@clinton-hall
Copy link
Owner

ok... I am guessing this didn't mount the .iso

Can you see any files in /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp?

what happens if you run the following command?

 mount -o loop /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp

@breezytm
Copy link
Author

breezytm commented Apr 6, 2019

that folder temp seems to be read only

WARNING: device write-protected, mounted read-only.

@breezytm
Copy link
Author

breezytm commented Apr 6, 2019

:sudo mount -o loop /opt/nzbget/downloads/completed/movie/Val erian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and .the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downl oads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-w ezjhOurBits/temp [sudo] password for system-admin: mount: /opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray .HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp: WARNING: device write-protected, mounted read-only. :/opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp$

@clinton-hall
Copy link
Owner

so this mounts ok when you use sudo, but not otherwise?

@breezytm
Copy link
Author

breezytm commented Apr 6, 2019

See below

`:/opt/nzbget/downloads/completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.US A.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/temp$ mount -o loop /opt/nzbget/downloads/completed/movie/Valerian .and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits/Valerian.and.the. City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhOurBits.iso /opt/nzbget/downloads/ completed/movie/Valerian.and.the.City.of.a.Thousand.Planets.2017.2160p.USA.UHD.Blu-ray.HEVC.TrueHD.Atmos.7.1-wezjhO urBits/temp

mount: only root can use "--options" option
`

@clinton-hall
Copy link
Owner

ok... I see 2 ways of working through this:
-1. create a python mount using ctypes.... this may or may not work without using root
e.g.

import ctypes
import ctypes.util
import os

libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)
libc.mount.argtypes = (ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.c_ulong, ctypes.c_char_p)

def mount(source, target, fs, options=''):
  ret = libc.mount(source, target, fs, 0, options)
  if ret < 0:
    errno = ctypes.get_errno()
    raise OSError(errno, "Error mounting {} ({}) on {} with options '{}': {}".
     format(source, fs, target, options, os.strerror(errno)))

mount('/dev/sdb1', '/mnt', 'ext4', 'rw')

-2. create an option for users to enter root password into config, and call sudo

clinton-hall added a commit that referenced this issue Apr 6, 2019
@clinton-hall
Copy link
Owner

if you update to latest then edit autoProcessMedia.cfg

[General]
    sudo_password = 

enter the sudo password and then test to see if this works to mount and transcode.

@breezytm
Copy link
Author

breezytm commented Apr 6, 2019

Before I do that, would you encrypt the password on the text .cfg file?

@breezytm
Copy link
Author

breezytm commented Apr 6, 2019

Also how come it is working for you but not working for me? Do you nzbget or whatever tool you use running as root? I have nzbget running as nzbget user

@clinton-hall
Copy link
Owner

fair call.. I currently don't have any encryption in the .cfg file... I'll need to do some research as to how to handle that... personally, my biggest concern is around giving the script sudo access (as opposed to the sudo password being stored within the .cfg). encrypting this will prevent anyone with access to the .cfg learning the password, and also protect people when they post their .cfg here asking for help.

For me this works becasue my NAS does not accept su or sudo. NZBGet runs as admin user, but the way the busybox commands work mount and umount work when called by admin.

@labrys
Copy link
Collaborator

labrys commented Apr 6, 2019

@clinton-hall From a security perspective I see this is as a very bad idea. The permissions of the user to run that command should be an environment configuration. For example you can use a combination of sudo and visudo to give passwordless access to the user for discretionary access to execute the command while not allowing other root commands. This should not be done by the application, there are too many ways it could be a security concern.

Edit: Just saw that this was related to mounting/unmounting - there's a good reason that mounting/unmounting requires root as it can be used for privilege escalation. I don't see an effective way of automating this without opening serious security concerns.

@clinton-hall
Copy link
Owner

@labrys
agreed... the 7zip methodology worked well, but the issue is 7zip does not support bluray .iso.
Likewise isoparser fails.

There has to be someway of using python to browse and extract from bluray .iso files, but this is way above my capabilities...

mount works fine for me on my NAS but obviously not on any system requiring sudo.

do you know if the ctypes library (snippet of code above that came from a sourceforge page) would allow mounting and parsing without requiring root privileges? or is that essentially going to emulate the mount command and have the same limitations?

@clinton-hall
Copy link
Owner

FWIW, I note a request on 7zip page to support BD iso. I have provided info there to see if this can be added.

@labrys
Copy link
Collaborator

labrys commented Apr 7, 2019

I'm fairly certain that mounting with ctypes requires root.

@labrys
Copy link
Collaborator

labrys commented Apr 7, 2019

Have you tried using rar? I'm fairly sure it can extract bluray isos.

@clinton-hall
Copy link
Owner

testing unrar at my end does not work. I have seen some indication online that winRAR does work, but unrar on linux does not.

I have provided more info to 7zip dev to see if 7zip can support UDF 2.50 and 2.60 iso files. If so, then I am confident the changes I have made will work for bluray images.

To Sum up:
-1. Bluray file structure is supported. - All video files are transoced in order (main movie followed by extra features)
-2. 7zip is called first for .iso files
-3. If 7zip failed to match DVD or Bluray file structure the script attempts to mount and then parse the folder structure.

So if mount works (some busybox systems or any system where the downloader runs as root) this will work.
Otherwise if/when 7zip is updated, this will work.

I'll remove the changes I have made for the sudo password...

clinton-hall added a commit that referenced this issue Apr 7, 2019
@breezytm
Copy link
Author

breezytm commented Apr 7, 2019

Sounds good. Is there a ticket on their side you can add to this thread for us to track the changes on there end.

@labrys
Copy link
Collaborator

labrys commented Apr 7, 2019

@clinton-hall can you detect that the reason for failure is a bluray iso? If so then possibly raise warning informing user to manually extract?

@clinton-hall
Copy link
Owner

clinton-hall commented Apr 7, 2019

Ticket at 7zip:
https://sourceforge.net/p/sevenzip/discussion/45797/thread/21f213e2/

I can't currently detect bluray iso as the reason for failure. In fact the issue is that 7zip actually does produce a file list, but it isn't correct (as per thread linked above Igor has indicated this is actually parsing an internal zip file) and therefore doesn't look like the expected DVD / BluRay structure.

@clinton-hall
Copy link
Owner

@breezytm for what it is worth, I transcoded that same movie to 1080p (so it should now be playable). Remembering that this is for the full 3+ hours of bonus features etc as well... the transcoding on my NAS took about 48 hours!

@breezytm
Copy link
Author

breezytm commented May 3, 2019

Hey I was out for a little while dealing with my mothers passing. Glad to see you open a case to get this included.

@clinton-hall
Copy link
Owner

@breezytm
Sorry to hear that. I hope you and your family are doing well.
As soon as I can get either 7zip that handles these, or another iso-parsing tool, I will update this further.

@clinton-hall clinton-hall mentioned this issue May 31, 2019
7 tasks
clinton-hall added a commit that referenced this issue May 31, 2019
* added code to extract bluray images and folder structure. #1588

* add Mounting of iso files as fall-back

* add new mkv-bluray default.

* clean-up fall-back for ffmpeg not accepting -show error
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants