This repository has been archived by the owner. It is now read-only.

BufferedOutputFileImpl corrupt ? #56

Open
wants to merge 2 commits into
from

Conversation

Projects
None yet
3 participants
@ms777

ms777 commented Jun 2, 2012

Hi,

I have been trying to view DVD disc structures on my hard disk with your excellent software ...

I am using a Windows7 PC, 4 physical cores, and XBMC on the same computer for testing

The only changes to the standard configuration I made are
MEncoder Expert Settings: container == iso :: -ovc copy -oac copy -forceidx -nosync
Common Transcode Settings: Chapter #Transcode# folder support / Interval in minutes: 2 (and active)

My XMBC actions:
I select the 00:00 chapter => everything is fine
I select the 02:00 chapter => sometimes the wrong part of the movie is played
I again select the 00:00 chapter => always the wrong part of the movie is played. After a while, the correct movie begins

To identify the root cause, I added logging of the read and write actions in BufferedOutputFileImpl (see the push request). All is written to a folder ps3_log on the desktop. I did not change anything in the logic od BufferedOutputFileImpl.
The ... write.dat are perfectly playable on MediaplayerClassic, and they contain what I expected to see.
The ... read.dat are perfectly playable on MediaplayerClassic, and they contain what I actually saw in XBMC.

BufferedOutputFileImpl, especially the buffer increase logic, and the dealing with offset in the write functions, are not easy to understand ... any help for further investigation would be very helpful

Cheers,

ms777

@Raptor399

This comment has been minimized.

Show comment Hide comment
@Raptor399

Raptor399 Jun 2, 2012

Member

Yes, BufferedOutputFileImpl is pretty complicated. If I remember correctly there already is a way to write a copy of the transcoded stream to a file....

Ah, yes. Take a look at PipeIPCProcess.java. If you add a line like debug = new FileOutputStream("debug.out");, you'll get a copy of the stream.

I found that useful to be able to compare the output of a manual command line execution and what PMS produces.

Member

Raptor399 commented Jun 2, 2012

Yes, BufferedOutputFileImpl is pretty complicated. If I remember correctly there already is a way to write a copy of the transcoded stream to a file....

Ah, yes. Take a look at PipeIPCProcess.java. If you add a line like debug = new FileOutputStream("debug.out");, you'll get a copy of the stream.

I found that useful to be able to compare the output of a manual command line execution and what PMS produces.

@ms777

This comment has been minimized.

Show comment Hide comment
@ms777

ms777 Jun 2, 2012

Hi Raptor399,

in BufferedOutputFileImpl there is a debug.out, which is activated. There you get a copy of what is written to the buffer.

The problem is that what you read from the buffer is not the same as what you write to it. You can see that from running the modified BufferedOutputFileImpl .java under the condiditions I described.

I believe something is wrong with the line
int mb = (int) (readCount % maxMemorySize);
in the write function. This assumes that the buffer always was maxMemorySize. This probably does not fail, when only one file is MEncoded and buffered at a time. Under XBMC, the old stream continues, while you browse for the next chapter, so there will be at least a short time, where there is not enouigh memory to grow the buffer to maxMemorySize.

ms777 commented Jun 2, 2012

Hi Raptor399,

in BufferedOutputFileImpl there is a debug.out, which is activated. There you get a copy of what is written to the buffer.

The problem is that what you read from the buffer is not the same as what you write to it. You can see that from running the modified BufferedOutputFileImpl .java under the condiditions I described.

I believe something is wrong with the line
int mb = (int) (readCount % maxMemorySize);
in the write function. This assumes that the buffer always was maxMemorySize. This probably does not fail, when only one file is MEncoded and buffered at a time. Under XBMC, the old stream continues, while you browse for the next chapter, so there will be at least a short time, where there is not enouigh memory to grow the buffer to maxMemorySize.

@ms777 ms777 closed this Jun 2, 2012

@ms777 ms777 reopened this Jun 2, 2012

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Jul 9, 2012

This one is only solution how to get FF/RW working what I found.
Newer PMS has already fixed vaule (37000) because 15000 is very CPU intensive also on quad core.
15000 = 30-35 Mbit
40000 = 80 Mbit
35000 = 70 Mbit
37000 = 75 Mbit
When disabled, non timeseek renderes (like PanTV) can't FF/RW. When enabled, time is converted to byte, CBR used instead of VBR so FF/RW works because position can be predicted.
I have to say that PanTV is very sensitive to size of "TranscodedVideoFileSize". Check my findings from the past at the end of Panasonic.conf.
When tested, there was no time2byteseek conversion so results are without it (but have to say latest version of Panasonic.conf file works for me including FF/RW except visible actual time position)

This one is only solution how to get FF/RW working what I found.
Newer PMS has already fixed vaule (37000) because 15000 is very CPU intensive also on quad core.
15000 = 30-35 Mbit
40000 = 80 Mbit
35000 = 70 Mbit
37000 = 75 Mbit
When disabled, non timeseek renderes (like PanTV) can't FF/RW. When enabled, time is converted to byte, CBR used instead of VBR so FF/RW works because position can be predicted.
I have to say that PanTV is very sensitive to size of "TranscodedVideoFileSize". Check my findings from the past at the end of Panasonic.conf.
When tested, there was no time2byteseek conversion so results are without it (but have to say latest version of Panasonic.conf file works for me including FF/RW except visible actual time position)

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Jul 9, 2012

Ignored when MediaInfo = true (default)

Ignored when MediaInfo = true (default)

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Jul 9, 2012

Never understood why "-endpos" is used??? When you rewind or play chapters, you don't need -endpos but -startpos only.
We alwyas want to play it to the end:-)

Never understood why "-endpos" is used??? When you rewind or play chapters, you don't need -endpos but -startpos only.
We alwyas want to play it to the end:-)

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Jul 9, 2012

Is it a specific workaround for Panasonic TV only or you found a bug so this fix works widely with all renderers?

Is it a specific workaround for Panasonic TV only or you found a bug so this fix works widely with all renderers?

@ms777

This comment has been minimized.

Show comment Hide comment
@ms777

ms777 Jul 9, 2012

Owner

Hi,

I am new to this and did this mainly for me, so please excuse any shortcomings ...

I use PMS only for watching DVD folders from my harddisk on my Panasonic TV. My connection is through powerline, so speed is well above what I need to watch without any compression. For quick testing I use XBMC on the same machine as PMS.

This is why I added the following to the MEncoder specific expert setting:
container == iso :: -ovc copy -oac copy -forceidx -nosync

Due to the non working FF and the lack of jumps to chpters / times, in the Common Transcode Settings I set the chapters interval to two minutes.

I observed the following:

  • last chapter did not play to the end
  • going to some chapters brought me to sometimes unpredictable places in the movie
  • often the movie stopped in the middle

I observed all kind of failures both on Panasonic and some of the failures (much reduced) on the XBMC.

For diagnosis, I wrote the RequestTestHandler.java and installed it into HttpServerPipelineFactory.java. This dumps all http traffic to disk.
I also modified BufferedOutputFileImpl.java to dump all input to and output from the buffer to disk. This was in an earlier commit (c4cdde8). By comparing the data written to the buffer (correct movie) and read from the buffer (often arbitrary position in the movie) I found that they were different. This is why I wrote BufferedOutputFileImpl2.java. I believe that the mistake in the old buffer implementation lies in the incorrect tracking of the read position, when the buffer size is increased. Note that (i) I did not yet include the shiftByTimeseek function and (ii) the growBuffer is written and exrternally tested, but not yet used.

With these changes, DVD are played nicely both on Panasonic and XBMC. FF is still missing, though.

One specific thing with Panasonic is that in the panasonic chapter menu, when browsing through the chapters, there is a small preview window showing the movie at chapter position (no still thumbnail or so). This leads to the start of several MEncoder process during browsing. With the old buffer implementation, PMS often was not able to allocate new buffer for the new MEncoder processes, which killed PMS.

So,

  • I believe there is a fundamental problem with the old BufferedOutputFileImpl.java
  • This problem shows up much stronger, when a new buffer is allocated while the old buffer is not yet released
  • This problem should persist under all compression settings. I have tested only -ovc copy -oac copy -forceidx -nosync, though.
  • My solution is not yet finished (no buffer growth implemented, shiftByTimeseek function ignored), but when completed, it should be a bug fix which works widely with all renderers.

I would really appreciate if the original author of BufferedOutputFileImpl.java could have a look into this ...

Cheers ms777

Owner

ms777 commented on 81e4b3f Jul 9, 2012

Hi,

I am new to this and did this mainly for me, so please excuse any shortcomings ...

I use PMS only for watching DVD folders from my harddisk on my Panasonic TV. My connection is through powerline, so speed is well above what I need to watch without any compression. For quick testing I use XBMC on the same machine as PMS.

This is why I added the following to the MEncoder specific expert setting:
container == iso :: -ovc copy -oac copy -forceidx -nosync

Due to the non working FF and the lack of jumps to chpters / times, in the Common Transcode Settings I set the chapters interval to two minutes.

I observed the following:

  • last chapter did not play to the end
  • going to some chapters brought me to sometimes unpredictable places in the movie
  • often the movie stopped in the middle

I observed all kind of failures both on Panasonic and some of the failures (much reduced) on the XBMC.

For diagnosis, I wrote the RequestTestHandler.java and installed it into HttpServerPipelineFactory.java. This dumps all http traffic to disk.
I also modified BufferedOutputFileImpl.java to dump all input to and output from the buffer to disk. This was in an earlier commit (c4cdde8). By comparing the data written to the buffer (correct movie) and read from the buffer (often arbitrary position in the movie) I found that they were different. This is why I wrote BufferedOutputFileImpl2.java. I believe that the mistake in the old buffer implementation lies in the incorrect tracking of the read position, when the buffer size is increased. Note that (i) I did not yet include the shiftByTimeseek function and (ii) the growBuffer is written and exrternally tested, but not yet used.

With these changes, DVD are played nicely both on Panasonic and XBMC. FF is still missing, though.

One specific thing with Panasonic is that in the panasonic chapter menu, when browsing through the chapters, there is a small preview window showing the movie at chapter position (no still thumbnail or so). This leads to the start of several MEncoder process during browsing. With the old buffer implementation, PMS often was not able to allocate new buffer for the new MEncoder processes, which killed PMS.

So,

  • I believe there is a fundamental problem with the old BufferedOutputFileImpl.java
  • This problem shows up much stronger, when a new buffer is allocated while the old buffer is not yet released
  • This problem should persist under all compression settings. I have tested only -ovc copy -oac copy -forceidx -nosync, though.
  • My solution is not yet finished (no buffer growth implemented, shiftByTimeseek function ignored), but when completed, it should be a bug fix which works widely with all renderers.

I would really appreciate if the original author of BufferedOutputFileImpl.java could have a look into this ...

Cheers ms777

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Jul 10, 2012

Contributor

Ok, long time ago I tested DVD so can't say now if FF/RW is possible as for other formats (MKV/DivX etc.) which works for me.
About the random skips I thought SubJunk found the culprit but it was reverted later...UniversalMediaServer/UniversalMediaServer@c0fe182
What I remember from past, only problem what I had was that titles were duplicated when browsed DVD folder more than ones.
Hopefully you will finish this bufferring fix so everybody will be happy.
Great work!
Thx

Contributor

ExSport commented Jul 10, 2012

Ok, long time ago I tested DVD so can't say now if FF/RW is possible as for other formats (MKV/DivX etc.) which works for me.
About the random skips I thought SubJunk found the culprit but it was reverted later...UniversalMediaServer/UniversalMediaServer@c0fe182
What I remember from past, only problem what I had was that titles were duplicated when browsed DVD folder more than ones.
Hopefully you will finish this bufferring fix so everybody will be happy.
Great work!
Thx

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Jul 12, 2012

Contributor

So good luck, until now nobody had itch to look into buffering code:-)
Seems promissing!
ExSport

Contributor

ExSport commented Jul 12, 2012

So good luck, until now nobody had itch to look into buffering code:-)
Seems promissing!
ExSport

@ExSport

This comment has been minimized.

Show comment Hide comment
@ExSport

ExSport Aug 5, 2012

Contributor

Any progress on this?
Thx

Contributor

ExSport commented Aug 5, 2012

Any progress on this?
Thx

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.