Data is corrupted when subsampling a large file (8Gb) on windows #15
Comments
|
Hi Jerome, That’s troubling. Just to confirm, if you do this: imagesc(mtData(:, :, 234)); there is no problem, but this: imagesc(mtData(1:5:end, 1:5:end, 234)); doesn’t work? Is there an error message, or is the data simply corrupted? Cheers, |
|
Dear Doctor Dylan, That is exactly what is happening, the data is corrupted after 2gb. Please refer to the attached images. gap = 5; val = 234;
This is what it should look like.
Thanks for the fast response. Best regards, |
|
Hi Jerome, I can't reproduce this with a 3Gb file in Thanks, |
|
Dear Dr. Muir, we started using your framework a while ago to analyze long video files as already mentioned by Jerome. Regarding your questions: Like you suggested, we tested the latets commit from here as well with the same result. Additionally, the effect was reproduced on two PCs with different compilers. From the observations, I would conclude that it is not dependent on the file size. What Jerome did not mention: We tried Windows 7 and Windows 8.1 64bit with Matlab 2015b 64bit. Do you need more information? Best regards, Michael |
|
Hi @mp1988, Thanks for the extra information. It may be a windows-specific issue. Could you please recompile the Thanks, |
|
Dear Dr. Muir, here are the outputs you asked for. Sincerely, pathname = D:\MatlabBa\MichaelVideo\40Hz_DC filename = PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001 mts: Called with command [open] ans = 4×4 uint16 matrix 24385 608 624 656 mts: Called with command [read_chunks] ans = 20×20 uint16 matrix Columns 1 through 16 24385 640 688 672 624 608 736 656 608 592 624 640 640 640 688 656 Columns 17 through 20 |
|
Hi @JeromeB211 @mp1988, I have just pushed an update to a new branch There were a couple of lines where non-64 bit versions of Best regards, |
DylanMuir
changed the title
|
Dear Dr. Muir, Thank you very much for the update. Unfortunately it did not solve the problems we had. The data is even corrupted at the first frame now. Although the values of the corrupted Data changed, they are still not correct. I ran the same code of which i sent you the output last time. Here are the results now. Sincerely , pathname = D:\MatlabBa\MichaelVideo\40Hz_DC filename = PG_Grasshopper3 GS3-U3-23S6M_15355972()_X1920_Y1200_BPP2_LA1_FPS10_TIME2017-01-18-16-57-49_001 mts: Called with command [open] ans = 4×4 uint16 matrix 24385 608 624 656 mts: Called with command [read_chunks] ans = 20×20 uint16 matrix Columns 1 through 16 24385 640 688 672 624 608 736 656 608 592 624 640 640 640 688 656 Columns 17 through 20 |
|
Hi, Sorry it's taken me so long to get back to this. It's difficult to debug because I can't reproduce the problem. When you say "the data is corrupted at the first frame", do you mean the first frame in the stack? Do you mean when skipping pixels (i.e. |
|
Hi,
Here, I am referring to what I wrote last time:
“
What works:
Loading and displaying a video frame by frame .
->Works in branch iss/15
What does not work:
1) Introducing subsampling steps >4.
2) Accessing a time-series (stack) of single pixels behind the 2GB border (of the whole file).
A =Mt(1,1,:)
A (1,1, 10000) -> incorrect value
B =Mt(1,1,10000) -> correct value
“
I cannot say for sure what Jerome meant, but I think he meant the case skipping pixels (i.e. 1:5:20) /subsampling.
Best regards
Michael
Von: Dylan Muir [mailto:notifications@github.com]
Gesendet: Donnerstag, 23. März 2017 10:35
An: DylanMuir/MappedTensor <MappedTensor@noreply.github.com>
Cc: Michael Paul <michael.paul1988@gmx.de>; Mention <mention@noreply.github.com>
Betreff: Re: [DylanMuir/MappedTensor] Data is corrupted when subsampling a large file (8Gb) on windows (#15)
Hi,
Sorry it's taken me so long to get back to this.
When you say "the data is corrupted at the first frame", do you mean the first frame in the stack? Do you mean when skipping pixels (i.e. 1:5:20) or when reading all pixels (i.e. 1:20), or both?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#15 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ_pvg1-gvMrECkPii0C6IfRM_m15tfMks5rojymgaJpZM4L1OsM> .
|
|
Hi, Please pull the latest commit to the mt = MappedTensor(<file details>);
a = mt(1:20, 1, end);
b = mt(1:5:20, 1, end);
[a(1:5:end) b] |
|
Dear Dylan,
I think you fixed it :) ( e = 0 ).
gap = 5
a = video_sequence(1:20, 1, end);
b = video_sequence(1:gap:20, 1, end);
c =[a(1:gap:end) b];
d =a(1:gap:end);
e =sum(d-b)
I also tried to access the tensor video_sequence with gap =32 which worked fine, too.
What did you have to change to make it work?
Best regards,
Michael
Von: Dylan Muir [mailto:notifications@github.com]
Gesendet: Freitag, 24. März 2017 11:02
An: DylanMuir/MappedTensor <MappedTensor@noreply.github.com>
Cc: Michael Paul <michael.paul1988@gmx.de>; Mention <mention@noreply.github.com>
Betreff: Re: [DylanMuir/MappedTensor] Data is corrupted when subsampling a large file (8Gb) on windows (#15)
Hi,
Please pull the latest commit to the iss/15 branch, recompile mapped_tensor_shim.c and try the following:
mt = MappedTensor(<file details>);
a = mt(1:20, 1, end);
b = mt(1:5:20, 1, end);
[a(1:5:end) b]
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#15 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ_pvpHnExVf4pxsKYz5BhNXBFKuQlX9ks5ro5SugaJpZM4L1OsM> .
|
|
Hi Michael, I'm glad this works for you now. I'll pull the changes into the |
Dear Doctor Muir,
I am having a problem displaying a sub-sampled version of a 8gb mapped tensor file with the "imagesc" fnction.
If i choose the x and y gap per frame to be equal or greater than 5, the image will be destroyed at the 234th frame, which is around 2gb of data.
It always happens at the 234th frame, regardless where i start displaying. For example if i start at frame 234 the image will be destroyed immediately.
Data type used was "single".
Trying to display all the frames with a sub-sampling of 4x4 or less will result in no errors at all.
The mex'ed version was used. Trying to use he non mex'ed version did not work.
I would be happy to get your insight into why this is happening.
Best regards,
Jerome
The text was updated successfully, but these errors were encountered: