Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upENH: Add vtkMRMLStreamingVolumeNode for compressed video #1028
Conversation
This comment has been minimized.
This comment has been minimized.
Looks nice. I wonder if the non-mrml code is a candidate for vtk proper or a vtk remote module. If so it could go into vtkAddons so people can use it in other vtk projects (vtkAddons is already set up as a vtk remote module). Also how does this work with Sequences? Is it easy to convert back and forth? It seems like we should add functionality at the vector volume node level to get a scalar volume version efficiently (basically add the pipelines from VectorToScalar Volume module) |
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
from
8d200d1
to
6b42d04
Oct 11, 2018
This comment has been minimized.
This comment has been minimized.
That's true, the non MRML classes should be able to live in vtkAddons. I've created another extension that I will publish soon that implements reading and writing MKV files using Sequences. Users can record and replay compressed video data that is streamed from OpenIGTLink and save it into an MKV. You can drag and drop any MKV file into Slicer as long as there is a supported codec registered. Here's an example video showing replay in Slicer using a random MKV that I found (960x400, encoded using VP9): |
This comment has been minimized.
This comment has been minimized.
leochan2009
commented
Oct 12, 2018
Hi Kyle, Thanks for pushing the video streaming forward! |
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
2 times, most recently
from
d8af813
to
0d866cf
Oct 12, 2018
This comment has been minimized.
This comment has been minimized.
The non MRML classes have been moved to vtkAddons. |
This comment has been minimized.
This comment has been minimized.
As a side note, |
This comment has been minimized.
This comment has been minimized.
This is an API that will allow us to use any codec, including those that are built into VTK. The API can do much more than simply read frames from file and write to file: it supports random seeking, in-memory editing, recoding, and storing metadata. Regarding VTK's FFMPEG support: FFMPEG is ridiculously large and complicated compared to what we need. We would need to a work a lot to make sure we disable all GPL parts and not use any questionable codecs in ffmpeg, figure out how to build ffmpeg with VP9 on all platforms, and access all the features through ffmpeg's API. So, for now we'll use VP9 codecs directly. |
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
2 times, most recently
from
70a4c1a
to
6246fed
Oct 15, 2018
lassoan
reviewed
Oct 16, 2018
Nice work! I added a number of comments inline, almost all of them just minor improvements or coding/naming style. |
//---------------------------------------------------------------------------- | ||
vtkMRMLNodeNewMacro(vtkMRMLStreamingVolumeNode); | ||
|
||
const int DEFAULT_NUMBER_OF_IMAGEDATACONNECTION_OBSERVERS = 1; |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 16, 2018
Contributor
Do we know which objects keep these observers? It would be nice to add it as comment.
It would be even better to determine this dynamically - can we check the number of observers right after creating the image and store that value?
vtkMRMLNodeNewMacro(vtkMRMLStreamingVolumeNode); | ||
|
||
const int DEFAULT_NUMBER_OF_IMAGEDATACONNECTION_OBSERVERS = 1; | ||
const int DEFAULT_NUMBER_OF_IMAGEDATA_OBSERVERS = 2; |
This comment has been minimized.
This comment has been minimized.
} | ||
|
||
//--------------------------------------------------------------------------- | ||
bool vtkMRMLStreamingVolumeNode::IsImageObserved() |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 16, 2018
Contributor
It would be good if we could include the "external" word in this name, for example HasExternalImageObserver.
if (this->Frame) | ||
{ | ||
vtkSmartPointer<vtkImageData> newImageData = vtkSmartPointer<vtkImageData>::New(); | ||
this->SetAndObserveImageData(newImageData); |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 16, 2018
Contributor
This may trigger updates everywhere where this node is observed (and those parts would then find a partially initialized/empty image data). We need to do all updates (preferably not just allocations but image content update) and set the new image data as the very last step.
//---------------------------------------------------------------------------- | ||
bool vtkStreamingVolumeCodecFactory::UnregisterStreamingCodecByClassName(const std::string& codecClassName) | ||
{ | ||
for (unsigned int i = 0; i < this->RegisteredCodecs.size(); ++i) |
This comment has been minimized.
This comment has been minimized.
/// \ingroup Volumes | ||
/// \brief Class that can create compresion device for streaming volume instances. | ||
/// | ||
/// This singleton class is a repository of all compression codecs for compressing volume . |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 16, 2018
Contributor
/// This singleton class is a repository of all compression codecs for compressing volume . | |
/// This singleton class is a repository of all compression codecs for compressing volume. |
vtkStreamingVolumeCodec* CreateCodecByFourCC(const std::string codecFourCC); | ||
|
||
/// Returns a list of all registered Codecs | ||
const std::vector<vtkSmartPointer<vtkStreamingVolumeCodec> >& GetStreamingCodecClasses(); |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 16, 2018
Contributor
It would be a bit nicer and safer if we returned the class names instead of classes themselves
{ | ||
public: | ||
|
||
enum |
This comment has been minimized.
This comment has been minimized.
vtkTypeMacro(vtkStreamingVolumeFrame, vtkObject); | ||
void PrintSelf(ostream& os, vtkIndent indent) VTK_OVERRIDE; | ||
|
||
vtkSetMacro(FrameType, int); |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
Could you please move documentation from member variables to get/set functions?
The reason is that when API documentation is generated, protected/private members (and their documentation) may not show up in the generated pages.
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
from
6246fed
to
3b6afff
Oct 17, 2018
This comment has been minimized.
This comment has been minimized.
@lassoan Changes made! |
lassoan
reviewed
Oct 17, 2018
/// 3. virtual std::string UpdateParameterInternal(std::string parameterName, std::string parameterValue); | ||
/// 4. virtual std::string GetFourCC(); | ||
/// 5. virtual std::string CreateCodecInstance(); // This can be overridden using vtkCodecNewMacro(className); | ||
class VTK_ADDON_EXPORT vtkStreamingVolumeCodec : public vtkObject |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
We should add a Raw or Trivial codec that just copies the data. It would serve as an example, it could be also used for testing, and for allowing reading/writing uncompressed data with the same API as compressed data.
lassoan
reviewed
Oct 17, 2018
/// For more information on frame types see: https://en.wikipedia.org/wiki/Video_compression_picture_types | ||
enum | ||
{ | ||
IFrame, // Uninterpolated keyframe |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
IFrame, // Uninterpolated keyframe | |
IFrame, ///< Uninterpolated keyframe |
lassoan
reviewed
Oct 17, 2018
enum | ||
{ | ||
IFrame, // Uninterpolated keyframe | ||
PFrame, // Frame interpolated from previous frames |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
PFrame, // Frame interpolated from previous frames | |
PFrame, ///< Frame interpolated from previous frames |
lassoan
reviewed
Oct 17, 2018
Thank you, it is almost there. Apart from the small nitpicks there would be one more important thing to do: add a simple test that exercises basic mechanisms (it serves as an example and makes sure the classes work as intended). Sorry for keep asking things, I should have no more extra requests. |
{ | ||
IFrame, // Uninterpolated keyframe | ||
PFrame, // Frame interpolated from previous frames | ||
BFrame, // Frame interpolated from previous and forward frames |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
BFrame, // Frame interpolated from previous and forward frames | |
BFrame, ///< Frame interpolated from previous and forward frames |
vtkTypeMacro(vtkStreamingVolumeFrame, vtkObject); | ||
void PrintSelf(ostream& os, vtkIndent indent) VTK_OVERRIDE; | ||
|
||
vtkSetMacro(FrameType, int); |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
Could you please move documentation from member variables to get/set functions?
The reason is that when API documentation is generated, protected/private members (and their documentation) may not show up in the generated pages.
vtkSetMacro(FrameType, int); | ||
vtkGetMacro(FrameType, int); | ||
|
||
void SetFrameData(vtkUnsignedCharArray* frameData) { this->FrameData = frameData; }; |
This comment has been minimized.
This comment has been minimized.
void SetFrameData(vtkUnsignedCharArray* frameData) { this->FrameData = frameData; }; | ||
vtkUnsignedCharArray* GetFrameData() { return this->FrameData; }; | ||
|
||
void SetPreviousFrame(vtkStreamingVolumeFrame* previousFrame) { this->PreviousFrame = previousFrame; }; |
This comment has been minimized.
This comment has been minimized.
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
from
3b6afff
to
d387cbd
Oct 17, 2018
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
from
d387cbd
to
1ff2055
Oct 17, 2018
This comment has been minimized.
This comment has been minimized.
@lassoan Done! |
lassoan
reviewed
Oct 17, 2018
vtkSmartPointer<vtkMRMLStreamingVolumeNode> streamingVolumeNode1 = vtkSmartPointer<vtkMRMLStreamingVolumeNode>::New(); | ||
streamingVolumeNode1->SetCodecFourCC("RV24"); | ||
streamingVolumeNode1->SetAndObserveImageData(imageData1); | ||
if (streamingVolumeNode1->GetFrame()) |
This comment has been minimized.
This comment has been minimized.
} | ||
streamingVolumeNode1->EncodeImageData(); | ||
|
||
vtkSmartPointer<vtkStreamingVolumeFrame> frameData = vtkSmartPointer<vtkStreamingVolumeFrame>::New(); |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
it is not useful to initialize frameData if you overwrite it in the next line
|
||
vtkSmartPointer<vtkStreamingVolumeFrame> frameData = vtkSmartPointer<vtkStreamingVolumeFrame>::New(); | ||
frameData = streamingVolumeNode1->GetFrame(); | ||
if (!frameData) |
This comment has been minimized.
This comment has been minimized.
|
||
// Calling GetImageData should decode the frame | ||
vtkSmartPointer<vtkImageData> imageData2 = streamingVolumeNode2->GetImageData(); | ||
if (!imageData2) |
This comment has been minimized.
This comment has been minimized.
EXERCISE_ALL_BASIC_MRML_METHODS(node1.GetPointer()); | ||
|
||
|
||
vtkSmartPointer<vtkStreamingVolumeCodecFactory> factory = vtkStreamingVolumeCodecFactory::GetInstance(); |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
We don't use factory explicitly. Is it necessary to instantiate it then? We can keep it but then use it (e.g., check if there are codecs registered, etc).
vtkStreamingVolumeFrame.h | ||
vtkStreamingVolumeCodecFactory.cxx | ||
vtkStreamingVolumeCodecFactory.h | ||
vtkRGBVolumeCodec.cxx |
This comment has been minimized.
This comment has been minimized.
lassoan
Oct 17, 2018
Contributor
It looks like some actual codec. It would be better to include uncompressed, raw, trivial, simple, plain or some similar word in the name.
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
3 times, most recently
from
87496ee
to
94606cd
Oct 17, 2018
This comment has been minimized.
This comment has been minimized.
@lassoan Ok, changes made! |
This comment has been minimized.
This comment has been minimized.
Thanks a lot @Sunderlandkyl - this looks good now! @jcfr Would it be possible to include this in the Slicer-4.10.0? |
This comment has been minimized.
This comment has been minimized.
Since the version has already been updated in the repo, and release on macOS and Windows is in progress. It will have to wait 4.10.1. |
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
from
94606cd
to
d671664
Oct 18, 2018
This comment has been minimized.
This comment has been minimized.
@lassoan Added GetAvailiableParameterNames() and GetParameterDescription() functions to vtkStreamingVolumeCodec. |
Sunderlandkyl
force-pushed the
Sunderlandkyl:video_streaming
branch
from
d671664
to
72561ea
Oct 23, 2018
This comment has been minimized.
This comment has been minimized.
Thank you very much for your contributions, it's been merged. |
Sunderlandkyl commentedOct 11, 2018
•
edited by cpinter
Co-authored-by: Longquan Chen leochan2009@hotmail.com
Developed based on the changes made by @leochan2009 in #917