Vapoursynth?

[cyaoeu]

Not a real issue with this project but could maybe be interesting.

Vapoursynth (http://www.vapoursynth.com) is a video manipulation software that uses python scripts to create video output (raw frames) that you can then pipe into ffmpeg or whatever encoder you like. After that you would need to mux the audio from the original video since vapoursynth doesn't support audio, but this is pretty easy (ffmpeg -i videowithouadio.mp4 -i videowithaudio.mp4 -map 0:v -map 1:a -c copy finalvideo.mp4 or something)

Using it would (theoretically) have some benefits:

• No need to save intermediate images to the disk and then use those for waifu2x, you could do that directly. The interpolation stuff should also be possible to do with vapoursynth functions (and/or plugins).
• Batching/multi threading.
• General speed, I think the vapoursynth plugins for waifu2x are more optimized for video than vanilla waifu2x at least. So would in combination with dandere2x be even faster.
• Ability to optionally do fancy vapoursynth video manipulation stuff like sharpening, edge cleaning, denoising and more.

There are two vapoursynth plugins that I know of:
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-Waifu2x-caffe and
https://github.com/kice/vs_mxnet .
Not sure if it would be easy (or even possible) to hook dandere2x into these but might be interesting to consider at least.

[akai-katto]

Sorry for the late response!

I've thought about and looked into the possibilities, and it's still on the table. I'll leave this open as a suggestion, but my main priority is rolling out bugs and getting a 'dandere2x' out of beta, and then after I can start looking into creating it into a plugin

[Tremeschin]

The only problem with this is the interaction between the Python, Waifu2x and Dandere2x_cpp. It's really tricky to explain here but let's try:

For the cpp part of dandere2x, it waits on three files AFAIK-And-Remember, the compressed_static, compressed_moving and the inputs folders on the workspace directory. Then it reads those files progressively and writes the vectors of the block matching algorithm into the pframe_data.

This is basically the CPP part of the code in a nutshell.

Then comes Python, it takes the pframes files and creates a residual file (the block differences between frame N and N+1) based on that pframe and inputs files and saves them into residual_images.

Waifu2x now upscales the residual_images dir and saves to residual_upscaled.

Python again, take the residual_upscaled and with the frame N in hands make a N+1 frame based on the blocks that were changed.

As you can see this process is very reliable on interactions between three different programs and you can probably tell it'll be easy to pipe everything from a program to another with ease.

Well that's technically true but it'll be some hard work since we would need to "port" Waifu2x and really remake the I/O structure of Dandere2x_cpp to accept pipes.

To be honest I haven't understood much about Vapoursynth and/or found it usefull (not the best word here) for current dandere2x structure or how it really works. I'm probably so wrong here but let's move on.

I thought about making temporary ramdisks to make the workspace dir "on the fly" directly into RAM but setting some limits in N frames ahead of the waifu2x process.

I did got ffmpeg piping working only on the merged content because it's being "processed" with Python and it's the final step in dandere2x.

It's on my experimental branch right now however this will be a feature of a future dandere2x pull request since it worked flawlessly, just needs some more research from aka and me into it.

Someday sometime I'll take another look at Vapoursynth when things get less nightly on d2x development.

Ah, if we could only pipe things to Waifu2x (preferably ncnn-vulkan) or / and d2x_cpp this will be solved forever!!