Video File Support for Cinematic Cutscenes #849
|
For those who want to make things simpler without using complex Storyboard files, why not add video format support for the intro and story mode cutscenes? GO seems to support something like that! |
Replies: 8 comments 8 replies
|
I like this idea! I got frustrated to capture footage frame by frame. |
|
Also the loading screen.. |
|
Something like that would be Great Cause If so custom framerate for storybiard would be Fantastic too |
|
Oh hey, this has a discussion already, saves me some time! Writing some notes here since this came up in the Discord tonight: So one of our developers (Sam) actually did put together a proof of concept branch a while ago that implements MP4 support, and code-wise it doesn't actually seem too hard to update it for our current codebase. 2022-10-25_01-43-18.mp4The main reason this wasn't finished up and made into a PR it seems (outside of Sam being inactive as far as Ikemen goes), is due to the size of the FFMPEG library that it used to decode video, and the resulting colossal number of codecs it supports. To quote the man himself:
I haven't built this proof-of-concept myself to see if it's actually that bad, but I wouldn't be surprised. I would definitely be down to look into seeing if I can figure out a way to limit what codecs we support and get this cleaned up and PR'd down the line, because I imagine a lot of creators are gonna find proper video support to be a killer feature. (Speaking as a video editor myself, I certainly agree!) |
|
Restricting the number of codecs has only advantages I think. Less code bloat. Less to maintain. |
|
Agreed. I'm in the camp of limiting it to maybe H.264 for video and AAC and MP3 for audio since those are what I typically use in the vast majority of my video files. Just not sure how to actually go about limiting that given how Sam's fork currently implements video decoding with an external lib. |
|
Me from the future here! (season's greetings, I'm writing this on Christmas Day LOL) Just now tried to build Sam's PoC after updating it to latest nightly and, unfortunately wasn't able to actually get anything to build. Something in Reisen seemed to really not like being compiled under Windows; depending on the version it either complains about a missing While I'm here, someone also mentioned this AV1 utilities library (among others; the same repo also contains a similar library for MP4s) that could theoretically be made into a video decoder. Unfortunately, I wasn't able to find any real documentation on this library or any simple examples (the only program that uses it is a web-streaming server, akin to Plex or Jellyfin). If there happens to be someone who has the know-how to turn this library into something that can spit out RGBA video frames from an input file like what Reisen does, this'd be pretty straightforward to do. |
|
Hello, I'm a pretty novice coder so my knowledge is pretty spotty But, I managed to get a version of Reisen to build with Ikemen on Windows, so I wanted to update people on that. To build I used Zimwips Reisen fork version 1.0.11. I had to edit the code so it would build, by removing lines 10-12 from "platform_windows.go" and according to Visual Studio Code, that function never got called anyways so it SHOULD be fine? (Novice coder opinion) After that and using the original tutorial to build Reisen for Windows I found that the program requires 97 .DLL's to be placed in the same directory as the .EXE file. The .DLL's add up to 125 MB of space. I personally am not sure if they could be moved to a folder where they would be less obtrusive. So that's 2 reasons why using the previously mentioned AV1 utilities library might be a better option. Also I wasn't actually able to get Sam's proof of concept working again, as the Ikemen code has since changed, and I don't understand stage rendering enough to get it working again. I'll keep trying but I'm not super confident in my skills. If anyone else wants to take a crack at it and needs help building on Windows I could try and help. Hopefully this information is helpful. UPDATE: So I got lucky and actually figured out how the code works KINDA. I wasn't able to get the background/stage video code working again. But I did write some LUA hooks in script.go and was able to launch videos via LUA script like a Story board. I also edited storyboard.lua and added code that checks if the path its being passed is a .Def or not. If not it assumes its a video and tries to play it as a video so I guess that means that any storyboard can be replaced with a video? I haven't tested it fully, but I know it works with intros. Also some interesting errors occurred. The first was that if the video had a different sample rate then the game it would cause audio issues, usually slowed down and pitched down audio. It was fixable by changing the games sample rate, changing the videos sample rate in hand break, or (probably the best option) using the "beep.Resample" function. To summarize, the code seems to still work, and I have made modifications to it to make it work differently and for it to auto resolve some of the issues I've encountered. BUT, I didn't find a way to do this with the AV1 utilities library, nor did I find a way to shrink the ffmpeg .Dll's so... not sure how helpful that all is. If my code would help I can share it, either with like a paste bin or by making an Ikemen fork (I don't actually know how to make a fork but I can hopefully figure it out.) or something. |
|
Good work! I imagine the bulk of those DLLs are just FFMPEG codecs, so if there's a way to rebuild FFMpeg and its DLLs to only support a subset (H.264 + MP3 or AAC audio, probably?), that would cut down massively on both file size and DLL count. |
|
Is video support added? I see nightly build has some commits fix: FFmpeg 7.1 and 8.0 support |
|
It seems so, at least as far as videos in the backgrounds go. I dunno about the other requested functionality. |
Oh my this is revolutionary. Instead of having 90-100 PNGs that can take the SFF 90 MB, a video can be much less like under 20 MB with similar compression. Not to say for story, good timing because I would otherwise have had to make a video and turn into sprite sequence when it can be a video. |
|
Yeah it's really cool! Were you able to find a way to set the videos as intros or storyboard files? I Haven't tried that yet because I got to excited and wanted to see if my edits to script.go would still work, and turns out they do after minor editing! the code lets you use videos as intros and storyboard files. not loading screens as I dunno how to interface with those. But compared to what you can do with the videos now I think its a little outdated. I wonder if I should update it and share it. I wonder how I could share it (I'm new and don't know what I'm doing) |
|
I also saw that video support was added, was I helpful with that? It would be cool if I was! Also did you guys run into the same Frame rate and Sample Rate problems I did? were you able to fix them? if not I can share my solutions. If sharing the code I wrote to make the videos play via LUA would help I would gladly do that, but if you got it than you got it. Besides all that, thanks for doing all that! it's a cool new feature! |
Yes, videos work as a new background type, so screenpacks, stages, and storyboards are fully supported.
Yes, I also encountered those problems in my implementation, but I was able to fix them.
It's possible to play videos when triggered from Lua via the existing So overall, video support in its current state is pretty much finished. There may still be some rough edges, so feel free to report any bugs if you find one. |
|
Oh I see now! That's amazing! I didn't even know you could use backgrounds in storyboards like that, I have more to learn. It finally clicked when I saw that example intro.def in "the-dev-room" on the discord, so I could stand to check that more as well. Thank you for the help, and thank you for doing all this! |
It seems so, at least as far as videos in the backgrounds go. I dunno about the other requested functionality.