More Import Plugins for Animation Software

[steveway]

So, Papagayo-NG is getting pretty good.
But to attract users this needs to be integrated in a workflow.
I've created a new save format, .pg2 which saves the project as a json object.
This should be easier to load and integrate into other software.
I've begun creating a script for blender (2.8) that can load these files and populates GreasePencil Objects with Layers and frames.
https://github.com/steveway/PapagayoNGPlugins
This is more of a prove of concept, creates only empty keyframes, has no UI etc...
But look at this filled timeline:

If anyone wants to extend that and create a real usable blender addon, go ahead.
Plugins for other applications would also be great if anyone wants to help.
EDIT: Moved Code from Gist to its own Repository.

[steveway]

Hi, I just updated this to a pretty much fully functional plugin.

[steveway]

For my Blender Greasepencil Plugin up there.
I did a small test today and added a list of used phonemes to the GUI of the Plugin.
Apparently the empty frames from my tests were a bug in the old Blender or I already fixed those back in September 2019 because with Blender 2.93 this works now!
I made a small video showing the function of this plugin.

papagayo_greasepencil_plugin.mp4

There is some wild gesturing in the middle after I click on "Select Papagayo-NG Project File", that is the file select dialog which OBS did not record.
There seems to be some lag with the playback, noticeable with the sound playing at the end, that is partially because I still have the Onion Skin enabled during Playback.
For now it only works with .pg2 files, but changing this to load .json export files correctly should not be too difficult.

[aziagiles]

@steveway wow. looks like a cool addon

[steveway]

No problem. you should now be able to also use the .json Files from Papagayo-NG. I updated it just now.

[aziagiles]

@steveway awesome

[steveway]

I just tested rendering that scene from my video earlier to a video animation directly with the sound.
On the resulting video the timing is correct, the lag is just in the preview in Blender.

animation_test_lame.mp4

There are for sure more improvements needed but for a first implementation most things are there.

[aziagiles]

@steveway wow. Did you use the Allosaurus breakdown for this?

[steveway]

That is still using the old handmade breakdown in this case.
But with Allosaurus and using the CMU39 phonemes it should produce an even more exact result.
And hopefully if you convert that to preston blair or the others the results should still look good using the new conversion.

[eameres]

OK, with Blender integration you definitely got my attention back! One of the only reasons I keep after effects around was for this purpose!

[steveway]

Great! I'll be glad if more people use this and Papagayo-NG!
Alright, I updated the Plugin a bit.
It should now try to load the sound file if you use a .pg2 file.
Also to help with the lag during playback which you saw in my video here, there are some sync settings for playback which you can change.
I would recommend setting it to "Sync to Audio":

[steveway]

Alright, this seems to be quite usable now.
I did some more testing in Blender.
You can animate moving the resulting animation for example.
So your talking character doesn't have to stand still, you can make it move around and other stuff without manually editing every keyframe.

[aziagiles]

Awesome, let me download the updated version and give it a try.

[steveway]

Alright, after some more testing I have some ideas for improvement.
The Plugin is already preparing a few things.
It creates a Greasepencil object for every voice with Layers for the used Phonemes.
I think it would make sense to create a collection in the scene for every voice and then to associate the corresponding Greasepencil Objects with them.
I think that should make some more sense this way.
I think that will be the next change I will do to the plugin.

[aziagiles]

Ok. that will be great.

[aziagiles]

Aww. I just downloaded the updated version, but it didn't work. I guess there is a bug.

[steveway]

That is strange. Can you tell what happens when you try to use this?

[aziagiles]

It worked yesterday with the version of yesterday, but today when I downloaded the upgraded version, there were issues. Anyways, let me re-download and try.

[aziagiles]

It's OK. It works now.

[aziagiles]

But I actually don't know how to use the addon properly. Please, can you provide a detail tutorial on the addon? I get stuck at the level of creating Grease pencil objects and adding to timeline.

[steveway]

Alright, I made some more updates.
Now it should be much more usable.
A few quick steps to get started:

• Create a new 2D Animation Project in Blender.
• Pull the Plugin Panel out, you can see in my video above where it is hidden by Blender.
• Click on the first button and select your prepared .pg2 or .json file from Papagayo-NG.
• Click on the "Create Grease Pencil Objects" Button.
• In the Scene Collection at the top right you should now have some Stroke Objects, and these already have Greasepencil Objects with the needed Layers for the phonemes added.
  
• If you click on those you can see on your timeline that these get selected.
• Now draw your mouthshapes onto the corresponding layers on frame 0, the keyframes should already exists there.
• After you've drawn the shapes you can press "Apply to Timeline" to combine these into a new layer.
• For playback, hide the Original layers (the eye symbol) and then press play.
• If you used a .pg2 file the sound should already be loaded and it should play together with the animation, otherwise add it like I've shown above.
  This should get you started for the beginning, there will be likely more improvements to the Plugin but for now it's kinda usable.

[steveway]

Alright, I added the sound file path to the .json export files.
The Papagayo-NG from my master will now write that extra info to the .json files on export.
And this plugin will now load the sound also from that.
I will likely add a checkbox so you can choose if it should try to load the sound on it's own.

[aziagiles]

That looks awesome. Let me give it a try.

[steveway]

I added the mentioned checkbox, the Plugin is now available in its own Repository instead of in a Gist:
https://github.com/steveway/PapagayoNGPlugins
EDIT: Renamed that repository, added the Plugin @Hunanbean updated as a submodule.

[aziagiles]

Ok

[aziagiles]

Ok I just tested it, but the ''PapagayoNGLipSync Importer for Blender'' addon file is absent. The directory containing it is empty.

[Hunanbean]

Very cool! That will make it much easier in the future for everyone. Thank you

@aziagiles I am not sure what the arrow symbol on the folder means, perhaps a symbolic link of sorts, but the files seem present in that folder at this point

Edit : Oh, the arrow is only present on the lipsync importer, but the files for both Grease Pencil plugin and Lipsync seem to be present now

[aziagiles]

@Hunanbean ok.that's awesome.

[steveway]

That arrow is because that is included as a submodule.
This way it just points to the other repository, that saves copying the files into several places and should help keep everything up to date.
If you download using git I believe on many versions of git you need to add the --recursive or the --recurse-submodules flag.
Apparently new versions of git will do this automatically.

I think a Plugin for Krita would be nice too.
You can create Plugins using Python to Krita nowadays and you can even use QT to integrate it into the GUI.
In the newest Beta for Krita 5 they introduced "Clone Frames", this sounds useful for this:
https://invent.kde.org/graphics/krita/-/merge_requests/469
Apparently you can only clone inside of a layer and not across layers.
But that shouldn't be a big problem, in that case we first duplicate from the original layer to the combined layer as they occur and then on that layer multiple occurrences of the same phoneme can be clones of each other.
This page seems to have some nice information about creating Plugins:
https://scripting.krita.org/lessons/introduction

[steveway]

Alright, I have a rough framework for a Krita Plugin.
There is no real functionality yet, only a test label.
But that should help with starting a real plugin.

[aziagiles]

@steveway Great. That should be awesome.

[steveway]

Alright, I've update the Krita Plugin.
It kinda works now.
I had to throw in some delays and some QApplication.processEvents() commands so it will not crash or hang.
It's slightly hacky, at the moment we are using the action command system to create and copy frames around.
This is because :

Currently, the scripting framework does not give access to the animation features.
As soon as there are some more methods for creating and copy-pasting frames this should be more stable.

[steveway]

About the Blender Plugin.
For more than a static Animation we want to move the mouth around.
This can be done pretty easily via an armature.
I made a little example video where I created an armature and assigned that to the resulting mouth layer.
If you move that around it will then move the mouth, even rotation works (not shown in the video).

greasepencil_armature.mp4

So, if you have a full character with an armature and bones, you should be able to assign this layer to the "mouth bone".
I guess I will add some functions to create or select existing bones and connect them from the Plugin.
In this example I connected the mouths to a specific bone which moves rotates around in 3D space while the main armature moves around as before in 2D Space. As you can see this works pretty well for animation.

2021-09-24.11-17-47.mp4