This repository contains various nodes for supporting Deforum-style animation generation with ComfyUI. I created these for my own use (producing videos for my "Alt Key Project" music - youtube channel), but I think they should be generic enough and useful to many ComfyUI users.
I have demonstrated the use of these custom nodes in this youtube video.
You can install Dream Project Animation Nodes using the ComfyUI Manager.
Run within (ComfyUI)/custom_nodes/ folder:
- git clone https://github.com/alt-key-project/comfyui-dream-project.git
- cd comfyui-dream-project
Then, if you are using the python embedded in ComfyUI:
- (ComfyUI)/python_embedded/python.exe -s -m pip install -r requirements.txt
With your system-wide python:
- pip install -r requirements.txt
Finally:
- Start ComfyUI.
After startup, a configuration file 'config.json' should have been created in the 'comfyui-dream-project' directory. Specifically check that the path of ffmpeg works in your system (add full path to the command if needed).
When upgrading, it is good to re-run the pip install command as specified in the install section. This will install any new dependencies.
Setting this to true will enable some trace-level logging.
Sets the output file extension and with that the envelope used.
Path to the ffmpeg executable or just the command if ffmpeg is in PATH.
The arguments sent to FFMPEG. A few of the values are provided by the node:
- %FPS% the target framerate
- %FRAMES% a frame ionput file
- %OUTPUT% output video file path
Sets the encoding quality of jpeg images.
Sets the name of the top level category on the menu. Set to empty string "" to remove the top level. If the top level is removed you may also want to disable the category icons to get nodes into existing category folders.
Flags to add a icon before and/or after the category name at each level.
Flags to add an icon before and/or after the node name.
Each key defines a unicode symbol as an icon used for the specified category.
This factor allows changing the bitrate to better fit the required quality and codec. A value of 1 is typically suitable for H.265.
Codec names as specified by ffmpeg. Some common options include "libx264", "libx264" and "mpeg2video".
Increasing the number of encoding threads in mpegCoder will generally reduce the overall encoding time, but it will also increase the load on the computer.
Sets the output file extension and with that the envelope used.
Sets the max-b-frames parameter for as specified in ffmpeg.
These are some concepts used in nodes:
The frame counter is an abstraction that keeps track of where we are in the animation - what frame is rendered and how does the current frame fit into the current animation.
A curve is simply a node that produces a value based on the frame counter (changing over time).
A palette is a collection of color values.
A sequence is a full set of animation frames and a corresponding timeline for these frames. The sequence is created by the 'Image Sequence Saver' node and it may be used to trigger post processing tasks such as generating the video file using ffmpeg. These nodes should be seen as a convenience and they are severely limited. Never put sequence nodes in parallel - they will not work as intended!
Output brightness, red, green and blue averages of a palette. Useful to control other processing.
Beat pattern curve with impulses at specified beats of a measure.
Switch nodes for different type for up to ten inputs.
Converts a boolean value to two different numeric values.
Weighted text prompt builder utility. Chain any number of these nodes and terminate with 'Finalize Prompt'.
Mathematical calculation node. Exposes most of the mathematical functions in the python math module, mathematical operators as well as round, abs, int, float, max and min.
Analyses two palettes and produces the quotient for each individual channel (b/a) and brightness.
CSV input curve where first column is frame or second and second column is value.
CSV output, mainly for debugging purposes. First column is frame number and second is value. Recreates file at frame 0 (removing and existing content in the file).
Utility for calculating good width/height based on common video dimensions.
Post processing for animation sequences calling FFMPEG to generate video files.
Post processing for animation sequences using the python module mpegCoder with ffmpeg library to generate video files.
Finds the number of files in a directory matching specified patterns.
Logging for float/int/string values.
Simple utility to calculate number of frames based on time and framerate.
Directory backed frame counter, for output directories.
Integer value used as frame counter. Useful for testing or if an auto-incrementing primitive is used as a frame counter.
Extracts information from the frame counter.
Adds an offset (in frames) to a frame counter.
Adds an offset in seconds to a frame counter.
Adjusts the brightness of an image by a factor.
Allows changing the colors of an image with a multiplier for each channel (RGB).
Adjusts the contrast of an image by a factor.
Node supporting zooming in/out and translating an image.
Post processing for animation sequences blending frame for a smoother blurred effect.
Loads a frame from a directory of images.
Saves a frame to a directory.
Post processing for animation sequences generating blended in-between frames.
Super-charged number generator for experimenting with ComfyUI.
Merges multiple log entries (reduces noodling).
The text logging facility for the Dream Project Animation nodes.
Linear interpolation between two values over the full animation.
Generates noise based on the colors of up to nine different palettes, each connected to position/area of the image. Although the palettes are optional, at least one palette should be provided.
Generates noise based on the colors in a palette.
Shifts the colors of one palette towards another target palette. If the alignment factor is 0.5 the result is nearly an average of the two palettes. At 0 no alignment is done and at 1 we get a close alignment to the target. Above one we will overshoot the alignment.
Multiplies the color values in a palette to shift the color balance or brightness.
Randomly samples a palette from an image based on pre-defined areas. The image is separated into nine rectangular areas of equal size and each node may sample one of these.
Randomly samples pixels from a source image to build a palette from it.
Saw wave curve.
Simple sine wave curve.
Single event/peak curve with a slight bell-shape.
Splits a text into tokens by a separator and returns one of the tokens based on a given index.
Triangle wave curve.
Single event/peak curve with triangular shape.
Use an uncompressed WAV audio file as a curve.
Many of the nodes found in 'WAS Node Suite' are useful the Dream Project Animation nodes - I suggest you install those custom nodes as well!
This example should be a starting point for anyone wanting to build with the Dream Project Animation nodes.
Same as above but with added color coherence through palettes.
motion-workflow-with-color-coherence
This flow demonstrates sampling image areas into palettes and generating noise for these areas.
This flow demonstrates prompt building with weights based on curves and brightness and contrast control.
This flow demonstrates use of the Laboratory and Logging nodes.
The call to FFMPEG currently in the default configuration (in config.json) does not seem to work for everyone. The good news is that you can change the arguments to whatever works for you - the node-supplied parameters (that probably all need to be in the call) are:
- -i %FRAMES% (the input file listing frames)
- -r %FPS% (sets the frame rate)
- %OUTPUT% (the path to the video file)
If possible, I will change the default configuration to one that more versions/builds of ffmpeg will accept. Do let me know what arguments are causing issues for you!
The mpegCoder library will always use variable frame rate encoding if it is available in the output format. With most outputs this means that your actual framerate will differ slightly from the requested one.