This is the code for my "This is Fine!" installation, first shown in the Elementen exhibition at Stichting KunstWerkt in Schiedam (NL).
The installation shows visitors of the exhibition engulfed in flames in images created with InvokeAI. It is a reference to the webmeme by the same name, which is based on a webcomic by K. C. Green.
In essence the installation shows images created with InvokeAI, based on the poses and a canny representation of a camera view. Pose estimations are shown in realtime as an overlay over the previously generated image until a timer ends, the pose is frozen and a new image is generated by InvokeAI. The main application has a very simple state machine with three states:
- Showing the latest image generation without distraction
- Looking for poses, showing them as an overlay over the previous image
- Waiting for the next image to finish generating while showing the frozen pose and a "throbber" animation
The application can be run on the same server that runs the InvokeAI instance, or on another server that can be accessed by the server that runs InvokeAI (and vice-versa). It is highly recommended that the application is run from its own virtual environment.
Windows:
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
python downloadlandmarker.py
python fine.py
Linux/MacOS:
python3 -m venv venv
source venv\bin\activate.sh
pip install -r requirements.txt
python3 downloadlandmarker.py
python3 fine.py
The frontend sends an InvokeAI workflow json file from the resources folder to the InvokeAI instance.
FauxpenPoseDetector is a realtime pose detector based on the mediapipe pose estimator, which is transmogrified into an OpenPose-alike result which can be used by the OpenPose Controlnet.
The FauxpenPoseDetector requires the pose_landmarker.task file placed in the resources folder. It can be downloaded from developers.google.com or using the downloadlandmarker.py script (with your choice of a lite, full or heavy model).
The FauxpenPoseDetector can be tested standalone in the commandline:
usage: fauxpenposedetector.py [-h] [--model MODEL] [--numPoses NUMPOSES] [--minPoseDetectionConfidence MINPOSEDETECTIONCONFIDENCE] [--minPosePresenceConfidence MINPOSEPRESENCECONFIDENCE]
[--minTrackingConfidence MINTRACKINGCONFIDENCE] [--fullscreen] [--hideVideo] [--cameraId CAMERAID] [--frameWidth FRAMEWIDTH] [--frameHeight FRAMEHEIGHT]
[--frameDelay FRAMEDELAY]
options:
-h, --help show this help message and exit
--model MODEL Name of the pose landmarker model bundle. (default: resources/pose_landmarker.task)
--numPoses NUMPOSES Max number of poses that can be detected by the landmarker. (default: 1)
--minPoseDetectionConfidence MINPOSEDETECTIONCONFIDENCE
The minimum confidence score for pose detection to be considered successful. (default: 0.5)
--minPosePresenceConfidence MINPOSEPRESENCECONFIDENCE
The minimum confidence score of pose presence score in the pose landmark detection. (default: 0.5)
--minTrackingConfidence MINTRACKINGCONFIDENCE
The minimum confidence score for the pose tracking to be considered successful. (default: 0.5)
--fullscreen Show result fullscreen. (default: False)
--hideVideo Show tracked landmarks on a black background. (default: False)
--cameraId CAMERAID Id of camera. (default: 0)
--frameWidth FRAMEWIDTH
Width of frame to capture from camera. (default: 0)
--frameHeight FRAMEHEIGHT
Height of frame to capture from camera. (default: 0)
--frameDelay FRAMEDELAY
The time in ms to wait between captures (default: 20)
A wrapper that runs the FauxpenPoseDetector in a thread and provides results of the detector to the frontend.
A simple Flask server which serves poses and canny images to InvokeAI, and which handles newly created images. It also contains a small utility method to convert workflow JSON files to a batch that gets enqueued to InvokeAI.
The application relies on communication via a simple REST interface between the This is Fine! frontend and the InvokeAI backend. It uses the Remote Image nodes. The provided workflow in the resources folder uses these nodes.
To facilitate running the installation as a projection on a large wall, a quad warp / corner pinning / keystone utility is provided. It can be calibrated using a small application
python warpcalibration.py
Select a corner with the keys 1, 2, 3 or 4 and use the cursor keys to move that corner.
Press s to save the calibration, r to reset to the full projection rectangle and l to restore a previousl saved calibration.
The This is Fine! application automatically loads a warpcalibration if it is present in the resources folder.