images refactor

[psychedelicious]

General design changes

• New ImageCategory model: images are designated as image (normal), control_image (processed controlnet image), or other (future use by community nodes). Any number of new categories can be added, these are just the ones I know we need now.
• Separate image concerns: "images" are now "image_files" (physical image files) and "image_records" (images in the db)

Refactor images service:

• Split the existing images service up into 3 4 low-level services and 1 high-level
• The low-level services are provided to the high-level service on its init
• ImageFileStorage: low-level service, responsible for image files, provided implementation uses disk storage
• ImageRecordStorage: low-level service, responsible for image records (ie the image tables in the db), provided implementation via the sqlite db
• UrlService: low-level service, responsible for generating URLs for images, provided implementation just formats path strings
• MetadataService: low-level service, responsible for generating metadata, provided implementation builds metadata the graph-traversal strategy described below
• ImagesService: high-level service, exclusively provides app access to the low-level services

Add images-related tables to db

• See image below for basics of the db, or image_records_storage.py for implementation
• Tables for image_types (existing ImageType model) and image_categories (new ImageCategory model) store possible values for their respective models. If the model is updated in the future, new values are added to the tables on startup.
• Tables to support future image gallery tagging and favorites

Metadata

• The execution graph is traversed when an image is saved and core generation parameters are picked out. It looks for the nearest predecessor t2l or l2l node and the noise and compel predecessor nodes from that. Metadata to be included are listed below
• The metadata is embedded in the image on save. However, the backend does not read metadata from the image or really do anything with it except store it in the database, and send it along with the rest of the image data when queried.
• Still some challenges here, see TODO

    # Included metadata
    positive_conditioning
    negative_conditioning
    width
    height
    seed
    cfg_scale
    steps
    scheduler
    model
    strength
    extra # placeholder for unvalidated PNG tEXt chunks from foreign uploaded images
    # Planned but not done yet:
    latents # init latents
    # Pending model refactor:
    # vae
    # unet
    # clip

Refactor web API routes:

TODO

• DONE Get image urls added to invocation_complete events (dirty hack in place now) instead, we make an extra round trip to retrieve an image and its metadata. We need to make the additional request to get the full metadata anyways so it's not really a waste. This is implemented.
• WIP: Migrate UI to use the new routes and models - images/thumbnails load but a lot of the surrounding functionality needs to be changed due to changes in API and data shapes
• DONE Hook up the metadata building functions (they are implemented and functional but not used)
• Figure out how to handle init images in metadata (tricky to find them...)
• WIP: Only the txt2img node is hooked up LatentsToImage is connected, but all nodes that output an image need to be hooked up
• Implement and test the tags and favorites, which currently have db tables but haven't been used yet
• Write tests for everything

TODO (in a future PR)

• Handle the objects now call latents (which, more generally, are pickle-able torch objects). These include image_latents and conditioning right now, but I think we will see a lot more types of these coming up. Probably need to give these objects similar treatment to images.

[psychedelicious]

I'd like to get a couple reviews of the overall design and changes to python code. The client side is still a WIP but the backend changes are almost fully implemented.

Marking this ready for review in hopes of getting a review or two before I finish polishing the UI, in case there is some major issue I've missed.

Besides the usual code review, this change needs careful thought. Once it rolls out to OSS and people are using it, we can't easily change db schemas. So long as there's no problem there, the rest is easy to fix.

It's possible to run this PR now, same way you'd run main with yarn dev. It will add tables to your invokeai.db file. The PR won't touch the existing tables at all, but to keep your invokeai.db safe, suggest moving it to another location and letting the app create a new one.

If you run this locally, you can only use the node editor, and only with certain nodes. This is bc I haven't changed all the image-outputting nodes to use the new service yet.

Suggest testing with exactly this graph layout for single images:

Or this one for multiples:

(you can change all parameters of course)

[psychedelicious]

@maryhipp

feat(nodes): address feedback
5668711

• Address database feedback:
  • Remove all the extraneous tables. Only an images table now:
  • image_type and image_category are unrestricted strings. When creating images, the provided values are checked to ensure they are a valid type and category.
  • Add updated_at and deleted_at columns. deleted_at is currently unused.
  • Use SQLite's built-in timestamp features to populate these. Add a trigger to update updated_at when the row is updated. Currently no way to update a row.
  • Rename the id column in images to image_name
• Rename ImageCategory.IMAGE to ImageCategory.GENERAL
• Move all exceptions outside their base classes to make them more portable.
• Add width and height columns to the database. These store the actual dimensions of the image file, whereas the metadata's width and height refer to the respective generation parameters and are nullable.
• Make deserialize_image_record take a dict instead of sqlite3.Row
• Improve comments throughout
• Tidy up unused code/files and some minor organisation

[psychedelicious]

Adjusted the full image route to provide a better user experience (for when you open the image in a new tab):

Here's the images table schema:

[hipsterusername]

@Kyle0654 @lstein - As mentioned in Discord, merging this in as this is needed to continue forward progress, but please feel free to provide feedback/thoughts to @psychedelicious where relevant