Proper rehauls to the internals with careful attention to detail.

[AbstractEyes]

Problem zones.

The internals of ComfyUI serve a serious purpose, and have many many TECH DEBTS and problems that cause expansion of those sections to be very difficult. I have identified at least 12 points of pain that can be streamlined, many code-ends eliminated, and many elements expanded to introduce new avenues of additional configuration with less effort.

My core principle is to enable MORE open-ended capability, not less. I want systems to function more robustly and more solidly. Professionalism and compartmentalization is a pain in the ass to maintain and often creates worse problems, however if this is handled carefully and with enough attention to detail - the core system can withstand a large flurry of changes, ranging from simple topical changes, to the entire core structure of the way we communicate to AI.

Rolling out big changes overnight is possible with a robust and carefully oriented structure.

Developer to developer, developer to user, user to user; I want this to be more user and developer friendly.

First things first; Rehauls. I'm already targeting the internals of Conditioning and Clip model management.

The loaders suffer from overuse and the funnel shows wear and tear, not to mention it has no legitimate filtration other than EVERYTHING GOES IN THE MODEL FOLDER. Which is fine, when there's I dunno 10-20 of them. Well, I've got a few hundred so the problems start to mount really quickly. Not to mention THOUSANDS of loras.

• model_manager Rehaul
• • This model manager is all over the board, it needs a bit of a facelift. I won't destroy any of the core structures or functional methods, but instead redirect them to more open-ended capability for "any" model; which will include a multitude of size checks for layer data, vram detections, onloading/offloading at runtime, dtype conversion curation, and more elements that will make it easier to snap new models into the model manager.
• Conditioning Rehaul
• • Additional conditioning controllers and nodes; such as performing formulas on those conditionings directly, segmented conditioning rather than fully pooled and concatenated, and so on.
• Clip Management
• • This one is already indev; which is a much more simple series of methodologies to handle clips internally - much of which is hardcoded to multiple structures hooked together and daisy chained with tape rather than wired correctly. The outcome will provide easier avenues to snap new clip models in from openai or open_clip with ease - provided they fit the dims, and if they don't the dimensions can be scaled to see what happens when you really stick the clip in the square hole. Not to mention actual text model and text encoder support that isn't just clip or openclip. I really don't see why these two... were jammed together... In any case I already have a plan for dealing with LLMs in association with CLIP and OpenCLIP.
• VAE Simplification
• • I'm sure I can speak for everyone when I say - VAE encoding and decoding can be done in a simpler way.
• VAE Configuration and additional capabilities
• • Inversely, the advanced angle and usage of VAE can be heavily expanded for further research potentials. This would fit in a similar encoder structure as the rehaul is already producing.
• More robust sampler configurations and additional capabilities
• • These are highly lackluster in the baseline setup. Res4Lyf have MANY, likely over a hundred fair and capable samplers. On top of that, an additionally large plethora of expanded elements and a more streamlined system.
• Additional nth hooks and hooking options
• • Currently the hooking system is fairly new but it has some capability. This could use shoring up and expansion; including additional hooks to load generics rather than specifics, modifying configurations at runtime, scheduling nth code instead of just hard-set code during inference, and more.
• Expanded API
• • Additional API access interface-points to externally communicate with ComfyUI, treated under the same security concern umbrella as currently determined.

So what do you think, which should I do FIRST!?