Towards recipe completeness

[liamhuber]

I'd like to do some work in this repo getting structured pydantic models (instead of nested dictionaries) for the prospective recipes. IMO this follows up directly with our discussion of universal decorators. @samwaseda and @XzzX, this pertains directly to concepts you've been working on; I'm not attempting to break a lot of new ground here, but to get rolling with my contributions to stuff we're all on the same page about. To that ends, here's my best as-of-now understanding and plan. I'll ping you for review on individual components, but also let me know if you see problems or available improvements to this roadmap.

Goals:

• Pydantic models to represent procedural programming "recipes" (function, macro, for, while, if & try)
• ast parsers for python code, and corresponding "universal" decorators for python functions that transforming python functions into recipes (and attach them to the function in the case of the decorator)
• Models here are serial-ready, and represent the pure "workflow recipe" per this discussion
  • E.g. when we reference a function for a function node, it's with the import information "module.qualname", not the actual function object itself
  • Save "live" recipes with actual python objects, and anything to do with types or data values for semantikon

Steps:

• A type enum alias for recipe element types
  • will need feedback on what to actually call things...
  • I definitely want everything to be a node, including graphs, so I want "node" as a generic base term and not the name for functions/leafs
• Pydantic models
  • I'll delay universal IDs for now; @samwaseda did some hashing work recently, and @XzzX has been thinking about this for the "node store" -- they will have better insight into details here
  • There are issues for "functions", "macros", "for", and "while" that I will revisit, but I believe I'm largely happy with them
  • I'm not so confident that we came to a good conclusion for "if" and will revisit the details in the corresponding issue
• Add a parser and corresponding decorator for functions
  • This currently doesn't exist at all, so it's a clean addition and an easy place to take a run at the agreed "universal decorator" approach
• Adapt and exploit tools in the workflow module to build a parser that returns the pydantic models instead of nested dictionaries
• Replace the FunctionWithWorkflow approach with a parser and decorator that add a recipe as an attribute to the given function
  • I expect this to deprecate some parts of the workflow module -- e.g., we won't need tools for .runing things, because the post-decoration objects are still just functions that can be called, but I expect lots of what's there can be used as-is or re-purposed
• Converter to/from python_workflow_definition
  • Fail cleanly on recipes with logical elements not supported there

[samwaseda]

I'll delay universal IDs for now; @samwaseda did some hashing work recently, and @XzzX has been thinking about this for the "node store" -- they will have better insight into details here

There are two types of hashing implemented in semantikon: one is for the entire workflow (with or without global inputs) and the other one is for each workflow node (not sure which one you are talking about above, because hashing doesn't appear in "Goals"). The first one is needed to instantiate the knowledge graph nodes for a given workflow. The second one is the one we talked about a lot; it might be helpful to not have to run the same node twice. The distinction between the two hashes, because two distinct workflows must instantiate different knowledge graph nodes even if the two workflows are the same up to a certain node.

I'm not so confident that we came to a good conclusion for "if" and will revisit the details in the corresponding issue

I'm also not so confident - as a matter of fact, I might rather prefer to exclude "if" at least in the current state at all, maybe even abandon it entirely, also because it's not really clear where it's needed in the workflow logic.

[liamhuber]

I'm not so confident that we came to a good conclusion for "if" and will revisit the details in the corresponding issue

I'm also not so confident - as a matter of fact, I might rather prefer to exclude "if" at least in the current state at all, maybe even abandon it entirely, also because it's not really clear where it's needed in the workflow logic.

Well, we're on the same page about the status at least 😂 I would anyhow like to take a crack at it, both because it's already possible in pyiron_workflow (and I'd rather not roll back the feature), and because it's the only outstanding part for the recipe system to fully capture procedural programming.