-
Notifications
You must be signed in to change notification settings - Fork 1
The Grand Refactor
Some long term goals we need to accomplish:
- Self Hosting: Madz hosted in itself.
- AI Planning: A very smart way process commands.
- Distribution: A way to distribute executables.
- Live Interaction: A way to interact with madz 'live', a persistent command line application.
There are a number of problems in madz, each of them prevalent through large sections of the codebase, that need to be solved to reach these goals:
- Implementation Leakage. At the moment the python implementation is leaking out of madz, this is fine from an external perspective, the problem is that internally this implementation leakage is tightly bound to implementation. We need to better separate these internally to prepare for self hosting.
- Tight binding. A lot of the code is tightly bound with other code. To prepare for self hosting, we will need to break these bindings.
- Procedural Design. Large portions of the system currently rely on the order in which things happen. These need to be changed to more declarative designs. Specifically for use by AI planning and live interaction.
- Housekeeping. We've changed some terminology (plugin -> module, in the general case) and some coding standards (camel case for many variables) that would be nice to go over all the code with. Move all madz stuff into single folders, etc. Github issue flags, etc.
- Single Process. Currently madz is a single process from command invocation to execution. We should separate these concerns for better stability, debugging, and live module updating.
- Limited Platform Information. We currently don't do a lot about platform information.
So here we describe the major re-factoring changes we should do to reach this new version of madz. This version of madz will be almost exactly the same, the only changes will be the way users interact with it, namely through the command line interface and config files.
We need to rebuild madz into a modular system that will eventually be represented as madz. To assist with it we will build a python library (of metaclasses and decarators) that will assist modules and their classes and functions to be easily integrated into the system.
Most importantly this will change the layout of madz. Besides the core system and utility library, every folder that does not contain a blank __init__.py will be considered a plugin.
Concepts are a late term feature for madz that we will proximate with custom metaclassed abstract base classes.
The core idea of concepts are to provide a semantic element which describes the features provided by the concept and a method to construct the concept. Concepts can be constructed in complex ways. For example, multiple objects can register to be potential parts of a concept, and the construction of the concept will use those parts to build what it needs.
The core system will be responsible for three main things:
- Managing the state of the system.
- Managing the capabilities of the system.
- Jumping off point for extensibility.
It will do this through the plugin modules of the madz system. The core system will be constructed and configured by the start up script. During this configuration phase conceptual objects of a couple major classes will be constructed:
- ConceptualCapability, represents some verb the system can do. This will be used for planning, live interaction, and separation of concerns. As a concept, a capability can be quite complex, with a variety of different modules registering with it.
- ConceptualState, represents some noun important to the system. States are affected by capabilities, generally states provide conditions, information, etc about some objects. They also provide information on the capabilities to effect change in the noun.
- ConceptualHelper, represents some helper which builds off of, and back into, the core system. As an example, the module abstraction is a conceptual helper used by nearly every madz module.
The new design will try to separate nearly everything into concepts, the core concepts below provide the ability to talk about modules and store/load their data:
- ConceptualState: Hook in point for new conceptual state.
- ConceptualCapability: Hook in point for new conceptual capabilities.
- ModuleIdentity[ConceptualState]: This is the state of all module identities that have been loaded or suggested.
- Identity Manager [Unique Objects]: Every identity is registered to an identity manager, which is responsible for storing and describing it.
- Identity Comparison Suite [MultiMethods]: A set of methods for comparing identities with identity managers as the polymorphic arguments.
- ModuleData[ConceptualState,ConceptualCapability]: This provides the ability to parse arbitrary module source folders, module binaries, or gather them from other locations. This also manages the store for where on disk data for the module can be found.
- Module Data Managers [Interface Objects]: Objects which can get stored data for a module (but not parse it). Can be a folder or repository, or some complex hybrid. Is responsible for transmitting and receiving data. Often caches parts of type information.
- Module Data Types [Interface Objects, Concept Hookin]: Objects describing data in modules, tied into the capabilities / concepts that use them. Includes parsing / writing if applicable. Used to determine what to package in some cases.
- Configuration[ConceptualState]: Manages and provides configuration information.
- Configuration Locations [Interface Objects]: Places to load configurations from.
- Configuration Domains [Interface Objects]: Domains of configuration, which cause application of different configurations. For example, languages, libraries, modules, compilers, etc. are all domains.
- Options [Interface Objects]: The options provided by the config (with information about valid domain contexts).
The extensible concepts that come with madz:
- ModuleDescription[ConceptualState]: This manages module contents descriptions (MDL), as well as the ability to parse and analyze them.
- Storage [Interface Objects]: Ways to describe module descriptions as a data format (node types).
- Parser [Interface Objects(Storage)]: A way to parse a specific storage type of MDL from a specific file format.
- Extensions [Interface Objects(Storage, Parser)]: List of all extensions for parser/storage combinations. A parser may follow specific a specific extension protocol that removes these polymorphic constraints.
- Language
- ModuleDescription[ConceptualState,ConceptualCapability]: This manages MDL and similar descriptions.
- ModuleWrapper[ConceptualCapability]: This provides the ability to turn module descriptions into source code.
Helpers:
The bootstrap will work as follows:
- Use the old version of the repository to generate a binary distribution of madz with the features used by madz startup script.
- Use the binary version of madz to load the madz startup script, which will request only plugins that are packed into the binary.
- Use the now loaded madz to generate a binary distribution.
To provide better live features, and better understanding of logged information in general, we should redesign logging for madz, and make it available as a module.
In addition to the log message and time we currently have, we should add tags. These tags would allow each log call to specify the information important to it. The logging level we currently have could be replaced with tags for example. Separate tags even (i.e. an importance value and a problem value). Tags could hold objects, for example the current madz module.
A presentation that would be favorable would be a hierarchical representation, this is mostly a client side thing, but the logging code would have to support creating the hierarchies. This is information not easily represented using tags, although tags would work well with such representations.
We need the ability for multiple threads to log, so the logger has to be thread safe, it may be useful to know which thread is logging, we could do this automatically with tags, or ignore the issue completely.
There are a number of features which we need to have better libraries for.
A common problem is creating general, or specific, preferences between objects. This library will allow those preferences to be expressed.
One of the core causes of the current procedural design is the lack of methods with multiple dispatch in python. This library will allow for the creation of multi-method objects, and the registering of multi method functions to those libraries.
A way to construct objects responsible for problems.