Macro + Compiler

[kurtbrose]

just expanding ideas, getting some terminology out that can nucleate further docs / cookbook items and guide future development --

a "macro" or "glomacro" in the glom context is a function or callable that:

• outputs a spec

• input may be anything (valid spec, or any python objects, or both)

• is meant to run once at spec definition time

a "compiler" or "glompiler" in a glom context is a function or callable that:

• takes a spec as input

• output may be anything (valid spec, or any python object)

• is meant to run once against a spec (since specs are meant to be small in number and global, things generated from specs should be similar)

both of these concepts could eventually be supported by official Macro and Compiler types; this would be a stepping stone to important tools like coverage checking

both of these concepts (but especially compilers) could be supported by glom specs that accept and/or output other glom specs, aka "meta-specs" or "glometas"

[kurtbrose]

practical use case of macro -- adding defaults to a dictionary before validation

>>> glom({'optional': 'val'}, (Fill([T, Literal(T)]), lambda t: Fill(t), lambda t: Merge(t)))
Merge(Fill([{'optional': 'val'}, T]), init=<class 'dict'>, op=<method 'update' of 'dict' objects>)

base version didn't work for some reason, we should look into this since it is a blocker for macros

>>> glom({'optional': 'val'}, (Fill([T, Literal(T)]), Fill, Merge))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\kurt\workspace\glom\glom\core.py", line 1764, in glom
    ret = _glom(target, spec, scope)
  File "C:\Users\kurt\workspace\glom\glom\core.py", line 1785, in _glom
    return scope[MODE](target, spec, scope)
  File "C:\Users\kurt\workspace\glom\glom\core.py", line 1794, in _glom_auto
    return _handle_tuple(target, spec, scope)
  File "C:\Users\kurt\workspace\glom\glom\core.py", line 1494, in _handle_tuple
    nxt = scope[glom](res, subspec, scope)
  File "C:\Users\kurt\workspace\glom\glom\core.py", line 1783, in _glom
    return spec.glomit(target, scope)
TypeError: glomit() missing 1 required positional argument: 'scope'

[kurtbrose]

for comparison, here is the non-marco version

# no macro
lambda defaults: Merge(Fill([defaults, T]))
# macro
lambda defaults: glom(defaults, (Fill([T, Literal(T)]), Fill, Merge))

[kurtbrose]

another practical use case -- and this works too

transform a django database configuration to a tableau REST api connection + datasource object

_BUILDER2ASSIGNER = Spec((
    Let(type=T[1]),
    T[0].items(),
    [Invoke(Assign).specs(
        lambda t: getattr(T[0], t[0]),
        Invoke(Spec).specs(T[1]))],
    Fill((
        Invoke(Fill).specs(Fill((
                Invoke(Invoke).specs(S['type']),
                Literal(T)
            ))),
        T,
        Literal(T[0])
        ))
    ))
"""
_BUILDER2ASSIGNER is a meta-spec to trasform

({'a': 'b', 'c': 'd'}, type)

to

(
    Fill( (Invoke(type), T) ),
    (
        Assign(T[0].a, T[1].b),
        Assign(T[0].c, T[1].d
    ),
    T[0]
)
"""


_DICT2LITERALS = Spec((
    T.items(),
    [Fill((T[0], Invoke(Literal).specs(T[1])))],
    dict))
"""
_DICT2LITERALS converts {key: val} to {key: Literal(val)}
"""



def django_db2tableau_conn(project_id, django_db):
    """
    convert a django database entry from settings.py into
    a tableau data-source with that connection
    """
    return glom(
        django_db,
        _BUILDER2ASSIGNER.glom((
            {
                'username': 'USER',
                'password': 'PASSWORD',
                'server_address': 'HOST',
                'server_port': 'PORT',
            },
            TSC.ConnectionItem)))

    return glom(
        {},
        _BUILDER2ASSIGNER.glom((
            _DICT2LITERALS.glom({
                'connections': [connection],
                'datasource_type': 'postgres',
                'name': django_db['NAME'],
            }),
            lambda: TSC.DatasourceItem(project_id)
            ))
        )

[kurtbrose]

alright, some real meta-glomming happening

one thing has become clear: for large-scale meta-glom to work well, it needs to be possible to determine the mode statically

this means one of two things:

1- 'compilation' is parallel to execution, with some kind of "test" object floated through the system; something kind of like T which will accept all operations

2- all modes are themselves compilation steps, which break the various constants out into their equivalent specs; we'd need a fleet of specs with names like auto_dict and fill_tuple and then meta-glom would refer to these