bug: SingerMapper sets mapping configuration provided from env to null

[jnv]

Meltano Version

3.4.0

Python Version

3.11

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Linux - Fedora 39

Description

I want to invoke meltano-map-transformer with stream_map provided via environment variable MELTANO_MAP_TRANSFORMER_STREAM_MAPS. However, the mapper never receives the configuration because the generated JSON file contains only null.

I tracked down this issue to before_configure hook in SingerMapper. Basically both before_configure hooks in SingerMapper and SingerPlugin get called and both write to the same configuration file. This works fine when the mapper configuration is provided via meltano.yml, but when I provide mapping via environment variable, the hook in SingerPlugin writes the config file correctly, but the hook in SingerMapper overwrites it to null.

Reproduction steps

I've prepared a repository for bug reproduction: https://github.com/jnv/meltano-bug-repro

There is the same mapping provided for meltano-map-transformer, once via meltano.yml as upper and once as MELTANO_MAP_TRANSFORMER_STREAM_MAPS in .env file.

Invoking via mapping works:

meltano invoke upper < data/animals.jsonl # works fine

Invoking meltano-map-transformer directly, with env variable, fails:

meltano invoke meltano-map-transformer < data/animals.jsonl

Debug log

meltano invoke meltano-map-transformer < data/animals.jsonl
2024-04-18T23:37:34.552597Z [debug    ] meltano 3.4.0, Linux          
2024-04-18T23:37:34.557497Z [info     ] Environment 'dev' is active   
2024-04-18T23:37:34.566582Z [debug    ] Creating DB engine for project at '/meltano-bug-repro' with DB URI 'sqlite://meltano-bug-repro/.meltano/meltano.db'
2024-04-18T23:37:34.592341Z [debug    ] Found plugin parent            parent=meltano-map-transformer plugin=meltano-map-transformer source=LOCKFILE
2024-04-18T23:37:34.615068Z [debug    ] Created configuration at /meltano-bug-repro/.meltano/run/meltano-map-transformer/mapper.247418c0-4992-43bc-88c5-f705ae91511a.config.json
2024-04-18T23:37:34.615279Z [debug    ] Created configuration          config_path=PosixPath('/meltano-bug-repro/.meltano/run/meltano-map-transformer/mapper.247418c0-4992-43bc-88c5-f705ae91511a.config.json') mapping_config=None mapping_name=None plugin_name=meltano-map-transformer
2024-04-18T23:37:34.616264Z [debug    ] Invoking: ['/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/bin/meltano-map-transform', '--config', '/meltano-bug-repro/.meltano/run/meltano-map-transformer/mapper.247418c0-4992-43bc-88c5-f705ae91511a.config.json']
Traceback (most recent call last):
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/bin/meltano-map-transform", line 8, in <module>
    sys.exit(StreamTransform.cli())
             ^^^^^^^^^^^^^^^^^^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/singer_sdk/plugin_base.py", line 80, in invoke
    return super().invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/singer_sdk/mapper_base.py", line 130, in invoke
    mapper = cls(
             ^^^^
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/meltano_map_transform/mapper.py", line 81, in __init__
    super().__init__(
  File "/meltano-bug-repro/.meltano/mappers/meltano-map-transformer/venv/lib/python3.11/site-packages/singer_sdk/plugin_base.py", line 150, in __init__
    config_dict.update(read_json_file(config_path))
TypeError: 'NoneType' object is not iterable
2024-04-18T23:37:35.091014Z [debug    ] Deleted configuration at /meltano-bug-repro/.meltano/run/meltano-map-transformer/mapper.247418c0-4992-43bc-88c5-f705ae91511a.config.json

The invoker definitely sees the env variable:

$ meltano invoke --print-var MELTANO_MAP_TRANSFORMER_STREAM_MAPS meltano-map-transformer < data/animals.jsonl  
MELTANO_MAP_TRANSFORMER_STREAM_MAPS={"animals": {"description": "_['description'].upper()"}}

But config dump prints just null:

$ meltano invoke --dump config meltano-map-transformer < data/animals.jsonl
null

While invoking the mapper works fine:

$ meltano invoke --dump config upper < data/animals.jsonl
{
  "stream_maps": {
    "animals": {
      "description": "_['description'].upper()"
    }
  }
}

[edgarrmondragon]

Thanks for reporting and digging! You might wanna see https://github.com/meltano/meltano/tree/edgarrmondragon/fix/mapper-env-vars

(fwiw I think this error would've been caught earlier if type checking was enabled on that module)

[jnv]

@edgarrmondragon Thanks so much for preparing a fix so quickly! I have tested your branch with my reproduction repo, and I can confirm that it indeed works. Is there some way I can help to get the PR merged?

[edgarrmondragon]

@edgarrmondragon Thanks so much for preparing a fix so quickly! I have tested your branch with my reproduction repo, and I can confirm that it indeed works. Is there some way I can help to get the PR merged?

@jnv Confirmation that the issue is fixed by the PR is pretty good actually 😄