"attribute missing" error when using TOML config file

[leoebfolsom]

When I try to run a diff using a TOML config file to compare two tables within one postgres database, I receive this error:

(datadiff) ➜  ~ data-diff --conf ~/config_files/datadiff.toml --run diff_test -v -k activity_id -w "event_timestamp < '2022-10-10'" -d
Traceback (most recent call last):
  File "/Users/leoebfolsom/virtualenvs/datadiff/bin/data-diff", line 8, in <module>
    sys.exit(main())
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/data_diff/__main__.py", line 194, in main
    kw = apply_config_from_file(conf, run, kw)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/data_diff/config.py", line 77, in apply_config_from_file
    return _apply_config(toml.load(f), run_name, kw)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/data_diff/config.py", line 42, in _apply_config
    raise ConfigParseError(f"Running 'run.{run_name}': Connection #{index} is missing attribute '{attr}'.")
data_diff.config.ConfigParseError: Running 'run.diff_test': Connection #1 is missing attribute 'database'.

Contents of TOML file:

# POSTGRES
# Specify the connection params to database 1.
[database.diff_test]
driver = "postgresql"
database = "diff_test"
user = "leoebfolsom"
password = "$PW_POSTGRES"

# Specify the default run params
[run.default]
# update_column = "timestamp"
verbose = true

[run.diff_test]
# Source 1 ("left")
database1 = "diff_test"
table1 = "org_activity_stream"

# Source 2 ("right")
database2 = "diff_test"
table2 = "org_activity_stream_backup"

I realized this problem is related to the args object being created as an empty dict ({}) because the run_args dictionary doesn't have a key of "1" (or "2").

I mocked up a superficial fix here, which gets my diff to run locally, but it doesn't pass CI/CD, indicating that there's a deeper problem.

Environment

• installed data-diff from github master at b9ce7ed
• macOS Monterey 12.6
• Python 3.9.6

[erezsh]

I don't think it's a bug. The correct syntax is

1.database = "diff_test"
1.table = "org_activity_stream"

# Source 2 ("right")
2.database = "diff_test"
2.table = "org_activity_stream_backup"

But we definitely need a better error.

[leoebfolsom]

That did it, thanks!