Skip to content
This repository was archived by the owner on May 17, 2024. It is now read-only.

Conversation

@leoebfolsom
Copy link
Contributor

The problem was that I was receiving this error:

(datadiff) ➜  ~ data-diff --conf ~/config_files/datadiff.toml --run diff_test -v -k activity_id -w "event_timestamp < '2022-10-10'"
Traceback (most recent call last):
  File "/Users/leoebfolsom/virtualenvs/datadiff/bin/data-diff", line 8, in <module>
    sys.exit(main())
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/data_diff/__main__.py", line 194, in main
    kw = apply_config_from_file(conf, run, kw)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/data_diff/config.py", line 83, in apply_config_from_file
    return _apply_config(toml.load(f), run_name, kw)
  File "/Users/leoebfolsom/virtualenvs/datadiff/lib/python3.9/site-packages/data_diff/config.py", line 45, in _apply_config
    raise ConfigParseError(f"Running 'run.{run_name}': Connection #{index} is missing attribute '{attr}'.")
data_diff.config.ConfigParseError: Running 'run.diff_test': Connection #1 is missing attribute 'database'.

I realized that the args object was an empty dict ({}) because it was created by trying to pop the value of the "1" key from the run_args dictionary, which is the first character in the string "12", which is being looped through. The run_args dictionary doesn't have a key of "1" (or "2"), which is why we get {}.

I guessed what was supposed to be happening: as we loop through the string "12" , we're supposed to create new dictionaries, args, with keys of database1, table1, and threads1 (and the same for 2); and then strip the 1 or 2 from the keys so that the rest of the code works.

With these changes, and some logging which I hope will demonstrate clearly what's going on, it works like this:

(datadiff) ➜  ~ data-diff --conf ~/config_files/datadiff.toml --run diff_test -v -k activity_id -w "event_timestamp < '2022-10-10'"
WARNING:root:LOGGING args_with_db_num
WARNING:root:{'database1': 'diff_test', 'table1': 'org_activity_stream', 'threads1': None}
WARNING:root:LOGGING args
WARNING:root:{'database': 'diff_test', 'table': 'org_activity_stream', 'threads': None}
WARNING:root:LOGGING args_with_db_num
WARNING:root:{'database2': 'diff_test', 'table2': 'org_activity_stream_backup', 'threads2': None}
WARNING:root:LOGGING args
WARNING:root:{'database': 'diff_test', 'table': 'org_activity_stream_backup', 'threads': None}
- 305205789902086336
+ 6424752678866087891

I don't necessarily expect this code change (in my initial commit) to be the solution, but it has allowed me to successfully run diff between two tables in the same postgres db locally.

@erezsh erezsh closed this Nov 1, 2022
@leoebfolsom leoebfolsom deleted the toml-db-config-args branch November 1, 2022 20:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants