[Bug] psycopg2 dependency is not installed correctly

[Auric-Manteo]

Is this a new bug in dbt-redshift?

• I believe this is a new bug in dbt-redshift
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When I install the following Pipfile and run dbt seed afterwards I get an import error.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
dbt-redshift = "==1.8.0"
sqlfluff = "~=2.2.0"
requests = "2.32.0"
jsonpath-ng="1.5.3"

[dev-packages]

[requires]
python_version = "3.9.9"

Expected Behavior

dbt seed should run successfully.

Steps To Reproduce

1. Run pipenv install in a directory with the above Pipfile.
2. Run dbt seed

Relevant log output

06:53:16  Running with dbt=1.8.1
06:53:16  Encountered an error:
No module named 'psycopg2'
06:53:16  Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 138, in wrapper
    result, success = func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 101, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/dbt/cli/requires.py", line 215, in wrapper
    profile = load_profile(flags.PROJECT_DIR, flags.VARS, flags.PROFILE, flags.TARGET, threads)
  File "/usr/local/lib/python3.9/site-packages/dbt/config/runtime.py", line 71, in load_profile
    profile = Profile.render(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 403, in render
    return cls.from_raw_profiles(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 369, in from_raw_profiles
    return cls.from_raw_profile_info(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 325, in from_raw_profile_info
    credentials: Credentials = cls._credentials_from_profile(
  File "/usr/local/lib/python3.9/site-packages/dbt/config/profile.py", line 149, in _credentials_from_profile
    cls = load_plugin(typename)
  File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 239, in load_plugin
    return FACTORY.load_plugin(name)
  File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 96, in load_plugin
    self.load_plugin(dep)
  File "/usr/local/lib/python3.9/site-packages/dbt/adapters/factory.py", line 68, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/dbt/adapters/postgres/__init__.py", line 4, in <module>
    from dbt.adapters.postgres.connections import PostgresConnectionManager, PostgresCredentials
  File "/usr/local/lib/python3.9/site-packages/dbt/adapters/postgres/connections.py", line 13, in <module>
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

Environment

- OS: MacOS and also tried on Ubuntu
- Python: 3.9.9
- dbt-core: 1.7.15
- dbt-redshift: 1.8.0

Additional Context

No response

[jtcohen6]

@Auric-Manteo On Ubuntu, I expect this would be a result of:

• [Regression] Forced psycopg2 dependency on Linux causes sudden build failures dbt-postgres#96

Which we're going to resolve by:

• installing psycopg2-binary by default, on all systems
• removing the dependency of dbt-redshift on dbt-postgres ([Feature] Remove dbt-postgres dependency #811)

However, on MacOS, I would expect the install to work successfully (with psycopg2-binary). Could you try:

python3 -m pip install --upgrade --force-reinstall dbt-core dbt-postgres dbt-redshift

[Auric-Manteo]

Oh, nice. It's inside a docker container which I guess is why it doesn't work on my Mac either.
I wasn't sure if that is relevant.

[jtcohen6]

Ah yes, that would explain it! The easiest way to fix this is by adding the psycopg2 build prerequisites to your docker image:

• https://www.psycopg.org/docs/install.html#build-prerequisites

I'm going to close this issue as a duplicate in the meantime

[Auric-Manteo]

Adding psycopg2 didn't work in my case, but reverting to dbt-redshift 1.7.7 does, which is fine for now.
Thanks for your help jtcohen6!