sql_database should give explicit credentials priority over secrets.toml

[djudjuu]

dlt version

1.9.0

Describe the problem

eventhough explicit credentials are defined in a ConnectionStringCredentials object, sql_database tries to instaniate the connection from the secrets.toml. (see code below)
As those have been defined but given default-values the error message is not helpful

Note: if given a string than it works

Expected behavior

should ignore secrets.toml

Steps to reproduce

• in a new venv
• dlt init sql_database duckdb
• don't touch secrets.toml they should have default placeholder values
• paste the following minimal example into sql_database_pipeline.py

# flake8: noqa
import humanize
from typing import Any
import os

import dlt
from dlt.common import pendulum
from dlt.sources.credentials import ConnectionStringCredentials

from dlt.sources.sql_database import sql_database, sql_table, Table

from sqlalchemy.sql.sqltypes import TypeEngine
import sqlalchemy as sa

import builtins

original_open = open

# helper to see file access to secrets.toml
original_open = open

def tracking_open(file, *args, **kwargs):
    if "secrets.toml" in file:
        print("secrets.toml was accessed!")
    return original_open(file, *args, **kwargs)

builtins.open = tracking_open



def my_connect():
    print('entered')
    # not working:
    credentials = ConnectionStringCredentials(
        "mysql+pymysql://rfamro@mysql-rfam-public.ebi.ac.uk:4497/Rfam"
    )
    # >> sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:drivername

    # works: 
    # credentials = "mysql+pymysql://rfamro@mysql-rfam-public.ebi.ac.uk:4497/Rfam"
    # >> get Access denied for user 'rfamro'@'62.214.81.158' (using password: YES) )

    source = sql_database(credentials).with_resources("family") 

if __name__ == "__main__":
    my_connect()

run python sql_database_pipeline.py

Operating system

Linux

Runtime environment

Local

Python version

3.12

dlt data source

sql_database

dlt destination

No response

Other deployment details

No response

Additional information

No response

[rudolfix]

@djudjuu this should be fixed on the newest devel, can you check?

[djudjuu]

yes it is indeed!