Allow setting enable_external_access while database is running

[ajfriend]

I'd like to be able to toggle enable_external_access while an in-memory database is running.

Currently, I'm getting errors like

InvalidInputException: Invalid Input Error: Cannot change enable_external_access setting while database is running

when I try to run code like:

import duckdb
con = duckdb.connect(
    database = ':memory:',
    config = {'enable_external_access': False},
)

con.execute('set enable_external_access to true')
rel = con.sql(" select * from 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2010-01.parquet' ")
con.execute('set enable_external_access to false')

# Do something else with `rel`, but protect me from accidentally doing a
# replacement scan on, for example, a `df` that I forgot to update in the query.
rel.aggregate('count(*)')

My use case is that I typically like to default to having enable_external_access as False, so that I don't accidentally do a replacement scan on a name that I didn't intend. This is because I'm often writing Python functions that take in a dataframe as an argument like df, use DuckDB to do some transformation, and return a new dataframe. Often, I'll unintentionally import a df from, say, an outer scope. So, instead, I like to block replacement scans and explicitly register the dataframes that I intend to work with in these functions.

Up until now, keeping enable_external_access = False has been fine, but recently, I've really liked the data ingestion affordances in DuckDB SQL like select * from '.../something.parquet' to easily import data locally or remotely.

It would be awesome if I could have it both ways! My plan was to control the replacement scans with a Python context manager that would resolve to something like the code above, which would turn the feature on only when I explicitly intend to use it. And that plan should work fine if I can toggle the setting.

Would it be possible to make it so that we can toggle enable_external_access while the database is running?

[ajfriend]

Alternatively, control of enable_external_access at the level of con.sql or con.execute could also work.

[ajfriend]

And one more use case: I would like to be able to install extensions even when enable_external_access is False, like

con.execute("""
INSTALL h3 FROM community;
LOAD h3;
""")

Currently, I get an error like:

PermissionException: Permission Error: Installing extensions is disabled through configuration

[Tishj]

Often, I'll unintentionally import a df from, say, an outer scope

That is no longer possible without enabling python_scan_all_frames

I don't think we can change external access, the idea being, is that you can somewhat safely create a connection with external access disabled and hand control of the connection over to a user of a service

[ajfriend]

Got it. Thanks! Is this frame scanning behavior covered in the docs? All I could find was https://duckdb.org/docs/configuration/pragmas#python-scan-all-dataframes, which mentions a change in the behavior, but I couldn't find a place where the current behavior is described.

[ajfriend]

That is no longer possible without enabling python_scan_all_frames

Actually, this doesn't seem to be behaving as I would expect. Am I doing something wrong?

import pandas as pd
import duckdb

df = pd.DataFrame({'A': [42]})

con = duckdb.connect()
con.execute('set python_scan_all_frames = false')
def foo():
    return con.sql('select * from df')

foo()

This code runs without error and returns the dataframe, but I would expect it to raise an error since df was defined outside of foo? Am I missing something obvious?

[Tishj]

If python can find it, we can find it
Scan all frames means we can see further back than python's regular variable resolution can.

As stated, this is an opt-in, setting it explicitly to false has no effect, its already false

[ajfriend]

Yeah, I was being dumb, but finally realized what's going on over here: #14041 (reply in thread) :)

Thanks for bearing with me!

[ajfriend]

I don't think we can change external access, the idea being, is that you can somewhat safely create a connection with external access disabled and hand control of the connection over to a user of a service

I'm not sure I understand what you mean here. Could you expand? What do you mean by "hand control of the connection over to a user of a service"?

[Tishj]

That is how enable_external_access is being used.
Your situation does not sound like the typical use case for disabling enable_external_access