Allow configuring storage as SETTINGS disk='<disk_name>' (instead of storage_policy) and explicit disk creation SETTINGS disk=disk(<disk_configuration>)

[kssenii]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Allow configuring storage as SETTINGS disk='<disk_name>' (instead of storage_policy) and with explicit disk creation SETTINGS disk=disk(type=s3, ...)

[vitlibar]

That can be dangerous. I mean what if somebody will write SETTINGS disk=disk(type='local', path='<some_system_path>')

[azat]

Maybe it should accept only disk name, and path can be extracted from ATTACH TABLE FROM <path> (now it looks at user_files) ?

[kssenii]

That can be dangerous. I mean what if somebody will write SETTINGS disk=disk(type='local', path='<some_system_path>')

and this is allowed, there are even tests for that, BUT it is allowed only if there is an explicit setting in server config that tells which base directory can be used for such out-of-config disks and in code all non-remote disks must lay in this directory.

[kssenii]

Maybe it should accept only disk name, and path can be extracted from ATTACH TABLE FROM (now it looks at user_files) ?

hm, this does not seem convenient for me

[UnamedRus]

Does it support named collections in disk definition (it make sense for s3)?

[kssenii]

Does it support named collections in disk definition (it make sense for s3)?

I do not see much sense for this, you can use disk = 'disk_name', in this case in my opinion. Or you mean it makes sense because now we can create named collections from sql? Then probably we should just allow to create disks from sql in a similar way instead.
Also I made disk configuration hidden in #46670.

[UnamedRus]

Or you mean it makes sense because now we can create named collections from sql?

It's more for:
Having per table s3 disk path. (so do per table disk)
But reusing the same s3 creds.

Then probably we should just allow to create disks from sql in a similar way instead.

Yeah, it would be nicely.
#29110

[kssenii]

Or you mean it makes sense because now we can create named collections from sql?
It's more for:
Having per table s3 disk path. (so do per table disk)
But reusing the same s3 creds.

ok, I see, could you please create an issue for this?

[azat]

It's more for:
Having per table s3 disk path. (so do per table disk)
But reusing the same s3 creds.

I thought about this too, but then decided that this could be a security breach, since you can override path/endpoint and change something that was not allowed.

[UnamedRus]

I thought about this too, but then decided that this could be a security breach, since you can override path/endpoint and change something that was not allowed.

But, arent it's already possible via feature like that:

<clickhouse>
        <s3>
                <datalake>
                        <endpoint>https://<BUCKET_NAME>.s3.eu-central-1.amazonaws.com</endpoint>
                        <use_environment_credentials>true</use_environment_credentials>
                       .......
                </datalake>
        </s3>
</clickhouse>

When we define creds per endpoint prefix.

[azat]

But, arent it's already possible via feature like that:

You mean if the endpoint does not limit to some path (i.e. use root bucket without using some prefix) ?
Then I would say that this is a user problem, or maybe it is not even a problem, maybe they have bucket per instance/load/env/...

I was worrying more about when endpoint=https://<BUCKET_NAME>.s3.eu-central-1.amazonaws.com/clickhouse, and this bucket for instance also has https://<BUCKET_NAME>.s3.eu-central-1.amazonaws.com/user_data and if you will allow overriding endpoint then you could allow to override anything in this bucket, including user_data.

[UnamedRus]

I see,

But it the same problem which will happen for PostgreSQL/MySQL creds already?

If you can override table for example