Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce deletion-mode for assets #421

Merged
merged 3 commits into from
Sep 18, 2023
Merged

Introduce deletion-mode for assets #421

merged 3 commits into from
Sep 18, 2023

Conversation

nicoloboschi
Copy link
Member

@nicoloboschi nicoloboschi commented Sep 15, 2023

Same of #418 but for assets.

new field deletion-mode: none|delete. Default is none

When deletion-mode=delete, the delete-statements field is required (list[] string) and all the statements are executed at deletion time.

assets:
- name: "vsearch-keyspace"
    asset-type: "cassandra-keyspace"
    creation-mode: create-if-not-exists
    deletion-mode: delete
    config:
      keyspace: "vsearch"
      datasource: "CassandraDatasource"
      create-statements:
        - "CREATE KEYSPACE vsearch WITH REPLICATION = {'class' : 'SimpleStrategy','replication_factor' : 1};"
      delete-statements:
        - "DROP KEYSPACE vsearch"

final String keySpace = getKeySpace();
List<String> statements =
ConfigurationUtils.getList("delete-statements", assetDefinition.getConfig());
if (statements.isEmpty()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a good idea, the table may have other dependencies like INDEXes it really depends on how you created it (and we have no control over the create-statements)

if there is no statement then we do nothing, like for create-statements

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not very intuitive.
You set deletion-mode: delete and then you have to provide the keyspace delete statement ?
It's much easier if you just delete it.

From my testing, the keyspace is being deleted even if not empty and it works well.

Do you think an INDEX would make the keyspace deletion fail ?

List<String> statements =
ConfigurationUtils.getList("delete-statements", assetDefinition.getConfig());
if (statements.isEmpty()) {
statements = List.of("DROP KEYSPACE IF EXISTS %s;".formatted(getKeyspace()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also here

Copy link
Member

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the auto delete command works with tables with indexed (for instance of Vector columns) then I am +1

as an alternative we can fail the deployment here

public class CassandraAssetsProvider extends AbstractAssetProvider {

in any case we must add the validation in CassandraAssetsProvider

@nicoloboschi nicoloboschi merged commit fc3aa7e into main Sep 18, 2023
8 checks passed
@nicoloboschi nicoloboschi deleted the assets-deletemode branch September 18, 2023 04:41
benfrank241 pushed a commit to vectorize-io/langstream that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants