Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql/schemachanger: enable declarative CREATE SEQUENCE #117793

Merged
merged 5 commits into from Feb 1, 2024

Conversation

fqazi
Copy link
Collaborator

@fqazi fqazi commented Jan 15, 2024

Previously, we did fully implement CREATE SEQUENCE in the declarative schema changer but had it disabled by default since we were missing this PR will do the following:

  • Enable CREATE SEQUENCE by default in the declarative schema changer
  • Add tests for transactionally creating / dropping sequences to confirm that the rules are sound
  • Address bugs with transactional creation / drops of relations, since we previously did not create data elements
  • Add tests for transactionally adding and using a sequence as a column
  • Skip back up and restore tests for newly created objects, since intermediate states are not backed up today (if we do CREATE SEQUENCE and ADD COLUMN in a single txn, we will intentionally not backup the sequence during the job phase)

Informs: #117648
Epic: CRDB-31282

Copy link

blathers-crl bot commented Jan 15, 2024

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@fqazi fqazi force-pushed the createSequenceFin branch 3 times, most recently from 9ccf6a1 to ba41c22 Compare January 17, 2024 00:21
@fqazi fqazi marked this pull request as ready for review January 17, 2024 14:25
@fqazi fqazi requested a review from a team as a code owner January 17, 2024 14:25
Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work! just had tiny nits

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @fqazi)


-- commits line 21 at r2:
are the next two commits after this examples of things that would have failed before?


pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go line 48 at r2 (raw file):

// descriptorDataIsBeingAdded indicates that descriptor data is being
// added and was not public earlier. This is mainly used to detect if an

this could lead to confusion. the comment has "descriptorDataIsBeingAdded indicates that descriptor data is being added" but the function is named descriptorDataIsNotBeingAdded. it looks like descriptorDataIsNotBeingAdded is correct.


pkg/sql/schemachanger/sctest/backup.go line 117 at r5 (raw file):

func backupSuccess(t *testing.T, factory TestServerFactory, cs CumulativeTestCaseSpec) {
	maybeRandomlySkip(t)
	// Skip comparing outputs, if there are any newly created objects.

why only for these objects? what about functions, schemas, types, and databases?


pkg/sql/schemachanger/sctest/backup.go line 482 at r5 (raw file):

					sqlutils.MatrixToStr(backupContents), pe.GetExplain())
			} else {
				t.Log("Skipping comparison of SQL afterward's as requested.")

super nit: "afterwards"

@fqazi fqazi force-pushed the createSequenceFin branch 2 times, most recently from 233d983 to 319323c Compare January 23, 2024 18:19
Copy link
Collaborator Author

@fqazi fqazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @rafiss)


-- commits line 21 at r2:

Previously, rafiss (Rafi Shamim) wrote…

are the next two commits after this examples of things that would have failed before?

Yes, and I re-ordered the commits to make enabling last.


pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go line 48 at r2 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

this could lead to confusion. the comment has "descriptorDataIsBeingAdded indicates that descriptor data is being added" but the function is named descriptorDataIsNotBeingAdded. it looks like descriptorDataIsNotBeingAdded is correct.

Done.

Good point, and reworded this to:

// descriptorDataIsNotBeingAdded indicates if we are operating on a descriptor
// that already exists and was not created in the current transaction. This is
// determined by detecting if the data element is public, and not going from
// absent to public which newly created descriptors will.


pkg/sql/schemachanger/sctest/backup.go line 117 at r5 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

why only for these objects? what about functions, schemas, types, and databases?

Good point add all other object types here.


pkg/sql/schemachanger/sctest/backup.go line 482 at r5 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

super nit: "afterwards"

Done.

@fqazi fqazi requested a review from rafiss January 25, 2024 03:54
Copy link
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained

Previously, when adding / dropping a relation in a single transaction,
we did not properly account for data elements. This could lead to DROP
operations doing work when they shouldn't in the same transaction. This
would happen because normally the DROP would no-op steps because they
were added by the create. To address this, this patch will add data
elements for CREATE SEQUENCE, and modify rules to skip the two version
invariant allowing these cases to work properly.

Release note: None
Previously, we only had coverage for a creating a sequence
on its own in the declarative schemachanger. This patch adds,
a test for creating / dropping sequences in a single txn.

Release note: None
Previously, we did not have a test case to cover the scenario where
a sequence was created and used by a column in a single txn. To address
this, this patch will add such a test.

Release note: None
Previously, the backup and restore tests ran on all of the end to end
tests file, however when we are creating new objects the intermediate
states are never backed up. As a result, these tests are expected to not
restore perfectly. To address this, this patch will avoid comparing
CREATE statements if we have a multi-statement end to end file where new
objects are made.

Release note: None
Previously, CREATE SEQUENCE was disabled in the declarative
schema changer because we were missing sufficient test coverage
and not comfortable enabling it by default. This patch enables,
CREATE SEQUENCE in the declarative schema changer again.

Release note (sql change): CREATE SEQUENCE is now enabled by default in
the declarative schema changer.
@fqazi
Copy link
Collaborator Author

fqazi commented Feb 1, 2024

@rafiss TFTR!

bors r+

@craig
Copy link
Contributor

craig bot commented Feb 1, 2024

Build succeeded:

@craig craig bot merged commit 420c8b2 into cockroachdb:master Feb 1, 2024
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants