Implement a schema repair mechanism #5337
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msullivan
force-pushed
the
repair-schema
branch
from
ef2318c
to
3d31a57
Compare
elprans
reviewed
Apr 12, 2023
|
|
||
| if not delta.is_data_safe(): | ||
| raise AssertionError( | ||
| 'Repair script for version upgrade is not data safe' |
There was a problem hiding this comment.
Would this be reraised as an ISE?
|
Yeah. We could do something else, but we do want to be generating a "file a
bug" message
…On Wed, Apr 12, 2023, 16:27 Elvis Pranskevichus ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In edb/server/compiler/ddl.py
<#5337 (comment)>:
> +
+ delta = s_ddl.delta_schemas(
+ schema,
+ reloaded_schema,
+ )
+ mismatch = bool(delta.get_subcommands())
+ if not mismatch:
+ return None
+
+ if debug.flags.delta_plan:
+ debug.header('Repair Delta')
+ debug.dump(delta)
+
+ if not delta.is_data_safe():
+ raise AssertionError(
+ 'Repair script for version upgrade is not data safe'
Would this be reraised as an ISE?
—
Reply to this email directly, view it on GitHub
<#5337 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACTC7OYGHBUR2242772GFLXA425VANCNFSM6AAAAAAW4JRP6I>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
That's exactly what we want to do, just making sure we are actually turning vanilla exceptions into ISEs in the bootstrap path. |
|
Oh sure enough, we are not actually |
msullivan
force-pushed
the
repair-schema
branch
from
4edf7fd
to
3d31a57
Compare
OK, yeah, it does not do anything nice. "Server fails to start and a traceback in the logs" is probably decent signal anyway, but I'm going to do a separate PR to put a better message in |
msullivan
force-pushed
the
repair-schema
branch
from
d597b53
to
1548b20
Compare
|
Hmmmmm. There's a danger with this scheme, though, which is: what to do if trying to reload the schema fails for a legitimate reason? After reviewing the 2.0 changelog, I'm less worried about this than I used to be. The only PR I spotted in a quick pass with a risk of this was #4616, which is pretty marginal. I know the 1.x series had some worse ones though It might be fine to just make the log message in that case say that you might need to fix-and-squash? |
As discussed in #5321, certain changes and bugfixes can result in the schema state being inconsistent, with inferred attributes that do not match what would be created if the schema was created freshly. Implement a repair system that works by serializing the current schema to DDL, generating a fresh schema from that, then comparing the current schema to this new target and applying the delta. For safety, we refuse if the change is not data-safe. This mechanism is exposed as a new "patch kind" for the patch mechanism, and as a test-mode ADMINISTER function. I tested this manually using #5180 and #5329. When we add 'repair' patches to stable branches we should endeavor to make sure that it actually gets exercised by the old versions CI job. Also tested with an edgeql+schema patch made after it. Fixes #5321.
msullivan
force-pushed
the
repair-schema
branch
from
1548b20
to
0b5afc8
Compare
msullivan
added a commit
that referenced
this pull request
Apr 20, 2023
As discussed in #5321, certain changes and bugfixes can result in the schema state being inconsistent, with inferred attributes that do not match what would be created if the schema was created freshly. Implement a repair system that works by serializing the current schema to DDL, generating a fresh schema from that, then comparing the current schema to this new target and applying the delta. For safety, we refuse if the change is not data-safe. This mechanism is exposed as a new "patch kind" for the patch mechanism, and as a test-mode ADMINISTER function. I tested this manually using #5180 and #5329. When we add 'repair' patches to stable branches we should endeavor to make sure that it actually gets exercised by the old versions CI job. Also tested with an edgeql+schema patch made after it. Fixes #5321.
msullivan
added a commit
that referenced
this pull request
Apr 21, 2023
As discussed in #5321, certain changes and bugfixes can result in the schema state being inconsistent, with inferred attributes that do not match what would be created if the schema was created freshly. Implement a repair system that works by serializing the current schema to DDL, generating a fresh schema from that, then comparing the current schema to this new target and applying the delta. For safety, we refuse if the change is not data-safe. This mechanism is exposed as a new "patch kind" for the patch mechanism, and as a test-mode ADMINISTER function. I tested this manually using #5180 and #5329. When we add 'repair' patches to stable branches we should endeavor to make sure that it actually gets exercised by the old versions CI job. Also tested with an edgeql+schema patch made after it. Fixes #5321.
raddevon
pushed a commit
that referenced
this pull request
May 8, 2023
As discussed in #5321, certain changes and bugfixes can result in the schema state being inconsistent, with inferred attributes that do not match what would be created if the schema was created freshly. Implement a repair system that works by serializing the current schema to DDL, generating a fresh schema from that, then comparing the current schema to this new target and applying the delta. For safety, we refuse if the change is not data-safe. This mechanism is exposed as a new "patch kind" for the patch mechanism, and as a test-mode ADMINISTER function. I tested this manually using #5180 and #5329. When we add 'repair' patches to stable branches we should endeavor to make sure that it actually gets exercised by the old versions CI job. Also tested with an edgeql+schema patch made after it. Fixes #5321.
raddevon
added a commit
that referenced
this pull request
May 15, 2023
* Document `edgedb watch` and new dev workflow * Fix numbering * Document nested modules (#5153) * Document set intersect, except operators (#5155) Co-authored-by: Victor Petrovykh <victor@edgedb.com> * config: Remove mandatory reliance on global config spec (#5357) Parts of config system innards are still exclusively reliant on global `spec`. Fix this and remove `config.set_settings()` from tests to avoid concurrency issues. * More cleanups in sertypes (#5359) Use helpers to encode strings and descriptor references, add comments, and clean things up a bit. * Type descriptor parsing cleanups (#5361) Apply the same treatment as for the marshalling side: use dispatch to parse, add context, and add helpers for common patterns. * Fix insert default cycle (#5355) * Analyze (explain) improvements (#5263) More comprehensive explain heuristics. That includes coarse-grained tree, fine-grained tree and debugging info. * Implement a schema repair mechanism (#5337) As discussed in #5321, certain changes and bugfixes can result in the schema state being inconsistent, with inferred attributes that do not match what would be created if the schema was created freshly. Implement a repair system that works by serializing the current schema to DDL, generating a fresh schema from that, then comparing the current schema to this new target and applying the delta. For safety, we refuse if the change is not data-safe. This mechanism is exposed as a new "patch kind" for the patch mechanism, and as a test-mode ADMINISTER function. I tested this manually using #5180 and #5329. When we add 'repair' patches to stable branches we should endeavor to make sure that it actually gets exercised by the old versions CI job. Also tested with an edgeql+schema patch made after it. Fixes #5321. * Optimize delta_objects by taking into account name information (#5351) The idea is simply that if an object named `Foo` exists in both the old and new schemas, compare those `Foo`s against each other and nothing else, saving the quadratic cross-product based comparison for objects that don't appear on both sides. On the pretty modestly sized `cards` database, this reduces the cost of the initial `describe current migration as json` for a trivial property addition from about 3.5s to 0.9s. Dropping the inoperative `hash_criteria` optimization shaves another 0.05s. Progress on #5322. * Doc v3 access policy errors (#5220) * Change warning with emoji to warning admonition * Change version warning to `versionadded` directive * Document custom access policy error messages * Enable `pg_dump` functionality. Provide SQL adapters that allow `pg_dump` to capture the contents of an EdgeDB database. This functionality is not meant to be used as backup (it will not be sufficient to restore the original EdgeDB database, use `edgedb dump` tools for that). It is instead meant to be used as a way to transfer data to a Postgres database potentially as a compatibility layer with other SQL tooling. In the current version the dump completely omits the following: - pg_stats - pg_cast - pg_proc - pg_operator - pg_trigger It also generally modifies some expressions such as defaults because they may contain references to omitted functions while not affecting the already stored data. The user-defined modules get mapped onto namespaces, however the built-ins and some supporting functionality reside in `edgedbstd` and `edgedb`. Co-authored-by: Aljaž Mur Eržen <aljaz.erzen@gmail.com> Co-authored-by: Michael J. Sullivan <sully@msully.net> * Small grammar fixes and wording suggestions (#5353) * Small grammar fixes and wording suggestions * Fix line overflow * Update docs/edgeql/paths.rst Co-authored-by: Devon Campbell <devon@edgedb.com> * Update docs/intro/schema.rst Co-authored-by: Devon Campbell <devon@edgedb.com> --------- Co-authored-by: Devon Campbell <devon@edgedb.com> * Fix test_docs to cope with missing test deps again (#5370) * Some fixes to database repairs (#5374) * Actually update the patch count in the database * Print log messages about the repairing * Write changelog for 2.14. (#5375) This has a bigger note than normal because I wanted to explain schema repair and suggest taking a dump before update. * Fix: eql compiler not using context correctly (#5371) * Forbid ranges of user-defined scalars (#5345) * Fix minor TypeScript syntax error in docs (#5378) * Document triggers (#5157) * Document triggers * Move introspection from "Guides" to "Schema" Guides will not be versioned in our documentation. We will separate versioned sections from non-versions sections in the UI to make it clear to users which sections are affected by the version selection dropdown and which are not. Introspection needs to be versioned, so it needs to be moved out of guides to make it clear once the UI change is implemented that it is part of the versioned documentation. This wouldn't have required changing the labels and references to labels for the introspection section, but I decided to do it anyway to keep the documentation consistent and to reduce any confusion for users who want to contribute and see that these labels do not follow our convention. * Add `for all` trigger example * Apply minor fixes * Explain __old__/__new__ * Explain handling triggers fired by other triggers * Update docs/reference/sdl/triggers.rst Co-authored-by: Zack Elan <zackelan@users.noreply.github.com> * Note that triggers run in a transaction w/ query * Update link label to reflect we are not linking direction to querybuilder docs but to the JS client's docs --------- Co-authored-by: Zack Elan <zackelan@users.noreply.github.com> * Updated elgohr/Publish-Docker-Github-Action to a supported version (v5) (#5262) * Added system config statement_cache_size (#5250) Used to control the size of Postgres prepared statement cache. * Also statically evaluate constraints on setting config properties * Replace v2 note with `versionadded` directive (#5369) * Clarification that backlinks are multi by default (#5387) * Clarification that backlinks are multi by default * rv separate 3.0 version (code identical) Co-authored-by: Devon Campbell <devon@edgedb.com> --------- Co-authored-by: Devon Campbell <devon@edgedb.com> * release: Remove Debian Stretch and Ubuntu Xenial (#5406) These reached EOL upstream and are no longer supported. * Fix references to single links on `__old__` in triggers (#5410) The issue was a subtle condition in path compilation where if a path was visible, but didn't have a source aspect (like `__old__`), *and* wasn't in the path_scope (which `__old__` isn't, but also it gets cleared in DML anyway), then the source rvar we create with ensure_source_rvar would get lost. Fixes #5384. * Fix dropping a pointer's constraint and making it computed at the same time (#5411) There were ordering issues in the SQL DDL we generated. Fixes #5381. * Don't claim that making a pointer computed is data-safe (#5412) The right way to do this is to use the AlterSpecialObjectField machinery, but that's a more involved change and I'm all about quick fixes this week. Fixes #5408. * Prohibit NUL character in query source (#5414) * Fix moving a trigger to a parent type (#5416) Two things are needed: * Make owned on triggers have compcoef 0, since set owned doesn't exist on triggers * Make DELETE + CREATE on parent pairs work right in ordering Fixes #5382. * Refactor EdgeQL AST to remove multiple inheritance (#5399) * inline LimitOffsetMixin * inline Clause abstract class * remove qlast.astmatch * inline ReturningMixin * inline FilterMixin * inline OrderByMixin * inline SubjectMixin * inject SelectClauseMixin * inline BaseAlias * nest SessionSetAliasDecl * ast refactor * inline Statement * rename mixins * make code darker * fix test suite * Parameterize schema for pg extension (uuid-ossp) (#5413) This supports the latest Heroku where extensions must be created under the schema "heroku_ext" instead of our preferred "edgedbext". Co-authored-by: Elvis Pranskevichus <elvis@edgedb.com> * analyze: fix introspection tables (#5418) * Specify instance-level config parameters as CLI arguments / env vars (#5266) * Extract and expose coerce_single_value() * Add new config layers through _edgecon_state The new "environment variable" and "command line" layers of config are set when starting the EdgeDB server through set_init_con_script_data(), and keeps static for the lifetime of the server process. Because one Postgres may have multiple EdgeDB servers, the temporary table _edgecon_state is reused to bridge the per-instance config set in environment variables or command-line arguments into the backend. Also fixed so that INIT_CON_SCRIPT is only built when apply_init_script. * Take and use cfg args from command-line and env The extra command-line arguments are parsed after pg cluster initialization when we have the config spec. If any of these config mirrors a backend setting, the pg cluster will be restarted to take that setting as a command-line argument, in order to match the priority of the origin. * Handle listen_addresses and listen_port * Fix bugs in setting Server._listen_hosts * `configure instance reset` only removes overriden values * Document mutation rewrites (#5324) * Document mutation rewrites * Apply suggestions from code review Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix line lengths * Add additional `__specified__` example Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix syntax --------- Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix tuples in global variables and do error checks on arrays (#5417) We need to recode tuples to inject OIDs, like we do with arrays and the like. Somewhat annoyingly, this ends up as a totally separate code path from the existing re-encoding of tuples for parameters. This is because for globals, which are schema declared, we have postgres composite types, and it would be wasteful and convoluted to split the tuples up instead of just using them. And then, once we are injecting the OIDs, we can do missing error checking that could allow invalid data in. (This was the main thing that motivated fixing this now instead of deffering.) Fixes #5356. * Fix rewrites inhereted from abstract types (#5419) * Add more 3.0 dump tests and fix reflected name of triggers (#5420) The *first* instance of triggers in the schema got it right, and then d4a5a16 factored out duplicated name handling but forgot to use the factored out macinery in triggers! * Increase the line length limit from 79 to 80 (#5422) This is very marginal, and because it so marginal and because 80 is a nice round number, I expect that nobody will spend any of their "one wild and precious life" arguing about it. * Allow `ADMINISTER repair_schema()` outside of test mode (#5421) Originally "all" ADMINISTER commands (really just ANALYZE) were hidden behind test mode, but we need to make case-by-case decisions. I feel pretty strongly that we need to allow `repair_schema()`, though I would refrain from *documenting* it yet. One main use case is if we ship a point relase that uses a "repair" patch, a user takes it and upgrades, and then runs into trouble and needs to downgrade. Now their schema might be out of sync, and doing a repair to undo the first repair would solve the problem. And, importantly, we can't just wait until we've seen users encounter this problem before doing something, because at that point it would be too late: they would need the feature present in an *older* version. But we can skip documenting it until it seems like it is coming up. * Fixup extension schema customization (#5423) Somehow this bit got messed up while I was merging #5413. * Fix SQL introspection on custom Postgres versions (#5380) * Avoid race in _early_introspect_db (#5426) * Fix migration that delete an link alias computed in a parent and child (#5428) Pointer.get_implicit_ancestors is supposed to only return "actual" ancestors of the pointer, not ancestors that occur because of computeds that directly reference another pointer. The logic for this didn't work right when there was another level of inheritance, though. Fixes #4769. * Try to make the NOCREATEDB test suite pass (#5424) Refresh the connection to avoid a state mismatch after restoring from dump. Fix the explain tests to not depend on a bunch of cost numbers that can't be relied on (though that just came up on heroku, not in non-heroku NOCREATEDB). Skip the SQL pg_dump tests. * More test suite fixing for NOCREATEDB (#5429) * Document SQL support (#5379) * Document SQL support Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Update docs/reference/sql_support.rst Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix line length * Clarify note about SSL encryption to note that this should be applied on the tool/client side, not the EdgeDB side * Add log replication footnote * Fix spelling * Fix SDL syntax in example * Break up large SQL block * Simplify page name * Explain why SSL is required * mimick -> mimic * Skip syntax checking for SQL * Add explicit instructions for connecting * Note in opening that queries are read-only --------- Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Document PG indexes (#5430) * Always disable the metapkg git cache (#5432) I am losing my mind. * Add 3.0 changelog (#5401) * Fix install link (#5434) * 3.0 beta changelog fixes (#5436) * Change cli upgrade to project upgrade * Correction + suggest nightly first if necessary * Looks like should be instance * Correction: first CLI + nightly 1st if necessary * Document 3.0 CLI changes * Note 3.0 features --------- Co-authored-by: Victor Petrovykh <victor@edgedb.com> Co-authored-by: Elvis Pranskevichus <elvis@edgedb.com> Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> Co-authored-by: Paul Colomiets <paul@edgedb.com> Co-authored-by: Michael J. Sullivan <sully@msully.net> Co-authored-by: Aljaž Mur Eržen <aljaz.erzen@gmail.com> Co-authored-by: Dave MacLeod <56599343+Dhghomon@users.noreply.github.com> Co-authored-by: Aron Adler <Arrow7000@users.noreply.github.com> Co-authored-by: Zack Elan <zackelan@users.noreply.github.com> Co-authored-by: Lars Gohr <elgohr@users.noreply.github.com> Co-authored-by: Fantix King <fantix.king@gmail.com>
msullivan
added a commit
that referenced
this pull request
May 15, 2023
* Document `edgedb watch` and new dev workflow * Fix numbering * Document nested modules (#5153) * Document set intersect, except operators (#5155) Co-authored-by: Victor Petrovykh <victor@edgedb.com> * config: Remove mandatory reliance on global config spec (#5357) Parts of config system innards are still exclusively reliant on global `spec`. Fix this and remove `config.set_settings()` from tests to avoid concurrency issues. * More cleanups in sertypes (#5359) Use helpers to encode strings and descriptor references, add comments, and clean things up a bit. * Type descriptor parsing cleanups (#5361) Apply the same treatment as for the marshalling side: use dispatch to parse, add context, and add helpers for common patterns. * Fix insert default cycle (#5355) * Analyze (explain) improvements (#5263) More comprehensive explain heuristics. That includes coarse-grained tree, fine-grained tree and debugging info. * Implement a schema repair mechanism (#5337) As discussed in #5321, certain changes and bugfixes can result in the schema state being inconsistent, with inferred attributes that do not match what would be created if the schema was created freshly. Implement a repair system that works by serializing the current schema to DDL, generating a fresh schema from that, then comparing the current schema to this new target and applying the delta. For safety, we refuse if the change is not data-safe. This mechanism is exposed as a new "patch kind" for the patch mechanism, and as a test-mode ADMINISTER function. I tested this manually using #5180 and #5329. When we add 'repair' patches to stable branches we should endeavor to make sure that it actually gets exercised by the old versions CI job. Also tested with an edgeql+schema patch made after it. Fixes #5321. * Optimize delta_objects by taking into account name information (#5351) The idea is simply that if an object named `Foo` exists in both the old and new schemas, compare those `Foo`s against each other and nothing else, saving the quadratic cross-product based comparison for objects that don't appear on both sides. On the pretty modestly sized `cards` database, this reduces the cost of the initial `describe current migration as json` for a trivial property addition from about 3.5s to 0.9s. Dropping the inoperative `hash_criteria` optimization shaves another 0.05s. Progress on #5322. * Doc v3 access policy errors (#5220) * Change warning with emoji to warning admonition * Change version warning to `versionadded` directive * Document custom access policy error messages * Enable `pg_dump` functionality. Provide SQL adapters that allow `pg_dump` to capture the contents of an EdgeDB database. This functionality is not meant to be used as backup (it will not be sufficient to restore the original EdgeDB database, use `edgedb dump` tools for that). It is instead meant to be used as a way to transfer data to a Postgres database potentially as a compatibility layer with other SQL tooling. In the current version the dump completely omits the following: - pg_stats - pg_cast - pg_proc - pg_operator - pg_trigger It also generally modifies some expressions such as defaults because they may contain references to omitted functions while not affecting the already stored data. The user-defined modules get mapped onto namespaces, however the built-ins and some supporting functionality reside in `edgedbstd` and `edgedb`. Co-authored-by: Aljaž Mur Eržen <aljaz.erzen@gmail.com> Co-authored-by: Michael J. Sullivan <sully@msully.net> * Small grammar fixes and wording suggestions (#5353) * Small grammar fixes and wording suggestions * Fix line overflow * Update docs/edgeql/paths.rst Co-authored-by: Devon Campbell <devon@edgedb.com> * Update docs/intro/schema.rst Co-authored-by: Devon Campbell <devon@edgedb.com> --------- Co-authored-by: Devon Campbell <devon@edgedb.com> * Fix test_docs to cope with missing test deps again (#5370) * Some fixes to database repairs (#5374) * Actually update the patch count in the database * Print log messages about the repairing * Write changelog for 2.14. (#5375) This has a bigger note than normal because I wanted to explain schema repair and suggest taking a dump before update. * Fix: eql compiler not using context correctly (#5371) * Forbid ranges of user-defined scalars (#5345) * Fix minor TypeScript syntax error in docs (#5378) * Document triggers (#5157) * Document triggers * Move introspection from "Guides" to "Schema" Guides will not be versioned in our documentation. We will separate versioned sections from non-versions sections in the UI to make it clear to users which sections are affected by the version selection dropdown and which are not. Introspection needs to be versioned, so it needs to be moved out of guides to make it clear once the UI change is implemented that it is part of the versioned documentation. This wouldn't have required changing the labels and references to labels for the introspection section, but I decided to do it anyway to keep the documentation consistent and to reduce any confusion for users who want to contribute and see that these labels do not follow our convention. * Add `for all` trigger example * Apply minor fixes * Explain __old__/__new__ * Explain handling triggers fired by other triggers * Update docs/reference/sdl/triggers.rst Co-authored-by: Zack Elan <zackelan@users.noreply.github.com> * Note that triggers run in a transaction w/ query * Update link label to reflect we are not linking direction to querybuilder docs but to the JS client's docs --------- Co-authored-by: Zack Elan <zackelan@users.noreply.github.com> * Updated elgohr/Publish-Docker-Github-Action to a supported version (v5) (#5262) * Added system config statement_cache_size (#5250) Used to control the size of Postgres prepared statement cache. * Also statically evaluate constraints on setting config properties * Replace v2 note with `versionadded` directive (#5369) * Clarification that backlinks are multi by default (#5387) * Clarification that backlinks are multi by default * rv separate 3.0 version (code identical) Co-authored-by: Devon Campbell <devon@edgedb.com> --------- Co-authored-by: Devon Campbell <devon@edgedb.com> * release: Remove Debian Stretch and Ubuntu Xenial (#5406) These reached EOL upstream and are no longer supported. * Fix references to single links on `__old__` in triggers (#5410) The issue was a subtle condition in path compilation where if a path was visible, but didn't have a source aspect (like `__old__`), *and* wasn't in the path_scope (which `__old__` isn't, but also it gets cleared in DML anyway), then the source rvar we create with ensure_source_rvar would get lost. Fixes #5384. * Fix dropping a pointer's constraint and making it computed at the same time (#5411) There were ordering issues in the SQL DDL we generated. Fixes #5381. * Don't claim that making a pointer computed is data-safe (#5412) The right way to do this is to use the AlterSpecialObjectField machinery, but that's a more involved change and I'm all about quick fixes this week. Fixes #5408. * Prohibit NUL character in query source (#5414) * Fix moving a trigger to a parent type (#5416) Two things are needed: * Make owned on triggers have compcoef 0, since set owned doesn't exist on triggers * Make DELETE + CREATE on parent pairs work right in ordering Fixes #5382. * Refactor EdgeQL AST to remove multiple inheritance (#5399) * inline LimitOffsetMixin * inline Clause abstract class * remove qlast.astmatch * inline ReturningMixin * inline FilterMixin * inline OrderByMixin * inline SubjectMixin * inject SelectClauseMixin * inline BaseAlias * nest SessionSetAliasDecl * ast refactor * inline Statement * rename mixins * make code darker * fix test suite * Parameterize schema for pg extension (uuid-ossp) (#5413) This supports the latest Heroku where extensions must be created under the schema "heroku_ext" instead of our preferred "edgedbext". Co-authored-by: Elvis Pranskevichus <elvis@edgedb.com> * analyze: fix introspection tables (#5418) * Specify instance-level config parameters as CLI arguments / env vars (#5266) * Extract and expose coerce_single_value() * Add new config layers through _edgecon_state The new "environment variable" and "command line" layers of config are set when starting the EdgeDB server through set_init_con_script_data(), and keeps static for the lifetime of the server process. Because one Postgres may have multiple EdgeDB servers, the temporary table _edgecon_state is reused to bridge the per-instance config set in environment variables or command-line arguments into the backend. Also fixed so that INIT_CON_SCRIPT is only built when apply_init_script. * Take and use cfg args from command-line and env The extra command-line arguments are parsed after pg cluster initialization when we have the config spec. If any of these config mirrors a backend setting, the pg cluster will be restarted to take that setting as a command-line argument, in order to match the priority of the origin. * Handle listen_addresses and listen_port * Fix bugs in setting Server._listen_hosts * `configure instance reset` only removes overriden values * Document mutation rewrites (#5324) * Document mutation rewrites * Apply suggestions from code review Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix line lengths * Add additional `__specified__` example Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix syntax --------- Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix tuples in global variables and do error checks on arrays (#5417) We need to recode tuples to inject OIDs, like we do with arrays and the like. Somewhat annoyingly, this ends up as a totally separate code path from the existing re-encoding of tuples for parameters. This is because for globals, which are schema declared, we have postgres composite types, and it would be wasteful and convoluted to split the tuples up instead of just using them. And then, once we are injecting the OIDs, we can do missing error checking that could allow invalid data in. (This was the main thing that motivated fixing this now instead of deffering.) Fixes #5356. * Fix rewrites inhereted from abstract types (#5419) * Add more 3.0 dump tests and fix reflected name of triggers (#5420) The *first* instance of triggers in the schema got it right, and then d4a5a16 factored out duplicated name handling but forgot to use the factored out macinery in triggers! * Increase the line length limit from 79 to 80 (#5422) This is very marginal, and because it so marginal and because 80 is a nice round number, I expect that nobody will spend any of their "one wild and precious life" arguing about it. * Allow `ADMINISTER repair_schema()` outside of test mode (#5421) Originally "all" ADMINISTER commands (really just ANALYZE) were hidden behind test mode, but we need to make case-by-case decisions. I feel pretty strongly that we need to allow `repair_schema()`, though I would refrain from *documenting* it yet. One main use case is if we ship a point relase that uses a "repair" patch, a user takes it and upgrades, and then runs into trouble and needs to downgrade. Now their schema might be out of sync, and doing a repair to undo the first repair would solve the problem. And, importantly, we can't just wait until we've seen users encounter this problem before doing something, because at that point it would be too late: they would need the feature present in an *older* version. But we can skip documenting it until it seems like it is coming up. * Fixup extension schema customization (#5423) Somehow this bit got messed up while I was merging #5413. * Fix SQL introspection on custom Postgres versions (#5380) * Avoid race in _early_introspect_db (#5426) * Fix migration that delete an link alias computed in a parent and child (#5428) Pointer.get_implicit_ancestors is supposed to only return "actual" ancestors of the pointer, not ancestors that occur because of computeds that directly reference another pointer. The logic for this didn't work right when there was another level of inheritance, though. Fixes #4769. * Try to make the NOCREATEDB test suite pass (#5424) Refresh the connection to avoid a state mismatch after restoring from dump. Fix the explain tests to not depend on a bunch of cost numbers that can't be relied on (though that just came up on heroku, not in non-heroku NOCREATEDB). Skip the SQL pg_dump tests. * More test suite fixing for NOCREATEDB (#5429) * Document SQL support (#5379) * Document SQL support Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Update docs/reference/sql_support.rst Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Fix line length * Clarify note about SSL encryption to note that this should be applied on the tool/client side, not the EdgeDB side * Add log replication footnote * Fix spelling * Fix SDL syntax in example * Break up large SQL block * Simplify page name * Explain why SSL is required * mimick -> mimic * Skip syntax checking for SQL * Add explicit instructions for connecting * Note in opening that queries are read-only --------- Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> * Document PG indexes (#5430) * Always disable the metapkg git cache (#5432) I am losing my mind. * Add 3.0 changelog (#5401) * Fix install link (#5434) * 3.0 beta changelog fixes (#5436) * Change cli upgrade to project upgrade * Correction + suggest nightly first if necessary * Looks like should be instance * Correction: first CLI + nightly 1st if necessary * Document 3.0 CLI changes * Note 3.0 features --------- Co-authored-by: Victor Petrovykh <victor@edgedb.com> Co-authored-by: Elvis Pranskevichus <elvis@edgedb.com> Co-authored-by: Aljaž Mur Eržen <aljazerzen@users.noreply.github.com> Co-authored-by: Paul Colomiets <paul@edgedb.com> Co-authored-by: Michael J. Sullivan <sully@msully.net> Co-authored-by: Aljaž Mur Eržen <aljaz.erzen@gmail.com> Co-authored-by: Dave MacLeod <56599343+Dhghomon@users.noreply.github.com> Co-authored-by: Aron Adler <Arrow7000@users.noreply.github.com> Co-authored-by: Zack Elan <zackelan@users.noreply.github.com> Co-authored-by: Lars Gohr <elgohr@users.noreply.github.com> Co-authored-by: Fantix King <fantix.king@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in #5321, certain changes and bugfixes can result in the
schema state being inconsistent, with inferred attributes that do not
match what would be created if the schema was created freshly.
Implement a repair system that works by serializing the current schema
to DDL, generating a fresh schema from that, then comparing the
current schema to this new target and applying the delta.
For safety, we refuse if the change is not data-safe.
This mechanism is exposed as a new "patch kind" for the patch mechanism,
and as a test-mode ADMINISTER function.
I tested this manually using #5180 and #5329. When we add 'repair'
patches to stable branches we should endeavor to make sure that it
actually gets exercised by the old versions CI job.
Also tested with an edgeql+schema patch made after it.
Fixes #5321.