Add update test for repair dimension slices #2824

mkindahl · 2021-01-14T09:25:18Z

In #1986, #1841, and a few others, there are tuple locking issues that can cause dimension slices to be removed when they are actually referenced by insert operations that later add constraints referring these slices.

In the fixes for this, we repair the dimension slice table by re-adding missing dimension slices referenced in the constraints, but we have no explicit test that this repair works for more complicated situations.

We should add update tests similar to those mentioned in #2065 but added to the update test. Note that the test should only break the dimension slice table for versions where the repair script is actually running during update.

NunoFilipeSantos · 2021-01-14T11:57:08Z

@mkindahl can you elaborate on what these complicated situations would be?

mkindahl · 2021-01-15T08:55:18Z

Constraints are parsed and are in several different forms. For example:

(("time" >= 3) AND ("time" < 6))
(_timescaledb_internal.get_partition_hash(tag) < 1073741823)

Since they can be of several different combinations, it is possible that errors are introduced in the parsing code that does not handle some specific situation. For example, the order of the comparisons, quotes or not quotes around column names, function calls on columns, different types of timestamps. This needs to be tested by constructing tables with different time types and different constraints, and then repair them.

This commit creates an update repair test that breaks a few tables for pre-2.0 versions to ensure that the repair script actually fixes them. The integrity check for the update tests already contain a check that dimension slices are valid, so there is no need to add a test for that. Fixes timescale#2824

This commit creates an update repair test that breaks a few tables for pre-2.0 versions to ensure that the repair script actually fixes them. The integrity check for the update tests already contain a check that dimension slices are valid, so there is no need to add a test for that. In addition, the commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what contraint had what type and generate an error. For the 2.0.0-rc1--2.0.0-rc2 repair script: if a partition constraint was broken, it would generate an error rather than repairing the dimension slices because `BIGINT_MIN` would be cast to a double float and then an attempt would be made to cast it back to `bigint`, causing an overflow error. Fixes timescale#2824

This commit creates an update repair test that breaks a few tables for pre-2.0 versions to ensure that the repair script actually fixes them. The integrity check for the update tests already contain a check that dimension slices are valid, so there is no need to add a test for that. This commit adds an extra dimension in the workflow to test updates with repair and run that separately. It also changes the update test scripts to by default run without repair tests and add the additional option `-r` for running repair tests in addition to the normal tests. In addition, the commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what contraint had what type and generate an error. For the 2.0.0-rc1--2.0.0-rc2 repair script: if a partition constraint was broken, it would generate an error rather than repairing the dimension slices because `BIGINT_MIN` would be cast to a double float and then an attempt would be made to cast it back to `bigint`, causing an overflow error. Fixes timescale#2824

The commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what constraint had what type and generate an error. For the 2.0.0-rc1--2.0.0-rc2 repair script: if a partition constraint was broken, it would generate an error rather than repairing the dimension slices because BIGINT_MIN would be cast to a double float and then an attempt would be made to cast it back to bigint, causing an overflow error. This commit also creates an update repair test that breaks a few tables for pre-2.0 versions to ensure that the repair script actually fixes them. The integrity check for the update tests already contain a check that dimension slices are valid, so there is no need to add a test for that. This commit adds an extra dimension in the workflow to test updates with repair and run that separately. It also changes the update test scripts to by default run without repair tests and add the additional option `-r` for running repair tests in addition to the normal tests. Fixes timescale#2824

The commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what constraint had what type and generate an error. For the 2.0.0-rc1--2.0.0-rc2 repair script: if a partition constraint was broken, it would generate an error rather than repairing the dimension slices because BIGINT_MIN would be cast to a double float and then an attempt would be made to cast it back to bigint, causing an overflow error. This commit also creates an update repair test that breaks a few tables for pre-2.0 versions to ensure that the repair script actually fixes them. The integrity check for the update tests already contain a check that dimension slices are valid, so there is no need to add a test for that. This commit adds an extra dimension in the workflow to test updates with repair and run that separately. It also changes the update test scripts to by default run without repair tests and add the additional option `-r` for running repair tests in addition to the normal tests. Fixes #2824

This is a backport to fix issue timescale#2824. The commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what constraint had what type and generate an error. This commit also creates an update repair test that breaks a few tables for pre-1.7.2 versions to ensure that the repair script actually fixes them. This commit adds an extra dimension in the workflow to test updates with repair and run that separately. It also changes the update test scripts to by default run without repair tests and add the additional option `-r` for running repair tests in addition to the normal tests.

This is a backport to fix issue timescale#2824. The commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what constraint had what type and generate an error. This commit also creates an update repair test that breaks a few tables for pre-1.7.2 versions to ensure that the repair script actually fixes them. This commit adds an extra dimension in the workflow to test updates with repair and run that separately. It also changes the update test scripts to by default run without repair tests and add the additional option `-r` for running repair tests in addition to the normal tests. It also updates the Travis workflow to run the update tests both with and without repair.

This is a backport to fix issue #2824. The commit fixes two bugs in the repair scripts that could prevent an update in rare circumstances. For the 1.7.1--1.7.2 repair script: if there were several missing dimension slices in different hypertables with the same column name, the repair script would be confused on what constraint had what type and generate an error. This commit also creates an update repair test that breaks a few tables for pre-1.7.2 versions to ensure that the repair script actually fixes them. This commit adds an extra dimension in the workflow to test updates with repair and run that separately. It also changes the update test scripts to by default run without repair tests and add the additional option `-r` for running repair tests in addition to the normal tests. It also updates the Travis workflow to run the update tests both with and without repair.

mkindahl mentioned this issue Jan 18, 2021

Add update test for repair table script #2838

Merged

mkindahl self-assigned this Jan 19, 2021

mkindahl closed this as completed in #2838 Jan 28, 2021

mkindahl mentioned this issue Feb 9, 2021

Fix repair in update scripts #2920

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add update test for repair dimension slices #2824

Add update test for repair dimension slices #2824

mkindahl commented Jan 14, 2021

NunoFilipeSantos commented Jan 14, 2021

mkindahl commented Jan 15, 2021

Add update test for repair dimension slices #2824

Add update test for repair dimension slices #2824

Comments

mkindahl commented Jan 14, 2021

NunoFilipeSantos commented Jan 14, 2021

mkindahl commented Jan 15, 2021