Skip to content

Commit

Permalink
Fix an issue within the E-Maj rollback processing when 1) two tables …
Browse files Browse the repository at this point in the history 
…of the rolled back tables groups are linked by a foreign key, 2) one of these tables has been assigned to its group after the other, and 3) an E-Maj rollback is executed targeting a mark set before the last table assignment. In this case, there was a risk of undetected referential integrity violation. This was introduced with E-Maj 4.0.0 that introduced the 'session_replication_role=replica' optimization in tables rollback processing. In this case, drop and re-add the FK or differ the FK checks at the transaction end, as it was done before the 'session_replication_role=replica' optimization. A test case has been added to the alter_logging.sql script.
  • Loading branch information
@beaud76
beaud76 committed Mar 27, 2023
1 parent 7ab5e7e commit c1a9cff
Show file tree
Hide file tree
Showing 28 changed files with 1,327 additions and 998 deletions.
5 changes: 5 additions & 0 deletions CHANGES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ E-Maj - Change log
group.

###Bug fixes:###
* Fix a bug in the rollback processing. When two tables of rolled back tables
group are linked by a foreign key, and one of these tables has been
assigned to the group after the other, an E-Maj rollback to a mark set
before the last table assignment may lead to an undetected referential
integrity violation.
* Fix a bug in both emaj_move_tables() and emaj_move_sequences() functions.
When several tables/sequences are moved at once to another tables group and
the destination group is the same as the source group for at least 2
Expand Down
15 changes: 10 additions & 5 deletions sql/emaj--4.1.0--devel.sql
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2367,15 +2367,17 @@ $_rlbk_planning$
) AS t
WHERE emaj._log_stat_tbl(t, greatest(v_markTimeId, lower(rel_time_range)), NULL) > 0;
GET DIAGNOSTICS v_effNbTable = ROW_COUNT;
-- For tables having all foreign keys linking tables in the rolled back groups, set the rlbp_is_repl_role_replica flag to TRUE.
-- Set the rlbp_is_repl_role_replica flag to TRUE for tables having all foreign keys linking tables:
-- 1) in the rolled back groups and 2) with the same rollback target mark.
-- This only concerns emaj installed as an extension because one needs to be sure that the _rlbk_tbl() function is executed with a
-- superuser role (this is needed to set the session_replication_role to 'replica').
v_isEmajExtension = EXISTS (SELECT 1 FROM pg_catalog.pg_extension WHERE extname = 'emaj');
IF v_isEmajExtension AND v_effNbTable > 0 THEN
WITH fkeys AS (
-- the foreign keys belonging to tables to rollback
SELECT rlbp_schema, rlbp_table, c.conname, nf.nspname, tf.relname, rel_group,
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups,
rlbp_target_time_id = greatest(v_markTimeId, lower(rel_time_range)) AS have_both_tables_the_same_target_mark
FROM emaj.emaj_rlbk_plan,
pg_catalog.pg_constraint c
JOIN pg_catalog.pg_class t ON (t.oid = c.conrelid)
Expand All @@ -2394,7 +2396,8 @@ $_rlbk_planning$
UNION
-- the foreign keys referencing tables to rollback
SELECT rlbp_schema, rlbp_table, c.conname, n.nspname, t.relname, rel_group,
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups,
rlbp_target_time_id = greatest(v_markTimeId, lower(rel_time_range)) AS have_both_tables_the_same_target_mark
FROM emaj.emaj_rlbk_plan,
pg_catalog.pg_constraint c
JOIN pg_catalog.pg_class t ON (t.oid = c.conrelid)
Expand All @@ -2413,7 +2416,8 @@ $_rlbk_planning$
), fkeys_agg AS (
-- aggregated foreign keys by tables to rollback
SELECT rlbp_schema, rlbp_table,
count(*) AS nb_all_fk, count(*) FILTER (WHERE are_both_tables_in_groups) AS nb_fk_groups_ok
count(*) AS nb_fk,
count(*) FILTER (WHERE are_both_tables_in_groups AND have_both_tables_the_same_target_mark) AS nb_fk_ok
FROM fkeys
GROUP BY 1,2
)
Expand All @@ -2424,7 +2428,8 @@ $_rlbk_planning$
AND rlbp_step IN ('RLBK_TABLE', 'LOCK_TABLE')
AND emaj_rlbk_plan.rlbp_table = fkeys_agg.rlbp_table
AND emaj_rlbk_plan.rlbp_schema = fkeys_agg.rlbp_schema
AND nb_all_fk = nb_fk_groups_ok -- if all fkeys are linking tables in the rolled back groups
AND nb_fk = nb_fk_ok -- if all fkeys are linking tables 1) in the rolled back groups
-- and 2) with the same rollback target mark
;
END IF;
--
Expand Down
15 changes: 10 additions & 5 deletions sql/emaj--devel.sql
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8233,15 +8233,17 @@ $_rlbk_planning$
) AS t
WHERE emaj._log_stat_tbl(t, greatest(v_markTimeId, lower(rel_time_range)), NULL) > 0;
GET DIAGNOSTICS v_effNbTable = ROW_COUNT;
-- For tables having all foreign keys linking tables in the rolled back groups, set the rlbp_is_repl_role_replica flag to TRUE.
-- Set the rlbp_is_repl_role_replica flag to TRUE for tables having all foreign keys linking tables:
-- 1) in the rolled back groups and 2) with the same rollback target mark.
-- This only concerns emaj installed as an extension because one needs to be sure that the _rlbk_tbl() function is executed with a
-- superuser role (this is needed to set the session_replication_role to 'replica').
v_isEmajExtension = EXISTS (SELECT 1 FROM pg_catalog.pg_extension WHERE extname = 'emaj');
IF v_isEmajExtension AND v_effNbTable > 0 THEN
WITH fkeys AS (
-- the foreign keys belonging to tables to rollback
SELECT rlbp_schema, rlbp_table, c.conname, nf.nspname, tf.relname, rel_group,
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups,
rlbp_target_time_id = greatest(v_markTimeId, lower(rel_time_range)) AS have_both_tables_the_same_target_mark
FROM emaj.emaj_rlbk_plan,
pg_catalog.pg_constraint c
JOIN pg_catalog.pg_class t ON (t.oid = c.conrelid)
Expand All @@ -8260,7 +8262,8 @@ $_rlbk_planning$
UNION
-- the foreign keys referencing tables to rollback
SELECT rlbp_schema, rlbp_table, c.conname, n.nspname, t.relname, rel_group,
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups,
rlbp_target_time_id = greatest(v_markTimeId, lower(rel_time_range)) AS have_both_tables_the_same_target_mark
FROM emaj.emaj_rlbk_plan,
pg_catalog.pg_constraint c
JOIN pg_catalog.pg_class t ON (t.oid = c.conrelid)
Expand All @@ -8279,7 +8282,8 @@ $_rlbk_planning$
), fkeys_agg AS (
-- aggregated foreign keys by tables to rollback
SELECT rlbp_schema, rlbp_table,
count(*) AS nb_all_fk, count(*) FILTER (WHERE are_both_tables_in_groups) AS nb_fk_groups_ok
count(*) AS nb_fk,
count(*) FILTER (WHERE are_both_tables_in_groups AND have_both_tables_the_same_target_mark) AS nb_fk_ok
FROM fkeys
GROUP BY 1,2
)
Expand All @@ -8290,7 +8294,8 @@ $_rlbk_planning$
AND rlbp_step IN ('RLBK_TABLE', 'LOCK_TABLE')
AND emaj_rlbk_plan.rlbp_table = fkeys_agg.rlbp_table
AND emaj_rlbk_plan.rlbp_schema = fkeys_agg.rlbp_schema
AND nb_all_fk = nb_fk_groups_ok -- if all fkeys are linking tables in the rolled back groups
AND nb_fk = nb_fk_ok -- if all fkeys are linking tables 1) in the rolled back groups
-- and 2) with the same rollback target mark
;
END IF;
--
Expand Down
15 changes: 10 additions & 5 deletions sql/emaj-devel.sql
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8240,15 +8240,17 @@ $_rlbk_planning$
) AS t
WHERE emaj._log_stat_tbl(t, greatest(v_markTimeId, lower(rel_time_range)), NULL) > 0;
GET DIAGNOSTICS v_effNbTable = ROW_COUNT;
-- For tables having all foreign keys linking tables in the rolled back groups, set the rlbp_is_repl_role_replica flag to TRUE.
-- Set the rlbp_is_repl_role_replica flag to TRUE for tables having all foreign keys linking tables:
-- 1) in the rolled back groups and 2) with the same rollback target mark.
-- This only concerns emaj installed as an extension because one needs to be sure that the _rlbk_tbl() function is executed with a
-- superuser role (this is needed to set the session_replication_role to 'replica').
v_isEmajExtension = EXISTS (SELECT 1 FROM pg_catalog.pg_extension WHERE extname = 'emaj');
IF v_isEmajExtension AND v_effNbTable > 0 THEN
WITH fkeys AS (
-- the foreign keys belonging to tables to rollback
SELECT rlbp_schema, rlbp_table, c.conname, nf.nspname, tf.relname, rel_group,
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups,
rlbp_target_time_id = greatest(v_markTimeId, lower(rel_time_range)) AS have_both_tables_the_same_target_mark
FROM emaj.emaj_rlbk_plan,
pg_catalog.pg_constraint c
JOIN pg_catalog.pg_class t ON (t.oid = c.conrelid)
Expand All @@ -8267,7 +8269,8 @@ $_rlbk_planning$
UNION
-- the foreign keys referencing tables to rollback
SELECT rlbp_schema, rlbp_table, c.conname, n.nspname, t.relname, rel_group,
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups
rel_group = ANY (v_groupNames) AS are_both_tables_in_groups,
rlbp_target_time_id = greatest(v_markTimeId, lower(rel_time_range)) AS have_both_tables_the_same_target_mark
FROM emaj.emaj_rlbk_plan,
pg_catalog.pg_constraint c
JOIN pg_catalog.pg_class t ON (t.oid = c.conrelid)
Expand All @@ -8286,7 +8289,8 @@ $_rlbk_planning$
), fkeys_agg AS (
-- aggregated foreign keys by tables to rollback
SELECT rlbp_schema, rlbp_table,
count(*) AS nb_all_fk, count(*) FILTER (WHERE are_both_tables_in_groups) AS nb_fk_groups_ok
count(*) AS nb_fk,
count(*) FILTER (WHERE are_both_tables_in_groups AND have_both_tables_the_same_target_mark) AS nb_fk_ok
FROM fkeys
GROUP BY 1,2
)
Expand All @@ -8297,7 +8301,8 @@ $_rlbk_planning$
AND rlbp_step IN ('RLBK_TABLE', 'LOCK_TABLE')
AND emaj_rlbk_plan.rlbp_table = fkeys_agg.rlbp_table
AND emaj_rlbk_plan.rlbp_schema = fkeys_agg.rlbp_schema
AND nb_all_fk = nb_fk_groups_ok -- if all fkeys are linking tables in the rolled back groups
AND nb_fk = nb_fk_ok -- if all fkeys are linking tables 1) in the rolled back groups
-- and 2) with the same rollback target mark
;
END IF;
--
Expand Down
8 changes: 4 additions & 4 deletions test/11/expected/adm1.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -214,19 +214,19 @@ select * from emaj.emaj_seq_hole;
select count(*) from emaj.emaj_rlbk;
count
-------
37
38
(1 row)

select count(*) from emaj.emaj_rlbk_session;
count
-------
37
38
(1 row)

select count(*) from emaj.emaj_rlbk_plan;
count
-------
187
203
(1 row)

select count(*) from emaj.emaj_rlbk_stat;
Expand Down Expand Up @@ -264,7 +264,7 @@ select hist_function, hist_event, hist_wording
hist_function | hist_event | hist_wording
-----------------+------------+--------------------------------------------------------------------------------------------------------------------------
PURGE_HISTORIES | BEGIN | Retention delay @ 0
PURGE_HISTORIES | | 20 emaj_hist rows deleted ; 138 relation history rows deleted ; 94 relation changes deleted ; 37 rollback events deleted
PURGE_HISTORIES | | 20 emaj_hist rows deleted ; 138 relation history rows deleted ; 94 relation changes deleted ; 38 rollback events deleted
PURGE_HISTORIES | END |
(3 rows)

Expand Down
3 changes: 2 additions & 1 deletion test/11/expected/adm3.out
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2112,6 +2112,7 @@ select rlbk_id, rlbk_groups, rlbk_mark, rlbk_time_id, rlbk_is_logged, rlbk_is_al
---------+-------------------------------+-----------------------+--------------+----------------+-----------------------------+-----------------+---------------+------------------+-------------------+----------------------+-------------+--------------------+--------------------+---------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10000 | {myGroup1,myGroup2} | after move | 10009 | t | f | 1 | 11 | 4 | 1 | 0 | COMMITTED | 10065 | public | t | [ts] | Notice: 1 / 11 tables effectively processed.#Notice: 0 / 4 sequences effectively processed.
10001 | {myGroup1,myGroup2} | before move | 10011 | f | t | 1 | 11 | 4 | 1 | 0 | COMMITTED | 10079 | public | t | [ts] | Notice: 1 / 11 tables effectively processed.#Notice: 0 / 4 sequences effectively processed.#Warning: The table myschema2."myTbl3" has only been rolled back to its latest group attachment (<timestamp>)#Warning: The sequence myschema2."myTbl3_col31_seq" has only been rolled back to its latest group attachment state (<timestamp>)#Warning: Tables group change not rolled back: log data tablespace for myschema2."myTbl3"#Warning: Tables group change not rolled back: log index tablespace for myschema2."myTbl3"#Warning: Tables group change not rolled back: E-Maj priority for myschema2."myTbl3"
10003 | {myGroup1} | Assign_mytbl4 | 10030 | f | f | 1 | 5 | 2 | 3 | 1 | COMMITTED | 10255 | public | t | [ts] | Notice: 3 / 5 tables effectively processed.#Notice: 1 / 2 sequences effectively processed.
12300 | {myGroup2} | M2 | 12300 | t | f | 1 | 6 | 2 | 3 | 1 | COMMITTED | 12300 | | f | [ts] | Notice: 3 / 6 tables effectively processed.#Notice: 1 / 2 sequences effectively processed.
12301 | {myGroup2} | M3 | 12302 | t | f | 1 | 6 | 2 | 3 | 1 | COMMITTED | 12318 | | f | [ts] | Notice: 3 / 6 tables effectively processed.#Notice: 1 / 2 sequences effectively processed.
12400 | {myGroup1} | M2 | 12400 | f | f | 1 | 5 | 2 | 2 | 0 | COMMITTED | 12400 | public | t | [ts] | Notice: 2 / 5 tables effectively processed.#Notice: 0 / 2 sequences effectively processed.
Expand Down Expand Up @@ -2146,7 +2147,7 @@ select rlbk_id, rlbk_groups, rlbk_mark, rlbk_time_id, rlbk_is_logged, rlbk_is_al
17602 | {grp_tmp,grp_tmp_3,grp_tmp_4} | Mk3 | 17628 | f | t | 1 | 8 | 3 | 3 | 0 | COMMITTED | 17793 | public | t | [ts] | Notice: 3 / 8 tables effectively processed.#Notice: 0 / 3 sequences effectively processed.#Warning: The table "phil's schema3"."myTbl2\" has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table "phil's schema3".mytbl4 has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table "phil's schema3"."phil's tbl1" has only been rolled back to its latest group attachment (<timestamp>)#Warning: The sequence "phil's schema3"."myTbl2\_col21_seq" has only been rolled back to its latest group attachment state (<timestamp>)#Warning: The sequence "phil's schema3"."phil's seq\1" has only been rolled back to its latest group attachment state (<timestamp>)#Warning: The table myschema4.mypartp2 has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table myschema4.mypartp3 has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table myschema4.mytblc1 has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table myschema4.mytblc2 has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table myschema4.mytblm has only been rolled back to its latest group attachment (<timestamp>)#Warning: The table myschema4.mytblr has only been rolled back to its latest group attachment (<timestamp>)#Warning: The sequence myschema4.mytblp_col3_seq has only been rolled back to its latest group attachment state (<timestamp>)#Warning: The sequence myschema4.mytblr_col1_seq has only been rolled back to its latest group attachment state (<timestamp>)#Warning: Tables group change not rolled back: log data tablespace for "phil's schema3"."phil's tbl1"#Warning: Tables group change not rolled back: E-Maj priority for "phil's schema3"."phil's tbl1"
18000 | {truncateTestGroup} | M1 | 18005 | t | f | 1 | 9 | 0 | 5 | 0 | COMMITTED | 18027 | public | t | [ts] | Notice: 5 / 9 tables effectively processed.
18001 | {truncateTestGroup} | M1 | 18007 | f | f | 1 | 9 | 0 | 5 | 0 | COMMITTED | 18050 | public | t | [ts] | Notice: 5 / 9 tables effectively processed.
(36 rows)
(37 rows)

select rlbs_rlbk_id, rlbs_session,
case when rlbs_end_datetime is null then 'null' else '[ts]' end as "end_datetime"
Expand Down

0 comments on commit c1a9cff

Please sign in to comment.