Fix deadlocks

[andrii-suse]

Attempt to reduce chance of deadlock by introducing ordered processing.

Case 1 should be fixed by commit 09ddc7b:

[14925]: [6-1] DETAIL:  Process holding the lock: 14684. Wait queue: .
[14925]: [7-1] CONTEXT:  while updating tuple (502148,7) in relation "filearr"
        SQL statement "UPDATE filearr 
                    SET mirrors = arr WHERE id = arg_fileid"
        PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement
[14925]: [8-1] STATEMENT:  SELECT COUNT(mirr_del_byid(52, id)) FROM temp1
[14925]: [9-1] ERROR:  deadlock detected
[14925]: [10-1] DETAIL:  Process 14925 waits for ShareLock on transaction 3473714341; blocked by process 14684.
        Process 14684 waits for ShareLock on transaction 3473714367; blocked by process 14925.
        Process 14925: SELECT COUNT(mirr_del_byid(52, id)) FROM temp1
        Process 14684: SELECT COUNT(mirr_del_byid(52, id)) FROM temp1
[14925]: [11-1] HINT:  See server log for query details.
[14925]: [12-1] CONTEXT:  while updating tuple (502148,7) in relation "filearr"
        SQL statement "UPDATE filearr 
                    SET mirrors = arr WHERE id = arg_fileid"
        PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement
[14925]: [13-1] STATEMENT:  SELECT COUNT(mirr_del_byid(52, id)) FROM temp1
[14684]: [9-1] LOG:  process 14684 acquired ShareLock on transaction 3473714367 after 86002.689 ms
[14684]: [10-1] CONTEXT:  while updating tuple (502209,46) in relation "filearr"
        SQL statement "UPDATE filearr 
                    SET mirrors = arr WHERE id = arg_fileid"
        PL/pgSQL function mirr_del_byid(integer,integer) line 21 at SQL statement
[14684]: [11-1] STATEMENT:  SELECT COUNT(mirr_del_byid(52, id)) FROM temp1

Cases 2 and 3 below should be fixed by ordered processing in rsync reader (currently order differs on different mirrors, which leads to deadlocks because commit happens after every 50 files). Identical order will make sure that in worse case one scanner will block another, not two scanners block each other.

[8055]: [4-1] LOG:  process 8055 detected deadlock while waiting for ShareLock on transaction 3406853688 after 30000.106 ms
[8055]: [5-1] DETAIL:  Process holding the lock: 8046. Wait queue: .
[8055]: [6-1] CONTEXT:  while updating tuple (499618,17) in relation "filearr"
        SQL statement "update filearr set mirrors = arr where id = fileid"
        PL/pgSQL function mirr_add_bypath(integer,text) line 21 at SQL statement
[8055]: [7-1] STATEMENT:  SELECT mirr_add_bypath($1, $2);
[8055]: [8-1] ERROR:  deadlock detected
[8055]: [9-1] DETAIL:  Process 8055 waits for ShareLock on transaction 3406853688; blocked by process 8046.
        Process 8046 waits for ShareLock on transaction 3406853619; blocked by process 8055.
        Process 8055: SELECT mirr_add_bypath($1, $2);
        Process 8046: SELECT mirr_add_bypath($1, $2);
[8055]: [10-1] HINT:  See server log for query details.
[8055]: [11-1] CONTEXT:  while updating tuple (499618,17) in relation "filearr"
        SQL statement "update filearr set mirrors = arr where id = fileid"
        PL/pgSQL function mirr_add_bypath(integer,text) line 21 at SQL statement
[8055]: [12-1] STATEMENT:  SELECT mirr_add_bypath($1, $2);
[8046]: [4-1] LOG:  duration: 29998.171 ms  execute dbdpg_p3074_1: SELECT mirr_add_bypath($1, $2);
[8046]: [5-1] DETAIL:  parameters: $1 = '114', $2 = 'history/20200211/tumbleweed/repo/oss/i586/k3b-19.12.2-1.1.i586.rpm'

[16253]: [4-1] LOG:  process 16253 detected deadlock while waiting for ShareLock on transaction 3354944753 after 30000.105 ms
[16253]: [5-1] DETAIL:  Process holding the lock: 16258. Wait queue: .
[16253]: [6-1] CONTEXT:  while inserting index tuple (16870,3) in relation "filearr_path_key"
        SQL statement "INSERT INTO filearr (path, mirrors) VALUES (arg_path, ARRAY[arg_serverid])"
        PL/pgSQL function mirr_add_bypath(integer,text) line 16 at SQL statement
[16253]: [7-1] STATEMENT:  SELECT mirr_add_bypath($1, $2);
[16253]: [8-1] ERROR:  deadlock detected
[16253]: [9-1] DETAIL:  Process 16253 waits for ShareLock on transaction 3354944753; blocked by process 16258.
        Process 16258 waits for ShareLock on transaction 3354944760; blocked by process 16253.
        Process 16253: SELECT mirr_add_bypath($1, $2);
        Process 16258: SELECT mirr_add_bypath($1, $2);
[16253]: [10-1] HINT:  See server log for query details.
[16253]: [11-1] CONTEXT:  while inserting index tuple (16870,3) in relation "filearr_path_key"
        SQL statement "INSERT INTO filearr (path, mirrors) VALUES (arg_path, ARRAY[arg_serverid])"
        PL/pgSQL function mirr_add_bypath(integer,text) line 16 at SQL statement
[16253]: [12-1] STATEMENT:  SELECT mirr_add_bypath($1, $2);
[16258]: [4-1] LOG:  duration: 29991.965 ms  execute dbdpg_p20347_1: SELECT mirr_add_bypath($1, $2);
[16258]: [5-1] DETAIL:  parameters: $1 = '169', $2 = 'repositories/devel:/languages:/ocaml/openSUSE_Tumbleweed/aarch64/ocaml-camlzip-test-1.10-1.d_l_ocaml.1.aarch64.rpm'