New sql schema

[darix]

New proposed database schema

for storing file information and mirror mapping

all the mirr_* functions are not ported yet.

For the details see https://github.com/openSUSE/mirrorbrain/wiki/Roadmap

[lrupp]

+1 from my side.
We started with the new Roadmap in 2019. Time to move forward...

[darix]

the comment patch is now already in the opensuse branch so it can be ignored here.

[andrii-suse]

Some comments are less important here, but we should build common understanding on some of the notes.

[andrii-suse]

I suggest going with bigint: currently max id in db is above 900M, which is close to 50% of int capacity. So limit may be reached in few years

[darix]

will do

[andrii-suse]

Following query should do the same job and is much lighter, because it doesn't calculate exact counts (m.id will be NULL only for those rows, which don't have any mirror):

DELETE FROM filemetadata 
USING filemetadata AS f
LEFT OUTER JOIN mirrors AS m ON filemetadata_id = f.id
WHERE filemetadata.id = f.id AND m.id is NULL AND f.mtime < (now()-'3 months'::interval)

I know that the counts are pre-calculated anyway - but maybe we can simplify workflow and do not maintain MATERIALIZED VIEW at all?

[darix]

explain (analyze, buffers) delete from filemetadata using filemetadata AS f LEFT OUTER JOIN mirrors AS m ON filemetadata_id = f.id WHERE filemetadata_id = f.id AND  m.server_id is NULL AND f.mtime < (now()-'3 months'::interval);
                                                                     QUERY PLAN                                                                     
----------------------------------------------------------------------------------------------------------------------------------------------------
 Delete on filemetadata  (cost=1.00..1116601.47 rows=2862434 width=18) (actual time=0.638..0.638 rows=0 loops=1)
   Buffers: shared read=4
   ->  Nested Loop  (cost=1.00..1116601.47 rows=2862434 width=18) (actual time=0.638..0.638 rows=0 loops=1)
         Buffers: shared read=4
         ->  Nested Loop  (cost=1.00..16.55 rows=1 width=12) (actual time=0.637..0.637 rows=0 loops=1)
               Buffers: shared read=4
               ->  Index Scan using idx_mirrors_server_id on mirrors m  (cost=0.56..8.08 rows=1 width=10) (actual time=0.636..0.636 rows=0 loops=1)
                     Index Cond: (server_id IS NULL)
                     Buffers: shared read=4
               ->  Index Scan using pk_filemetadata on filemetadata f  (cost=0.44..8.46 rows=1 width=10) (never executed)
                     Index Cond: (id = m.filemetadata_id)
                     Filter: (mtime < (now() - '3 mons'::interval))
         ->  Seq Scan on filemetadata  (cost=0.00..888591.96 rows=22799296 width=6) (never executed)
 Planning Time: 5.143 ms
 JIT:
   Functions: 14
   Options: Inlining true, Optimization true, Expressions true, Deforming true
   Timing: Generation 1.544 ms, Inlining 0.000 ms, Optimization 0.000 ms, Emission 0.000 ms, Total 1.544 ms
 Execution Time: 2.353 ms
(19 rows)

[andrii-suse]

oh it probably should be WHERE filemetadata.id instead of WHERE filemetadata_id

[andrii-suse]

We should use INT instead of SMALLINT, which is mileted to 32K : zsync algorithm doesn't have practical limit for block size and e.g. block size of 1M is much suitable for huge files (see #47 )

[andrii-suse]

we need path column here as well, so the view can be queried without join

[darix]

I thought about it. but right know it is 2 queries anyway.
https://github.com/openSUSE/mirrorbrain/blob/opensuse/mod_mirrorbrain/mod_mirrorbrain.c#L139..L163

[andrii-suse]

I mean instead of subquery in the hash query

mirrorbrain/mod_mirrorbrain/mod_mirrorbrain.c

Lines 157 to 159 in f33ca26

	"WHERE file_id = (SELECT id " \
	"FROM filearr " \
	"WHERE path = %s) " \

we can just have WHERE path = %s if the view would have path

[andrii-suse]

This is how til will look if the column is inside view https://github.com/openSUSE/mirrorbrain/pull/52/files#diff-a44eaf51cd9831129dc4b515db0bd2aeL157-R156

[darix]

all your changes should be addressed