caching problem for non-ports

[dlangille]

Commits against MOVED, and probably all non-ports, are not being cleared after a commit.

This arose from #467

[dlangille]

This is found I found in the processing logs for https://cgit.freebsd.org/ports/commit/?id=1a34bf36e06cbf94c3e5f303c41b50dbf4eb2f93

Observer has noticed that the update for 1a34bf36e06cbf94c3e5f303c41b50dbf4eb2f93 has finished.
Observer will clear the following ports from cache after the commit:
net/openmpi3
Observer will clear the following files from cache after the commit:
we are ignoring /ports/head/MOVED for cache clearing
we are ignoring /ports/head/net/Makefile for cache clearing
we are ignoring /ports/head/net/openmpi3/Makefile for cache clearing
we are ignoring /ports/head/net/openmpi3/distinfo for cache clearing
we are ignoring /ports/head/net/openmpi3/files/patch-opal_mca_pmix_pmix2x_pmix_src_mca_pshmem_mmap_pshmem__mmap.c for cache clearing
we are ignoring /ports/head/net/openmpi3/pkg-descr for cache clearing
we are ignoring /ports/head/net/openmpi3/pkg-plist for cache clearing
*** end of items to be cleared

[dlangille]

After my code changes:

Observer will clear the following files from cache after the commit:
/ports/head/MOVED
we are ignoring /ports/head/net/Makefile for cache clearing
we are ignoring /ports/head/net/openmpi3/Makefile for cache clearing
we are ignoring /ports/head/net/openmpi3/distinfo for cache clearing
we are ignoring /ports/head/net/openmpi3/files/patch-opal_mca_pmix_pmix2x_pmix_src_mca_pshmem_mmap_pshmem__mmap.c for cache clearing
we are ignoring /ports/head/net/openmpi3/pkg-descr for cache clearing
we are ignoring /ports/head/net/openmpi3/pkg-plist for cache clearing
*** end of items to be cleared

[dlangille]

The problem was caused by full pathnames, with leading /

The old code: my ($subtree, $category_name, $port_name, $extra) = split/\//,$filename, 4;

For a file like /ports/head/MOVED this became:

$subtree=''`
$category_name='ports'
$port_name='head'
$extra='MOVED'

I am now very hazy as to why this would ignore all files. However, the new code does it right.

# if we're on head, we figure out the path components differently than on a branch
$on_head = $filename =~ m/^$FreshPorts::Constants::Ports_HEAD_commit/;

# see above for a breakdown on what this does
# because the path always starts with a leading /, the first component pulled out is always an empty string
if ($on_head) {
	($null, $subtree, $head, $category_name, $extra) = split/\//,$filename, 5;
} else {
	($null, $subtree, $branches, $branch, $category_name, $extra) = split/\//,$filename, 6;
}

[dlangille]

Next step: clear the cache on the front end for MOVED.

[dlangille]

Committed revision 5914. for this.

Next step: add entries to the new table.

[dlangille]

SQL for the table:

-- Table: public.cache_clearing_files

-- DROP TABLE IF EXISTS public.cache_clearing_files;

CREATE TABLE IF NOT EXISTS public.cache_clearing_files
(
    id integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
    pathname text COLLATE pg_catalog."default" NOT NULL,
    date_added timestamp without time zone NOT NULL DEFAULT now()
)

TABLESPACE pg_default;

ALTER TABLE IF EXISTS public.cache_clearing_files
    OWNER to postgres;

REVOKE ALL ON TABLE public.cache_clearing_files FROM commits;
REVOKE ALL ON TABLE public.cache_clearing_files FROM listening;
REVOKE ALL ON TABLE public.cache_clearing_files FROM reporting;
REVOKE ALL ON TABLE public.cache_clearing_files FROM rsyncer;
REVOKE ALL ON TABLE public.cache_clearing_files FROM www;

GRANT SELECT, UPDATE, INSERT ON TABLE public.cache_clearing_files TO commits;

GRANT DELETE, SELECT ON TABLE public.cache_clearing_files TO listening;

GRANT ALL ON TABLE public.cache_clearing_files TO postgres;

GRANT SELECT ON TABLE public.cache_clearing_files TO reporting;

GRANT SELECT ON TABLE public.cache_clearing_files TO rsyncer;

GRANT SELECT, INSERT ON TABLE public.cache_clearing_files TO www;
-- Index: cache_clearing_files_id_idx

-- DROP INDEX IF EXISTS public.cache_clearing_files_id_idx;

CREATE UNIQUE INDEX IF NOT EXISTS cache_clearing_files_id_idx
    ON public.cache_clearing_files USING btree
    (id ASC NULLS LAST)
    TABLESPACE pg_default;
-- Index: cache_clearing_files_pathname_idx

-- DROP INDEX IF EXISTS public.cache_clearing_files_pathname_idx;

CREATE UNIQUE INDEX IF NOT EXISTS cache_clearing_files_pathname_idx
    ON public.cache_clearing_files USING btree
    (pathname COLLATE pg_catalog."default" ASC NULLS LAST)
    TABLESPACE pg_default;

COMMENT ON INDEX public.cache_clearing_files_pathname_idx
    IS 'No sense clearing the same file multiple times.';
```-- Table: public.cache_clearing_files

-- DROP TABLE IF EXISTS public.cache_clearing_files;

CREATE TABLE IF NOT EXISTS public.cache_clearing_files
(
    id integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
    pathname text COLLATE pg_catalog."default" NOT NULL,
    date_added timestamp without time zone NOT NULL DEFAULT now(),
    CONSTRAINT cache_clearing_files_pathname UNIQUE (pathname)
)

TABLESPACE pg_default;

ALTER TABLE IF EXISTS public.cache_clearing_files
    OWNER to postgres;

REVOKE ALL ON TABLE public.cache_clearing_files FROM commits;
REVOKE ALL ON TABLE public.cache_clearing_files FROM listening;
REVOKE ALL ON TABLE public.cache_clearing_files FROM reporting;
REVOKE ALL ON TABLE public.cache_clearing_files FROM rsyncer;
REVOKE ALL ON TABLE public.cache_clearing_files FROM www;

GRANT SELECT, UPDATE, INSERT ON TABLE public.cache_clearing_files TO commits;

GRANT DELETE, SELECT ON TABLE public.cache_clearing_files TO listening;

GRANT ALL ON TABLE public.cache_clearing_files TO postgres;

GRANT SELECT ON TABLE public.cache_clearing_files TO reporting;

GRANT SELECT ON TABLE public.cache_clearing_files TO rsyncer;

GRANT SELECT, INSERT ON TABLE public.cache_clearing_files TO www;
-- Index: cache_clearing_files_id_idx

-- DROP INDEX IF EXISTS public.cache_clearing_files_id_idx;

CREATE UNIQUE INDEX IF NOT EXISTS cache_clearing_files_id_idx
    ON public.cache_clearing_files USING btree
    (id ASC NULLS LAST)
    TABLESPACE pg_default;

[dlangille]

Table created on all databases

[dlangille]

py39-freshports-fp-listen-git-1.0.8 contains the new cache clearing code. This runs on all FreshPorts www nodes.

[dlangille]

I think this now works on dev. My tests are successful.

Next: wait for a commit to MOVED / UPDATING and see if these pages are up to date:

• https://dev.freshports.org/MOVED
• https://dev.freshports.org/UPDATING

[dlangille]

I think this now works on dev. My tests are successful.

Next: wait for a commit to MOVED / UPDATING and see if these pages are up to date:

• https://dev.freshports.org/MOVED
• https://dev.freshports.org/UPDATING

Seems good now on dev. Also, on test and stage, but I suspect those two are because the results were not cached, thus, it was not a valid test.

Next: promote from dev to test, then wait for another MOVED / UPDATING` commit.

[dlangille]

Now in test

[dlangille]

Failed on test - found I had uncommitted code on dev. Committed and promoted to test.

[dlangille]

promoted to stage, where https://stage.freshports.org/MOVED is showing Sun, 22 Oct 2023 as the latest update.

and https://stage.freshports.org/UPDATING is dated Fri, 27 Oct 2023, which is correct for today.

Now waiting for the next commit to MOVED

[dlangille]

Good. https://stage.freshports.org/MOVED now shows Sat, 28 Oct 2023 - good. So does https://stage.freshports.org/UPDATING - this is correct. This is ready for production.

[dlangille]

Promoted to production.

[dlangille]

Worked fine for this commit: https://dev.freshports.org/commit.php?category=devel&port=py-subversion&files=yes&message_id=914eca41bf74707f7c59ee8958861b12e77e2243