automatically know the list of valid ABI #508

dlangille · 2023-12-08T15:44:37Z

          > The goal: automatically know the list of valid ABI. I like to do as little as possible.

Compiling that list from a script (/bin/sh, python, or perl) would be helpful.

That script would make a great standalone project.

This just in, via: https://lists.freebsd.org/archives/freebsd-hackers/2023-December/002733.html

# fetch https://pkg.freebsd.org/index.html
index.html                                            3606  B  103 MBps    00s

# grep FreeBSD: index.html | sed -e 's@.*\(FreeBSD:[^ <]*\).*@\1@' | sort
FreeBSD:12:aarch64
FreeBSD:12:amd64
FreeBSD:12:armv6
FreeBSD:12:armv7
FreeBSD:12:i386
FreeBSD:13:aarch64
FreeBSD:13:amd64
FreeBSD:13:armv6
FreeBSD:13:armv7
FreeBSD:13:i386
FreeBSD:13:powerpc
FreeBSD:13:powerpc64
FreeBSD:13:powerpc64le
FreeBSD:14:aarch64
FreeBSD:14:amd64
FreeBSD:14:armv6
FreeBSD:14:armv7
FreeBSD:14:i386
FreeBSD:14:powerpc
FreeBSD:14:powerpc64
FreeBSD:14:powerpc64le
FreeBSD:15:aarch64
FreeBSD:15:amd64
FreeBSD:15:armv6
FreeBSD:15:armv7
FreeBSD:15:i386
FreeBSD:15:powerpc
FreeBSD:15:powerpc64
FreeBSD:15:powerpc64le

Originally posted by @dlangille in #505 (comment)

The text was updated successfully, but these errors were encountered:

dlangille · 2024-01-02T15:43:39Z

On 2023-12-31, FreeBSD 12 was removed from the list. Now we have:

[15:42 empty dan ~/tmp] % fetch https://pkg.freebsd.org/index.html
index.html                                            3068  B   37 MBps    00s
[15:42 empty dan ~/tmp] % grep FreeBSD: index.html | sed -e 's@.*\(FreeBSD:[^ <]*\).*@\1@' | sort
FreeBSD:13:aarch64
FreeBSD:13:amd64
FreeBSD:13:armv6
FreeBSD:13:armv7
FreeBSD:13:i386
FreeBSD:13:powerpc
FreeBSD:13:powerpc64
FreeBSD:13:powerpc64le
FreeBSD:14:aarch64
FreeBSD:14:amd64
FreeBSD:14:armv6
FreeBSD:14:armv7
FreeBSD:14:i386
FreeBSD:14:powerpc
FreeBSD:14:powerpc64
FreeBSD:14:powerpc64le
FreeBSD:15:aarch64
FreeBSD:15:amd64
FreeBSD:15:armv7
FreeBSD:15:powerpc
FreeBSD:15:powerpc64
FreeBSD:15:powerpc64le

dlangille · 2024-01-02T18:38:39Z

Hint for future me:

construct queries from scripts. Dump it to a file. use psql -f

re: https://www.postgresql.org/docs/current/app-psql.html

dlangille · 2024-01-02T19:32:20Z

[19:31 dev-ingress01 dan ~/scripts] % cat add-new-abi.sh
#!/bin/sh

ABI=$(./current_list_of_valid_abi.sh)

in_list=''
#echo $ABI
for abi in ${ABI}
do
#  echo $abi
  if [ "$in_list" == "" ]
  then
    in_list="('${abi}')"
  else
    in_list="${in_list}, ('${abi}')"
  fi
done

#echo $in_list

query="
with new_and_old as (
WITH current_abi_list as (
select name 
  from (VALUES $in_list)
   v(name))
select current_abi_list.name as name, abi.name as missing from current_abi_list left outer join abi on current_abi_list.name = abi.name) 
insert into abi (name)
select name from new_and_old where missing is null;"

echo $query

[19:31 dev-ingress01 dan ~/scripts] % ./add-new-abi.sh 
with new_and_old as ( WITH current_abi_list as ( select name from (VALUES ('FreeBSD:13:aarch64'), 
('FreeBSD:13:amd64'), ('FreeBSD:13:armv6'), ('FreeBSD:13:armv7'), ('FreeBSD:13:i386'), ('FreeBSD:13:powerpc'),
 ('FreeBSD:13:powerpc64'), ('FreeBSD:13:powerpc64le'), ('FreeBSD:14:aarch64'), ('FreeBSD:14:amd64'),
 ('FreeBSD:14:armv6'), ('FreeBSD:14:armv7'), ('FreeBSD:14:i386'), ('FreeBSD:14:powerpc'), ('FreeBSD:14:powerpc64'),
 ('FreeBSD:14:powerpc64le'), ('FreeBSD:15:aarch64'), ('FreeBSD:15:amd64'), ('FreeBSD:15:armv7'),
 ('FreeBSD:15:powerpc'), ('FreeBSD:15:powerpc64'), ('FreeBSD:15:powerpc64le')) v(name)) 
select current_abi_list.name as name, abi.name as missing
from current_abi_list left outer join abi on current_abi_list.name = abi.name) 
insert into abi (name) 
select name from new_and_old where missing is null;

dlangille · 2024-01-02T19:34:51Z

Running that:

freshports.dev=# begin;
BEGIN
freshports.dev=*# with new_and_old as ( WITH current_abi_list as ( select name from (VALUES ('FreeBSD:13:aarch64'), 
('FreeBSD:13:amd64'), ('FreeBSD:13:armv6'), ('FreeBSD:13:armv7'), ('FreeBSD:13:i386'), ('FreeBSD:13:powerpc'),
 ('FreeBSD:13:powerpc64'), ('FreeBSD:13:powerpc64le'), ('FreeBSD:14:aarch64'), ('FreeBSD:14:amd64'),
 ('FreeBSD:14:armv6'), ('FreeBSD:14:armv7'), ('FreeBSD:14:i386'), ('FreeBSD:14:powerpc'), ('FreeBSD:14:powerpc64'),
 ('FreeBSD:14:powerpc64le'), ('FreeBSD:15:aarch64'), ('FreeBSD:15:amd64'), ('FreeBSD:15:armv7'),
 ('FreeBSD:15:powerpc'), ('FreeBSD:15:powerpc64'), ('FreeBSD:15:powerpc64le')) v(name)) select current_abi_list.name
 as name, abi.name as missing from current_abi_list left outer join abi on current_abi_list.name = abi.name) insert into abi
 (name) select name from new_and_old where missing is null;
INSERT 0 0
freshports.dev=*# rollback;
ROLLBACK
freshports.dev=#

dlangille · 2024-01-02T19:36:15Z

[19:32 dev-ingress01 dan ~/scripts] % cat ./delete-depcreated-abi.sh
#!/bin/sh

ABI=$(./current_list_of_valid_abi.sh)

in_list=''
#echo $ABI
for abi in ${ABI}
do
#  echo $abi
  if [ "$in_list" == "" ]
  then
    in_list="'${abi}'"
  else
    in_list="${in_list}, '${abi}'"
  fi
done

#echo $in_list

query="DELETE FROM abi where name not in (${in_list})"

echo $query
[19:35 dev-ingress01 dan ~/scripts] % ./delete-depcreated-abi.sh 
DELETE FROM abi where name not in ('FreeBSD:13:aarch64', 'FreeBSD:13:amd64', 'FreeBSD:13:armv6', 
'FreeBSD:13:armv7', 'FreeBSD:13:i386', 'FreeBSD:13:powerpc', 'FreeBSD:13:powerpc64', 'FreeBSD:13:powerpc64le', 
'FreeBSD:14:aarch64', 'FreeBSD:14:amd64', 'FreeBSD:14:armv6', 'FreeBSD:14:armv7', 'FreeBSD:14:i386', 
'FreeBSD:14:powerpc', 'FreeBSD:14:powerpc64', 'FreeBSD:14:powerpc64le', 'FreeBSD:15:aarch64', 'FreeBSD:15:amd64', 
'FreeBSD:15:armv7', 'FreeBSD:15:powerpc', 'FreeBSD:15:powerpc64', 'FreeBSD:15:powerpc64le')
[19:35 dev-ingress01 dan ~/scripts] %

dlangille · 2024-01-02T19:37:06Z

Running that, we get:

freshports.dev=*# rollback;
ROLLBACK
freshports.dev=# select * from abi order by name;
 id |          name          | active 
----+------------------------+--------
  2 | FreeBSD:13:aarch64     | t
 11 | FreeBSD:13:amd64       | t
 20 | FreeBSD:13:armv6       | t
 21 | FreeBSD:13:armv7       | t
 10 | FreeBSD:13:i386        | t
 23 | FreeBSD:13:mips64      | t
 58 | FreeBSD:13:powerpc     | t
 24 | FreeBSD:13:powerpc64   | t
 57 | FreeBSD:13:powerpc64le | t
 34 | FreeBSD:13:riscv64     | t
 26 | FreeBSD:14:aarch64     | t
 27 | FreeBSD:14:amd64       | t
 28 | FreeBSD:14:armv6       | t
 29 | FreeBSD:14:armv7       | t
 30 | FreeBSD:14:i386        | t
 32 | FreeBSD:14:mips64      | t
 59 | FreeBSD:14:powerpc     | t
 33 | FreeBSD:14:powerpc64   | t
 56 | FreeBSD:14:powerpc64le | t
 35 | FreeBSD:14:riscv64     | t
 54 | FreeBSD:15:aarch64     | t
 47 | FreeBSD:15:amd64       | t
 48 | FreeBSD:15:armv6       | t
 49 | FreeBSD:15:armv7       | t
 50 | FreeBSD:15:i386        | t
 51 | FreeBSD:15:mips64      | t
 60 | FreeBSD:15:powerpc     | t
 52 | FreeBSD:15:powerpc64   | t
 55 | FreeBSD:15:powerpc64le | t
 53 | FreeBSD:15:riscv64     | t
(30 rows)

freshports.dev=# begin;
BEGIN
freshports.dev=*# DELETE FROM abi where name not in ('FreeBSD:13:aarch64', 'FreeBSD:13:amd64', 'FreeBSD:13:armv6', 
'FreeBSD:13:armv7', 'FreeBSD:13:i386', 'FreeBSD:13:powerpc', 'FreeBSD:13:powerpc64', 'FreeBSD:13:powerpc64le', 
'FreeBSD:14:aarch64', 'FreeBSD:14:amd64', 'FreeBSD:14:armv6', 'FreeBSD:14:armv7', 'FreeBSD:14:i386', 
'FreeBSD:14:powerpc', 'FreeBSD:14:powerpc64', 'FreeBSD:14:powerpc64le', 'FreeBSD:15:aarch64', 'FreeBSD:15:amd64', 
'FreeBSD:15:armv7', 'FreeBSD:15:powerpc', 'FreeBSD:15:powerpc64', 'FreeBSD:15:powerpc64le')
freshports.dev-*# ;
DELETE 8
freshports.dev=*# select * from abi order by name;
 id |          name          | active 
----+------------------------+--------
  2 | FreeBSD:13:aarch64     | t
 11 | FreeBSD:13:amd64       | t
 20 | FreeBSD:13:armv6       | t
 21 | FreeBSD:13:armv7       | t
 10 | FreeBSD:13:i386        | t
 58 | FreeBSD:13:powerpc     | t
 24 | FreeBSD:13:powerpc64   | t
 57 | FreeBSD:13:powerpc64le | t
 26 | FreeBSD:14:aarch64     | t
 27 | FreeBSD:14:amd64       | t
 28 | FreeBSD:14:armv6       | t
 29 | FreeBSD:14:armv7       | t
 30 | FreeBSD:14:i386        | t
 59 | FreeBSD:14:powerpc     | t
 33 | FreeBSD:14:powerpc64   | t
 56 | FreeBSD:14:powerpc64le | t
 54 | FreeBSD:15:aarch64     | t
 47 | FreeBSD:15:amd64       | t
 49 | FreeBSD:15:armv7       | t
 60 | FreeBSD:15:powerpc     | t
 52 | FreeBSD:15:powerpc64   | t
 55 | FreeBSD:15:powerpc64le | t
(22 rows)

freshports.dev=*# rollback;
ROLLBACK
freshports.dev=#

dlangille · 2024-01-03T15:51:22Z

Here we go. I amended the scripts above to take the list of ABI from a file. Then output full SQL.

The result is this script, which does the refresh:

[15:48 dev-ingress01 dan ~/scripts] % cat refresh-abi.sh  
#!/bin/sh

# Ensure the ABI table contains the right list of values.

if [ ! -f config.sh ]
then
	echo "config.sh not found by missing-port-categories.sh..."
	exit 1
fi

. config.sh

export PGDATABASE=$DB
export PGHOST=$HOST
export PGUSER=$DBUSER_ABI

# see also PGSSLMODE and PGSSLROOTCERT in config.sh


valid=$(mktemp  ${SPOOLINGDIR}/abi-valid.XXXXXX)
sql=$(mktemp ${SPOOLINGDIR}/abi-sel.XXXXXX)

# fetch and extract the list of valid ABI
./current-list-of-valid-abi.sh > $valid

# build the SQL commands
echo 'BEGIN;'                           >  $sql

# ABI to be deleted
./delete-depcreated-abi.sh     < $valid >> $sql

# ABI to be added
echo                                    >> $sql
./add-new-abi.sh               < $valid >> $sql

# and we end
echo                                    >> $sql
echo 'ROLLBACK;'                          >>  $sql

# run the SQL
psql -f $sql 


# if all good, remove the files. If not, leave the files
# the output of this script will be saved and an error flagged
if [ $? == 0 ]
then
  # commit taken
  rm $valid $sql
else
  # raise an error, somewhere, somehow
fi
[15:48 dev-ingress01 dan ~/scripts] %

dlangille · 2024-01-03T21:23:50Z

This is now ready to run overnight on dev.

dlangille · 2024-01-07T14:02:43Z

From /var/log/freshports/freshports.log when there is no change in ABI:

Jan  7 13:58:00 dev-ingress01 FreshPorts[7047]: into /usr/local/etc/periodic/daily/330.refresh-abi
Jan  7 13:58:00 dev-ingress01 dan[7056]: ./refresh-abi.sh starts
Jan  7 13:58:00 dev-ingress01 dan[7082]: ./refresh-abi.sh nothing to change
Jan  7 13:58:00 dev-ingress01 dan[7088]: ./refresh-abi.sh finishes

When an ABI goes away:

Jan  7 13:58:18 dev-ingress01 FreshPorts[7669]: into /usr/local/etc/periodic/daily/330.refresh-abi
Jan  7 13:58:18 dev-ingress01 dan[7679]: ./refresh-abi.sh starts
Jan  7 13:58:18 dev-ingress01 dan[7699]: Sun Jan 7 13:58:18 UTC 2024
Jan  7 13:58:18 dev-ingress01 dan[7704]: BEGIN DELETE 1 INSERT 0 0 COMMIT
Jan  7 13:58:18 dev-ingress01 dan[7711]: ./refresh-abi.sh finishes

You would also find this entry in /var/log/freshports/abi.log which is rotated very slowly based on size.

Sun Jan 7 13:58:18 UTC 2024
BEGIN
DELETE 1
INSERT 0 0
COMMIT

dlangille · 2024-01-12T18:18:48Z

now packaged.

Installed on test, dev, and stage.

dlangille self-assigned this Dec 8, 2023

dlangille added the ABI label Dec 8, 2023

dlangille closed this as completed Jan 12, 2024

dlangille mentioned this issue Mar 16, 2024

Do not display the quarterly branch for -CURRENT - that is never built #554

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

automatically know the list of valid ABI #508

automatically know the list of valid ABI #508

dlangille commented Dec 8, 2023

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024 •

edited

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024

dlangille commented Jan 3, 2024

dlangille commented Jan 3, 2024

dlangille commented Jan 7, 2024

dlangille commented Jan 12, 2024

automatically know the list of valid ABI #508

automatically know the list of valid ABI #508

Comments

dlangille commented Dec 8, 2023

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024 • edited

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024

dlangille commented Jan 2, 2024

dlangille commented Jan 3, 2024

dlangille commented Jan 3, 2024

dlangille commented Jan 7, 2024

dlangille commented Jan 12, 2024

dlangille commented Jan 2, 2024 •

edited