Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test dependencies are missing #411

Closed
yurivict opened this issue Dec 19, 2022 · 50 comments
Closed

Test dependencies are missing #411

yurivict opened this issue Dec 19, 2022 · 50 comments

Comments

@yurivict
Copy link
Contributor

The PHP code doesn't show them.
I am not sure if the database contains them.

@dlangille
Copy link
Contributor

I'll bet the code doesn't extract them.

@dlangille
Copy link
Contributor

Here is an example:

[pkg01 dan ~/ports/head/sysutils/py-honcho] % make -V TEST_DEPENDS                               22:30:33
py39-mock>0:devel/py-mock@py39  py39-pytest>0:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39

@dlangille
Copy link
Contributor

Nope, the code does not extract that.

[dev-ingress01 dan ~/scripts] % grep -r TEST_DEPENDS *                                           22:31:03
[dev-ingress01 dan ~/scripts] % cd ~/modules                                                     22:31:05
[dev-ingress01 dan ~/modules] % grep -r TEST_DEPENDS *                                           22:31:11
grep: config.pm: Permission denied
[dev-ingress01 dan ~/modules] %                                                                  22:31:13

@dlangille
Copy link
Contributor

It's pretty straight forward to modify the code and update the database for this. It takes about an hour. Then we need to add a one-time script to update all the ports. We also have to display the TEST_DEPENDS. And allow searching on that field.

I can do all the backend stuff soon. That's the scripts and modules directories you see above. That's in a repo here in my basement. It's not made the transition to git yet, but hopefully soon.

@dlangille
Copy link
Contributor

dlangille commented Dec 19, 2022

dev.freshports.org has the code to read and store test_depends into the database. Once I have it working, I'll update your database.

@dlangille
Copy link
Contributor

Now the front end at dev.freshports.org should display test_depends when some turn up.

@yurivict
Copy link
Contributor Author

math/py-numpy has TEST_DEPENDS but dev.freshports.org doesn't show them: https://dev.freshports.org/math/py-numpy

@dlangille
Copy link
Contributor

dlangille commented Dec 20, 2022

The current code will populate ports as they are updated (i.e. when a new commit is processed). A script will be written to populate test_depends of all ports.

In the meantime, these ports have updates and have values for test_depends:

freshports.devgit=# select id, element_pathname(element_id), test_depends from ports where test_depends is not null;
   id   |                  element_pathname                  |                                 test_depends                                 
--------+----------------------------------------------------+------------------------------------------------------------------------------
  49513 | /ports/head/devel/py-intbitset                     | py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
  53715 | /ports/head/devel/folly                            | googletest>0:devel/googletest
 109140 | /ports/head/www/py-aiostream                       | /usr/local/bin/python3.9:lang/python39
 109139 | /ports/head/cad/py-gmsh                            | /usr/local/bin/python3.9:lang/python39
 109141 | /ports/head/cad/py-pygmsh                          | py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
  49508 | /ports/head/devel/py-boolean.py                    | /usr/local/bin/python3.9:lang/python39
  49516 | /ports/head/textproc/py-pyahocorasick              | /usr/local/bin/python3.9:lang/python39
  97333 | /ports/head/www/py-dj-database-url                 | /usr/local/bin/python3.9:lang/python39
  97104 | /ports/head/textproc/py-mkdocs-material-extensions | /usr/local/bin/python3.9:lang/python39
  97103 | /ports/head/textproc/py-pymdown-extensions         | /usr/local/bin/python3.9:lang/python39
  97105 | /ports/head/textproc/py-mkdocs-material            | /usr/local/bin/python3.9:lang/python39
  36283 | /ports/head/deskutils/py-vdirsyncer                | /usr/local/bin/python3.9:lang/python39
(12 rows)

freshports.devgit=# 

See https://dev.freshports.org/devel/py-intbitset/#requiredtest for example. I'm sure the others are also correct. I will commit my changes soon.

@dlangille
Copy link
Contributor

I have updated your database instance:

freshports.org=# begin;
BEGIN
freshports.org=*# ALTER TABLE IF EXISTS public.ports
    ADD COLUMN test_depends text COLLATE pg_catalog."default";
ALTER TABLE
freshports.org=*# commit;
COMMIT

@dlangille
Copy link
Contributor

And set one value for testing:

freshports.org=# begin;
BEGIN
freshports.org=*# update ports set test_depends = 'py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39' where id = 49513;
UPDATE 1
freshports.org=*# commit;
COMMIT
freshports.org=# select id, element_pathname(element_id), test_depends from ports where test_depends is not null;
  id   |        element_pathname        |                                 test_depends                                 
-------+--------------------------------+------------------------------------------------------------------------------
 49513 | /ports/head/devel/py-intbitset | py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
(1 row)

freshports.org=# 

@dlangille
Copy link
Contributor

@dlangille
Copy link
Contributor

dlangille commented Dec 20, 2022

If you pull my recent changes, and clear server-side cache, you should see test_depends at the above URL.

@dlangille
Copy link
Contributor

Your database is being refreshed with test_depends values now.

tail -F /jails/ingress01/tmp/refresh-all-active-ports.log from the host will show progress.

@yurivict
Copy link
Contributor Author

Thanks!

@dlangille
Copy link
Contributor

The script finished. However, running it was premature. It didn't have the new code from dev.freshports.org.

@dlangille
Copy link
Contributor

New code packaged and installed on host. Script running again.

@dlangille
Copy link
Contributor

It's worse than I thought. The ingress01 jail does not have a freshports jail configured.

@dlangille
Copy link
Contributor

Jail configured. Update script running again.

@dlangille
Copy link
Contributor

From dev, this total seems very low.

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
    44
(1 row)

freshports.devgit=# 

grep finds many more than that.

[dev-ingress01 dan /jails/freshports/usr/ports] % grep -rl TEST_DEPENDS *  | sort -u | wc -l
    2332

@dlangille
Copy link
Contributor

Here is a short list for my future reference:

[dev-ingress01 dan /jails/freshports/usr/ports] % grep -rl TEST_DEPENDS *                                                                                                             15:27:22
CHANGES
Mk/bsd.java.mk
Mk/Uses/ghostscript.mk
Mk/Uses/magick.mk
Mk/Uses/python.mk
Mk/Uses/perl5.mk
Mk/Uses/pyqt.mk
Mk/Uses/pytest.mk
Mk/Uses/angr.mk
Mk/bsd.port.mk
archivers/p5-IO-Compress-Lzf/Makefile
archivers/p5-IO-Compress-Lzop/Makefile
archivers/py-libarchive-c/Makefile
archivers/py-borgbackup11/Makefile
archivers/py-lz4/Makefile
archivers/p5-Archive-SimpleExtractor/Makefile
archivers/p5-IO-Compress-Lzma/Makefile
archivers/R-cran-zip/Makefile
archivers/p5-Compress-Raw-Bzip2/Makefile
archivers/p5-Compress-Raw-Zlib/Makefile
archivers/p5-Archive-Libarchive/Makefile
archivers/p5-IO-Compress/Makefile
archivers/py-borgbackup/Makefile
archivers/gcpio/Makefile
archivers/p5-Compress-Raw-Lzma/Makefile
archivers/py-zstd/Makefile
archivers/gtar/Makefile
archivers/p5-Archive-Tar-Wrapper/Makefile
archivers/p5-Archive-Any-Lite/Makefile
archivers/p5-Archive-Extract-Libarchive/Makefile
archivers/py-xopen/Makefile
archivers/p5-Archive-Any/Makefile
astro/p5-Starlink-AST/Makefile
astro/py-skyfield-data/Makefile
astro/py-spktype21/Makefile
astro/py-spacetrack/Makefile
astro/p5-Astro-satpass/Makefile
astro/p5-Astro-Coords/Makefile
astro/p5-Astro-SIMBAD-Client/Makefile
astro/p5-Astro-Constants/Makefile
astro/py-jplephem/Makefile
astro/p5-Astro-PAL/Makefile
audio/libmysofa/Makefile
audio/py-opuslib/Makefile
audio/flacon/Makefile
audio/py-gtts/Makefile
audio/p5-Audio-MPD/Makefile
audio/py-aubio/Makefile
audio/moodbar/Makefile
audio/py-soundcloud/Makefile
audio/p5-Audio-Scan/Makefile
audio/sonivox/Makefile
audio/p5-Net-DAAP-DMAP/Makefile
benchmarks/py-locust/Makefile
benchmarks/libcelero/Makefile
biology/p5-Bio-NEXUS/Makefile
biology/p5-Bio-GFF3/Makefile
biology/p5-Bio-DB-NCBIHelper/Makefile
biology/bcftools/Makefile
biology/bedtools/Makefile
biology/spoa/Makefile
biology/stringtie/Makefile
biology/p5-Bio-DB-EMBL/Makefile
biology/p5-Bio-MAGETAB/Makefile
biology/biosoup/Makefile
biology/htslib/Makefile
biology/bioparser/Makefile
biology/p5-Bio-Phylo/Makefile
biology/metaeuk/Makefile
biology/p5-BioPerl/Makefile
biology/samtools/Makefile
biology/py-cutadapt/Makefile
biology/py-ont-fast5-api/Makefile
biology/unikmer/Makefile
cad/graywolf/Makefile
cad/py-ezdxf/Makefile
cad/xyce/Makefile
cad/nvc/Makefile
cad/py-gdspy/Makefile
cad/yosys/Makefile
cad/cascade-compiler/Makefile
comms/p5-Nexmo-SMS/Makefile
comms/py-fritzconnection/Makefile
comms/py-xmodem/Makefile
comms/py-esptool/Makefile
converters/p5-JSON-MaybeXS/Makefile
converters/osm2pgsql/Makefile
converters/p5-Sereal-Encoder/Makefile
converters/p5-Encode-IMAPUTF7/Makefile
converters/py-bencode.py/Makefile
converters/p5-Sereal/Makefile
converters/p5-JSON-Any/Makefile
converters/p5-Unicode-Stringprep/Makefile
converters/py-svglib/Makefile
converters/p5-Encode-Base32-Crockford/Makefile
converters/p5-Sereal-Decoder/Makefile
converters/p5-JSON-Tiny/Makefile
converters/py-text-unidecode/Makefile
converters/p5-Cpanel-JSON-XS/Makefile
converters/p5-Encode-Base32-GMP/Makefile
converters/p5-Encode-Base58-GMP/Makefile
converters/p5-Convert-BinHex/Makefile
databases/p5-Search-Xapian/Makefile
databases/p5-DBM-Deep/Makefile
databases/p5-Class-DBI-Plugin-Pager/Makefile
databases/p5-DBIx-Class-DateTime-Epoch/Makefile
databases/p5-CHI-Driver-Memcached/Makefile
databases/p5-DBIx-Abstract/Makefile
databases/py-fakeredis/Makefile
databases/py-pgcli/Makefile
databases/p5-DBIx-Class-Tree-NestedSet/Makefile
databases/p5-DBIx-Class-Candy/Makefile
databases/p5-DBIx-Class/Makefile
databases/p5-DBIx-Class-Schema-Config/Makefile
databases/p5-DBIx-Wrapper/Makefile
databases/p5-SQL-Abstract-Limit/Makefile
databases/py-python-binary-memcached/Makefile
databases/p5-DBICx-TestDatabase/Makefile
databases/py-sqlalchemy11/Makefile
databases/p5-SQL-Translator/Makefile
databases/p5-Class-DBI-Plugin-Iterator/Makefile
databases/p5-HTML-FormHandler-Model-DBIC/Makefile
databases/p5-App-Sqitch/Makefile
databases/p5-DBIx-Class-Storage-TxnEndHook/Makefile
databases/p5-Protocol-CassandraCQL/Makefile
databases/p5-DBIx-Class-DynamicDefault/Makefile
databases/p5-DBIx-XHTML_Table/Makefile
databases/cassandra3/Makefile
databases/ldb22/Makefile
databases/p5-Dancer2-Session-DBIC/Makefile
databases/p5-Rose-DB-Object/Makefile
databases/p5-Class-DBI-Plugin-RetrieveAll/Makefile
databases/p5-DBIx-Class-Schema-PopulateMore/Makefile
databases/p5-ORLite/Makefile
databases/p5-DBIx-Class-Schema-Loader/Makefile
databases/p5-Redis-Fast/Makefile
databases/pglogical/Makefile
databases/p5-DBIx-DBHResolver/Makefile
databases/p5-DBIx-Inspector/Makefile
databases/p5-DBIx-Class-Migration/Makefile
databases/p5-DBD-ODBC/Makefile
databases/py-sqlite-utils/Makefile
databases/p5-DBIx-Class-QueryProfiler/Makefile
databases/p5-DBIx-Skinny-Pager/Makefile
databases/p5-DBIx-QuickDB/Makefile
databases/p5-DBIx-Handler/Makefile
databases/p5-Test-mysqld/Makefile
databases/p5-DBIx-SearchBuilder/Makefile
databases/py-databases/Makefile
databases/p5-DBIx-Class-ResultSet-RecursiveUpdate/Makefile
databases/p5-DBIx-Tracer/Makefile
databases/p5-Dancer-Plugin-Database/Makefile
databases/py-marshmallow-sqlalchemy/Makefile
databases/py-cassandra-driver/Makefile
databases/p5-DBICx-Sugar/Makefile
databases/py-redis2/Makefile
databases/py-duckdb/Makefile
databases/p5-Test-Fixture-DBI/Makefile
databases/p5-SQL-Abstract/Makefile
databases/p5-BSON/Makefile
databases/p5-DBIx-Admin-CreateTable/Makefile
databases/p5-SQL-Abstract-More/Makefile
databases/p5-Cache-Memcached-Fast/Makefile
databases/py-sqlalchemy12/Makefile
databases/py-lmdb/Makefile
databases/p5-DBIx-Class-Helpers/Makefile
databases/p5-AnyEvent-Redis/Makefile
databases/p5-Class-DBI-Plugin-Type/Makefile
databases/p5-CHI-Driver-Redis/Makefile
databases/p5-Test-postgresql/Makefile
databases/py-Pyrseas/Makefile
databases/p5-DBIx-Class-TimeStamp/Makefile
databases/p5-BerkeleyDB/Makefile
databases/p5-DBIx-ContextualFetch/Makefile
databases/py-aesqlapius/Makefile
databases/p5-DBIx-Class-AsFdat/Makefile
databases/p5-DBIx-Class-BitField/Makefile
databases/p5-DBIx-Sunny/Makefile
databases/p5-Net-Async-CassandraCQL/Makefile
databases/p5-DBIx-Class-MooseColumns/Makefile
databases/leveldb/Makefile
databases/p5-DBIx-Class-Cursor-Cached/Makefile
databases/libmemcached/Makefile
databases/py-ormar/Makefile
databases/p5-Text-Query-SQL/Makefile
databases/p5-DBIx-Class-Fixtures/Makefile
databases/py-peewee_migrate/Makefile
databases/p5-DBIWrapper/Makefile
databases/py-pony/Makefile
databases/p5-Data-Page/Makefile
databases/p5-Search-Xapian12/Makefile
databases/p5-Amon2-DBI/Makefile
databases/p5-Class-DBI/Makefile
databases/p5-DBICx-TxnInsert/Makefile
databases/p5-SQL-NamedPlaceholder/Makefile
databases/p5-Interchange6-Schema/Makefile
databases/p5-AnyEvent-DBD-Pg/Makefile
databases/p5-DBIx-Skinny-InflateColumn-DateTime/Makefile
databases/p5-DBIx-Class-ResultSet-HashRef/Makefile
databases/p5-Class-DBI-DDL/Makefile
databases/p5-Teng/Makefile
databases/p5-DBD-mysql/Makefile
databases/p5-DBIx-Class-InflateColumn-IP/Makefile
databases/p5-RedisDB/Makefile
databases/py-sqlalchemy10/Makefile
databases/p5-DBIx-Class-Tree/Makefile
databases/py-alembic14/Makefile
databases/p5-DBD-Multi/Makefile
databases/p5-DBIx-Class-Loader/Makefile
databases/p5-Cache-Memcached-Managed/Makefile
databases/p5-Scope-Container-DBI/Makefile
databases/p5-Class-DBI-Plugin-DeepAbstractSearch/Makefile
databases/p5-Dancer2-Plugin-Database/Makefile
databases/p5-MongoDB/Makefile
databases/p5-DBIx-QueryLog/Makefile
databases/py-alembic/Makefile
databases/p5-Dancer-Plugin-DBIC/Makefile
databases/p5-DBIx-Class-InflateColumn-Serializer/Makefile
databases/p5-DBIx-Admin-DSNManager/Makefile
databases/p5-Jifty-DBI/Makefile
databases/p5-Redis/Makefile
databases/py-txredisapi/Makefile
databases/p5-DBIx-Class-InflateColumn-FS/Makefile
databases/p5-BSON-XS/Makefile
databases/py-sqlalchemy14/Makefile
databases/p5-Dancer2-Plugin-DBIC/Makefile
databases/p5-DBIx-Class-AuditLog/Makefile
databases/p5-CHI-Driver-TokyoTyrant/Makefile
databases/py-redis/Makefile
databases/p5-DBIx-Class-EncodedColumn/Makefile
databases/p5-Dancer-Plugin-Database-Core/Makefile
databases/p5-Oryx/Makefile
databases/p5-DBIx-Lite/Makefile
databases/sqlitecpp/Makefile
databases/p5-POE-Component-EasyDBI/Makefile
databases/py-pylibmc/Makefile
databases/p5-DBIx-Simple/Makefile
databases/p5-DBD-MariaDB/Makefile
databases/p5-SQL-Maker/Makefile
databases/py-pymemcache/Makefile
databases/p5-Prophet/Makefile
databases/p5-DBIx-NoSQL/Makefile
databases/py-sqlalchemy13/Makefile
databases/p5-DBIx-Tree/Makefile
databases/p5-DBIx-Skinny-Schema-Loader/Makefile
databases/p5-RedisDB-Parser/Makefile
databases/p5-DBIx-Introspector/Makefile
databases/p5-DBIx-DisconnectAll/Makefile
databases/p5-CouchDB-View/Makefile
databases/py-minidb/Makefile
databases/p5-Class-DBI-Untaint/Makefile
deskutils/py-taskw/Makefile
deskutils/p5-Data-ICal-DateTime/Makefile
deskutils/p5-Data-ICal/Makefile
deskutils/taskwarrior/Makefile
deskutils/py-bugwarrior/Makefile
devel/p5-XS-Parse-Sublike/Makefile
devel/p5-Test-Kwalitee/Makefile
devel/p5-Time-Duration/Makefile
devel/py-botocore/Makefile
devel/p5-self-init/Makefile
devel/p5-Term-Form/Makefile
devel/p5-Data-Section/Makefile
devel/p5-IPC-PerlSSH/Makefile
devel/p5-FFI-Platypus-Type-PtrObject/Makefile
devel/p5-Directory-Scratch-Structured/Makefile
devel/p5-Perl-PrereqScanner-NotQuiteLite/Makefile
devel/pkgconf/Makefile
devel/p5-Test-DependentModules/Makefile
devel/p5-Class-Roles/Makefile
devel/py-jinja2-time/Makefile
devel/R-cran-lintr/Makefile
devel/p5-Test-Base/Makefile
devel/p5-File-Util/Makefile
devel/py-moto/Makefile
devel/R-cran-sessioninfo/Makefile
devel/p5-DateTime-TimeZone/Makefile
devel/p5-Data-Hierarchy/Makefile
devel/p5-Git-Repository/Makefile
devel/p5-Config-Auto/Makefile
devel/p5-File-Write-Rotate/Makefile
devel/p5-Test-WWW-Mechanize-Catalyst/Makefile
devel/py-genson/Makefile
devel/py-rubymarshal/Makefile
devel/p5-Sub-Name/Makefile
devel/p5-Config-Tiny/Makefile
devel/py-omnijson/Makefile
devel/py-pylama/Makefile
devel/p5-Hash-Slice/Makefile
devel/p5-DynaLoader-Functions/Makefile
devel/py-typeguard/Makefile
devel/p5-IO-Prompt-Tiny/Makefile
devel/py-xdoctest/Makefile
devel/p5-Test-Prereq/Makefile
devel/py-columnize/Makefile
devel/trompeloeil/Makefile
devel/p5-MooseX-LazyRequire/Makefile
devel/py-iso8601/Makefile
devel/p5-Data-Serializable/Makefile
devel/py-adb/Makefile
devel/p5-Data-Object-Types/Makefile
devel/p5-Data-Object-Role-Formulatable/Makefile
devel/p5-POE-Quickie/Makefile
devel/p5-github_creator/Makefile
devel/py-rebulk/Makefile
devel/p5-MooseX-Meta-TypeConstraint-Mooish/Makefile
devel/py-jsonschema/Makefile
devel/R-cran-rappdirs/Makefile
devel/p5-IOC/Makefile
devel/py-conditional/Makefile
devel/p5-POEx-Types/Makefile
devel/py-pyperf/Makefile
devel/p5-Proc-Guard/Makefile
devel/p5-Parallel-ForkManager/Makefile
devel/p5-MooseX-Params-Validate/Makefile
devel/p5-Tickit/Makefile
devel/p5-Algorithm-Evolutionary/Makefile
devel/p5-Type-Tiny/Makefile
devel/py-pyhcl/Makefile
devel/py-rauth/Makefile
devel/py-deepdiff/Makefile
devel/py-yappi/Makefile
devel/p5-Log-Dispatch-File-Stamped/Makefile
devel/py-jaraco.functools/Makefile
devel/py-dj40-django-rq/Makefile
devel/py-pudb/Makefile
devel/p5-Data-Validator/Makefile
devel/p5-Test-Mock-LWP-Dispatch/Makefile
devel/py-pytest-subtests/Makefile
devel/p5-Test-File/Makefile
devel/p5-Dist-Metadata/Makefile
devel/p5-Hash-AutoHash-Args/Makefile
devel/p5-Test-Inter/Makefile
devel/py-glob2/Makefile
devel/p5-Data-Object-Role-Dumpable/Makefile
devel/p5-Clone/Makefile
devel/p5-Test-Moose-More/Makefile
devel/jansson/Makefile
devel/p5-Future-IO/Makefile
devel/p5-MooseX-Types-Set-Object/Makefile
devel/rapidcheck/Makefile
devel/p5-Test-MockModule/Makefile
devel/meson/Makefile
devel/py-jdcal/Makefile
devel/p5-MooseX-MetaDescription/Makefile
devel/py-pathspec/Makefile
devel/py-pep8-naming/Makefile
devel/ghq/Makefile
devel/p5-Parse-CPAN-Packages/Makefile
devel/p5-List-SomeUtils-XS/Makefile
devel/p5-BorderStyleBase/Makefile
devel/p5-Module-Build-XSUtil/Makefile
devel/p5-MooseX-Extended/Makefile
devel/p5-MouseX-NativeTraits/Makefile
devel/entt/Makefile
devel/py-fastnumbers/Makefile
devel/p5-File-Flock/Makefile
devel/tcl-trf/Makefile
devel/ros-rosdistro/Makefile
devel/py-extras/Makefile
devel/p5-Sub-Exporter-ForMethods/Makefile
devel/py-line-profiler/Makefile
devel/p5-App-CLI/Makefile
devel/R-cran-ini/Makefile
devel/p5-Devel-MAT/Makefile
devel/p5-Test-DBIC-ExpectedQueries/Makefile
devel/date/Makefile
devel/py-molecule/Makefile
devel/p5-FFI-Platypus-Declare/Makefile
devel/py-buildbot/Makefile
devel/p5-Class-C3-Adopt-NEXT/Makefile
devel/p5-Class-Accessor-Fast-Contained/Makefile
devel/py-stack-data/Makefile
devel/p5-Path-Resource/Makefile
devel/p5-Data-Object-Class/Makefile
devel/py-jupyter-core/Makefile
devel/py-astroid/Makefile
devel/R-cran-mockery/Makefile
devel/p5-Module-Install-ReadmeFromPod/Makefile
devel/p5-Data-Object-Data/Makefile
devel/p5-File-BaseDir/Makefile
devel/py-defusedxml/Makefile
devel/p5-Devel-NYTProf/Makefile
devel/p5-CHI/Makefile
devel/py-Jinja2/Makefile
devel/p5-Test-File-ShareDir/Makefile
devel/py-ubelt/Makefile
^C

@dlangille
Copy link
Contributor

Testing.

[dev-ingress01 dan /jails/freshports/usr/ports] % cd archivers/p5-IO-Compress-Lzf
[dev-ingress01 dan /jails/freshports/usr/ports/archivers/p5-IO-Compress-Lzf] % make -V TEST_DEPENDS
p5-Test-Pod>=1:devel/p5-Test-Pod
% grep TEST_DEPENDS Makefile
TEST_DEPENDS=	p5-Test-Pod>=1:devel/p5-Test-Pod

But, nothing in the database:

freshports.devgit=# select id, element_pathname(element_id), test_depends from ports where id in (select id from ports_active where name = 'p5-IO-Compress-Lzf') order by element_pathname(element_id);
  id   |             element_pathname             | test_depends 
-------+------------------------------------------+--------------
 28244 | /ports/head/archivers/p5-IO-Compress-Lzf | 
(1 row)

freshports.devgit=# 

@dlangille
Copy link
Contributor

Let's modify the update script to do just one port:

[dev-ingress01 dan ~/modules/Verify] % svn di
Index: refresh-all-active-ports.pl
===================================================================
--- refresh-all-active-ports.pl	(revision 5871)
+++ refresh-all-active-ports.pl	(working copy)
@@ -44,6 +44,7 @@
          PA.name,
          PA.pathname as port_pathname
     FROM ports_active PA
+    WHERE name = 'p5-IO-Compress-Lzf'
 ORDER BY category, name;
 ";
 
[dev-ingress01 dan ~/modules/Verify] % 

@dlangille
Copy link
Contributor

[dev-ingress01 dan ~/modules/Verify] % echo perl ./refresh-all-active-ports.pl | sudo su -fm freshports > /tmp/refresh-p5-IO-Compress-Lzf.log
[dev-ingress01 dan ~/modules/Verify] % 

@dlangille
Copy link
Contributor

So that worked....

freshports.devgit=# select id, element_pathname(element_id), test_depends from ports where id in (select id from ports_active where name = 'p5-IO-Compress-Lzf') order by element_pathname(element_id);
  id   |             element_pathname             |           test_depends           
-------+------------------------------------------+----------------------------------
 28244 | /ports/head/archivers/p5-IO-Compress-Lzf | p5-Test-Pod>=1:devel/p5-Test-Pod
(1 row)

freshports.devgit=# 

@dlangille
Copy link
Contributor

A revised script is running on dev now. I see progress in the count():

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
    53
(1 row)

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
    95
(1 row)

freshports.devgit=# 

@dlangille
Copy link
Contributor

Progress so far:

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
   730
(1 row)

@dlangille
Copy link
Contributor

The grep above, found TEST_DEPENDS in 2332 Makefiles - the script has found 3171 - Perhaps that's from slave ports?

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
  3171
(1 row)

@yurivict
Copy link
Contributor Author

1242 ports origins have TEST_DEPENDS.

@dlangille
Copy link
Contributor

1242 ports origins have TEST_DEPENDS.

Please show your work. ;)

I want to know how that compares to what I found in #411 (comment)

@yurivict
Copy link
Contributor Author

  1. git clone https://github.com/yurivict/freebsd-portsdb.git portsdb
  2. cd portsdb
  3. ./import.sh # this imports /usr/ports into ports.sqlite
  4. sqlite3 ports.sqlite < sql-misc-queries/find-pkgorigins-that-have-test-depends.sql # this finds all pkgorigins with TESTS_DEPENDS

@yurivict
Copy link
Contributor Author

Sorry, the correct number for pkgorigins with TEST_DEPENDS is 6342.
The original query counted them in reverse.

@dlangille
Copy link
Contributor

still going:

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
  3823
(1 row)

freshports.devgit=# 

FYI, regular commit processing is underway while I do this. That's a first. Sometimes it breaks the script because they are using the same git clone instance. When FreshPorts goes a git pull, sometimes a Makefile is not available. For example:

checking to make sure /jails/freshports/usr/ports/graphics/ocaml-cairo/Makefile exists
 * * * * not found.  WTF?
makecommand = /usr/local/bin/sudo /usr/sbin/jexec freshports /make-port.sh /usr/ports graphics/ocaml-cairo 2>/tmp/FreshPorts.graphics.ocaml-cairo.make-error.2022.12.21.22.35.35.81480
Result = 512
size is 123
getting error message from temp file
$result='-1'
$showconfig=''
That make failed:

'This command (FreshPorts code 1):

/usr/local/bin/sudo /usr/sbin/jexec freshports /make-port.sh /usr/ports graphics/ocaml-cairo 2>/tmp/FreshPorts.graphics.ocaml-cairo.make-error.2022.12.21.22.35.35.81480

produced this error:

Error message is: cd: /usr/ports/graphics/ocaml-cairo: No such file or directory
make: cannot open /usr/ports/graphics/ocaml-cairo/Makefile.
Make results are : 
make: stopped in /

@dlangille
Copy link
Contributor

Looks like the script finished about 9 hours ago:

[dev-ingress01 dan /tmp] % ls -l refresh-all-active-ports.log
-rw-r--r--  1 dan  wheel  609731323 2022.12.22 06:07 refresh-all-active-ports.log

@dlangille
Copy link
Contributor

dlangille commented Dec 22, 2022

We have about 5600 ports with TEST_DEPENDS. They are recorded at https://gist.github.com/dlangille/c797354c61ac2f9778fc3c0010243f50

@dlangille
Copy link
Contributor

Taking one at random, I see: https://dev.freshports.org/devel/py-inifile/

@dlangille
Copy link
Contributor

NOTES for me: when updating a database:

begin;
drop view ports_all;
drop view ports_active;
\i updates-2022-12-19-test-depends.ddl 
\i views.sql 
\i permissions.sql 
commit;

@dlangille
Copy link
Contributor

After more jail configuration this morning, the update script is running:

[r720-02-ingress01 dan /usr/local/etc/sudoers.d] % tail -F /tmp/refresh-all-active-ports.log | grep skipp
skipping update - no change in test_depends
skipping update - no change in test_depends
skipping update - no change in test_depends
skipping update - no change in test_depends

@dlangille
Copy link
Contributor

Guess who didn't start this in tmux... restarting.

@dlangille
Copy link
Contributor

This isn't adding up. It's as if the script does not complete.

freshports.org=# select count(*) from ports where test_depends is not null;
 count 
-------
   155
(1 row)
[r720-02-ingress01 dan /jails/freshports/usr/ports] % grep -lr TEST_DEPENDS * | wc -l]\
    2332

@dlangille
Copy link
Contributor

It didn't get past audio/rezound

[r720-02-ingress01 dan /jails/freshports/usr/ports] % tail /tmp/refresh-all-active-ports.log
RepoType='git'
Repository='head'
CommitBranch='0'
calling FreshPorts::Branches::GetPathToRepoForBranch
CommitBranch:   '0
REPODIR:        '/usr/ports'
REPODIR_CHROOT: '/usr/ports'
checking to make sure /jails/freshports/usr/ports/audio/rezound/Makefile exists
Phew.  It's here.  Moving on....
makecommand = /usr/local/bin/sudo /usr/sbin/jexec freshports /make-port.sh /usr/ports audio/rezound 2>/tmp/FreshPorts.audio.rezound.make-error.2022.12.22.19.5.43.25716
[r720-02-ingress01 dan /jails/freshports/usr/ports] % 

@dlangille
Copy link
Contributor

Still running, looking better:

freshports.org=# select count(*) from ports where test_depends is not null;
 count 
-------
  3143

@dlangille
Copy link
Contributor

This started at 0208 and it's now 1957:

[r720-02-ingress01 dan ~] % ps auwwx
USER         PID %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
freshports 89298  2.2  0.0 174740 121384  2  S+J  02:18   11:16.88 perl refresh-all-active-ports.pl

@dlangille
Copy link
Contributor

dlangille commented Dec 23, 2022

the backend and frontend code for this are both in production. The test_depends field is not yet populated. New commits should populate it if there is a value specified.

@dlangille
Copy link
Contributor

Done.


freshports.org=# select count(*) from ports where test_depends is not null;
 count 
-------
  6329
(1 row)

freshports.org=# 

Over to you. :) Merry Christmas.

@dlangille
Copy link
Contributor

dlangille commented Dec 24, 2022

Full list, with test_depends values, at https://gist.github.com/dlangille/ef20e4729ac89231d7df32a5ca0968e7

@dlangille
Copy link
Contributor

However, https://r720-02.freshports.org/x11/xpra/ does not show any test_depends

https://dev.freshports.org/astro/py-astral/#dependencies does show it.

Perhaps your code needs a pull from the repo for classes/port-display - 4b90a28

@dlangille
Copy link
Contributor

@yurivict your test instance now processes live commits. It should always be up to date with prod.

@dlangille
Copy link
Contributor

@yurivict Hope all is well. :)

@dlangille
Copy link
Contributor

@yurivict this is in production now. Shall we close this issue?

@yurivict
Copy link
Contributor Author

Yes, closing it.
Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants