Search: Makefile (ports only): results for a website or domain in the context of WWW #406

grahamperrin · 2022-11-10T07:11:19Z

After raising FreeBSD bug 267686 – emulators/qemu, emulators/qemu-devel, emulators/qemu-user-static: review WWW, I aimed to seek any other port for which the WWW is currently:

https://wiki.qemu.org/Main_Page

Seeking that string, verbatim, did not have the required effect (no real surprise). Instead, I sought:

wiki.qemu.org

https://www.freshports.org/search.php?stype=makefile&method=match&query=wiki.qemu.org&num=30&orderby=category&orderbyupdown=asc&search=Search&format=html&branch=head finds only two of the three ports that were identified in bug 267686.

Please, is this an issue with FreshPorts, or am I overlooking something?

Not found: emulators/qemu-user-static

https://cgit.freebsd.org/ports/tree/emulators/qemu-user-static/Makefile#n10:

WWW= https://wiki.qemu.org/Main_Page

The text was updated successfully, but these errors were encountered:

dlangille · 2022-11-10T14:01:52Z

It is not found because the database contains stale data for that column:

freshports.devgit=# select makefile from ports where id in (select id from ports_active where name = 'qemu-user-static') limit 10;
                                          makefile                                          
--------------------------------------------------------------------------------------------
 
 
 
 # Created by: nox@FreeBSD.org                                                             +
 # $FreeBSD: head/emulators/qemu-user-static/Makefile 441455 2017-05-22 13:17:38Z linimon $+
                                                                                           +
 MAINTAINER=     sbruno@FreeBSD.org                                                        +
 COMMENT=        QEMU CPU Emulator github bsd-user branch - static user targets            +
 PKGNAMESUFFIX=  -user-static                                                              +
                                                                                           +
 QEMU_USER_STATIC=       yes                                                               +
 PKGMESSAGE=     /dev/null                                                                 +
 DESCR=          ${.CURDIR}/pkg-descr                                                      +
 OPTIONS_SLAVE=  STATIC_LINK BSD_USER X86_TARGETS                                          +
 OPTIONS_EXCLUDE=SAMBA X11 GTK2 OPENGL GNUTLS SASL JPEG PNG CURL \                         +
                 CDROM_DMA PCAP USBREDIR GNS3 DOCS                                         +
 MASTERDIR=      ${.CURDIR}/../qemu-sbruno                                                 +
 CONFIGURE_ARGS+=        --extra-cflags=-D_IFI_OQDROPS                                     +
                                                                                           +
 NOT_FOR_ARCHS=          powerpc64                                                         +
 NOT_FOR_ARCHS_REASON=   error: Unhandled abi                                              +
...

dlangille · 2022-11-10T14:38:05Z

I first suspected this was related to the two mega-commits surrounding the move of WWW from pkg-descr to Makefile:

It took a while to catch up and one of the things I missed was updating the Makefile entry. This is updated on every commit but for ports with no subsequent commits it will remain old.

Then I looked at another port:

freshports.devgit=# select makefile from ports where id in (select id from ports_active where name = 'anvil');
                                         makefile                                         
------------------------------------------------------------------------------------------
 # $FreeBSD: head/sysutils/anvil/Makefile 451386 2017-10-06 15:05:48Z dvl $              +
                                                                                         +
 PORTNAME=       anvil                                                                   +
 PORTVERSION=    0.0.11                                                                  +
 PORTREVISION=   1                                                                       +
 CATEGORIES=     sysutils                                                                +
                                                                                         +
...

That is also outdated. Now I suspect Makefile is not being updated properly at all.

dlangille · 2022-12-08T14:27:22Z

I think this confirms the 'makefile not being updates' theory.

freshports.devgit=# select id, makefile from ports where id in (select id from ports_active where name = 'ot-recorder');
  id   | makefile 
-------+----------
 54134 | 
(1 row)

freshports.devgit=#

dlangille · 2022-12-08T16:33:23Z

Seems fixed in dev for new commits. A script will need to be run to fix up head for all ports. This is the diff:

[dev-ingress01 dan ~/modules] % svn di
Index: port.pm
===================================================================
--- port.pm	(revision 5841)
+++ port.pm	(working copy)
@@ -85,6 +85,7 @@
 	$this->{conflicts_install}    = '';
 	$this->{options_name}         = '';
 	$this->{generate_plist}       = '';
+	$this->{makefile}             = '';
 
 	$this->{categories}           = '';
 	$this->{element_pathname}     = '';
@@ -146,6 +147,7 @@
 	$this->{conflicts_install}     = $row->{conflicts_install};
 	$this->{options_name}          = $row->{options_name};
 	$this->{generate_plist}        = $row->{generate_plist};
+	$this->{makefile}              = $row->{makefile};
 
 	$this->{categories}            = $row->{categories};
 	$this->{last_commit_id}        = $row->{last_commit_id};
@@ -278,6 +280,7 @@
        conflicts_build      = " . FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{conflicts_build})        . ", 
        conflicts_install    = " . FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{conflicts_install})      . ", 
        options_name         = " . FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{options_name})           . ", 
+       makefile             = " . FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{makefile})               . ", 
        categories           = " . FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{categories});
 
 
@@ -510,6 +513,8 @@
 	my $Repository   = shift;
 	my $CommitBranch = shift;
 
+	my $makefile; # contents of Makefile for this port
+
 	my $result;
 	my $makecommand;
 	my $ErrorMessage = '';	# stores the result of the latest make command
@@ -918,7 +923,21 @@
 		}
 		chomp($distinfo); # get rid of the trailing whitespace.
 
+		print "extracting Makefile contents for full test searching\n";
+		# this is relative to the host root, not the ports jail root
+		print "\$Makefile='$Makefile'\n";
+		# Let's pull in the Makefile file for full text searching
+		if (defined($Makefile) && $Makefile) {
+		  print "invoking _GetFileContents() for Makefile with '$Makefile'\n";
+		  $makefile = $this->_GetFileContents($Makefile);
+#		  print "back from _GetFileContentsFromJail with '$makefile'\n";
+		} else {
+		  print "Sorry kids, '$Makefile' doesn't look good to me\n";
+		}
+		chomp($makefile); # get rid of the trailing whitespace.
 
+
+
 		# extract the generate_plist contents
 		my $configure_plist_command = "/usr/local/bin/sudo /usr/sbin/jexec $FreshPorts::Config::JailName $FreshPorts::Config::JailConfigurePlist $REPODIR_CHROOT $this->{category}/$this->{name} 2>$TmpFile";
 
@@ -958,8 +977,9 @@
 
 
 		# show some results
-		print "12x \$generate_plist      = '$generate_plist\n'";
-		print "12y \$package_flavors     = '$package_flavors\n'";
+		print "12x \$generate_plist      = '$generate_plist'\n";
+		print "12y \$package_flavors     = '$package_flavors'\n";
+		print "12z \$makefile            = '$makefile'\n";
 
 		print "12 \$shortdescription     = '$shortdescription'\n";
 		print "13 \$longdescription      = '$longdescription'\n";
@@ -1068,6 +1088,7 @@
 		$this->{options_name}	        = $options_name;
 		$this->{conflicts_build}	= $conflicts_build;
 		$this->{generate_plist}		= $generate_plist;
+		$this->{makefile}		= $makefile;
 		# convert all whitespace to a single space
 		# This arose from 200609130717.k8D7HpNc057638@repoman.freebsd.org
 		#
@@ -1177,14 +1198,23 @@
 sub _GetFileContentsFromJail($) {
 	my $this = shift;
 	my $file = shift;
+
+	return $this->_GetFileContents($FreshPorts::Config::JailBaseDir . $file);
+}
+
+
+# =================================
+sub _GetFileContents($) {
+	my $this = shift;
+	my $file = shift;
 	my $filecontents;
 
-	print "about to read from " . $FreshPorts::Config::JailBaseDir . $file . "\n";
+	print "about to read from " . $file . "\n";
 
 	$filecontents = "";
 	# this needs to open relative to the jail root.
 	# to be pure, we should do this as a script in the jail-root
-	if (open (F, $FreshPorts::Config::JailBaseDir . $file))
+	if (open (F, $file))
 	{
 	
 	  while(<F>){

dlangille · 2022-12-08T20:05:49Z

changes packaged and installed on test.

dlangille · 2022-12-08T22:04:00Z

test is good. Now on staging.

dlangille · 2022-12-10T15:19:47Z

Confirmed good on staging.

dlangille · 2022-12-10T15:21:05Z

Confirmed good on prod.

dlangille · 2022-12-10T15:21:42Z

The next step, run a one-off script to refresh Makefile for each port on head.

dlangille · 2022-12-10T16:17:33Z

Now updating dev:

[dev-ingress01 dan ~/modules/Verify] % echo perl ./refresh-all-active-ports.pl | sudo su -fm freshports > /tmp/refresh-all-active-ports.log

ETA 0000-0400 UTC

dlangille · 2022-12-10T18:59:02Z

Restarting after dealing with #407

dlangille · 2022-12-10T22:48:08Z

Let's say one second per port, 45000/60/60 or 12.5 hours...

dlangille · 2022-12-11T13:15:00Z

Finished at about 2022.12.11 12:53

dlangille · 2022-12-11T13:24:16Z

The original search phrase is no longer valid. The Makefiles have since been modified. Now the WWW value is www.qemu.org. That search finds 7 ports on dev:

emulators/qemu
emulators/qemu-devel
emulators/qemu-guest-agent
emulators/qemu-user-static
emulators/qemu-user-static-devel
emulators/qemu6
emulators/qemu70

dlangille · 2022-12-11T13:25:17Z

While waiting for @grahamperrin to confirm, I'll run the update script on both test and stage concurrently.

Adding full text searches on Makefile remains future consideration.

grahamperrin · 2022-12-13T02:53:30Z

… finds 7 ports on dev: …

The same for me, thanks.

https://dev.freshports.org/search.php?stype=makefile&method=match&query=www.qemu.org&num=100&orderby=category&orderbyupdown=asc&search=Search&format=html&branch=head

From the foot of text-only https://dev.freshports.org/search.php?stype=makefile&method=match&query=www.qemu.org&num=100&orderby=category&orderbyupdown=asc&search=Search&format=plaintext&branch=head:

here we are2NumFetches = 7WHAT IS THIS?emulators/qemu
emulators/qemu-devel
emulators/qemu-guest-agent
emulators/qemu-user-static
emulators/qemu-user-static-devel
emulators/qemu6
emulators/qemu70

stage and www also find seven:

emulators/qemu
emulators/qemu-devel
emulators/qemu-guest-agent
emulators/qemu-user-static
emulators/qemu-user-static-devel
emulators/qemu6
emulators/qemu70

dlangille · 2022-12-13T02:54:49Z

Yeah dev has some debug on.

dlangille · 2022-12-14T01:25:20Z

@grahamperrin prod is done.

grahamperrin · 2022-12-14T23:11:22Z

Excellent, thanks.

(I see the 2022-12-14 0110 UTC message at the production site.)

Close?

yurivict · 2022-12-15T01:54:37Z

I've developed PortsDB that rebuilds in 20 minutes and contains all essential information about ports.

It builds the SQLite DB: https://people.freebsd.org/~yuri/ports.sqlite

Maybe FreshPorts can consider using it some time in the future.

FreshPorts can, for example, show what other ports depend on a particular port, like Spack does.

dlangille · 2022-12-15T13:03:27Z

I've developed PortsDB that rebuilds in 20 minutes and contains all essential information about ports.

I've been looking, it seems to do this in parallel? I ask because I see -j $(sysctl -n hw.ncpu) in functions.sh under ports_tree_traverse()

What is make -I doing?

Where is data extracted from the ports tree for a specific port? I could not find that.

I have thought about making my maintenance scripts run in parallel instead of serial. My main concern: updating master ports first before any slave ports.

It builds the SQLite DB: https://people.freebsd.org/~yuri/ports.sqlite

Maybe FreshPorts can consider using it some time in the future.

FreshPorts can, for example, show what other ports depend on a particular port, like Spack does.

FreshPorts does not have a user interface like Spack, but it does show the dependencies. For example, at https://www.freshports.org/shells/bash/#dependencies the run time dependencies are shown. Both the forward (what does this port need (RUN_DEPENDS)) and the reverse (what ports require this port) are shown.

yurivict · 2022-12-15T18:09:07Z

I've been looking, it seems to do this in parallel? I ask because I see -j $(sysctl -n hw.ncpu) in functions.sh under ports_tree_traverse()

It uses 'make' to parallelize port tree scan.

What is make -I doing?

It creates the shadow ports tree from the main tree which it scans. -I is to specify it. This is because it needs an additional patch in the tree.

Where is data extracted from the ports tree for a specific port? I could not find that.

It is written first to the SQL script, which is later executed and creates SQLite records.

dlangille self-assigned this Nov 10, 2022

dlangille added the bug label Nov 10, 2022

dlangille mentioned this issue Dec 10, 2022

ports_active should honor branch #407

Closed

dlangille closed this as completed Dec 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Search: Makefile (ports only): results for a website or domain in the context of WWW #406

Search: Makefile (ports only): results for a website or domain in the context of WWW #406

grahamperrin commented Nov 10, 2022 •

edited

dlangille commented Nov 10, 2022 •

edited

dlangille commented Nov 10, 2022

dlangille commented Dec 8, 2022

dlangille commented Dec 8, 2022 •

edited

dlangille commented Dec 8, 2022

dlangille commented Dec 8, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 11, 2022

dlangille commented Dec 11, 2022

dlangille commented Dec 11, 2022 •

edited

grahamperrin commented Dec 13, 2022

dlangille commented Dec 13, 2022

dlangille commented Dec 14, 2022

grahamperrin commented Dec 14, 2022

yurivict commented Dec 15, 2022

dlangille commented Dec 15, 2022

yurivict commented Dec 15, 2022

Search: Makefile (ports only): results for a website or domain in the context of WWW #406

Search: Makefile (ports only): results for a website or domain in the context of WWW #406

Comments

grahamperrin commented Nov 10, 2022 • edited

dlangille commented Nov 10, 2022 • edited

dlangille commented Nov 10, 2022

dlangille commented Dec 8, 2022

dlangille commented Dec 8, 2022 • edited

dlangille commented Dec 8, 2022

dlangille commented Dec 8, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 10, 2022

dlangille commented Dec 11, 2022

dlangille commented Dec 11, 2022

dlangille commented Dec 11, 2022 • edited

grahamperrin commented Dec 13, 2022

dlangille commented Dec 13, 2022

dlangille commented Dec 14, 2022

grahamperrin commented Dec 14, 2022

yurivict commented Dec 15, 2022

dlangille commented Dec 15, 2022

yurivict commented Dec 15, 2022

grahamperrin commented Nov 10, 2022 •

edited

dlangille commented Nov 10, 2022 •

edited

dlangille commented Dec 8, 2022 •

edited

dlangille commented Dec 11, 2022 •

edited