dman support

[anarcat]

one of the great uses of manpages.d.o is for installs that do not have man installed. for this, the ubuntu people wrote this clever little script called dman which fetches the raw pages to display locally. it's pretty neat!

but out of the box, it doesn't work with the debiman layout. i had to modify the script so that it works with manpages.debian.org - here's the diff:

--- dman	2016-04-13 10:47:42.000000000 -0400
+++ dman.debian	2017-01-31 13:31:49.110251075 -0500
@@ -26,7 +26,11 @@
 ###############################################################################
 
 
-. /etc/lsb-release
+if [ -r /etc/lsb-release ] ; then
+    . /etc/lsb-release
+else
+    DISTRIB_CODENAME=$(lsb_release -c -s)
+fi
 while true; do
 	case "$1" in
 		--release)
@@ -44,21 +48,27 @@
 # Mirror support of man's languages
 if [ ! -z "$LANG" ]; then
 	LOCALE="$LANG"
+    LOCPATH="-$LANG"
+    LOCDOT=".$LANG"
 fi
 if [ ! -z "$LC_MESSAGES" ]; then
 	LOCALE="$LC_MESSAGES"
+    LOCPATH="-$LOCALE"
+    LOCDOT=".$LOCALE"
 fi
-if echo $LOCALE | grep -q "^en"; then
+if echo $LOCALE | grep -E -q "^(C|en)"; then
 	LOCALE=""
+    LOCPATH=""
+    LOCDOT=".en"
 fi
 
-URL="http://manpages.ubuntu.com/manpages.gz/"
+URL="http://manpages.debian.org"
 
 mandir=`mktemp -d dman.XXXXXX`
 trap "rm -rf $mandir" EXIT HUP INT QUIT TERM
 for i in `seq 1 9`; do
 	man="$mandir/$i"
-	if wget -O "$man" "$URL/$DISTRIB_CODENAME/$LOCALE/man$i/$PAGE.$i.gz" 2>/dev/null; then
+	if wget -O "$man" "$URL/$DISTRIB_CODENAME/manpages$LOCPATH/$PAGE.$i$LOCDOT.gz" 2>/dev/null; then
 		man $MAN_ARGS -l "$man" || true
 	fi
 	rm -f "$man"

notice how I had to dance around the locale stuff... i posted the full script in this paste bin: http://paste.debian.net/911957/

also, the above doesn't actually work in Debian stretch because it looks for https://manpages.debian.org/stretch/manpages/intro.1.en.gz - which is a 302 into the redirector, which redirects to the HTML page:

anarcat@curie:~$ curl -I https://manpages.debian.org/stable/manpages/intro.1.en.gz     
HTTP/1.1 302 Found
Date: Tue, 31 Jan 2017 18:42:00 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000
Public-Key-Pins: pin-sha256="m2r9mfIa+ot6bIIC0bCt/7KZ1ych/f8QY3gk9cqUWqs="; pin-sha256="35f/cSfa9he3sUJgp1wZT9gzbI7/zH10hlT/utpEziU="; max-age=5184000
Location: https://dyn.manpages.debian.org/stable/manpages/intro.1.en.gz?
Cache-Control: max-age=3600
Expires: Tue, 31 Jan 2017 19:42:00 GMT
Content-Type: text/html; charset=iso-8859-1

anarcat@curie:~$ curl -I https://dyn.manpages.debian.org/stable/manpages/intro.1.en.gz?     
HTTP/1.1 307 Temporary Redirect
Date: Tue, 31 Jan 2017 18:41:54 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000
Public-Key-Pins: pin-sha256="3T9ypiCPJdEeUOpKSooGJ1IpFbKsl/ktH0dV/wygJMk="; pin-sha256="xV7KmbTUH6WeUjOC5Tv7gsOpie45AvOH8/vjaIBsBxk="; max-age=5184000
Content-Length: 68
Content-Type: text/html; charset=utf-8
X-Clacks-Overhead: GNU Terry Pratchett
Location: https://manpages.debian.org/jessie/manpages/intro.1.en.html
Content-Language: en

so to fix this, we'd need to:

• ship the dman script along with other assets for people's easy deployment and joy
• fix the redirector so it supports codenames (e.g. stretch) along with symbolic names (e.g. testing), maybe related to should use /stretch instead of /testing #54?
• ideally, make this part of the builtin man binary and/or package this as a separate dman binary package that can be installed for real (see also package for Debian #1 for debian packaging) was made part of the debian-goodies package
• maybe rewrite dman in go while we're at it? :p - moved to package for Debian #1

[anarcat]

redirectory fixed in ae44c70, thanks!

[anarcat]

i just noticed there's a debman command in the debian-goodies package. That command can show the manpage within a .deb or it will download the .deb with debget if missing.

Sounds like another thing we could hijack. :) debman is a shellscript as well.

[anarcat]

in the meantime, could we ship the dman script in the assets and link it somewhere? maybe in the about section?

i'd make a PR if that's okay with you.

[stapelberg]

I’d prefer it if we could package up the dman script in a debian package (probably to experimental as we’re currently frozen). I think it’d be more confusing than useful to have different distribution channels for software, and Debian packages are definitely the best way to distribute software to Debian users :)

[anarcat]

Okay well, the problem is this will take forever to trickle down into a stable release... How about this - we figure out a debian package for our little shell script refugee and link to its source code in the FAQ?

In fact, I have created a collab-maint branch for dman in debian-goodies called debiman-support which adds a dman command to the package. We could then link to the source code when/if this is accepted by the package maintainer...

Here's the source code now:

https://anonscm.debian.org/git/collab-maint/debian-goodies.git/tree/dman?h=debiman-support

How does that sound?

[anarcat]

The alternative to this, of course, is to package debiman directly (see #1) and make this a distinct binary package people can just install directly. Given the complexity (or I guess my unfamiliarity) with golang + debian packaging, I am tempted to just stick with this easier option, as it also doesn't involve a rewrite in go. ;)

in other news, i asked to merge the branch in debian-goodies in #860920. let's see how that flies.

[stapelberg]

Linking to the source of a Debian package is fine with me.

[xtaran]

debian-goodies 0.70 with dman included has just been uploaded to Debian experimental.

[jbicha]

@dustinkirkland See above and https://tracker.debian.org/news/844034

[anarcat]

alright, I've linked to dman in the FAQ - we just need to merge the branch to complete this issue I believe - see #1 for the pipe dreams of packaging debiman and rewriting dman and so on. ;)

[dustinkirkland]

Hey team -- first off, I'm super pleased that you find dman useful! I love it, as well.

I would be very happy for Debian to ship dman in a Debian package, of course. I would just politely ask that you make it work with both Debian and Ubuntu, with a simple conditional that you can key directly off of the info you get out of the lsb-release.

As such, we'll be glad to use the same dman in Ubuntu as you use in Debian.

Thanks!
@dustinkirkland

[anarcat]

On 2017-04-28 12:18:14, Dustin Kirkland wrote: Hey team -- first off, I'm super pleased that you find dman useful! I love it, as well. I would be very happy for Debian to ship dman in a Debian package, of course. I would just politely ask that you make it work with *both* Debian and Ubuntu, with a simple conditional that you can key directly off of the info you get out of the lsb-release. As such, we'll be glad to use the same dman in Ubuntu as you use in Debian.

Hi! I think that's a great idea. Unfortunately, I made some changes to the script and I am not sure it still works against manpages.ubuntu.com right now. I am certain, however, that the debian-goodies maintainer will happily welcome patches to fix that! The source code is here: https://anonscm.debian.org/git/collab-maint/debian-goodies.git/tree/dman And you can file a pull request / send a patch on the Debian-goodies package: https://tracker.debian.org/pkg/debian-goodies Thanks!

[xtaran]

@anarcat, @dustinkirkland: Yes, actually I had this issue in my mind already, too. Patches are indeed very welcome.

[anarcat]

i tried this simple patch to try and support ubuntu (based on the no-suite branch):

diff --git a/dman b/dman
index a44ecf1..fea73f7 100755
--- a/dman
+++ b/dman
@@ -61,7 +61,14 @@ if echo $LOCALE | grep -E -q "^(C|en)"; then
        LOCDOT=".en"
 fi
 
-BASE_URL="https://manpages.debian.org"
+case $(lsb_release -s -i) in
+    Ubuntu)
+        BASE_URL="https://manpages.ubuntu.com"
+        ;;
+    *)
+        BASE_URL="https://manpages.debian.org"
+        ;;
+esac
 
 mandir=`mktemp -d dman.XXXXXX`
 trap "rm -rf $mandir" EXIT HUP INT QUIT TERM

unfortunately that doesn't work, because we look into debiman-specific paths and ubuntu's manpage service lacks the required redirections:

./dman: not found: https://manpages.ubuntu.com/stretch/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/stretch/crontab.gz
./dman: not found: https://manpages.ubuntu.com/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/crontab.gz
./dman: Could not fetch manpage from given locations.

even when using the manpages.gz prefix, this fails:

./dman: not found: https://manpages.ubuntu.com/manpages.gz/stretch/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/manpages.gz/stretch/crontab.gz
./dman: not found: https://manpages.ubuntu.com/manpages.gz/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/manpages.gz/crontab.gz
./dman: Could not fetch manpage from given locations.

the original dman had quite a bit of logic to guess the filesystem paths on the other end. we got rid of a lot of that for performance reasons: in the worst case, the original version was doing 9 requests to the server - we lowered that to 2, and jumped back to 4 to support multiple suites.

so i'm not sure how to deal with this. the logic will get really hairy if we need to support site-specific patterns like this. all i can think of now is rewrite the thing in python to support parameter expansion - e.g. https://%(base_url)s/%(suite)s/%(man)s.%(lang)s.gz is debiman, and ubuntu is https://%(base_url)s/%(lang)s/man%(section)s/%(man)s.%(section)s.gz... maybe this could be done in bash too, but you'd get nasty eval()s...

[anarcat]

we have opened #861611 for Ubuntu support in dman, so this is pretty much complete here. i have opened a separate issue (#79) for some optimizations that could make dman perform a little better as well.