Ubuntu installation of critcl is broken

[esternin]

Ubuntu depositories seem to have critcl packaged and available, so all one needs (Under Ubuntu 20.04 LTS) is to

sudo apt-get install critcl tcllib-critcl
...
The following additional packages will be installed:
  tcl-trf
...

and then

$ tclsh
% package require critcl
3.1.17
%

seems to be just fine. Unfortunately, no calls to critcl (for example, critcl::clibraries -Lwhatever) work, because /usr/share/tcltk/critcl3.1.17/critcl.tcl contains this line: package require critcl_md5c, and this package is nowhere to be found.

Are my expectations unreasonable? Is the only way to get critcl is to build it from github sources? is there a built version of critcl_md5c somewhere that I can download and install?

[andreas-kupries]

Checking the sources here, the package sources are present at https://github.com/andreas-kupries/critcl/tree/master/lib/critcl-md5c

Checking the build.tcl script this package is built properly by it.
See

critcl/build.tcl

Lines 488 to 520 in 98ce8e1

	# Special package: critcl_md5c
	# Local MD5 hash implementation.
	puts "\nInstalled C package:\tcritcl::md5c"
	# It is special because it is a critcl-based package, not pure
	# Tcl as everything else of critcl. Its installation makes it
	# the first package which will be compiled with critcl on this
	# machine. It uses the just-installed application for
	# that. This is package-mode, where MD5 itself is not used, so
	# there is no chicken vs. egg.
	set src [file join $selfdir lib critcl-md5c md5c.tcl]
	set version [version $src]
	set name critcl_md5c
	set dst [file join $dstl [pkgdirname $name $version]]
	set cmd {}
	lappend cmd exec >@ stdout 2>@ stderr
	lappend cmd [info nameofexecutable]
	lappend cmd $theapp
	if {$target ne {}} {
	lappend cmd -target $target
	}
	lappend cmd -libdir [file join $dstl tmp] -pkg $src
	puts [list executing $cmd]
	eval $cmd
	file delete -force $dst
	file rename [file join $dstl tmp md5c] $dst
	file delete -force [file join $dstl tmp]
	puts "${prefix}Installed package: $dst"

It is basically the first critcl package which is built using critcl itself.

Given that I am Ubuntu 20.04 LTS myself I used wajig list-files critcl (*) to get the list of installed files and filtering for package index files I confirm that critcl_md5c is not installed.

I also confirm that the package critcl::callback is not installed either.
This is the second critcl package the build.tcl script creates using the core critcl itself.
See

critcl/build.tcl

Lines 525 to 559 in 98ce8e1

	puts "\nInstalled C package:\tcritcl::callback"
	# It is special because it is a critcl-based package, not pure
	# Tcl as everything else of critcl. Its installation makes it
	# the second package which will be compiled with critcl on this
	# machine. It uses the just-installed application for
	# that.
	set src [file join $selfdir lib critcl-callback callback.tcl]
	set version [version $src]
	set name critcl_callback
	set dst [file join $dstl $name$version]
	set dsth [file join $dsti $name]
	set cmd {}
	lappend cmd exec >@ stdout 2>@ stderr
	lappend cmd [info nameofexecutable]
	lappend cmd $theapp
	if {$target ne {}} {
	lappend cmd -target $target
	}
	set dstl_tmp [file join $dstl tmp]
	lappend cmd -libdir $dstl_tmp
	lappend cmd -includedir $dstl_tmp
	lappend cmd -pkg $src
	eval $cmd
	file delete -force $dst $dsth
	file rename [file join $dstl tmp callback] $dst
	file rename [file join $dstl tmp critcl_callback] $dsth
	file delete -force $dstl_tmp
	puts "${prefix}Installed package: $dst"
	puts "${prefix}Installed headers: [
	file join $dsti critcl_callback]"

This looks to me to be an issue with how Ubuntu is building the package.
IMHO you should report this to Ubuntu so that they can fix their build process.

While I am the maintainer of Critcl I am not involved in how Ubuntu is building it.

---

(*) Where wajig is a nice wrapper around the lot of debian/apt package tools.

[esternin]

I am not sure how an outsider to the code can ask anything of "Ubuntu", this may have to come from you.

So is there a workaround? (like an alternate critcl.deb) I manually installed the files from github into /usr/share/tcltk/critcl-md5c directory and added pkgIndex.tcl to it, but this did not solve the problem.

By the way, I think there might be a good reason why md5c package is failing: the "5" in the name may be a problem, as it gets mistaken for a version number, see here, under Problems No.1.

[esternin]

I downloaded the latest tar and tried to build from source. build.tcl install immediately fails on md5 for me. There must be something on your system that is different.

$ sudo apt remove critcl tcllib-critcl
$ find /usr/share/tcltk -name "*critcl*"
/usr/share/tcltk/tcllib1.20/pt/pt_cparam_config_critcl.tcl
/usr/share/tcltk/tcllib1.20/pt/rde_critcl
$ uname -a
Linux XXXX 5.4.0-97-generic #110-Ubuntu SMP Thu Jan 13 18:22:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
...
$ tclsh build.tcl install -target linux-64-* ../critcl3.18
Installing into:
	Packages:	../critcl3.18
	Application:	./bin
	Headers:	./include

Installed package:      ../critcl3.18/critcl-app3.1.18
Installed package:      ../critcl3.18/critcl3.1.18
Installed package:      ../critcl3.18/critcl-bitmap1.0.1
Installed package:      ../critcl3.18/critcl-class1.1
Installed package:      ../critcl3.18/critcl-cutil0.2
Installed package:      ../critcl3.18/critcl-emap1.2
Installed package:      ../critcl3.18/critcl-enum1.1
Installed package:      ../critcl3.18/critcl-iassoc1.1
Installed package:      ../critcl3.18/critcl-literals1.3
Installed package:      ../critcl3.18/critcl-platform1.0.15
Installed package:      ../critcl3.18/critcl-util1.1
Installed package:      ../critcl3.18/dict841
Installed package:      ../critcl3.18/lassign841.0.1
Installed package:      ../critcl3.18/lmap841
Installed package:      ../critcl3.18/stubs_container1
Installed package:      ../critcl3.18/stubs_gen_decl1
Installed package:      ../critcl3.18/stubs_gen_header1
Installed package:      ../critcl3.18/stubs_gen_init1
Installed package:      ../critcl3.18/stubs_gen_lib1
Installed package:      ../critcl3.18/stubs_gen_macro1
Installed package:      ../critcl3.18/stubs_gen_slot1
Installed package:      ../critcl3.18/stubs_genframe1
Installed package:      ../critcl3.18/stubs_reader1
Installed package:      ../critcl3.18/stubs_writer1
geedonk {. .. critcl3.18}
Installed application:  ./bin/critcl

Installed C package:	critcl::md5c
executing {exec >@ stdout 2>@ stderr /usr/bin/tclsh ./bin/critcl -target linux-64-* -libdir ../critcl3.18/tmp -pkg /home/xxxx/andreas-kupries-critcl-74c550f/lib/critcl-md5c/md5c.tcl}
can't find package critcl::app
    while executing
"package require critcl::app"
    (file "./bin/critcl" line 13)
child process exited abnormally
    while executing
"exec >@ stdout 2>@ stderr /usr/bin/tclsh ./bin/critcl -target linux-64-* -libdir ../critcl3.18/tmp -pkg /home/xxxx/andreas-kupries-critcl-74c550f/lib/..."
    ("eval" body line 1)
    invoked from within
"eval $cmd"
    invoked from within
"_install -target linux-64-* ../critcl3.18"
    ("eval" body line 1)
    invoked from within
"eval _$argv"

[esternin]

Alas, build.tcl fails even faster if I use an absolute path for the destination, it tries to create /bin, /include, etc.:

$ tclsh build.tcl install /home/xxxx/critcl3.1.18
can't create directory "/include": permission denied
    while executing
"file mkdir $dsta $dsti"
    (procedure "_install" line 17)
    invoked from within
"_install /home/edik/critcl3.1.18"
    ("eval" body line 1)
    invoked from within
"eval _$argv"

I see that build.tcl has not changed on github in 2 years. Are you maybe using a private copy of it that is different?

[andreas-kupries]

I am not sure how an outsider to the code can ask anything of "Ubuntu", this may have to come from you.

Ubuntu does not have an official ticket system where you can report issues with the distribution ?

I manually installed the files from github into[...] and added pkgIndex.tcl

This is underspecified in what files where copied, and the exact actions taken. I suspect missing files, and an incorrect pkgIndex.tcl. My package directory (generated from build.tcl) looks like

hephaistos:(511) ~/Development/0.Stable/Critcl/w-main > find ~/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/ -print0 | xargs -0 ls -ld
drwxr-xr-x 3 aku aku  4096 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/
-rw-r--r-- 1 aku aku 11028 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/critcl-rt.tcl
-rw-r--r-- 1 aku aku    14 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/license.terms
drwxr-xr-x 2 aku aku  4096 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/linux-x86_64
-rwxr-xr-x 1 aku aku 14632 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/linux-x86_64/md5c.so
-rw-r--r-- 1 aku aku   457 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/pkgIndex.tcl
-rw-r--r-- 1 aku aku   595 Nov 26  2020 /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/teapot.txt

hephaistos:(512) ~/Development/0.Stable/Critcl/w-main > cat ~/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12/pkgIndex.tcl 
if {![package vsatisfies [package provide Tcl] 8.4]} {return}
package ifneeded critcl_md5c 0.12 "[list proc __critcl_load__ {dir} { ;     source [file join $dir critcl-rt.tcl] ;     set path [file join $dir [::critcl::runtime::MapPlatform]] ;     set ext [info sharedlibextension] ;     set lib [file join $path "md5c$ext"] ;     load $lib Md5c ;     package provide critcl_md5c 0.12 ;     catch {rename __critcl_load__ {}}}] ; [list __critcl_load__ $dir]"

The problem you are referring to wrt the 5 in the package name is more a problem with the pkg_mkIndex command pretty much nobody uses anymore these days, AFAIK.

the latest tar

That is 3.1.18.1 ?

private copy of it that is different?

I am working with commit 98ce8e1 which is a bit beyond 3.1.18.1.

From a checkout of that commit:

1. Cleanup of existing installation

hephaistos:(515) ~/Development/0.Stable/Critcl/w-main > ./build.tcl drop
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-app3.1.18
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl3.1.18
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-bitmap1.0.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-class1.1.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-cutil0.2.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-emap1.2
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-enum1.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-iassoc1.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-literals1.3
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-platform1.0.15
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl-util1.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/dict841
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/lassign841.0.1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/lmap841
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_container1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_gen_decl1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_gen_header1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_gen_init1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_gen_lib1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_gen_macro1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_gen_slot1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_genframe1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_reader1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/stubs_writer1
Removed package:     /home/aku/opt/ActiveTcl/lib/tcl8.6/critcl_md5c0.12
Removed application: /home/aku/opt/ActiveTcl/bin/critcl

Then

hephaistos:(516) ~/Development/0.Stable/Critcl/w-main > ./build.tcl install |& tee install.log

Attached: install.log

tclsh build.tcl install -target linux-64-* ../critcl3.18

I do not believe that there is a need to specify the -target. Leaving it out on my system it auto-selects linux-x86_64-gcc.

can't find package critcl::app
while executing
"package require critcl::app"

Ah. The directory ../critcl3.18, which install the Tcl parts into is not a proper package directory for the /usr/bin/tclsh.
I suspect you are using your own directory because if you don't the build.tcl tries to install under /usr, and you do not have permissions and/or do not wish to sudo.

In my case I am ok without an explicit directory because I use Tcl installation in my home directory, and not the system tclsh.

Trying build.tcl install ../foo fails for me also ...
And I confirm the same issue you have for an absolute path.

Well, it looks as if I have to revisit this stuff yet another time ...

Thank you for bringing this to my attention, and in persisting.

In your mind, what is the ../critcl3.18 directory exactly ?
AFAICT you are treating it as the directory where to install

[andreas-kupries]

While I have to go to sleep now I see that the installed critcl app actually contains code to find its packages, regardless of where it is installed. That code however looks to be broken in at least 2 ways. One I was able to fix, the other not yet. A bad calculation of a relative path from app to libdir ...

I will continue tomorrow, likely in the evening ... Rebuild the logic from scratch, with lots of comments, serving as spec.

[esternin]

Thank you for your patience with my prodding.

In general, I prefer to (1)use the distributed "stable" versions, so I download .tars and work from there. The latest pointer on your site is to a tar ball that opened up into andreas-kupries-critcl-74c550f, as you could see in my original post. I later tried with your latest commit, 98ce8e1, with the same results.

I also (2) prefer to first install a new package in my user space, to test and review. No disrespect, but giving sudo privileges to a .tcl script I just downloaded from somewhere feels like a dangerous thing to do. I believed I was following your manual when I specified a destination directory as the fourth command-line parameter.

I have since read the build.tcl script and ran the command with privileges; it worked without a hitch. So there is a path parsing bug somewhere, all right.

I had some trouble with critcl::clibraries afterwards, as the compiler would not find the requested .so library, despite several clean-ups and re-installs, and ldconfigs. I ended up moving the library into a default location for the gcc, and then it worked.

I am very keen to get things tested. The goal for me is to provide tcl with hardware access through a manufacturer-supplied library, for a high-resolution deep-sky camera. Ideally, I would simply bind each of the C subroutines to its equivalent tcl function. I have not been getting very much traction based on just the posted API, I found it difficult to figure out, especially for passing/returning strings and structures (all your examples are about passing ints and floats). Hardware-support libraries are usually full of such stuff, so if you know of some working examples, I would appreciate a pointer!

Maybe I'll open a new issue on this: "need more and simple examples"?

[esternin]

In your mind, what is the ../critcl3.18 directory exactly ?
AFAICT you are treating it as the directory where to install

Yes, of course. Isn't this correct? I am confused. From tclsh build.tcl help:

      /home/edik/critcl/build.tcl install ?-target T? ?destination?
	Install all packages, and application.
	destination = path of package directory, default [info library].

How else could one interpret these instructions? "Build and install into the destination directory", which may be different for different architectures, for example.

[andreas-kupries]

So there is a path parsing bug somewhere, all right.

Having a fresh brain this morning, and from last looks just before going to bed I believe it is the findlib procedure in build.tcl which makes the trouble. If the specified path is of the form /path/to/lib/foo/ it is used to find the /path/to/lib and use that as directory to place bin and include as siblings to. IIRC this code comes from Pyorick. Things is, it looks to be off the rails if there is no lib/ segment in the path, like for ../foo. Of course, I accepted this thing at the time, so I take part in the mess.

.. install operations ... permissions ...

Yes, all agreed.

critcl::clibraries

Will revisit the docs on expected arguments. It should accept something like -lzstd. ... This may assume that the library is in some standard place for the compiler to find. ... It may accept -Lpath also to extend the set of such paths, or also -l/path/to/shlib.so. Currently not recalling.

I know that the -lzstd worked for me in some personal code. ... This code is inofficially available at http://akupries.tclers.tk/tmp/CritclExamples.tgz ... Wrappers for zstd, a timezone database, libcorrect, and cdc-rabin fingerprinting.

Maybe I'll open a new issue on this: "need more and simple examples"?

Yes, please. I dislike holes in the examples ... and I see you added #116 already. Thank you.

In the meantime please see the above link, if these four wrappers are helpful to you. All of them are used in a personal project. (Consider them BSD-licensed) (Ah, the various package prolog code has references to other packages in that project which are not in the tarball. The commands ak is ensemble ... and ak is tracing ... will not work, among others).

And now I have to go and work for money ;) ...

[andreas-kupries]

🤦 ... I have #114 sitting there for most of a year now.

I will rebase this to master, and then check if its behaviour is sensible. Defaults, derivations, etc. Absolute/relative paths.
Especially test that it can build the md5 and callback packages properly.

If yes then I would propose to go and merge this to let build.tcl have path handling in line with configure and the like.

@pooryorick I remember that you fiddled with the build.tcl path handling quite a bit. Given this I ask you to review the #114 and see if it would work for you and your cases of installing critcl. Please respond quickly, and especially if you do not have the time to review and test.

[andreas-kupries]

Rebase done, fixed some nits, and added a paths method to check what paths are generated from the options and environment before committing to an active operation.

Still to check if the processing and insertion of paths into the generated critcl app is correct. I suspect not.

[andreas-kupries]

Pushed fixes for that also. The #114 PR should now be in a workable state.

[esternin]

critcl::clibraries

Will revisit the docs on expected arguments. It should accept something like -lzstd. ... This may assume that the library is in some standard place for the compiler to find. ... It may accept -Lpath also to extend the set of such paths, or also -l/path/to/shlib.so. Currently not recalling.

Ah, that's interesting. The way I interpreted the documentation was to use two calls, one to set -L, one to set -l. Should they have been combined, as the last one replaces the previous? For multi-platform tcl code it would be handy to separate the two, as the library name is always the same, but the path in the SDK and the file extensions may be different (e.g. .so vs .dll), but no biggie if it must be a single line.

package require critcl
critcl::cheaders /opt/ASI_linux_mac_SDK_V1.21/include/ASICamera2.h
critcl::clibraries -L/opt/ASI_linux_mac_SDK_V1.21/lib/x64
critcl::clibraries -lASICamera2

This should probably be in #116

[andreas-kupries]

https://andreas-kupries.github.io/critcl/doc/files/critcl_pkg.html#24

The important part there is that Calls to this command are cumulative.

[andreas-kupries]

Lets keep the entire discussion wrt documentation on #116 .

Can you check if the modified build.tcl on the head of #114 is suitable ? I.e. working for you ?

[esternin]

--dest-dir worked as expected, using both relative and absolute addresses, on RedHat 7 and on Ubuntu 20.04 LTS.

Did not test other flags, but they should allow for a flexible install/rebuild for Ubuntu distribution.

One thing: I would not have chosen the tcl8.6/ as the place to put all this, since it would then have to be duplicated for tcl8.7/ or whatever, when it arrives. On Ubuntu, packages go into /usr/share/tcltk/ directly, next to where /usr/share/tcltk/tcl8.6 and potentially other versions live. Maybe also group all of the critcl-* and snub-* installs under one umbrella directory, critcl, to keep all critcl packages together. Just a suggestion.

[andreas-kupries]

... .../tcl8.6/

The new build.tcl it does not use [info library] as default, i.e. no .../tclX.y for libdir, except if the user explicitly specifies it as such.

[andreas-kupries]

I will now wait until Friday for @pooryorick to also weigh in on #114. If he does not object this will be merged.
Although I plan to separate the doc work out of that PR so that #116 has its own dedicated PR.

[andreas-kupries]

Split the doc work to #117 .

[esternin]

Andreas, I know you closed this, but I found myself installing on another computer (a Chromebook, in a Crostini Linux container) and realized that, just like for Ubuntu, the default destination for libs is not quite right.

All tcl add-ons should be in /usr/share/tcltk, and not in the /usr/lib as build.tcl defaults. They will not be found without some path modification on the user's part. I do not know if your default is a remnant from a previous era, or if there is a distro out there that uses /usr/lib for add-ons, but you might want to re-consider this.

The new switches allow one to overcome this, of course, so what I did on Ubuntu, and now on Debian - arguably the two most popular distros, was:

tclsh build.tcl install --lib-dir /usr/share/tcltk

You may want to include it as instructions for Ubuntu/Debian users, if you want to keep your default.

[andreas-kupries]

Note added to the how to install.