Let iconv build on Cygwin without installing Cygwin libiconv package

[jpflori]

Crafting a somehow minimal Cygwin install, I got the libiconv2 package with binaries installed but the libiconv one with dev things is not.

Because of that building our libiconv spkg fails because it uses the system libintl which points to libiconv files provided in the dev libiconv package (and not the libiconv2).

Maybe the easiest solution is to prereq the installation of the dev libiconv package, but we can also go by passing --disable-nls to configure.

Use spkg at
http://boxen.math.washington.edu/home/jpflori/iconv-1.13.1.p4.spkg

(I'll investigate in a follow-up ticket if building our own libiconv is actually needed on Cygwin nowadays.)

CC: @kcrisman

Component: packages: standard

Keywords: cygwin iconv spkg

Author: Jean-Pierre Flori

Reviewer: Jeroen Demeyer

Merged: sage-5.7.beta0

Issue created by migration from https://trac.sagemath.org/ticket/13912

[jdemeyer]

Reviewer: Jeroen Demeyer

[jdemeyer]

comment:2

I cannot test on Cygwin, but I assumes this solves your problem. Anyway, the patch itself looks good.

[jdemeyer]

comment:3

Hang on, isn't this a typo:

CYGGWIN

[jdemeyer]

comment:4

Also, the URL gives 403 Forbidden

[jpflori]

comment:5

Indeed, I'll fix both issues.

[jpflori]

Attachment: iconv-1.13.1.p4.diff.gz

Spkg diff, for review only.

[jpflori]

comment:6

Done, diff and spkg updated.

[kcrisman]

comment:7

It will take me too long at this point of the semester to be able to check whether this is necessary to review it, but it would be nice if you could post a link to exactly what this change does in the configuration and why it works; otherwise Jeroen's review is fine with me.

[jpflori]

comment:8

Basically this just adds --disable-nls (we don't use nls anyway) to configure so that we don't link against libintl which would imply to install the ... libiconv devel package.
So I think disabling nls is more proper.

[kcrisman]

comment:9

Okay, I just don't know what nls is, which is what I was really asking. After a significant amount of searching, I found this which explained it, and I definitely agree that we don't need internationalization - well, not at this time, though it would be awesome and amazing if someone somehow internationalized all of Sage. Which sounds like a very large job.

[jdemeyer]

Merged: sage-5.7.beta0

[dimpase]

comment:11

I don't know why it only hit at beta1, not beta0, but rebasing has kind of destroyed my cygwin install, as now even bash breaks with fork errors at libiconv*.dll. My only guess it could be several instances of dlls with the same name...

[dimpase]

comment:12

I'd like to reopen this ticket. The same issue as here applies to the rest of Sage packages that create dlls with names clashing with dlls from Cygwin packages (or perhaps other ones). Creating a dll which has the same name as a Cygwin dll is asking for big trouble.
You might read on for details here in this long thread.

In a nutshell, Windows is so dump that an application request to load a dll with a certain name is handled by searching certain sub-directories, in certain order. (Some of these subdirectories are actually specifies in Windows registry). In some cases the order of sub-directores in the PATH matters. I.e. when you link to a dlls, no full path information on the dll is stored.
All you have is the PATH, registry entries, and few other things like current directory, to play with.

[jdemeyer]

comment:13

Replying to @dimpase:

I'd like to reopen this ticket.

You mean unmerge it?

[dimpase]

comment:14

Replying to @jdemeyer:

Replying to @dimpase:

I'd like to reopen this ticket.

You mean unmerge it?

It probably does not matter too much, whether it is unmerged immediately or not, as there are more tickets like this. It's the whole Cygwin port strategy that needs to be re-evaluated.

By the way, a typical symptom of this problem, dll name clash, is a Cygwin fork failure...

[kcrisman]

comment:15

In other words, we should go for maximal Cygwin packages (more like Sage-on-Gentoo) and not minimal (like the usual Sage Linux Distro)?

[kcrisman]

comment:16

I don't know why it only hit at beta1, not beta0, but rebasing has kind of destroyed my cygwin install, as now even bash breaks with fork errors at libiconv*.dll. My only guess it could be several instances of dlls with the same name...

This could also be the problem I start encountering after a while...

[dimpase]

comment:17

Replying to @kcrisman:

In other words, we should go for maximal Cygwin packages (more like Sage-on-Gentoo) and not minimal (like the usual Sage Linux Distro)?

this would be the most obvious solution to this, although not the only one possible; and actually it's not quite a solution, as Sage still has to include some dlls which can potentially conflict with ones from Cygwin, e.g. Singular and ECL are available as Cygwin packages, and there could (potentially) be naming clashes.

Another way is to take more care with PATH contents, although this would require a better understanding of the issue, and in particular how rebaseall affects this (and whether rebaseall can handle this). There is a certain database of Cygwin dlls maintained by rebase/rebaseall (as I gather from e.g. here), and we need to understand what is stored there etc. Our situation reminds me of the one discussed here, but I don't understand the details. Note that there are lots of dlls that are produced on the fly when building Python/Cython extensions, and these can potentially create dll name conflicts.

We can also considering a special dll naming scheme. E.g. we can give all the dlls produced by Sage a certain prefix, e.g. "sagecyg", just as Cygwin's dlls all get prefix "cyg" (much more work, but it looks like a really watertight way to deal with this).

The biggest problem now seems that we don't (at least I don't) have a good understanding how Windows loader, Cygwin rebase package (and that mysterios to me database), and Cygwin itself work together with dlls and where exactly the name clashes problem can hit.

A related unanswered question is whether the 32-bit emulation on 64-bit Windows systems in the only environment in which Sage can be made to reliably work with the present Cygwin. (Leaving ancient Windows versions like XP alone).
My understanding is that in this case the address space available for Cygwin dlls is much bigger than on "proper" 32-bit systems, and this is very important for Sage, which has a huge requirement for such a space, if a static non-overlapping memory layout for dlls is required for proper functioning.

[jpflori]

comment:18

Replying to @dimpase:

I'd like to reopen this ticket. The same issue as here applies to the rest of Sage packages that create dlls with names clashing with dlls from Cygwin packages (or perhaps other ones). Creating a dll which has the same name as a Cygwin dll is asking for big trouble.
You might read on for details here in this long thread.

In a nutshell, Windows is so dump that an application request to load a dll with a certain name is handled by searching certain sub-directories, in certain order. (Some of these subdirectories are actually specifies in Windows registry). In some cases the order of sub-directores in the PATH matters. I.e. when you link to a dlls, no full path information on the dll is stored.
All you have is the PATH, registry entries, and few other things like current directory, to play with.

I'm aware of this problem, but I think you're being a little dramatic here.
In some cases it is indeed really tricky, e.g. the ecl.dll of the Sage library resolved linking to the ecl library (whose dll name was ecl.dll before some ticket here) to ... itself.
That's indeed really dumb and in such a case I don't think tweaking PATH can change anything because the current directory will always come first.

And I think that on linux we still have libraries without RPATH so that LD_LIBRARY_PATH is used and that's not that worse than the situation on windows.

[jpflori]

comment:19

Replying to @kcrisman:

I don't know why it only hit at beta1, not beta0, but rebasing has kind of destroyed my cygwin install, as now even bash breaks with fork errors at libiconv*.dll. My only guess it could be several instances of dlls with the same name...

This could also be the problem I start encountering after a while...

Even on my seemingly lucky install, when I do install and reinstall too much things reabsing can go crazy and everything get useless and mined by fork errors, but I really fear its only a name clash issue, that just the forking hell which is clearly described on Cygwin pages.

[jpflori]

comment:20

Replying to @dimpase:

Replying to @kcrisman:

In other words, we should go for maximal Cygwin packages (more like Sage-on-Gentoo) and not minimal (like the usual Sage Linux Distro)?

this would be the most obvious solution to this, although not the only one possible; and actually it's not quite a solution, as Sage still has to include some dlls which can potentially conflict with ones from Cygwin, e.g. Singular and ECL are available as Cygwin packages, and there could (potentially) be naming clashes.

I don't think name clash is the main issue.

Another way is to take more care with PATH contents, although this would require a better understanding of the issue, and in particular how rebaseall affects this (and whether rebaseall can handle this). There is a certain database of Cygwin dlls maintained by rebase/rebaseall (as I gather from e.g. here), and we need to understand what is stored there etc. Our situation reminds me of the one discussed here, but I don't understand the details. Note that there are lots of dlls that are produced on the fly when building Python/Cython extensions, and these can potentially create dll name conflicts.

Our PATH looks pretty sane now, we look first in the Sage directories where the libraries we linked to are installed.

I think the problem of forking, address clash issues, arises with or without name clashes.
The last one I got where caused by libraries with so creepy names (random numbers and strange case) that I doubt we were colliding with the Cygwin namespace.
It was rather that the addressing sapce was full and the hackish rebasing process failed.

I also encountered such situations when I was tweaking max heap memory with peflags to let PARI's doctests pass.
If I allocated too much memory to the python process, then loading Sage failed with forking errors, and putting it back down to the default value let Sage start again (and the doctests fail :)) without any other intervention.

We can also considering a special dll naming scheme. E.g. we can give all the dlls produced by Sage a certain prefix, e.g. "sagecyg", just as Cygwin's dlls all get prefix "cyg" (much more work, but it looks like a really watertight way to deal with this).

The biggest problem now seems that we don't (at least I don't) have a good understanding how Windows loader, Cygwin rebase package (and that mysterios to me database), and Cygwin itself work together with dlls and where exactly the name clashes problem can hit.

Yep, that would make things easier hopefully.

A related unanswered question is whether the 32-bit emulation on 64-bit Windows systems in the only environment in which Sage can be made to reliably work with the present Cygwin. (Leaving ancient Windows versions like XP alone).

I don't really get what you're thinking about.
You mean running a 32 bits Windows?
If you have a virtual machine in mind then a virtual Linux is the way to go.

My understanding is that in this case the address space available for Cygwin dlls is much bigger than on "proper" 32-bit systems, and this is very important for Sage, which has a huge requirement for such a space, if a static non-overlapping memory layout for dlls is required for proper functioning.

[dimpase]

comment:21

Replying to @jpflori:

Replying to @dimpase:

Replying to @kcrisman:

In other words, we should go for maximal Cygwin packages (more like Sage-on-Gentoo) and not minimal (like the usual Sage Linux Distro)?

this would be the most obvious solution to this, although not the only one possible; and actually it's not quite a solution, as Sage still has to include some dlls which can potentially conflict with ones from Cygwin, e.g. Singular and ECL are available as Cygwin packages, and there could (potentially) be naming clashes.

I don't think name clash is the main issue.

An alternative theory is that my Cygwin install got nuked as I did not remove the previously built version of Sage, and the address space for the Cygwin dlls was not able to accommodate that much (my Cygwin install is almost full, I think).

So I have removed /etc/rebase* database files and the previous Sage installs, and after re-running cygwin setup I was able to run rebaseall without errors.
Now buiding the latest beta, will see how far I get this time.