packaging 1.0.0-rc3 (was: packaging 1.0.0-rc2)

[slayoo]

@gnudatalanguage/gdlteam help needed!

Here's how the current situation with GDL packages looks like:

• Debian (1.0.0-rc.1, Dec 2019)

• Ubuntu (0.9.9, Nov 2018)

• Fedora (0.9.9, Nov 2018)

• FreeBSD (0.9.9, Nov 2018)

• Aur (0.9.9, Nov 2018)

• Homebrew (0.9.7, Jan 2017)

• Gentoo (0.9.6, Dec 2015)

We have 1.0.0-rc.2 since March 2020.

Help or even hints how to speed the process up very welcome!

[GillesDuvert]

@slayoo I am in the dark, too.
This looks more like a communication/advertizing problem than genuine build difficulties, no? Apart on MacOS?

[thierry-FreeBSD]

On FreeBSD, issue #757 is blocking (I do not want to switch to Gcc).
I have not yet found a fix for it.

[slayoo]

@thierry-FreeBSD thanks, perhaps it would be a good idea then to temporarily skip compiling Python support?

[thierry-FreeBSD]

Yes, that's right: when Python is disabled, everything is fine: all tests in test_suite.pro pass, and no error is detected during some sessions.

[slayoo]

Thanks! Note that, despite the name, test_suite.pro does not check much besides very basic syntax elements, please run the full set of tests with "make check" to test the build

[GillesDuvert]

see comments about PR #763 in #757. Hope it helps.

[thierry-FreeBSD]

Thanks! Note that, despite the name, test_suite.pro does not check much besides very basic syntax elements, please run the full set of tests with "make check" to test the build

Hmm, not bad but

98% tests passed, 5 tests failed out of 232

Total Test time (real) = 130.29 sec

The following tests FAILED:
        107 - test_file_test.pro (Failed)
        110 - test_fix.pro (Failed)
        145 - test_memory.pro (Failed)
        197 - test_simplex.pro (Failed)
        223 - test_wait.pro (Failed)

Now I have to check these 5 failures!

[opoplawski]

Fedora is running into #677

[alaingdl]

Sorry, a stupid question ! Why not directly a 1.0.0 version. I do consider most of the mandatory issues have been solved, and moving to 1.0.0 should ensure some reaction from the distros.
Next version will be later 1.0.1 or 1.1 with READS for complex, revisited Widgets, ...

[slayoo]

I do vote for 1.0.0

[slayoo]

Thank you @olebole !
1.0.0-rc2 in Debian!
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961037

[slayoo]

We have also a pull request with an update to Macports: macports/macports-ports#7091

[olebole]

Currently, my builds are running on Debian experimental; see here for the results. The builds run the tests, but do not fail if the tests don't succeed. I will report back on failures.

One problem here is that it is for us packagers very difficult to see and to follow, which failures are acceptable somehow, and which are critical. It would be great if the tests where the developers expect them to fail would be marked somehow (if this is possible with the current test framework), or put into a separate test suite.

[GillesDuvert]

I'm lost.
That was the idea between the release candidates, no, that we have as many rc as necessary to get GDL OK on all platforms?
IMHO having GDL updated for already installed GDLs (I install dozen of updates every week with my Mageia) is more important than the name of the version.
If the packagers update GDL as long as they can build it, then there is no problem, the OSes/platforms where it can be updated will be updated. Some will be very outdated, and too bad but we'll never be able to solve all the problems ourselves, and hopefully somebody will volunteer to do the job for a specific platform, see what happened with Windows.
As we have a continuous integration on windows and good natured intel/amd 32/64 bits little endian architecture its no wonder some distros are more sucessful than others... Having continuous integration on big-endian machines and bsd would be a must.

[olebole]

Here is a table of the platforms that RC2 was attempted to build so far, together with the results. I will update this when new builds happen.

Architecture	Failures
amd64	test_fft_leak (#147), test_file_test (#534), test_make_dll, test_simplex
arm64	test_byte_conversion, test_bytscl, test_fft_leak (#147), test_file_test (#534), test_fix, test_formats (#144), test_make_dll, test_rounding, test_simplex
armel	build failure (#677)
armhf	build failure (#677)
i386	test_byte_conversion, test_fft_leak (#147), test_file_test (#534), test_fix, test_formats (#144), test_l64, test_make_dll
mips64el	test_bytscl, test_fft_leak (#147), test_file_test (#534), test_finite, test_fix, test_formats (#144), test_make_dll, test_simplex
mipsel	test_bytscl, test_fft_leak (#147), test_file_test (#534), test_finite, test_fix, test_formats (#144), test_l64, test_make_dll, test_simplex
powerpc	test_deriv, test_fft_leak (#147), test_file_test (#534), test_fix, test_formats (#144), test_l64, test_make_dll, test_simplex, test_window_background
ppc64el	test_byte_conversion, test_bytscl, test_fft_leak (#147), test_file_test (#534), test_finite, test_fix, test_formats (#144), test_make_dll, test_rounding, test_simplex
alpha	test_container, test_fft_leak (#147), test_file_test (#534), test_fix, test_formats (#144), test_image_statistics, test_make_dll, test_n_tags, test_rounding, test_simplex
riscv64	test_byte_conversion, test_bytscl, test_fft_leak (#147), test_file_test (#534), test_finite, test_fix, test_formats (#144), test_make_dll, test_rounding, test_simplex

I marked the failures that are disabled in Travis as italic and those which don't have an open issue bold, for the other I mentioned the corresponding issue. The architecture name links to the full build log. Note that some tests were switched off since they take too long on slow processors.

[olebole]

@GillesDuvert

If the packagers update GDL as long as they can build it, then there is no problem, the OSes/platforms where it can be updated will be updated.

We have CI at least on some platforms. When the tests fail on any of them, the new version will not migrate on all patforms. Therefore it is essential for me to have the tests marked that may fail without being release critical.

[slayoo]

I'm lost.
That was the idea between the release candidates, no, that we have as many rc as necessary to get GDL OK on all platforms?

[slayoo]

Apparently, the "-rc" suffix seems to be interpreted as a signal not to try it

[GillesDuvert]

@olebole this is immensely useful!

[GillesDuvert]

@gnudatalanguage/gdlmaintainers

• test_fft_leak: we probably cannot do a thing, and who cares?

• test_file_test if I'm not mistaken fails on subtle architecture dependent details and should not be used, even if it is a useful test.

• test_simplex, on the contrary should work, even if it is such a rare thing. Probably the simplex, hence the use of glpk, should be deactivated on such platforms. Until cured of course.

• test_formats is bound to fail on non-64 bit machines or big-endian ones, it may even fail with IDL itself. Of course it would be important to have identical formats between platforms, but this is far from a priority.

[GillesDuvert]

@olebole how is it that compiling problem #677 appears on armel and not on arm64? different gcc versions?

[opoplawski]

Apparently, the "-rc" suffix seems to be interpreted as a signal not to try it

Not as far as Fedora is concerned. I routinely attempt to build "rc"s and even git master. But it ain't building on Fedora and so won't be updated there until it does whether it's called 1.0.0 or not.

[GillesDuvert]

@gnudatalanguage/gdlteam, @olebole (an other dears packagers) :
#749 questioned me, so I installed the Ubuntu 20.04 on a VM and installed gnudatalanguage. Apparently Ubuntu is, as far as gdl is concerned, a copy of Debian.
I was able to reproduce the problem encountered by @mvandam .
I see 3 problems:

• gdl is linked with wxwidgets but the installer does not install the plplot-wxwidgets driver. It must.
• the installed version has an issue with the plplot driver that I've patched on github last year (28 Aug 2019), and it has not yet percolated to Ubuntu 20.04 ? (20.04 is, what?, 2 month old ? @slayoo what should we do to have our patches put in updates of distros? We should have updated the 0.9.9 then, i.e., update the last version quite often)
• finally and probably worse, the installer does not install all the ancillary files, that should go in share/gnudatalanguage , especially those in the 'resource' directory, but also the testsuite. We should insure these go with the installers, apt get, urpmi or whatsit. Or we can check that by a procedure at the first use of gdl and upload all these files. If we do not have that, no wonder we'll continue to have those frustrating negative reports...

@gnudatalanguage/gdlteam I suggest to remove the 5 or so tests that cannot possibly pass:
test_fft_leak (all), test_file_test(all), test_formats(32bits machines or strange architectures), test_wait...
then add a specific test to force a plot in a widget (? not sure of how to do that), another to insure that the low and high-resolution maps are present in 'resource' and probably also enforce the presence of the testsuite/demo_*** procedures...

[slayoo]

@slayoo what should we do to have our patches put in updates of distros? We should have updated the 0.9.9 then, i.e., update the last version quite often)

Well, I guess it was a rhetorical question :) Yes, the only way to go seems to release more often...

[slayoo]

On FreeBSD, issue #757 is blocking (I do not want to switch to Gcc).
I have not yet found a fix for it.

@thierry-FreeBSD Thierry, this is fixed now

[olebole]

@GillesDuvert

Generally, the best way to report packaging related problems is to open a bug in the distribution (Debian or Ubuntu). This makes it sure that it doesn't get lost.

@gnudatalanguage/gdlteam, @olebole (an other dears packagers) :
#749 questioned me, so I installed the Ubuntu 20.04 on a VM and installed gnudatalanguage. Apparently Ubuntu is, as far as gdl is concerned, a copy of Debian.
I was able to reproduce the problem encountered by @mvandam .
I see 3 problems:

• gdl is linked with wxwidgets but the installer does not install the plplot-wxwidgets driver. It must.

Is the driver really allways needed? Also for tasks without graphics? I am asking since we distinguish between an "absolute" and a "strong, but not absolute" dependency. And what is the relation to the plplot-xwin driver? During the build, I only install the latter one, and the tests pass. And the xwin driver is also a "strong, but not absolute" dependency (aka "Recommends") of the GDL package.

• the installed version has an issue with the plplot driver that I've patched on github last year (28 Aug 2019), and it has not yet percolated to Ubuntu 20.04 ? (20.04 is, what?, 2 month old ? @slayoo what should we do to have our patches put in updates of distros? We should have updated the 0.9.9 then, i.e., update the last version quite often)

The Ubuntu version of GDL is 0.9.9; I usually update it only when a new GDL version is out (unless there is a bug reported in the Ubuntu or Debian bugtracker). So, Github commits normally don't automatically migrate to Ubuntu.

Ubuntu freezes its imort from Debian ~2 months before the release (February for 20.04). Updating the published version of Ubuntu is rather difficult and restricted, as they don't have much manpower to review changes. So, we will have the 1.0 version probably in Ubuntu 20.10.

• finally and probably worse, the installer does not install all the ancillary files, that should go in share/gnudatalanguage , especially those in the 'resource' directory, but also the testsuite. We should insure these go with the installers, apt get, urpmi or whatsit. Or we can check that by a procedure at the first use of gdl and upload all these files. If we do not have that, no wonder we'll continue to have those frustrating negative reports...

For 0.9.9, is was still unclear what license these files have; so I couldn't include them in the package. For 1.0RC, they are included, as one can see in the file list.

[alaingdl]

Thanks @olebole

• this is a serious philosophical-like issue with this plplot-xwplplot-xwin problem. If we see the packaging as an utility for end-users, yes it is just mandatory. Easy access to graphics is a strong feature of GDL (more important than fast computation !). For computing nodes on HPC, for sure we will compile GDL without X11 ! But we are suppose to do it by our-self. And we received that much questions on that problem then maybe it is time to change our mind ?

• I would say that 99% of the *.pro files in the testsuite/ are under GNU GPL. Few files are generated by our codes (xdr, txt, h5 ...). I do not remember the origin of Saturn.jpg but it is not a problem.

[GillesDuvert]

We should just keep it simple: everybody that installs GDL with apt_get, brew, urmpi etc ... needs nothing else than the the full-featured thing, and that includes X11 (so plplot-xwin) and widgets (so, wxWidgets and plplot-wxwidgets), the ancillary files. Documentation if there was one, and test files lies traditionaly in an additional package.
So "Recommend" is what we need.
The Debian packaging in 3 parts, shared library, regular binary and python is very clever and we should also do it by default when compiling.

Non-X11, MPI features should be available by self-compilation only, since they are not user oriented. We should emphasize somewhere (in the README etc) that for "server" use, best results are attained with a customized and optimized recompilation. Some code parts are going to run faster with sse, avx... and also without DEBUG & asserts.

[GillesDuvert]

@olebole thank so much for the clarification. Too bad we did not report our bug solves earlier in the Debian bug tracker. Let's do it from now.

[thierry-FreeBSD]

For FreeBSD, plplot is not built with wxwidgets by default, but I filled a PR to change it:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247181

[slayoo]

For FreeBSD, plplot is not built with wxwidgets by default, but I filled a PR to change it:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247181

Thank you!

[slayoo]

@olebole thank so much for the clarification. Too bad we did not report our bug solves earlier in the Debian bug tracker. Let's do it from now.

Wouldn't "release often" be a better solution? In my understanding, including custom patches in Debian scripts is only meant for critical updates that cannot wait till next release of the software being packaged, right?

[olebole]

I would support that. For me, backporting patches to the Debian release is always some effort, and I would do that only for critical bugs where upstream is unable to create a new version by itself.

What would be the reason to not make a new (bugfix) version of GDL when a critical bug was fixed?

[maynardGK]

With regard to the original issue, that of a plplotwxwidget library, it should be noted that the normal wxwidget driver compiled in plplot uses the new methods, incompatible with the old, which is why we use -DWXWIDGETS=OLD (or something like that) to build a properly functioning widget interface. This will cause no problem for most platforms (where PLPLOT is used in only one application) and so a separate insertion would work ok, but the plplot team will not agree to regress for our sake. Probably, the mention will make the OLD option disappear faster from PLPLOT.

Greg

[opoplawski]

The plplot option is -DOLD_WXWIDGETS=ON. The Fedora plplot package is not compiled with that option, nor will it be. So, with that in mind - does it still make sense to ship a configuration that we expect to be broken?

[GillesDuvert]

I think GDL does not need the old driver since some time already. At least it's OK on my machine with out-of-the box plplotwxwidgets 5.14.0 (Mageia 7). Apparently, there are still problems on Windows of obscure origin but this is not the concern here.

[GillesDuvert]

BTW, if bugfixes builds are not too demanding for packagers, we should do that more often.

[slayoo]

@maynardGK @GillesDuvert @opoplawski
Could you clarify the OLD_WXWIDGETS issue? I'm completely lost here.
Is it related with this patch: https://github.com/gnudatalanguage/gdl/blob/master/scripts/patch/wxWidgets-3.0.2.patch ?

[GillesDuvert]

Sorry, I cannot say more: works for me with a plplot install from Mageia that for sure did not use 'old' driver.

[maynardGK]

OLD refers to the driver used in the plplot build which was radically changed about v 5.11; any effects would be found in plots to a widget window.

…

On Thu, Jun 18, 2020, 7:20 AM GillesDuvert ***@***.***> wrote: Sorry, I cannot say more: works for me with a wxWidgets install from Mageia that for sure did not use 'old' driver. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#761 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB22KZN3SJNFLN7WM4YDENDRXIPEBANCNFSM4NFEKEMA> .

[GillesDuvert]

yes, and windows build has 1 issue related to use of plplot wxidgets (see below).

probably related is the fact that the graphic window has no 'handle'.

[slayoo]

It's June 2020 and users do struggle with 2015-vintage GDL on Gentoo: https://forums.gentoo.org/viewtopic-t-1114956-highlight-gdl.html

We need to do something! :)

(for the record, there is a bug report for making a package update: https://bugs.gentoo.org/704026)

Any Gentoo users among @gnudatalanguage/gdlteam ?

[slayoo]

For the record: 1.0.0-rc.3 just released: https://github.com/gnudatalanguage/gdl/releases/tag/v1.0.0-rc.3

[olebole]

I just uploaded 1.0rc3 to Debian, and found almost all archs built (and tested by the new LIST).
Exceptions are PowerPC 64 bit little Endian (#530) and ARM 32 bit (#677). For PPC, I think we will find a solution or workaround, but the ARM failure worries me a bit. That is the Raspberry, which is in use by a number of people.
One remark, however: Some tests were skipped because they take too long on some archs (mainly MIPS); see also #557:

• test_angles
• test_bug_3104209
• test_bug_3104326
• test_bug_3147733
• test_bug_n000720
• test_idlneturl
• test_indgen
• test_matrix_multiply
• test_random

[GillesDuvert]

#530 we indeed have a solution.
#677 is identical to #734 --- linker does not appreciate duplicate symbols (that exist indeed) on these machines exclusively. Need a special liker switch? A solution would be to revert to previous state, on these architecture only, where all templates were put into a unique file. Very annoying as this quadruples the compilation time.
for long tests, this would need to check inside whhere time is spent.

[slayoo]

Thank you @olebole !

[slayoo]

It's June 2020 and users do struggle with January-2017 GDL 0.9.7 on Homebrew: https://stackoverflow.com/questions/62354528/gdl-not-found-after-installing-with-homebrew

Any Homebrew users among @gnudatalanguage/gdlteam ?

[GillesDuvert]

At least point them on this forum 😄 .
GDL means other things than our "GDL" ...

[GillesDuvert]

Today's #967 merging results in full-fledged GDL working nice on modern linux, OsX and windows.
This was obtained by installing on a fresh distribution using .ci/build_gdl.sh .
On 64 bit distributions only, I do not expect GDL pass all tests on alll platforms.
I hope this will help to get excellent distros for our coming v1.0

[alerque]

Hey guys, with all due respect you are doing this wrong.

Help or even hints how to speed the process up very welcome!

If you want to speed up packaging, cut a release!

The -rc label is a flat out road block to packaging for many/most distros. If you want adoption to move forward you need to tag a release. You can always keep fixing issues and putting out patch releases, but not having a release tag will hinder adoption. An Arch user brought this up on the mailing list but the official rule is to not package RC builds unless absolutely necessary.

As long as only an RC exists but no actual release the assumption is that there is some blocker and that it should not be distributed or used in production.

[slayoo]

Thank you @alerque. We indeed understood that introducing -rc's was a totally bad idea. A 1.0.0 will be released soon. Let's then close this issue.

[jkohnert]

Hey @alerque, that was me. 🙂 I prepared a PKGBUILD for 1.0.0-rc.3, but did not (yet) push it to the aur, since there's different opinions on whether in this particular case the (heavy)patching rule might come into play (we had issues in GraphicsMagick and Python3 support, I included slightly modified patches already in 1.0.0-rc.X). But good to know, there'll be a release soon.