Load PrimGrp, SmallGrp and TransGrp packages if available #1714

olexandr-konovalov · 2017-09-12T19:53:13Z

This is a modification of some changes attempted in #1650 (part of the plan attempted in #1650 in order to make the transition less disruptive).

If any of these packages is not available, GAP will read corresponding directory (prim, small or trans).

Dependency on TRANSGrp defined in the package required reading lib/galois.g* files in another place (in read4.g). In its turn, they require IsIrreducible which is defined in overload.g and delegates to IsIrreducibleRingElement, hence the latter was used.

Since the CI scripts are note yet using packages-master.tar.gz, this will now be tested for the setting in which none of the three new packages is available.

The next step would be to switch to packages-master.tar.gz and run CI tests with two of the packages - PrimGrp and TransGrp.

When this will be merged, we will be able to tell developers to get the new packages in question, either installing them individually or by removing pkg directory and calling make bootstrap-pkg-full to get the latest collection of GAP packages for the master branch.

After that, assuming that developers obtained relevant packages, we may proceed with next steps (could be done for one package at a time):

removing prim/small/trans directories as soon and taking out of the manuals those sections which have been incorporated into packages
replacing temporary code to ensure that GAP loads the packages by adding packages to GAPInfo.Dependencies.NeededOtherPackages.

fingolfin · 2017-09-12T20:32:23Z

hpcgap/lib/package.gi

@@ -1824,6 +1824,19 @@ BindGlobal( "BANNER", false );

    GAPInfo.delayedImplementationParts:= [];

+    # Ensure GAP loads PrimGrp, SmallGrp and TransGrp packages
+    # To be migrared to lib/system.g after the replacement of


Typo: migrated

fingolfin · 2017-09-12T20:33:45Z

hpcgap/lib/read.g

+## callable, even if the library is unavailable.
+if TestPackageAvailability("PrimGrp")=fail then
+  InstallGlobalFunction(PrimitiveGroupsAvailable,deg->false);
+fi;


Shouldn't there be something like this for smallgrp, too? @markuspf is there? (Not on my computer, so I can't check right now)

It's already there, above those lines:

if TestPackageAvailability("smallgrp")=fail then ReadSmall( "readsml.g","small groups" ); fi;

Its currently called SMALL_AVAILABLE and t akes as an argument a number and either returns a record or fail.

@alex-konovalov no, that's something else

@markuspf right, I know about that one, but I was thinking about something more user friendly and actually documented (SMALL_AVAILABLE is an internal function). I've submit this as gap-packages/smallgrp#20

fingolfin · 2017-09-12T20:34:26Z

hpcgap/lib/galois.gi

@@ -546,7 +546,7 @@ local f,n,i,sh,fu,ps,pps,ind,keineu,ba,bk,j,k,a,anz,pm,
  Info(InfoPerformance,2,"Using Transitive Groups Library");
  f:=arg[1];
  f:=f/LeadingCoefficient(f);
-  if not(IsIrreducible(f)) then
+  if not(IsIrreducibleRingElement(f)) then


Since you are changing this anyway, perhaps remove the superfluous parenthesis?

codecov · 2017-09-13T00:17:51Z

Codecov Report

Merging #1714 into master will decrease coverage by 14.15%.
The diff coverage is 80%.

@@             Coverage Diff             @@
##           master    #1714       +/-   ##
===========================================
- Coverage   63.43%   49.28%   -14.16%     
===========================================
  Files        1000      459      -541     
  Lines      322760   239320    -83440     
  Branches    13015    10724     -2291     
===========================================
- Hits       204757   117959    -86798     
- Misses     115361   118627     +3266     
- Partials     2642     2734       +92

Impacted Files	Coverage Δ
lib/galois.gi	`0% <0%> (-50.46%)`	⬇️
lib/package.gi	`35.96% <88.88%> (-0.6%)`	⬇️
lib/attr.gi	`0% <0%> (-100%)`	⬇️
lib/teachm2.g	`0% <0%> (-100%)`	⬇️
lib/session.g	`0% <0%> (-90.33%)`	⬇️
lib/contfrac.gi	`0% <0%> (-89.48%)`	⬇️
lib/wpobj.gi	`8.69% <0%> (-89.14%)`	⬇️
grp/imf.gi	`0% <0%> (-84.96%)`	⬇️
lib/proto.gi	`0% <0%> (-83.34%)`	⬇️
lib/ctblauto.gi	`0% <0%> (-82.41%)`	⬇️
... and 862 more

olexandr-konovalov · 2017-09-13T20:27:02Z

HPC-GAP fails with

Syntax warning: Unbound global variable in /home/travis/build/gap-system/gap/h\
pcgap/lib/galois.gi:503
 atomic TRANSREGION do
                     ^

I will investigate. Maybe the order in which files are read is different.

hulpke

Looks OK to me.

olexandr-konovalov · 2017-09-16T21:37:52Z

Hm, unlike prim, the hpcgap/trans directory has two files. I will look how to refactor this.

fingolfin · 2017-09-18T22:03:06Z

lib/read4.g

+
+if not IsBound(HPCGAP) then
+  # Galois group identification relies on the list of transitive groups
+  # (in HPC-GAP these file are read in read.g)


But why is this different between GAP and HPC-GAP? The comment should explain that (which is the difficult, non-trivial part -- in contrast figuring out where GAP vs. HPC-GAP loads something can be figured with a simple "grep" in a few seconds)

And independently of that: Why does regular GAP put this into read4.g? I'd expect this to be in read3.g and read5.g, for the .gd and .gi respectively.

@fingolfin: Because we want to please both installations with and without new packages. Starting GAP with -D option, one can see that the relative order in which these files are read is different. If there is no transgrp package, it is

#I READ_GAP_ROOT: loading 'trans/trans.gd' as GAP file #I READ_GAP_ROOT: loading 'trans/trans.grp' as GAP file #I READ_GAP_ROOT: loading 'trans/trans.gi' as GAP file #I READ_GAP_ROOT: loading 'lib/galois.gd' as GAP file #I READ_GAP_ROOT: loading 'lib/galois.gi' as GAP file

With it we have

#I READ_GAP_ROOT: loading 'lib/trans.gd' as GAP file #I READ_GAP_ROOT: loading 'lib/galois.gd' as GAP file #I Read( "/Users/alexk/GITREPS/gap/pkg/transgrp/lib/trans.grp" ) #I Read( "/Users/alexk/GITREPS/gap/pkg/transgrp/lib/trans.gi" ) #I READ_GAP_ROOT: loading 'lib/galois.gi' as GAP file

Furthermore, TRANSREGION is defined in hpcgap/trans/trans.gd - when the package is present, this will not be read, so I define it in hpcgap/lib/read.g.

I was testing this with packages and without package, and found this "equilibrium". It works now while we need to cater for both setups. As soon as we will go to the next stage and will remove trans and hpcgap/trans, I will look again and will try to converge handling of this in GAP and HPC-GAP.

@alex-konovalov I still don't understand. You described the effect, telling me that the load order is different between GAP and HPC-GAP -- and, well, of course it is, as e.g. the ReadLib calls in a different order. But that still doesn't give me a logical reason for why we do it like that.

My POV is this: Either there are strong reasons why it must be different, then we should document them in comments quite clearly. Or else, there are purely technical obstacles which we can and should remove, to make it possible to align this code.

The TRANSREGION sounds like the latter to me. Indeed, this is completely missing from the transgrp package -- so I guess before we can fix it, we must make transgrp HPC-GAP ready.

I'll look into that now.

@fingolfin it's not only that the load order is different between GAP and HPC-GAP - it is also different dependently on whether the data library is a package or directory.

I have been trying to resolve all problems without making any changes in transgrp, assuming that those can be made eventually - transgrp is not hosted under gap-packages and has a longer release cycle. If you could look at that, please do, of course.

But transgrp does not contain any HPC-GAP specific changes. So with this PR, support for transitive groups in HPC-GAP will be completely broken, no? And probably also for smallgrp, unless that contains HPC-GAP specific changes (I didn't see any on a quick glance just now... @markuspf ?)

So, for now it might be best to leave out the LoadPackage bits for HPC-GAP, until those packages contain the HPC-GAP specific changes, no?

fingolfin · 2017-09-19T10:05:11Z

hpcgap/lib/read.g

 ##
+## Assign TransitiveGroupsAvailable to a dummy function to make it
+## callable, even if the library is unavailable.
+if TestPackageAvailability("TransGrp")=fail then


can we please change this to "transgrp", to match the other places (same for PrimgGrp and SmallGrp below). Or, if you prefer, change all other places to use TransGrp. Just as long as it is consistent.

Agree. I think because LoadPackage and friends is case-insensitive, lowercase everywhere for all packages would be consistent (I do not use uppercase when requesting a package, and I think the same applies to most of users).

The canonical NAME of the package is the other thing. Here authors may combine upper and lower case as they wish. My preferred spelling would be mixed case, like PrimGrp, SmallGrp, TransGrp in this case.

If you'd like the packages to use that spelling for their canonical name, then feel free to submit pull requests to change their PackageInfo.g files accordingly -- right now, though, transgrp and smallgrp use an all-lowercase spelling, only PrimGrp does not.

If any of these packages is not available, GAP will read corresponding directory (`prim`, `small` or `trans`). Also replaced the call of `IsIrreducible` in `lib/galois.g` by `IsIrreducibleRingElement` to allow reading it earlier than `overload.g`, and moved definition of `TRANSREGION` out of `trans` directory to make it working both with `trans` and TransGrp package.

olexandr-konovalov · 2017-09-29T14:40:27Z

This has been rebased. Tests without and with prim and trans should be passing (hopefully).

lib/read.g and hpcgap/lib/read.g now have no diffs
definition of TRANSREGION moved out of trans directory to read4.g make it working both with trans and TransGrp package.
Both ReadLib("galois.gd"); and ReadLib("galois.gi"); are currently read in lib/read4.g, so there is no difference in the place where they are read in GAP and HPC-GAP.

As for whether ReadLib("galois.gd"); and ReadLib("galois.gi"); should be read in different places, it is tricky now to please both trans directory and transgrp package because of TRANSProperties.

It is defined in lib/grpperm.gd as a placeholder. Then it is unbound in trans/trans.gd to be redefined in trans/trans.grp or pkg/transgrp/lib/trans.grp as an actual function.

When trans directory is loaded, trans/trans.grp is read immediately after trans/trans.gd, so TRANSProperties does not "vanish". But when the package is loaded, if I move ReadLib("galois.gi"); to lib/read5.g, it is being read between trans/trans.gd and trans/trans.grp, so I have a warning about unbound TRANSProperties.

olexandr-konovalov · 2017-09-29T14:48:17Z

testpackages in Travis fails because bin/BuildPackages.sh fails to build NormalizInterface-1.0.0. We did not see on travis before since we were using packages from GAP 4.8.8 distribution, while this PR tries to use more recent packages archive.

I can update merged packages archive to rollback to the previous NormalizInterface or if @sebasguts will be able to investigate, wait for the fix.

olexandr-konovalov · 2017-10-02T12:26:19Z

I will now rollback to the stable version of NormalizInterface while @sebasguts will prepare a new one. This should make tests pass.

olexandr-konovalov · 2017-10-02T23:24:49Z

Tests are now passing. But I've downloaded and installed smallgrp package, and in HPC-GAP I have this:

testing: /Users/alexk/GITREPS/gap/tst/testinstall/hpc/task.tst
########> Diff in /Users/alexk/GITREPS/gap/tst/testinstall/hpc/task.tst:55
# Input is:
CallAsTask(SmallGroup,256,1);
# Expected output:
<pc group of size 256 with 8 generators>
# But found:
fail
########

Will investigate tomorrow.

olexandr-konovalov · 2017-10-05T06:39:48Z

The third package - SmallGrp - has now been picked up for the redistribution, so this PR is now fully tested and we can merge it now.

Next step would be to instruct everyone to download and install the three new packages - PrimGrp, TransGrp, SmallGrp (individually or via make bootstrap-pkg-full). In the meantime, I will rebase and update #1650 which will remove prim, trans and small directories and clean up documentation. I suggest to ask everyone to be ready for #1650 by November 1st.

After that, when #1650 will be merged, GAP will not start without the three packages, so everyone who will not switch by that time, would have to download and install them.

PR gap-system#1714 added some code needed to load either a new package for a data library, if present, or load the existing prim, small or trans directory otherwise. This code is not needed any more, since we remove prim, trans, and small directories. Also make PrimGrp, SmallGrp and TransGrp packages needed to run GAP. This ensures that a proper error message will be displayed in case any of these packages is missing.

PR #1714 added some code needed to load either a new package for a data library, if present, or load the existing prim, small or trans directory otherwise. This code is not needed any more, since we remove prim, trans, and small directories. Also make PrimGrp, SmallGrp and TransGrp packages needed to run GAP. This ensures that a proper error message will be displayed in case any of these packages is missing.

olexandr-konovalov requested review from fingolfin and hulpke September 12, 2017 19:57

fingolfin reviewed Sep 12, 2017

View reviewed changes

hulpke approved these changes Sep 16, 2017

View reviewed changes

olexandr-konovalov force-pushed the swap-libs-pkgs branch from 63b4a5a to cd19dd6 Compare September 16, 2017 21:33

fingolfin reviewed Sep 18, 2017

View reviewed changes

fingolfin reviewed Sep 19, 2017

View reviewed changes

fingolfin mentioned this pull request Sep 20, 2017

Make Revisions obsolete, and sync hpcgap/lib/init.g #1734

Merged

This was referenced Sep 21, 2017

Setup daily docker builds of the GAP repository, for use in package's Travis tests #1732

Closed

add two tests for trivial cases which cause errors when nilmat is loaded #632

Merged

olexandr-konovalov force-pushed the swap-libs-pkgs branch 2 times, most recently from 44cc775 to 837d829 Compare September 29, 2017 12:27

Alexander Konovalov added 2 commits September 29, 2017 15:11

Use package archives for the master branch for bootstrapping

3af6942

olexandr-konovalov force-pushed the swap-libs-pkgs branch from 837d829 to 3af6942 Compare September 29, 2017 14:18

olexandr-konovalov mentioned this pull request Oct 3, 2017

Removed SmallGroup call from HPC-GAP test #1763

Merged

olexandr-konovalov mentioned this pull request Oct 5, 2017

Removing prim, small and trans directories #1650

Merged

stevelinton merged commit 7662892 into gap-system:master Oct 6, 2017

olexandr-konovalov deleted the swap-libs-pkgs branch October 17, 2017 22:16

fingolfin added release notes: added PRs introducing changes that have since been mentioned in the release notes topic: library topic: packages issues or PRs related to package handling, or specific to a package (for packages w/o issue tracker) labels Dec 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load PrimGrp, SmallGrp and TransGrp packages if available #1714

Load PrimGrp, SmallGrp and TransGrp packages if available #1714

olexandr-konovalov commented Sep 12, 2017 •

edited

Loading

fingolfin Sep 12, 2017

fingolfin Sep 12, 2017

olexandr-konovalov Sep 12, 2017

markuspf Sep 12, 2017

fingolfin Sep 13, 2017

fingolfin Sep 12, 2017

codecov bot commented Sep 13, 2017 •

edited

Loading

olexandr-konovalov commented Sep 13, 2017

hulpke left a comment

olexandr-konovalov commented Sep 16, 2017

fingolfin Sep 18, 2017

olexandr-konovalov Sep 18, 2017

fingolfin Sep 19, 2017

olexandr-konovalov Sep 19, 2017

fingolfin Sep 19, 2017

fingolfin Sep 19, 2017

olexandr-konovalov Sep 19, 2017

fingolfin Sep 19, 2017

olexandr-konovalov commented Sep 29, 2017

olexandr-konovalov commented Sep 29, 2017

olexandr-konovalov commented Oct 2, 2017

olexandr-konovalov commented Oct 2, 2017

olexandr-konovalov commented Oct 5, 2017

Load PrimGrp, SmallGrp and TransGrp packages if available #1714

Load PrimGrp, SmallGrp and TransGrp packages if available #1714

Conversation

olexandr-konovalov commented Sep 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 13, 2017 • edited Loading

Codecov Report

olexandr-konovalov commented Sep 13, 2017

hulpke left a comment

Choose a reason for hiding this comment

olexandr-konovalov commented Sep 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olexandr-konovalov commented Sep 29, 2017

olexandr-konovalov commented Sep 29, 2017

olexandr-konovalov commented Oct 2, 2017

olexandr-konovalov commented Oct 2, 2017

olexandr-konovalov commented Oct 5, 2017

olexandr-konovalov commented Sep 12, 2017 •

edited

Loading

codecov bot commented Sep 13, 2017 •

edited

Loading