Make Generic.MatSpace truly generic

[fingolfin]

Closes #974
Resolves #911

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.31%. Comparing base (b2544d9) to head (67bda8b).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1318      +/-   ##
==========================================
+ Coverage   87.28%   87.31%   +0.03%     
==========================================
  Files         116      116              
  Lines       29736    29892     +156     
==========================================
+ Hits        25954    26100     +146     
- Misses       3782     3792      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fingolfin]

Note: this is actually a bug fix, so even if we reject this PR here, this part of it should be merged... The bug is that by aliasing some entries, some code won't work right if it uses functions like add! on matrix entries... I found out because this PR now uses this constructor, so the issue became apparent; in master it lurks hidden.

[thofma]

Yes, the code in AA assumes that one cannot do add! on matrix entries per se. If we want to change this, this needs further discussion and maybe do it in a separate PR.

[thofma]

See also https://nemocas.github.io/Nemo.jl/stable/developer/topics/#Aliasing-rules.

[fingolfin]

So are you saying that code I found which failed when I aliased the zeros is wrong?

Otherwise, that link says that matrix elements may alias; but it does not prescribe whether e.g. zero_matrix may produce a matrix with aliased entries; or whether it perhaps must produce one with non-aliased entries?

For now I went with the latter because it fixed a bug, and it also preserves the current behavior, so is less likely to introduce regressions.

[thofma]

Yes, it is probably wrong. But I would have to see it.

[fingolfin]

I forgot which one it was, but basically any of the functions in src/Matrix.jl which invoke e.g. addeq! likely will not work right when there is aliasing, such as fflu!, used to implement det.

[fingolfin]

if I restore the code with fill then test/generic/MatrixAlgebra-test.jl:1187 fails:

   R, x = polynomial_ring(ZZ, "x")
   U, z = polynomial_ring(R, "z")
   T = matrix_space(R, 6, 6)

   M = T()
   for i = 1:3
      for j = 1:3
         M[i, j] = rand(R, -1:2, -10:10)
         M[i + 3, j + 3] = deepcopy(M[i, j])
      end
   end

   p1 = charpoly(U, M)

   for i = 1:10
      similarity!(M, rand(1:6), R(rand(R, 0:2, -3:3)))
   end

   p2 = charpoly(U, M)

   @test p1 == p2

So maybe only similarity! is broken?

[thofma]

Hm, maybe the same error as in #955 :) There Claus made things also correct according to the documentation, but it exposed some other bugs.

I will have a look

[thofma]

I wrote something about this proposed change in #1000

[fingolfin]

This is now ready to review.

[fingolfin]

of course here we also have aliased entries...

[fingolfin]

I've been wondering: right now, this PR is completely geared towards providing a unified MatrixSpace implementation for types returned by dense_matrix_type(R) for a given ring R. Which it does, and which is already simplifying things a lot for implementors.

But what if there are additional matrix types for the same ring R, e.g. sparse ones, or whatever else? Shouldn't these also have a matrix space parent? If yes, of course they are not worse off with this PR than without it. But we could extend things (in a future PR) to cover those types as well: if we made the matrix type a second type parameter for struct Generic.MatSpace, then we could handle those types as well. Thoughts?

[thofma]

I understand how this could be useful, but at the moment I do not see the end. For example, the sparse matrices are covered by https://github.com/thofma/Hecke.jl/blob/master/src/HeckeTypes.jl#L409. At the moment I do not see that we will have plenty of other matrix types, so I would put this on the back burner.

[fingolfin]

Yeah, it's not something I'd wish to tackle in this PR... and if we ever want it, it shouldn't be too hard to add.

Once the OscarCI woes are resolved (hoping PR #1345 will do that), I'll re-run CI here, and make sure all tests pass, and then I hope we can merge?

[thofma]

I still am a bit skeptical about those changes with the non-aliasing of the entries. I works around a real bug, by making the code slower and more inconsistent. Inconsistent, because we are adding more code that assumes that entries do not alias each other, while there is also a lot of code which does not assume this.

[fingolfin]

Re: unaliasing: so just to be clear, this is about restoring the behavior to how it was a few weeks ago: I changed it accidentally as a side effect of some other re-factoring.

If you think the accidental introducing of the aliasing I made is a desirable optimization, we can reintroduce it in a dedicated PR, together with similar changes to a bunch of other places where they could obviously be made. But it seems undesirable to have this kind of change as an accident in a commit / PR that does not even mention it (namely PR #1324). In particular this optimization is in 0.29.3 and 0.29.4 and in no other AA version.

[fingolfin]

I've removed the fill change / (un)aliasing change now. While I still think it was a mistake to introduce it "accidentally" in PR #1324 it seems equally problematic to remove it in this PR. Also doesn't seem worth our time to debate it overly here

[thofma]

OK. The downstream error is expected? If so, I will mark this as breaking

[fingolfin]

Better hold it for now, I may need to reconsider some things sigh

[fingolfin]

Hrm, the tests against Nemo release fail because it has silly tests of the form

   m = one(Generic.MatSpace{Nemo.fpFieldElem}(Z13, 2, 2))
   for n = (m, -m, m*m, m+m, 2m)
      @test n isa Generic.MatSpaceElem{Nemo.fpFieldElem}
   end

which now fail because because the "generic" matspace and the regular matspace are the same.

I would still argue this does not constitute a true breaking change -- arguably that test is the problem, no real code relies on this.

[lgoettgens]

I would argue that we should now export MatSpace both from Generic and AbstractAlgebra such that downstream one does not need to write AbstractAlgebra.Generic.MatSpace but just MatSpace

[lgoettgens]

I would argue that we should now export MatSpace both from Generic and AbstractAlgebra such that downstream one does not need to write AbstractAlgebra.Generic.MatSpace but just MatSpace

Nevermind, this needs a breaking release since one needs to merge the abstract and the concrete type for this.

[lgoettgens]

Hrm, the tests against Nemo release fail because it has silly tests of the form

   m = one(Generic.MatSpace{Nemo.fpFieldElem}(Z13, 2, 2))
   for n = (m, -m, m*m, m+m, 2m)
      @test n isa Generic.MatSpaceElem{Nemo.fpFieldElem}
   end

which now fail because because the "generic" matspace and the regular matspace are the same.

I would still argue this does not constitute a true breaking change -- arguably that test is the problem, no real code relies on this.

I think that changing this behaviour would lead to Nemocas/Nemo.jl#651 again. I didn't quite understand what'S going on there, but maybe @thofma remembers.

.

[fingolfin]

I don't think this would lead to Nemocas/Nemo.jl#651 again. That test there was about verifying that adding generic matrices again produces generic matrices. Removing the test won't change that. The problem with that test (and its many variants, all of which are removed by Nemocas/Nemo.jl#1738) is that it tries to instantiate the generic matrix via the generic matrix space, which now is gone.

We could change this to try to instantiate the generic matrices directly:

julia> m = Generic.MatSpaceElem{ZZRingElem}(ZZ, 2, 2)
[0   0]
[0   0]

julia> typeof(m)
AbstractAlgebra.Generic.MatSpaceElem{ZZRingElem}

Those generic matrices over e.g. ZZ or QQ of course then wouldn't have a valid parent type anymore, because the change in this PR ensure there is one matrix space per tuple (ring,n,m) (my proposal to parametrize the matrix space by the matrix type was rejected):

julia> typeof(parent(m))
AbstractAlgebra.Generic.MatSpace{ZZRingElem}

julia> typeof(zero(parent(m)))
ZZMatrix

Therefore I think such test would be somewhat besides the point -- generic matrices over ZZ, QQ, ... should simply not be used anymore. Those tests verified that arithmetic on generic matrices over rings were we have "better" "native" matrix types does not "accidentally" produce matrices of a different type. That is still the case:

julia> typeof(m + m)
AbstractAlgebra.Generic.MatSpaceElem{ZZRingElem}

Anyway, the test needs to be adjusted, either by removing it (as I do now) or by changing how the initial generic matrix is generated (as done above). What do you prefer, @thofma ?

[thofma]

Yes, the tests are obsolete or wrong with the changes here (which we want!).

[lgoettgens]

The failures in HeckeCI seem like something real. Or do we want to abandon the kwarg cached for these?

[fingolfin]

On the long run the cached kwarg for matrix_space could go away as it now returns singletons anyway.

But the PR here does not actually remove a cached kwarg from AbstractAlgebra.matrix_space. Rather that was removed in PR #1313 from March 2023. But this was "hidden" because the custom matrix spaces in Nemo all came with custom matrix_space methods so we didn't notice...

As a pragmatic solution, I'll add back the cached kwarg here. Then once the dust has settled (i.e. this PR merged and released, and likewise its Nemo counterpart) we could remove its use(s) in Hecke, and at some point remove it again in a breaking PR from AA (of course that breaking PR could be open right away, no need to delay it)

[fingolfin]

The HeckeCI changes are resolved. Shall we merge this before making 0.41.4 then?

@thofma @lgoettgens

[lgoettgens]

For future bisecting/CI reasons, it would be better to first delete/adapt the tests in Nemo, release a new Nemo, then merge this, release a new AA, and then merge the rest in Nemo.

But if this is too much overhead and you are sure that Nemocas/Nemo.jl#1738 can be merged immediately after releasing AA and fixes CI and everything, that's fine for me as well.

[lgoettgens]

For future bisecting/CI reasons, it would be better to first delete/adapt the tests in Nemo, release a new Nemo, then merge this, release a new AA, and then merge the rest in Nemo.

I created Nemocas/Nemo.jl#1748 to adapt the Nemo tests.

[lgoettgens]

The HeckeCI changes are resolved. Shall we merge this before making 0.41.4 then?

@thofma @lgoettgens

Everything is green now. No objections from my side