Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unique representation for homsets #14793

Closed
nthiery opened this issue Jun 21, 2013 · 64 comments
Closed

Unique representation for homsets #14793

nthiery opened this issue Jun 21, 2013 · 64 comments

Comments

@nthiery
Copy link
Contributor

nthiery commented Jun 21, 2013

The unique representation of homsets is taken care of by Hom. What's missing is:

  • Fast hash and comparison by id. This can be implemented by having Homset inherit from WithEqualityById
  • Pickling by construction, calling back Hom(domain, codomain, category)

CC: @sagetrac-sage-combinat @simon-king-jena @jpflori

Component: categories

Author: Simon King

Branch/Commit: e2d2f16

Reviewer: Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/14793

@nthiery nthiery added this to the sage-6.1 milestone Jun 21, 2013
@nthiery nthiery self-assigned this Jun 21, 2013
@nthiery
Copy link
Contributor Author

nthiery commented Jun 21, 2013

comment:1

To discuss: the exact semantic for homsets of non unique parents

@simon-king-jena
Copy link
Member

comment:2

Observation: If one uses WithEqualityById, many tests in sage.modular and sage.schemes fail with unprintable assertion errors.

We need to discuss whether we "simply" fix them, or better continue to compare homsets by equality (not identity) of domain and codomain.

In other words, we might end up with just providing a default pickling for homsets, which is currently missing.

@simon-king-jena
Copy link
Member

comment:3

When using WithEqualityByid and a default __reduce__ method, we'd get (sorted by "section"):

Schemes:

sage -t sage/schemes/elliptic_curves/padic_lseries.py  # 1 doctest failed
sage -t sage/schemes/elliptic_curves/ell_curve_isogeny.py  # 18 doctests failed
sage -t sage/schemes/elliptic_curves/ell_point.py  # 8 doctests failed
sage -t sage/schemes/elliptic_curves/ell_finite_field.py  # 1 doctest failed
sage -t sage/schemes/projective/projective_space.py  # 2 doctests failed
sage -t sage/schemes/elliptic_curves/isogeny_class.py  # 1 doctest failed
sage -t sage/schemes/affine/affine_space.py  # 1 doctest failed
sage -t sage/schemes/generic/homset.py  # 2 doctests failed

Modular:

sage -t sage/modular/local_comp/local_comp.py  # 5 doctests failed
sage -t sage/modular/abvar/abvar.py  # 2 doctests failed
sage -t sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t sage/modular/abvar/homspace.py  # 1 doctest failed
sage -t sage/modular/modsym/subspace.py  # 2 doctests failed
sage -t sage/modular/hecke/ambient_module.py  # 1 doctest failed
sage -t sage/modular/abvar/finite_subgroup.py  # 2 doctests failed
sage -t sage/modular/modform/cuspidal_submodule.py  # 1 doctest failed
sage -t sage/modular/abvar/homology.py  # 2 doctests failed
sage -t sage/modular/abvar/abvar_newform.py  # 1 doctest failed
sage -t sage/modular/abvar/cuspidal_subgroup.py  # 2 doctests failed
sage -t sage/modular/hecke/degenmap.py  # 1 doctest failed
sage -t sage/modular/modsym/element.py  # 1 doctest failed
sage -t sage/modular/hecke/homspace.py  # 1 doctest failed
sage -t sage/modular/abvar/lseries.py  # 2 doctests failed
sage -t sage/modular/hecke/morphism.py  # 1 doctest failed

Rings:

sage -t sage/rings/morphism.pyx  # 7 doctests failed
sage -t sage/rings/quotient_ring.py  # 1 doctest failed
sage -t sage/rings/finite_rings/homset.py  # 3 doctests failed
sage -t sage/rings/number_field/morphism.py  # 4 doctests failed
sage -t sage/rings/homset.py  # 2 doctests failed

Modules

sage -t sage/modules/fg_pid/fgp_morphism.py  # 4 doctests failed
sage -t sage/modules/quotient_module.py  # 2 doctests failed
sage -t sage/modules/vector_space_homspace.py  # 1 doctest failed

Others:

sage -t sage/doctest/forker.py  # 1 doctest failed
sage -t sage/doctest/control.py  # 1 doctest failed
sage -t sage/structure/parent.pyx  # 1 doctest failed
sage -t sage/categories/hecke_modules.py  # 1 doctest failed
sage -t sage/homology/simplicial_complex_homset.py  # 1 doctest failed

Getting failures in sage/doctest seems amazing.

What to do?

@simon-king-jena
Copy link
Member

comment:4

Just a crazy idea (brain storm):

We do not want WithEqualityById by default, because it makes no sense to have Homset be a unique parent, if domain and codomain are no unique parents.

However, if both domain and codomain are unique parents, it is a good idea to use WithEqualityById.

So, we could check whether both domain and codomain inherit from UniqueRepresentation. If they do, then they are unique parents, and then we could create a dynamic class out of WithEqualityById and the class of the homset, and overload the __class__ of the homset.

In that way, we would automatically have fast homset comparison and hash, for those parts of sage that use unique representation.

I am running tests with an according patch now...

@simon-king-jena
Copy link
Member

comment:5

With what I described in my previous post, I get these errors.

sage -t sage/schemes/elliptic_curves/padic_lseries.py  # 1 doctest failed
sage -t sage/schemes/elliptic_curves/ell_point.py  # 8 doctests failed
sage -t sage/schemes/generic/homset.py  # 1 doctest failed
sage -t sage/schemes/elliptic_curves/ell_curve_isogeny.py  # 8 doctests failed
sage -t sage/schemes/elliptic_curves/ell_finite_field.py  # 1 doctest failed
sage -t sage/schemes/elliptic_curves/isogeny_class.py  # 1 doctest failed
sage -t sage/schemes/projective/projective_space.py  # 2 doctests failed
sage -t sage/schemes/affine/affine_space.py  # 1 doctest failed

Modular:

sage -t sage/modular/hecke/morphism.py  # 1 doctest failed
sage -t sage/modular/modsym/element.py  # 1 doctest failed
sage -t sage/modules/quotient_module.py  # 2 doctests failed
sage -t sage/modular/abvar/lseries.py  # 2 doctests failed
sage -t sage/modular/abvar/homology.py  # 2 doctests failed
sage -t sage/modular/abvar/abvar_newform.py  # 1 doctest failed
sage -t sage/modular/abvar/cuspidal_subgroup.py  # 2 doctests failed
sage -t sage/modular/abvar/finite_subgroup.py  # 2 doctests failed
sage -t sage/modular/modform/cuspidal_submodule.py  # 1 doctest failed
sage -t sage/modular/abvar/abvar.py  # 2 doctests failed
sage -t sage/modular/local_comp/local_comp.py  # 5 doctests failed
sage -t sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t sage/modular/abvar/homspace.py  # 1 doctest failed
sage -t sage/modular/modsym/subspace.py  # 2 doctests failed
sage -t sage/modular/hecke/ambient_module.py  # 1 doctest failed
sage -t sage/modular/hecke/degenmap.py  # 1 doctest failed

and others:

sage -t sage/structure/parent.pyx  # 1 doctest failed
sage -t sage/rings/quotient_ring.py  # 1 doctest failed
sage -t sage/modules/fg_pid/fgp_morphism.py  # 3 doctests failed
sage -t sage/rings/morphism.pyx  # 3 doctests failed
sage -t sage/rings/homset.py  # 1 doctest failed

Are these less than before?

@simon-king-jena
Copy link
Member

comment:6

Let's look at the errors more closely:

sage -t sage/modular/hecke/morphism.py
**********************************************************************
File "sage/modular/hecke/morphism.py", line 110, in sage.modular.hecke.morphism.HeckeModuleMorphism_matrix.__init__
Failed example:
    t == loads(dumps(t))
Exception raised:
    Traceback (most recent call last):
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 486, in _run
        self.execute(example, compiled, test.globs)
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 845, in execute
        exec compiled in globs
      File "<doctest sage.modular.hecke.morphism.HeckeModuleMorphism_matrix.__init__[2]>", line 1, in <module>
        t == loads(dumps(t))
      File "sage_object.pyx", line 1236, in sage.structure.sage_object.loads (build/cythonized/sage/structure/sage_object.c:11044)
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/homset.py", line 291, in Hom
        cat_X = X.category()
      File "module.pyx", line 45, in sage.modules.module.Module_old.category (build/cythonized/sage/modules/module.c:1501)
      File "classcall_metaclass.pyx", line 330, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1224)
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/modules.py", line 108, in __classcall_private__
        result = super(Modules, cls).__classcall__(cls, base_ring)
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/category.py", line 468, in __classcall__
        return super(Category, cls).__classcall__(cls, *args, **options)
      File "cachefunc.pyx", line 992, in sage.misc.cachefunc.WeakCachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:5394)
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 447, in __classcall__
        instance = typecall(cls, *args, **options)
      File "classcall_metaclass.pyx", line 518, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:1586)
      File "/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/category_types.py", line 326, in __init__
        assert base in Rings(), "base must be a ring"
    AssertionError: <unprintable AssertionError object>
**********************************************************************
1 item had failures:
   1 of   4 in sage.modular.hecke.morphism.HeckeModuleMorphism_matrix.__init__
    [19 tests, 1 failure, 0.23 s]
----------------------------------------------------------------------
sage -t sage/modular/hecke/morphism.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 0.3 seconds
    cpu time: 0.2 seconds
    cumulative wall time: 0.2 seconds

and this actually looks like it is due to the __reduce__ method, not due to the WithEqualityById.

@simon-king-jena
Copy link
Member

comment:7

Interesting! This

            sage: M = ModularSymbols(6)
            sage: t = M.Hom(M)(matrix(QQ,3,3,srange(9)), name="spam"); t
            Hecke module morphism spam defined by the matrix
            [0 1 2]
            [3 4 5]
            [6 7 8]
            Domain: Modular Symbols space of dimension 3 for Gamma_0(6) of weight ...
            Codomain: Modular Symbols space of dimension 3 for Gamma_0(6) of weight ...
            sage: t == loads(dumps(t))
            True

works on the command line, but fails in a doctest. :-/

@simon-king-jena
Copy link
Member

comment:8

I think I've hit this one before when working on homsets. Perhaps I'll find my old solution somewhere.

@simon-king-jena
Copy link
Member

comment:9

No, I did not find the solution.

The problem seems to be that the modular symbols space M is pickled the OLD python way, hence, its __dict__ is pickled. When M is unpickled, some homspace accessible from an attribute of M is unpickled. Codomain and domain of this homspace happen to be M --- and if the Hom function is called with an uninitialised version of M, then stuff fails.

Solution: Provide a "proper" pickling for modular symbols.
Problem: I don't understand the modular symbols code.

@simon-king-jena
Copy link
Member

comment:10

If I recall correctly, people do explicitly not want to use a cache (such as: UniqueRepresentation) on modular symbols. They have some cache, but make a point of being able to disable it.

Here is a minimal example, that fails when using a proper __reduce__ method for homsets:

            sage: sage.modular.hecke.morphism.is_HeckeModuleMorphism_matrix(ModularSymbols(6).hecke_operator(7).matrix_form().hecke_module_morphism())
            True
            sage: M = ModularSymbols(6)
            sage: t = M.Hom(M)(matrix(QQ,3,3,srange(9)), name="spam"); t
            Hecke module morphism spam defined by the matrix
            [0 1 2]
            [3 4 5]
            [6 7 8]
            Domain: Modular Symbols space of dimension 3 for Gamma_0(6) of weight ...
            Codomain: Modular Symbols space of dimension 3 for Gamma_0(6) of weight ...
            sage: t == loads(dumps(t))
            True

The first line actually is important. With the first line, we get this error in the last line:

Traceback (most recent call last)
<ipython-input-11-a4f55210dece> in <module>()
----> 1 t == loads(dumps(t))

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/structure/sage_object.so in sage.structure.sage_object.loads (build/cythonized/sage/structure/sage_object.c:11044)()

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/homset.pyc in Hom(X, Y, category)
    289 
    290     # Determines the category
--> 291     cat_X = X.category()
    292     cat_Y = Y.category()
    293     if category is None:

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/modules/module.so in sage.modules.module.Module_old.category (build/cythonized/sage/modules/module.c:1501)()

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/misc/classcall_metaclass.so in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1224)()                                                                                                                                                                                 

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/modules.pyc in __classcall_private__(cls, base_ring, dispatch)
    106             if base_ring in _Fields:
    107                 return VectorSpaces(base_ring, check=False)
--> 108         result = super(Modules, cls).__classcall__(cls, base_ring)
    109         result._reduction[2]['dispatch'] = False
    110         return result

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/category.pyc in __classcall__(cls, *args, **options)
    466         if isinstance(cls, DynamicMetaclass):
    467             cls = cls.__base__
--> 468         return super(Category, cls).__classcall__(cls, *args, **options)
    469 
    470     def __init__(self, s=None):

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/misc/cachefunc.so in sage.misc.cachefunc.WeakCachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:5394)()

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/structure/unique_representation.pyc in __classcall__(cls, *args, **options)
    445 
    446         """
--> 447         instance = typecall(cls, *args, **options)
    448         assert isinstance( instance, cls )
    449         if instance.__class__.__reduce__ == CachedRepresentation.__reduce__:

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/misc/classcall_metaclass.so in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:1586)()                                                                                                                                                                                                    

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/category_types.pyc in __init__(self, base, name)
    324     def __init__(self, base, name=None):
    325         from sage.categories.rings import Rings
--> 326         assert base in Rings(), "base must be a ring"
    327         Category_over_base.__init__(self, base, name)
    328 

<type 'str'>: (<type 'exceptions.AttributeError'>, AttributeError('ModularSymbolsAmbient_wt2_g0_with_category' object has no attribute '_HeckeModule_generic__level',))

Hooray, we have a reproducible failure.

@simon-king-jena
Copy link
Member

comment:11

Better:

sage: sage.modular.hecke.morphism.is_HeckeModuleMorphism_matrix(ModularSymbols(6).hecke_operator(7).matrix_form().hecke_module_morphism())
True
sage: M = ModularSymbols(6)
sage: loads(dumps(M)) is M
Traceback (most recent call last)
<ipython-input-3-7b45a005bcab> in <module>()
----> 1 loads(dumps(M)) is M

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/structure/sage_object.so in sage.structure.sage_object.loads (build/cythonized/sage/structure/sage_object.c:11044)()

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/homset.pyc in Hom(X, Y, category)
    289 
    290     # Determines the category
--> 291     cat_X = X.category()
    292     cat_Y = Y.category()
    293     if category is None:

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/modules/module.so in sage.modules.module.Module_old.category (build/cythonized/sage/modules/module.c:1501)()

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/misc/classcall_metaclass.so in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1224)()                                                                                                                                                                                 

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/modules.pyc in __classcall_private__(cls, base_ring, dispatch)
    106             if base_ring in _Fields:
    107                 return VectorSpaces(base_ring, check=False)
--> 108         result = super(Modules, cls).__classcall__(cls, base_ring)
    109         result._reduction[2]['dispatch'] = False
    110         return result

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/category.pyc in __classcall__(cls, *args, **options)
    466         if isinstance(cls, DynamicMetaclass):
    467             cls = cls.__base__
--> 468         return super(Category, cls).__classcall__(cls, *args, **options)
    469 
    470     def __init__(self, s=None):

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/misc/cachefunc.so in sage.misc.cachefunc.WeakCachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:5394)()

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/structure/unique_representation.pyc in __classcall__(cls, *args, **options)
    445 
    446         """
--> 447         instance = typecall(cls, *args, **options)
    448         assert isinstance( instance, cls )
    449         if instance.__class__.__reduce__ == CachedRepresentation.__reduce__:

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/misc/classcall_metaclass.so in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:1586)()                                                                                                                                                                                                    

/home/simon/SAGE/prerelease/sage-5.10.rc1/local/lib/python2.7/site-packages/sage/categories/category_types.pyc in __init__(self, base, name)
    324     def __init__(self, base, name=None):
    325         from sage.categories.rings import Rings
--> 326         assert base in Rings(), "base must be a ring"
    327         Category_over_base.__init__(self, base, name)
    328 

<type 'str'>: (<type 'exceptions.AttributeError'>, AttributeError('ModularSymbolsAmbient_wt2_g0_with_category' object has no attribute '_HeckeModule_generic__level',))

So, this should be analysable.

@simon-king-jena
Copy link
Member

comment:12

Here is the problem and a potential solution.

While unpickling of the modular symbols M, we need to construct a homset with domain and codomain M. At this point, calling M.category() results in an error, since M.base() returns None and M.category() wants to return Modules(M.base()).

Hence, in the code of the Hom function from sage.categories.homset, the lines

            cat_X = X.category()
            cat_Y = Y.category()

crash.

The point is: When Hom is called in the process of unpickling a homset, then the correct category is supplied. Hence, the argument "category" of Hom is not None. And I think if in this situation X.category() raises an error then we can simply ignore the error and skip the consistency check cat_X.is_subcategory(category).

Hence, I'd do:

    if category is None:
        category = X.category()._meet_(Y.category())
        # Recurse to make sure that Hom(X, Y) and Hom(X, Y, category) are identical
        H = Hom(X, Y, category)
    else:
        # while unpickling, it is possible that the
        # categories of X and Y are not initialised
        # and a default category can't be determined.
        # Since cat_X/Y.is_subcategory(category) is just
        # a sanity check anyway, we will ignore an error
        # that is raised when determining cat_X/cat_Y
        if not isinstance(category, Category):
            raise TypeError, "Argument category (= %s) must be a category."%category
        try:
            cat_X = X.category()
        except BaseException:
            cat_X = category
        try:
            cat_Y = Y.category()
        except BaseException:
            cat_Y = category
        if not cat_X.is_subcategory(category):
            raise TypeError, "%s is not in %s"%(X, category)
        if not cat_Y.is_subcategory(category):
            raise TypeError, "%s is not in %s"%(Y, category)

        # Construct H
        try: # _Hom_ hook from the parent
            H = X._Hom_(Y, category)
...

Do you think this is a feasible idea?

@nbruin
Copy link
Contributor

nbruin commented Jun 22, 2013

comment:13

Replying to @simon-king-jena:

Here is the problem and a potential solution.

While unpickling of the modular symbols M, we need to construct a homset with domain and codomain M. At this point, calling M.category() results in an error, since M.base() returns None and M.category() wants to return Modules(M.base()).

Is None ever a valid value for M.base()? At this point, is there enough information available on M to derive what base() should return? In that case, I'd think the cleanest way would be to make base() a caching routine: return a stored value if available and otherwise derive the correct value, store it, and return that. Whenever someone asks for base they're probably not interested in an invalid value. Or is computing base possibly expensive and not really necessary for the unpickling?

[...]

Do you think this is a feasible idea?

If you can't easily fix the category determination then, yes. However, what you're proposing is a hack, so solving it properly should really be preferred. (Other people more knowledgeable on category stuff will probably have a more informed opinion than this generic remark)

@simon-king-jena
Copy link
Member

comment:14

Replying to @nbruin:

While unpickling of the modular symbols M, we need to construct a homset with domain and codomain M. At this point, calling M.category() results in an error, since M.base() returns None and M.category() wants to return Modules(M.base()).

Is None ever a valid value for M.base()?

I don't know if it is valid, but at least it is easily possible to get None:

sage: Parent().base() is None
True

At this point, is there enough information available on M to derive what base() should return?

I don't think so---unless modular symbols are always defined over the rationals. That's not my field of mathematical expertise.

In that case, I'd think the cleanest way would be to make base() a caching routine: return a stored value if available and otherwise derive the correct value, store it, and return that. Whenever someone asks for base they're probably not interested in an invalid value. Or is computing base possibly expensive and not really necessary for the unpickling?

base is usually set when you call Parent.__init__.

Anyway. If it is really the case that the base of modular symbols will always be the rational field, then it is fine.

Best regards,
Simon

@simon-king-jena
Copy link
Member

comment:15

Not good. The documentation states:

    - ``base_ring`` - the base ring. Defaults to `\QQ` if no character
      is given, or to the minimal extension of `\QQ` containing the
      values of the character.

I guess the proper solution would be to provide a __reduce__ method for modular symbols. Since modular symbols come in many different flavours, this might be a bit awkward.

@simon-king-jena
Copy link
Member

comment:16

Perhaps the following idea works and is clean.

What does Python currently do when unpickling a modular symbol? Well, it creates a new instance of the underlying class, and fills its __dict__.

I guess it is possible to compute the base ring from the contents of the __dict__ (need to find out how, but I am sure it is encoded there somewhere. Indeed, we have

sage: M = ModularSymbols(6)
sage: M._ModularSymbolsSpace__character.base_ring()
Rational Field
sage: M.base()
Rational Field

And that means, we could create an unpickling function doing the following, where dct is the __dict__ and cls is the class of the modular symbol to be unpickled:

    M = cls.__new__(cls)
    ParentWithAdditiveAbelianGens.__init__(M,base=dct['_ModularSymbolsSpace__character'].base_ring(), category=Modules(dct['_ModularSymbolsSpace__character'].base_ring()))
    M.__dict__ = copy(dct)
    return M

Well, I will try it later...

@nbruin
Copy link
Contributor

nbruin commented Jun 22, 2013

comment:17

Replying to @simon-king-jena:

An example in the documentation:

sage: G = DirichletGroup(13,GF(4,'a'));
sage: e = G.list()[2];
sage: M = ModularSymbols(e,4);
sage: M.base()
Finite Field in a of size 2^2
sage: M._ModularSymbolsSpace__character.base_ring()
Finite Field in a of size 2^2

The base ring can (in principle at least) be pretty much anything, and the way you're proposing to retrieve it seems to work OK (it seems the character is always there)

@simon-king-jena
Copy link
Member

comment:18

Replying to @nbruin:

The base ring can (in principle at least) be pretty much anything, and the way you're proposing to retrieve it seems to work OK (it seems the character is always there)

I'm afraid it does not work. The problem is: We need the homset to finish initialisation of the parent, and Python decides to try and create the homset rather early in the reconstruction of the parent. Hence, when we need the character or the base ring or any other information, it is simply not available yet.

I tried to provide a __reduce__ method for HeckeModule_generic, which solved the problem of pickling modular symbols. But then, a different problem arose: An infinite loop when pickling an abelian variety (to be precise: of ModularAbelianVariety_modsym). And it is not clear to me how to write a __reduce__ method for abelian varieties such that the loop does not occur.

Probable I should provide the code that I have so far. In about 24 hours, I guess...

And a different approach: We should try to understand why

sage: M = ModularSymbols(6)
sage: loads(dumps(M)) == M
True

works, while

sage: sage.modular.hecke.morphism.is_HeckeModuleMorphism_matrix(ModularSymbols(6).hecke_operator(7).matrix_form().hecke_module_morphism())
True
sage: M = ModularSymbols(6)
sage: loads(dumps(M)) == M
True

crashes.

@simon-king-jena
Copy link
Member

comment:19

This works:

sage: M = ModularSymbols(6)
sage: f = M.hecke_operator(7).matrix_form()
sage: loads(dumps(M)) == M
True

Continuing the example, this makes it fail:

sage: phi = f.hecke_module_morphism()
sage: loads(dumps(M)) == M
Traceback (most recent call last):
...
<type 'str'>: (<type 'exceptions.AttributeError'>, AttributeError('ModularSymbolsAmbient_wt2_g0_with_category' object has no attribute '_HeckeModule_generic__level',))

Continuing the example, this makes it work again:

sage: del M._HeckeModule_generic__hecke_algebra
sage: loads(dumps(M)) == M
True

Hence, I think the following happens.

  • s = dumps(M) pickles the content of M.__dict__. loads(s) then starts with unpickling the stored content of M.__dict__, after creating a would-be-copy N of M, which is not initialised at this point (this is what Python does in those cases). The unpickled content of M.dict would later be assigned to N.dict
  • The Hecke algebra of M is stored as an attribute of M. Hom(M,M) somehow becomes stored as an attribute of M.hecke_algebra() when calling .hecke_module_morphism(). Hence, when deleting the attribute containing M.hecke_algebra(), all is good.
  • When unpickling the content of M.__dict__, Python eventually tries to unpickle Hom(M,M), but replaces M with its would-be-copy N. But since N is not initialised, Hom(N,N) fails.

This suggests a way out: We could implement "proper" unpickling of M.hecke_algebra(), so that Hom(M,M) will not be called prematurely.

It might also be a good idea to use @cached_method when a method is cached. Namely, I often see stuff like

    def free_module(self):
        """
        Return the free module underlying this ambient Hecke module (the
        forgetful functor from Hecke modules to modules over the base ring)

        EXAMPLE::

            sage: ModularForms(59, 2).free_module()
            Vector space of dimension 6 over Rational Field
        """
        try:
            return self.__free_module
        except AttributeError:
            M = sage.modules.all.FreeModule(self.base_ring(), self.rank())
            self.__free_module = M
            return M

which uses __dict__ and hence will result in errors during unpickling.

@simon-king-jena
Copy link
Member

comment:20

Arrgh. The obvious approach

class HeckeAlgebra_base(sage.rings.commutative_algebra.CommutativeAlgebra):
...
    def __reduce__(self):
        return HeckeAlgebra, (self.__M,)

fails, because again for unpickling the Hecke algebra, one needs that M is sufficiently initialised, because otherwise HeckeAlgebra(M) fails.

@simon-king-jena
Copy link
Member

comment:21

It is a can of worms. The problem will always arise under the following conditions:

  • We have Homset.__reduce__ returning Hom, (self._domain, self._codomain, self.__category)
  • We have a parent X whose pickling is done via pickling __dict__ using the default Python way
  • Hom(X,X) (or any other homset involving X) is stored in X.__dict__.

In this situation, unpickling X means that we have an uninitialised copy Y of X and want to call Hom(Y,Y).

My previous suggestion was: "In Hom(X,Y,cat), if X.category() fails and cat is given, then trust that X will eventually become sufficiently initialised to see that X is in cat".

Perhaps a modified suggestion is better: "In Hom(X,Y,cat), if X.category() fails, then at least test isinstance(X, cat.parent_class)." This test is almost as good as X in cat, since the two tests are actually equivalent, unless a base ring is involved. IF a base ring is involved, then X is in cat up to a change of the base ring.

Do you have a better suggestion? I really would not like to implement __reduce__ methods everywhere in sage.schemes and sage.modular.

@simon-king-jena
Copy link
Member

comment:22

PS: With the idea exposed in the previous post, the following previously failing examples would work:

age: M = ModularSymbols(6)
sage: sage.modular.hecke.morphism.is_HeckeModuleMorphism_matrix(ModularSymbols(6).hecke_operator(7).matrix_form().hecke_module_morphism())
True
sage: loads(dumps(M)) == M
True
sage: A = J0(33)
sage: D = A.decomposition(); D
[
Simple abelian subvariety 11a(1,33) of dimension 1 of J0(33),
Simple abelian subvariety 11a(3,33) of dimension 1 of J0(33),
Simple abelian subvariety 33a(1,33) of dimension 1 of J0(33)
]
sage: loads(dumps(D))
[
Simple abelian subvariety 11a(1,33) of dimension 1 of J0(33),
Simple abelian subvariety 11a(3,33) of dimension 1 of J0(33),
Simple abelian subvariety 33a(1,33) of dimension 1 of J0(33)
]

@simon-king-jena
Copy link
Member

comment:23

Some thoughts:

  • Another plus of my suggestion is that it would encourage people to properly initialise the category (since otherwise X.class would not be a subclass of the category's parent class).

  • The negative of my suggestion: If one programs in Cython, then X.__class__ will not become a sub-class of the category's parent class, even if initialisation of the category was done properly.

  • The positive in the negative of my suggestion: If one programs in Cython, then the problem is not relevant anyway. The problem I'm addressing will only appear if one relies on Python's way of pickling, without implementing __reduce__.

@simon-king-jena
Copy link
Member

comment:24

I have attached an experimental patch that implements the new suggestion. With it, the only error in sage.modular is:

sage -t sage/modular/abvar/homspace.py  # 1 doctest failed

which is some automatic test suite.

The situation in sage.schemes is still not good, as we get:

sage -t sage/schemes/elliptic_curves/ell_curve_isogeny.py  # 8 doctests failed
sage -t sage/schemes/elliptic_curves/ell_point.py  # 8 doctests failed
sage -t sage/schemes/elliptic_curves/ell_finite_field.py  # 1 doctest failed
sage -t sage/schemes/elliptic_curves/isogeny_class.py  # 1 doctest failed
sage -t sage/schemes/projective/projective_space.py  # 2 doctests failed
sage -t sage/schemes/affine/affine_space.py  # 1 doctest failed
sage -t sage/schemes/generic/homset.py  # 1 doctest failed

If we are lucky, this boils down to an improper initialisation of the category of some schemes.

@simon-king-jena
Copy link
Member

comment:25

Indeed:

simon@linux-sqwp:~/SAGE/prerelease/sage-5.10.rc1/devel/sage> grep "def category" -R sage/schemes/
sage/schemes/generic/morphism.py:    def category(self):

Overloading CategoryObject.category smells like an improper use of the category framework in sage.schemes.

@simon-king-jena
Copy link
Member

comment:37

Travis, how difficult has it been to apply the old patch? Certainly there have been a bunch of conflicts.

@simon-king-jena
Copy link
Member

comment:38

For the record: I am happy with your changes, Travis. To be on the safe side, I am running tests on my laptop now, but after all you are the reviewer... And sorry that I forgot to add tests for some new methods.

@tscrim
Copy link
Collaborator

tscrim commented Dec 27, 2013

comment:39

Replying to @simon-king-jena:

Travis, how difficult has it been to apply the old patch? Certainly there have been a bunch of conflicts.

None; there were no conflicts. Although it might conflict with #10963..... gulp

Actually, I noticed that I missed an added __reduce__() in categories/homset.py -- it has a docstring but no doctests. Could you add one since I won't have access to a computer with Sage for another 6-8 hours (I will do it then if you haven't done it)? Thanks.

@simon-king-jena
Copy link
Member

comment:40

Replying to @tscrim:

Actually, I noticed that I missed an added __reduce__() in categories/homset.py -- it has a docstring but no doctests. Could you add one since I won't have access to a computer with Sage for another 6-8 hours (I will do it then if you haven't done it)? Thanks.

I see your post only now---and 8 hours are past :-)

Anyway, I am adding a test for the __reduce__ method now. We should then have another look at the "patch" and see what hasn't been tested.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 28, 2013

Changed commit from ea3ecfb to e2d2f16

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Dec 28, 2013

Branch pushed to git repo; I updated commit sha1. New commits:

e2d2f16Trac 14793: Add a missing doctest

@simon-king-jena
Copy link
Member

comment:42

Test added.

@simon-king-jena
Copy link
Member

comment:43

PS: I verified that the homsets in the test really use this and no other method for pickling.

@simon-king-jena
Copy link
Member

comment:44

Good news: #10963 merges cleanly.

@tscrim
Copy link
Collaborator

tscrim commented Dec 28, 2013

comment:45

I put sleep > sage last night, so :p and thanks for adding that. It's also good to know that there are no conflicts with #10963. I've gone through your changes, tests pass, and everything is documented. The TestSuite and things is #14279. What else is left here to review?

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 2, 2014

Changed commit from e2d2f16 to 17fe8ee

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 2, 2014

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

17fe8eeStarted transition.
0ae3289Initial attempt at _internal_coerce_map_from_() method.
45597e1Merge branch 'u/jpflori/ticket/14711' of trac.sagemath.org:sage into public/structure/unique_repr_homsets-14793
0af59eaReviewer changes. Mostly formatting.
37bf59eMerge branch 'develop' into ticket/14711, resolving conflicts with Trac 12217
ee30c20Address the "check" keyword of scheme morphisms by name, not position
d68c5dfMinor fixes, that became needed since #14711 was not merged quickly enough
c42b539Merge branch 'master' into ticket/14711
23f18f2Merge branch 'master' into ticket/14711
364b985Add warning to string repr of weakened maps. Implement copying for *all* kinds of maps.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 2, 2014

Branch pushed to git repo; I updated commit sha1. This was a forced push. Recent commits:

0af59ea Reviewer changes. Mostly formatting.
37bf59e Merge branch 'develop' into ticket/14711, resolving conflicts with Trac 12217
ee30c20 Address the "check" keyword of scheme morphisms by name, not position
d68c5df Minor fixes, that became needed since #14711 was not merged quickly enough
c42b539 Merge branch 'master' into ticket/14711

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 2, 2014

Changed commit from 17fe8ee to 0af59ea

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 2, 2014

Branch pushed to git repo; I updated commit sha1. This was a forced push. Recent commits:

e2d2f16 Trac 14793: Add a missing doctest
ea3ecfb Some review changes.
1411319 #14793: Use `reduce` for pickling homsets, and use WithEqualityById for them

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jan 2, 2014

Changed commit from 0af59ea to e2d2f16

@tscrim
Copy link
Collaborator

tscrim commented Jan 2, 2014

comment:49

I'm an idiot and merged the wrong branches...

@simon-king-jena
Copy link
Member

comment:50

Replying to @tscrim:

I'm an idiot and merged the wrong branches...

Doesn't this qualify as "changing history"? You bad boy ;-)

@tscrim
Copy link
Collaborator

tscrim commented Jan 2, 2014

comment:51

It's not changing history, just taking a different path :p.

@tscrim
Copy link
Collaborator

tscrim commented Jan 9, 2014

comment:52

I think we're good to go here unless there's something else you can see or think of?

@simon-king-jena
Copy link
Member

comment:53

Replying to @tscrim:

I think we're good to go here unless there's something else you can see or think of?

Just to make sure: We have three commits, namely one that corresponds to the original patch, your review changes, and then my commit adding one doctest?

For the record, I agree with your review changes. But after all, you are the reviewer, not I. If you think the code is fine and if all tests pass after merging the develop branch, I'd not oppose to let this be positively reviewed.

@tscrim
Copy link
Collaborator

tscrim commented Jan 9, 2014

comment:54

Replying to @simon-king-jena:

Just to make sure: We have three commits, namely one that corresponds to the original patch, your review changes, and then my commit adding one doctest?

Correct.

For the record, I agree with your review changes. But after all, you are the reviewer, not I. If you think the code is fine and if all tests pass after merging the develop branch, I'd not oppose to let this be positively reviewed.

I think everything is good, so positive review. Thank you for your work on this Simon. Now #14279 and its dependency (after we finish #10963 and the weak coercions).

@vbraun
Copy link
Member

vbraun commented Mar 3, 2014

Changed branch from public/structure/unique_repr_homsets-14793 to e2d2f16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants