Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mathics crashes with KeyError: 'System`MakeBoxes' #1169

Closed
vsht opened this issue Mar 2, 2021 · 33 comments
Closed

mathics crashes with KeyError: 'System`MakeBoxes' #1169

vsht opened this issue Mar 2, 2021 · 33 comments
Labels

Comments

@vsht
Copy link

vsht commented Mar 2, 2021

Hi,

I apologize if I overlooked something, but on my system (Fedora 32) upon compiling Mathics
from git via pip3 install -e . --user, running mathics leads to the following error

$ mathics
/media/Data/Projects/Other/Mathics/mathics/builtin/datentime.py:130: DeprecationWarning: invalid escape sequence \$
  """
Traceback (most recent call last):
  File "/home/vs/.local/bin/mathics", line 11, in <module>
    load_entry_point('Mathics3', 'console_scripts', 'mathics')()
  File "/media/Data/Projects/Other/Mathics/mathics/main.py", line 280, in main
    definitions = Definitions(add_builtin=True, extension_modules=extension_modules)
  File "/media/Data/Projects/Other/Mathics/mathics/core/definitions.py", line 77, in __init__
    contribute(self)
  File "/media/Data/Projects/Other/Mathics/mathics/builtin/__init__.py", line 137, in contribute
    builtins["System`MakeBoxes"].contribute(definitions)
KeyError: 'System`MakeBoxes'

The same happens if I install mathics directly via pip using

pip install Mathics3  --user
pip install Mathics-Django  --user
pip install mathicsscript  --user
pip install pymathics-natlang  --user
pip install pymathics-graph  --user

which is what I initially did. Am I doing something wrong?

Cheers,
Vladyslav

@rocky
Copy link
Member

rocky commented Mar 2, 2021

TL;DR remove all mathics packages before installing first.

pip uninstall pymathics-graph
pip uninstall pymathics-natlang
pip uninstall mathicsscript
pip uninstall Mathics-Django
pip uninstall Mathics3

then check that everything is gone including what is in /local/Data/Projects/Other.

I will guess that you have a mixture of mathics installed using --user. Under what directory is this installing to?

I see above /home/vs/.local and /media/Data/Projects/Other. And if you are getting a deprecation warning you may have an older version. I am not getting a deprecation warning. What version of Python do you have installed? python -V will show that.

In the short term there are going to be API mismatches as we try to reorganize things. In theory the package versioning should catch this. In practice things are a bit too complex and mistakes in the past have been made (which can't be fixed if you have those installed). Therefore it is best to delete old versions before installing a newer or different version.

@vsht
Copy link
Author

vsht commented Mar 2, 2021

Sorry for the confusion. I didn't have any Mathics versions installed before installing via pip.
That already gave me the KeyError. Then I uninstalled Mathics via pip and tried to install the git version
which again gave me the same error.

I'm fairly sure that there is no mixture of mathics versions on my machine. /media/Data/Projects/Other/Mathics is the directory where I cloned the git version. Building it with pip3 install -e . --user means that the compiled versions is installed to
/home/vs/.local/lib/python3.8/site-packages. I checked several times, purging everything related to Mathics in /home/vs/.local
and then compiling the git version, but the error remains. The Python version I'm using is

Python 3.8.8 (default, Feb 20 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux

Here is the log output when building mathics

Obtaining file:///media/Data/Projects/Other/Mathics
Requirement already satisfied: cython>=0.15.1 in /usr/lib64/python3.8/site-packages (from Mathics3==2.0.1.dev0) (0.29.14)
Requirement already satisfied: Mathics_Scanner<1.1.0,>=1.0.0 in /home/vs/.local/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (1.0.0)
Requirement already satisfied: sympy<=1.8dev,>=1.7 in /home/vs/.local/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (1.7.1)
Requirement already satisfied: mpmath>=1.1.0 in /usr/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (1.1.0)
Requirement already satisfied: numpy in /usr/lib64/python3.8/site-packages (from Mathics3==2.0.1.dev0) (1.18.4)
Requirement already satisfied: palettable in /home/vs/.local/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (3.3.0)
Requirement already satisfied: pint in /home/vs/.local/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (0.16.1)
Requirement already satisfied: python-dateutil in /usr/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (2.8.0)
Requirement already satisfied: llvmlite in /home/vs/.local/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (0.35.0)
Requirement already satisfied: requests in /usr/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (2.22.0)
Requirement already satisfied: scikit-image in /usr/lib64/python3.8/site-packages (from Mathics3==2.0.1.dev0) (0.16.2)
Requirement already satisfied: wordcloud in /home/vs/.local/lib/python3.8/site-packages (from Mathics3==2.0.1.dev0) (1.8.1)
Requirement already satisfied: PyYAML in /usr/lib64/python3.8/site-packages (from Mathics3==2.0.1.dev0) (5.4.1)
Requirement already satisfied: ujson in /home/vs/.local/lib/python3.8/site-packages (from Mathics_Scanner<1.1.0,>=1.0.0->Mathics3==2.0.1.dev0) (4.0.2)
Requirement already satisfied: chardet in /usr/lib/python3.8/site-packages (from Mathics_Scanner<1.1.0,>=1.0.0->Mathics3==2.0.1.dev0) (3.0.4)
Requirement already satisfied: packaging in /usr/lib/python3.8/site-packages (from pint->Mathics3==2.0.1.dev0) (20.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.8/site-packages (from python-dateutil->Mathics3==2.0.1.dev0) (1.14.0)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python3.8/site-packages (from requests->Mathics3==2.0.1.dev0) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.8/site-packages (from requests->Mathics3==2.0.1.dev0) (1.25.7)
Requirement already satisfied: networkx>=2.0 in /usr/lib/python3.8/site-packages (from scikit-image->Mathics3==2.0.1.dev0) (2.5)
Requirement already satisfied: pillow>=4.3.0 in /usr/lib64/python3.8/site-packages (from scikit-image->Mathics3==2.0.1.dev0) (7.0.0)
Requirement already satisfied: imageio>=2.3.0 in /home/vs/.local/lib/python3.8/site-packages (from scikit-image->Mathics3==2.0.1.dev0) (2.9.0)
Requirement already satisfied: PyWavelets>=0.4.0 in /usr/lib64/python3.8/site-packages (from scikit-image->Mathics3==2.0.1.dev0) (1.0.3)
Requirement already satisfied: matplotlib in /usr/lib64/python3.8/site-packages (from wordcloud->Mathics3==2.0.1.dev0) (3.2.2)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/lib/python3.8/site-packages (from packaging->pint->Mathics3==2.0.1.dev0) (2.4.7)
Requirement already satisfied: decorator>=4.3.0 in /usr/lib/python3.8/site-packages (from networkx>=2.0->scikit-image->Mathics3==2.0.1.dev0) (4.4.0)
Requirement already satisfied: cycler>=0.10 in /usr/lib/python3.8/site-packages (from matplotlib->wordcloud->Mathics3==2.0.1.dev0) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/lib64/python3.8/site-packages (from matplotlib->wordcloud->Mathics3==2.0.1.dev0) (1.3.1)
Installing collected packages: Mathics3
  Found existing installation: Mathics3 2.0.1.dev0
    Uninstalling Mathics3-2.0.1.dev0:
      Successfully uninstalled Mathics3-2.0.1.dev0
  Running setup.py develop for Mathics3
Successfully installed Mathics3

@rocky
Copy link
Member

rocky commented Mar 3, 2021

Well, I am at a loss here. I have uninstalled and installed from the released sources with Python 3.8.8 and I don't get what you see.

Let's start out with :

$ mathics
/media/Data/Projects/Other/Mathics/mathics/builtin/datentime.py:130: DeprecationWarning: invalid escape sequence \$
  """

When you run Python 3.8.8 on that file alone: /media/Data/Projects/Other/Mathics/mathics/builtin/datentime.py you get the Deprecation warning? If so compare with https://github.com/mathics/Mathics/blob/2.0.0/mathics/builtin/datentime.py to make sure these have the same contents. I am guessing that https://github.com/mathics/Mathics/blob/2.0.0/mathics/builtin/datentime.py#L152 might be the offending line.

I can get this kind of failure if I remove builtin/inout.py so this could happen if somehow you don't have this file. As a generalization of that, if Python is looking in the wrong place for builtins this might happen. Some path variable is bolixed which could be a weakness in handling your environment, or it could be a bug in your environment.

Other than that, my recommendation is to use the docker images, or the condaforge images if PyPI isn't working out.

One other thing to note. pip isn't always the same as pip3 and you seemed to mix both in the initial report. Switching between the two can sometimes cause problems, especially when installed scripts (like mathics ) are involved.

@vsht
Copy link
Author

vsht commented Mar 4, 2021

Sorry for the late reply
python /media/Data/Projects/Other/Mathics/mathics/builtin/datentime.py
runs without any issues.

Just to understand it better, what is the preferred way to install the dev version locally?

Running make install as in the development guide I get

python3 setup.py install
/usr/lib/python3.8/site-packages/setuptools/dist.py:472: UserWarning: Normalizing '2.0.1.dev' to '2.0.1.dev0'
  warnings.warn(
running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib64/python3.8'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/local/lib64/python3.8/site-packages/

This directory does not currently exist.  Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).

make: *** [Makefile:46: install] Fehler 1

which is expected I guess, since I'm not going to run that with superuser permissions.

This is why I used pip with the --user key in the first place.

@rocky
Copy link
Member

rocky commented Mar 4, 2021

python /media/Data/Projects/Other/Mathics/mathics/builtin/datentime.py
runs without any issues.

I guess that confirms that when you ran mathics you were picking up some old code from somwhere.

(I should mention that in a number of the earlier reports while having some details haven't had enough details. Decades ago Richard Stallman wrote something really good about how to report a bug. See for exmple. https://www.gnu.org/software/gnats/doc/gnats-4.1.999/html_node/Helpful-hints.html)

Just to understand it better, what is the preferred way to install the dev version locally?

I don't know that there is an "offical" way. Personally I use pyenv:
https://github.com/pyenv/pyenv-installer

Finally, I should mention that interestingly I got that same exact "Key Error" message the other day. But what was going on there was that one of the python modules under bultins had some stray characters I inadvertently added, so some of the builtin modules didn't load. There was no error given in module loading - modules get discovered in the filesystem and are loaded automatically without having an explicit "import" of fixed file filenames. So instead of a Python syntax error message, instead I got this error.

Generalizing from this, if one of the "builtin" modules is mismatched in source code (it might be valid Python in some context but not valid from the context from which it was loaded though) then you can get this error.

@mmatera
Copy link
Contributor

mmatera commented Mar 4, 2021

python /media/Data/Projects/Other/Mathics/mathics/builtin/datentime.py
runs without any issues.

I guess that confirms that when you ran mathics you were picking up some old code from somwhere.

(I should mention that in a number of the earlier reports while having some details don't either haven't given enough details. Decades ago Richard Stallman wrote something really good about how to report a bug. See for exmple. https://www.gnu.org/software/gnats/doc/gnats-4.1.999/html_node/Helpful-hints.html)

Just to understand it better, what is the preferred way to install the dev version locally?

I don't know that there is an "offical" way. Personally I use pyenv:
https://github.com/pyenv/pyenv-installer

Finally, I should mention that interestingly I got that same exact "Key Error" message the other day. But what was going on there was that one of the python modules under bultins had some stray characters I inadvertently added, so some of the builtin modules didn't load. There was no error given in module loading - modules get discovered in the filesystem and are loaded automatically without having an explicit "import" of fixed file filenames. So instead of a Python syntax error message, instead I got this error.

Generalizing from this, if one of the "builtin" modules is mismatched in source code (it might be valid Python in some context but not valid from the context from which it was loaded though) then you can get this error.

This makes me think that it would be a good idea to add some checks on the mechanism of loading builtin modules, at least to show a warning when a module is not loaded, or mismatch the version.

@rocky
Copy link
Member

rocky commented Mar 4, 2021

Yep, and it might already - I was in a rush so I didn't have time to investigate. It is just that this is what thought was probably going on.

@vsht
Copy link
Author

vsht commented Mar 4, 2021

I really don't know what "old code from somwhere" is being picked up here, but if the problem
is on my side and is a nonobvious one, then I guess it's too tedious to debug it here. Thanks
for your help and feel free to close the issue.

@mmatera
Copy link
Contributor

mmatera commented Mar 4, 2021

@vsht, we have been doing a huge refactory of the project, so several pieces that used to be in someplace, now are in other places. When you download and install the code several times, often it happens that different (incompatible) versions coexist in your system. It can happen then that the Python interpreter gets confused, and load parts of incompatible libraries at a time, without a warning. We hope before the next release we are going to be able to handle this kind of issue.

@vsht
Copy link
Author

vsht commented Mar 4, 2021

@mmatera That's fine, I'll figure something out. At least I can now run Mathics in a virtual Ubuntu machine, but I'm still confused
why it doesn't work on Fedora. But it's already a start.

I've just tried to load our FeynCalc package into Mathics, which resulted into a crash :). Now I see that it's because the
FileNames routine is not implemented yet. Furthermore, Mathics identifies itself as Mma 6, so I had to remove the corresponding
version check from the FeynCalc code.

I guess a lot of those missing functions could be acutally implemented using the existing WL routines already available in Mathics.
This sound like fun.

@mmatera
Copy link
Contributor

mmatera commented Mar 4, 2021

@mmatera That's fine, I'll figure something out. At least I can now run Mathics in a virtual Ubuntu machine, but I'm still confused
why it doesn't work on Fedora. But it's already a start.

I've just tried to load our FeynCalc package into Mathics, which resulted into a crash :). Now I see that it's because the
FileNames routine is not implemented yet. Furthermore, Mathics identifies itself as Mma 6, so I had to remove the corresponding
version check from the FeynCalc code.

I guess a lot of those missing functions could be acutally implemented using the existing WL routines already available in Mathics.
This sound like fun.

Actually, we are working on that. If you have a list of missing symbols needed for that package, please let us know as a new issue.

@vsht
Copy link
Author

vsht commented Mar 4, 2021

If I find some time, I'll try to write a small "bootstrap" package to load the main package. Would be cool to have
FeynCalc running on Mathics (at least to some extent). Anyway, this is not related to the original issue any more,
so I'm closing this one.

@vsht vsht closed this as completed Mar 4, 2021
@rocky
Copy link
Member

rocky commented Mar 4, 2021

Mathics identifies itself as Mma 6,

This is a part of a broader problem I would like to see fixed. See: #1130

Also, overall I would like to see more packages like FeynCalc able to use Mathics. At some point I'd like to start a list registry somehwere of packages we can use. And maybe even hook into http://packagedata.net

Unfortunately, almost always we just don't have enough done yet.

So thanks for undertaking this! And thanks for not giving up on us just yet.

@rocky
Copy link
Member

rocky commented Mar 4, 2021

I'm still confused why it doesn't work on Fedora. But it's already a start.

If there is a way you can allow me to login to that computer, I'll debug the setup to see what's wrong some weekend.

You can contact me privately at rocky@gnu.org and and/or add this public key to to an authorized_keys file to allow ssh access.

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBihEdIAEu3JQ1e2h8eSrqZb378z6P43fI702Fwz3ptfrTdr5FUxFGGmoQGA+5XNFeEFBPII4VbWp2iMzLThTxM= rocky@gnu.org

@soehms
Copy link

soehms commented Mar 23, 2021

Hi,

I've got the exactly same issue trying to install Mathics into Sage via pip (under WSL Ubuntu):

sage -pip install Mathics3
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.3.beta9, Release Date: 2021-03-14               │
│ Using Python 3.9.2. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: from mathics.session import MathicsSession
sage: session = MathicsSession()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-ee4ac55fc782> in <module>
----> 1 session = MathicsSession()

~/devel/sage/local/lib/python3.9/site-packages/mathics/session.py in __init__(self, add_builtin, catch_interrupt, form)
      7 class MathicsSession:
      8     def __init__(self, add_builtin=True, catch_interrupt=False, form="InputForm"):
----> 9         self.definitions = Definitions(add_builtin)
     10         self.evaluation = Evaluation(definitions=self.definitions, catch_interrupt=catch_interrupt)
     11         self.form = form

~/devel/sage/local/lib/python3.9/site-packages/mathics/core/definitions.py in __init__(self, add_builtin, builtin_filename, extension_modules)
     75                     loaded = True
     76             if not loaded:
---> 77                 contribute(self)
     78                 for module in extension_modules:
     79                     try:

~/devel/sage/local/lib/python3.9/site-packages/mathics/builtin/__init__.py in contribute(definitions)
    135 def contribute(definitions):
    136     # let MakeBoxes contribute first
--> 137     builtins["System`MakeBoxes"].contribute(definitions)
    138     for name, item in builtins.items():
    139         if name != "System`MakeBoxes":

KeyError: 'System`MakeBoxes'

So it obviously is not just relateted to Fedora 32. Is there already a known workaround?

@rocky
Copy link
Member

rocky commented Mar 23, 2021

What we have come to learn is that this message generally indicates that one of the Python builtin modules failed to load. And there are many reasons a there are reasons why Python might not be able to import a module (which is what is in fact what is going on underneath).

In #1175 @mmatera improved error messaging when things fail to load. So if you can try with a development version we might be able to see more information about what went wrong here.

In not too long a time, we should have another release out.

Going in the other direction, going back a version to 1.1.1 (asssuming you were installing Mathics 2.0.0) might work since life was simpler then.

@soehms
Copy link

soehms commented Mar 24, 2021

If you still need to debug this: The issue is reproducible on Ubuntu systems, as well (for example LinuxMint 19.1). Especially you can have a look at it using the Sage development Docker image

docker run -it sagemath/sagemath-dev:develop

It will drop you in a bash shell where you can run Sage 9.3.beta8 (just type sage, a pdb is included). If you proceed from there as shown in my previous comment the bug will show.

BTW: The background for my attempts comes from a question I was asked last week when I talked about Sage in a knot theorists seminary at the University of Regensburg. One classical software package for knot theorists is the Mathematica package KnotTheory. The question was, if it can be used inside Sage, as well. This reminded me in your Talk on the Sage Days 110.

I must confess that I have absolutely no experiences with Mathematica since I haven't access to a licensed system, right now. So I have no idea if it is realistic to have this package running on Mathics.

After the problem I reported here, I tried to read the package into Mathics using your Docker image mathicsorg/mathics:latest.

In[1]:= << KnotTheory`
Set::wrsym: Symbol $Path is Protected.
Set::wrsym: Symbol $Path is Protected.
Loading KnotTheory` version of September 6, 2014, 13:37:37.2841.
        Read more at http://katlas.org/wiki/KnotTheory.
......

SetDelayed::write: Tag Condition in DT4Knots[n_, t_] /; 12 <= n <= 16 is Protected.
SetDelayed::write: Tag Condition in DTCode[Knot[n_, t_, k_]] /; 12 <= n <= 16 is Protected.
SetDelayed::write: Tag Condition in NameString[Knot[n_Integer ? (#1 <= 10&), k_Integer]] /; k <= NumberOfKnots[n] is Protected.
SetDelayed::write: Tag Condition in NameString[Knot[n_Integer ? (#1 >= 11&), Alternating, k_Integer]] /; k <= NumberOfKnots[n, Alternating] is Protected.
SetDelayed::write: Tag Condition in NameString[Knot[n_Integer ? (#1 >= 11&), NonAlternating, k_Integer]] /; k <= NumberOfKnots[n, NonAlternating] is Protected.
SetDelayed::write: Tag Condition in NameString[Link[n_Integer, Alternating, k_Integer]] /; k <= NumberOfLinks[n, Alternating] is Protected.
SetDelayed::write: Tag Condition in NameString[Link[n_Integer, NonAlternating, k_Integer]] /; k <= NumberOfLinks[n, NonAlternating] is Protected.
StringSplit::strse: String or list of strings expected at position 1 in StringSplit[S].
StringSplit::strse: String or list of strings expected at position 1 in StringSplit[S].
Traceback (most recent call last):
...
  File "/usr/local/lib/python3.8/site-packages/mathics/builtin/comparison.py", line 375, in do_cmp
    if s1.is_Float and s2.is_Float:
AttributeError: 'NoneType' object has no attribute 'is_Float'

Are these just minor problems, or is there no chance?

@mmatera
Copy link
Contributor

mmatera commented Mar 24, 2021

If you still need to debug this: The issue is reproducible on Ubuntu systems, as well (for example LinuxMint 19.1). Especially you can have a look at it using the Sage development Docker image

docker run -it sagemath/sagemath-dev:develop

It will drop you in a bash shell where you can run Sage 9.3.beta8 (just type sage, a pdb is included). If you proceed from there as shown in my previous comment the bug will show.

I will have a look at this, to see the problem. I am not familiarized with the loading mechanism in sage. However, if it uses MathicsSession, the problem could be there: this is a recently implemented feature, and it could have several new bugs. I will check it out.

BTW: The background for my attempts comes from a question I was asked last week when I talked about Sage in a knot theorists seminary at the University of Regensburg. One classical software package for knot theorists is the Mathematica package KnotTheory. The question was, if it can be used inside Sage, as well. This reminded me in your Talk on the Sage Days 110.

I must confess that I have absolutely no experiences with Mathematica since I haven't access to a licensed system, right now. So I have no idea if it is realistic to have this package running on Mathics.

After the problem I reported here, I tried to read the package into Mathics using your Docker image mathicsorg/mathics:latest.

In[1]:= << KnotTheory`
Set::wrsym: Symbol $Path is Protected.
Set::wrsym: Symbol $Path is Protected.
Loading KnotTheory` version of September 6, 2014, 13:37:37.2841.
        Read more at http://katlas.org/wiki/KnotTheory.
......

SetDelayed::write: Tag Condition in DT4Knots[n_, t_] /; 12 <= n <= 16 is Protected.
SetDelayed::write: Tag Condition in DTCode[Knot[n_, t_, k_]] /; 12 <= n <= 16 is Protected.
SetDelayed::write: Tag Condition in NameString[Knot[n_Integer ? (#1 <= 10&), k_Integer]] /; k <= NumberOfKnots[n] is Protected.
SetDelayed::write: Tag Condition in NameString[Knot[n_Integer ? (#1 >= 11&), Alternating, k_Integer]] /; k <= NumberOfKnots[n, Alternating] is Protected.
SetDelayed::write: Tag Condition in NameString[Knot[n_Integer ? (#1 >= 11&), NonAlternating, k_Integer]] /; k <= NumberOfKnots[n, NonAlternating] is Protected.
SetDelayed::write: Tag Condition in NameString[Link[n_Integer, Alternating, k_Integer]] /; k <= NumberOfLinks[n, Alternating] is Protected.
SetDelayed::write: Tag Condition in NameString[Link[n_Integer, NonAlternating, k_Integer]] /; k <= NumberOfLinks[n, NonAlternating] is Protected.
StringSplit::strse: String or list of strings expected at position 1 in StringSplit[S].
StringSplit::strse: String or list of strings expected at position 1 in StringSplit[S].
Traceback (most recent call last):
...
  File "/usr/local/lib/python3.8/site-packages/mathics/builtin/comparison.py", line 375, in do_cmp
    if s1.is_Float and s2.is_Float:
AttributeError: 'NoneType' object has no attribute 'is_Float'

Are these just minor problems, or is there no chance?
The messages seem to come from minor bugs in Mathics, that could be fixed easily. If the package does not use many undefined symbols, it should load after fixing these bugs.

@rocky
Copy link
Member

rocky commented Mar 24, 2021

@soehms I installed sage from docker as you suggested but instead of the last released version of Mathics3 (2.0.0) I used the github version like this:

sage -pip install git+git://github.com/mathics/Mathics.git#egg=Mathics3

So now we get more information about what is going wrong:

sage@bf11a1dadd11:~/sage$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.3.beta8, Release Date: 2021-03-07               │
│ Using Python 3.9.1. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: session = MathicsSession()                                                
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load comparison. Check your installation.
    mathics.builtin loads from /home/sage/sage/local/lib/python3.9/site-packages/mathics/builtin/
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load constants. Check your installation.
    mathics.builtin loads from /home/sage/sage/local/lib/python3.9/site-packages/mathics/builtin/
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load inout. Check your installation.
    mathics.builtin loads from /home/sage/sage/local/lib/python3.9/site-packages/mathics/builtin/

So now the question is why do we get this inside the docker environment?

Some other miscellaneous remarks.

First of all, when you install the latest version of Mathics3, you are just getting the core library and the most miniimal CLI script, whereas in version 1.1.1 you also the web GUI. In the sage docker image this seems to be preferred because the sage docker image doesn't have the OS support to run mysql and that is needed in the Django interface for it to save its data.

If you want to get the maximum number of add-ons installed (which will of course be the hardest since it requires all dependencies) then pip install Mathics-omnibus

Second, I as mentioned in the Sage talk, I am (still) very interested in integrating better with Sage. So would like to see this work inside the docker image. It might mean beefing up the docker image in sage, or in the Mathics-omnibus project which builds its own docker image to include a version of sage (or both).

Another way to interact might be to add sage as a pymathics module so that inside Mathics we could

  • use sage libaries (which are really a bundling from another library) to implement WL routines
  • Be able inside Mathics to drop into Python, sympy or sage, which suggests:
  • being able to convert Mathics queries into sage.

@vsht
Copy link
Author

vsht commented Mar 24, 2021

I think that I've found a solution. First of all, thanks to @mmatera for implementing a debugging output
when the loading mechanism fails. This gave me

odule 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load comparison. Check your installation.
    mathics.builtin loads from /media/Data/Projects/Other/Mathics/mathics/builtin/
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load inout. Check your installation.
    mathics.builtin loads from /media/Data/Projects/Other/Mathics/mathics/builtin/
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load constants. Check your installation.
    mathics.builtin loads from /media/Data/Projects/Other/Mathics/mathics/builtin/
Traceback (most recent call last):
  File "/home/vs/.local/bin/mathics", line 12, in <module>
    load_entry_point('Mathics3', 'console_scripts', 'mathics')()
  File "/media/Data/Projects/Other/Mathics/mathics/main.py", line 281, in main
    definitions = Definitions(add_builtin=True, extension_modules=extension_modules)
  File "/media/Data/Projects/Other/Mathics/mathics/core/definitions.py", line 77, in __init__
    contribute(self)
  File "/media/Data/Projects/Other/Mathics/mathics/builtin/__init__.py", line 147, in contribute
    _builtins["System`MakeBoxes"].contribute(definitions)
KeyError: 'System`MakeBoxes'

Now I certainly do have the mpmath library installed, but Fedora ships version 1.1.0, which is apparently insufficient for Mathics.
After changing "mpmath>=1.1.0" to "mpmath>=1.2.0" in setup.py of Mathics I rerun

pip install -e . --user

which forced pip to fetch mpmath 1.2.0. After that I finally was able to load Mathics on Fedora! Yay!

@rocky
Copy link
Member

rocky commented Mar 24, 2021

@vsht This is great! I will change the minimum dependency for mpmath as suggested above.

rocky added a commit that referenced this issue Mar 24, 2021
Improve installation on Fedora and sage's docker environment.

Fixes #1169 (again)
@rocky
Copy link
Member

rocky commented Mar 24, 2021

@soehms using mmpath>=1.2.0 (or the current github master) should allow you to use mathics inside of the sage docker container as well. (Or at least it worked for me.)

@vsht
Copy link
Author

vsht commented Mar 24, 2021

Many thanks! BTW, speaking of running FeynCalc on Mathics, it seems that Mathics doesn't support the
Hold[Expression] argument in Read, cf.

https://reference.wolfram.com/language/ref/Read.html

You can use Read to get objects to insert into any expression structure, not necessarily a list. Example: Read[stream,Hold[Expression]] gets an expression and places it inside Hold.

Here is a short example

stream = StringToStream["x=2; y=3;"];
out = Read[stream, Hold[Expression]]
Close[stream]
(*Hold[x=2;y=3;] *)
(*String*)
out // FullForm
ReleaseHold[out]
{x, y}
(* Hold[CompoundExpression[Set[x,2],Set[y,3],Null]] *)
(* {2,3} *)

This part is needed in the package loader code

https://github.com/FeynCalc/feyncalc/blob/d98bfc526768c5ee66408ae2b9762ca08ba1c158/FeynCalc/FeynCalc.m#L166

which essentially reads definitions from many different files and puts them into the FeynCalc context.

PS Probably I should better create a separate issue for FeynCalc on Mathics where one can discuss other things missing and possible workarounds.

@rocky
Copy link
Member

rocky commented Mar 24, 2021

PS Probably I should better create a separate issue for FeynCalc on Mathics where one can discuss other things missing and possible workarounds.

@vsht - @mmatera has already thought of that. See #1174

@vsht
Copy link
Author

vsht commented Mar 24, 2021

Ok, then I'll continue there.

@soehms
Copy link

soehms commented Mar 25, 2021

@soehms using mmpath>=1.2.0 (or the current github master) should allow you to use mathics inside of the sage docker container as well. (Or at least it worked for me.)

It seems that I need more advice to be successful, as well. In the Docker container of Sage I invoked the installation of Mathics in exactly the same way, as you did. According to the messages mpmath 1.1.0 has been upgraded to 1.2.1. But the bug still shows:

sage@8a26c30692b7:~/sage$ sage -pip install git+git://github.com/mathics/Mathics.git#egg=Mathics3
Collecting Mathics3
  Cloning git://github.com/mathics/Mathics.git to /tmp/pip-install-sq_hqcu9/mathics3_e0e4109018ee4a2c9903b81b7f2815a7
...
Collecting mpmath>=1.2.0
  Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
     |████████████████████████████████| 532 kB 360 kB/s
...
Installing collected packages: ujson, tifffile, PyWavelets, mpmath, imageio, chardet, wordcloud, scikit-image, pint, palettable, Mathics-Scanner, llvmlite, Mathics3
  Attempting uninstall: mpmath
    Found existing installation: mpmath 1.1.0
    Uninstalling mpmath-1.1.0:
      Successfully uninstalled mpmath-1.1.0
Successfully installed Mathics-Scanner-1.0.0 Mathics3-2.0.1.dev0 PyWavelets-1.1.1 chardet-4.0.0 imageio-2.9.0 llvmlite-0.36.0 mpmath-1.2.1 palettable-3.3.0 pint-0.17 scikit-image-0.18.1 tifffile-2021.3.17 ujson-4.0.2 wordcloud-1.8.1
...
sage: from mathics.session import MathicsSession
sage: session = MathicsSession()
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load comparison. Check your installation.
    mathics.builtin loads from /home/sage/sage/local/lib/python3.9/site-packages/mathics/builtin/
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load inout. Check your installation.
    mathics.builtin loads from /home/sage/sage/local/lib/python3.9/site-packages/mathics/builtin/
module 'mpmath.ctx_mp_python' has no attribute 'mpf'
    Not able to load constants. Check your installation.
    mathics.builtin loads from /home/sage/sage/local/lib/python3.9/site-packages/mathics/builtin/
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
...
--> 147     _builtins["System`MakeBoxes"].contribute(definitions)
    148     for name, item in _builtins.items():
    149         if name != "System`MakeBoxes":

KeyError: 'System`MakeBoxes'

Indeed, the attribute mpf seems to be missing in that version, too:

sage: import mpmath.ctx_mp_python
sage: hasattr(mpmath.ctx_mp_python, 'mpf')
False
sage: hasattr(mpmath.ctx_mp_python, '_mpf')
True

What did I do wrong? What else should I do?

If you want to get the maximum number of add-ons installed (which will of course be the hardest since it requires all dependencies) then pip install Mathics-omnibus

Once the Docker version is up and running I'll try running Mathics on my own Sage installs and checking these things out

Second, I as mentioned in the Sage talk, I am (still) very interested in integrating better with Sage ....

That sounds interesting. I am not a core developer of Sage (I am contributing to Sage just since 2018 in my leisure-time), but I appreciate everything that improves the interaction between open source CAS. As soon as I'll be able to learn more about Mathics (and Wolfram Language), I will see how I can support this.

From the viewpoint of Sage an interface to Mathics in a similar way as for example to Gap or Singular could be a good enhancement, too. If this will successfully enable integration of functionality from packages written in Wolfram Language such at the mentioned KnotTheory this would convince the community, I think.

@rocky
Copy link
Member

rocky commented Mar 25, 2021

I can't spend time on this today, but it is clear that this is a Sage and mpmath interaction problem. If you go into the Python 3.9 from the docker container and import mpmath you won't have a problem:

sage@938a0ca56786:~/sage$ ./local/bin/python3
Python 3.9.1 (default, Mar  8 2021, 09:12:04) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mpmath
>>> mpmath.ctx_mp_python.mpf
<class 'mpmath.ctx_mp_python.mpf'>

And if you look at /home/sage/sage/local/lib/python3.9/site-packages/mpmath/libmp/libmpf.py, you'll find customization for the "sage" backend:

grep -C 3 sage /home/sage/sage/local/lib/python3.9/site-packages/mpmath/libmp/libmpf.py
getrandbits = None

from .backend import (MPZ, MPZ_TYPE, MPZ_ZERO, MPZ_ONE, MPZ_TWO, MPZ_FIVE,
    BACKEND, STRICT, HASH_MODULUS, HASH_BITS, gmpy, sage, sage_utils)

from .libintmath import (giant_steps,
    trailtable, bctable, lshift, rshift, bitcount, trailing,
--
#   2: pickle doesn't work for gmpy mpzs
# Both problems are solved by using hex()

if BACKEND == 'sage':
    def to_pickable(x):
        sign, man, exp, bc = x
        return sign, hex(man), exp, bc
--
    _normalize = gmpy._mpmath_normalize
    _normalize1 = gmpy._mpmath_normalize

if BACKEND == 'sage':
    _normalize = _normalize1 = sage_utils.normalize
...

I am not sure what the BACKEND == 'sage' means but I guess it doesn't use or provide the "mpf" class.

I'd ask on Sage to get more understanding of what's up and what should be done.

@soehms
Copy link

soehms commented Mar 25, 2021

I'd ask on Sage to get more understanding of what's up and what should be done.

see this sage-devel thread.

@rocky
Copy link
Member

rocky commented Mar 26, 2021

It seems that I need more advice to be successful, as well. In the Docker container of Sage I invoked the installation of Mathics in exactly the same way, as you did. According to the messages mpmath 1.1.0 has been upgraded to 1.2.1. But the bug still shows:

@soehms In commit f81d0c5 the problems Mathics was having using sage mpmath from inside sage should be addressed.

So if you run sage -pip install git+git://github.com/mathics/Mathics.git#egg=Mathics3, as you did before, hopefully you should be able to get a mathics session.

@soehms
Copy link

soehms commented Mar 27, 2021

@soehms In commit f81d0c5 the problems Mathics was having using sage mpmath from inside sage should be addressed.

So if you run sage -pip install git+git://github.com/mathics/Mathics.git#egg=Mathics3, as you did before, hopefully you should be able to get a mathics session.

Yes, it works! Great, then I can check it out next week! Many thanks!

@soehms
Copy link

soehms commented May 5, 2021

From the viewpoint of Sage an interface to Mathics in a similar way as for example to Gap or Singular could be a good enhancement, too. If this will successfully enable integration of functionality from packages written in Wolfram Language such at the mentioned KnotTheory this would convince the community, I think.

@rocky

FYI: I made a start to have such an interface realized (see Trac Ticket #31778). I'll be waiting for your comments in that Trac ticket before I will announce it on sage-devel.

@rocky
Copy link
Member

rocky commented May 5, 2021

As I wrote in https://trac.sagemath.org/ticket/31778#comment:3 :

Thanks for undertaking this, Sebastian Oehms!

Know that I'll help out whereever and however I can.

@soehms
Copy link

soehms commented May 5, 2021

Thanks for undertaking this, Sebastian Oehms!
Know that I'll help out whereever and however I can.

Thanks for your immediate answer and the offer to help. But please, just call me Sebastian!

I just started a sage-devel thread about this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants