gd does not build after relocation

[ayyer]

Hi,

I'm having trouble upgrading from Sage-6.4 to Sage-6.7 on my Macbook Pro. The installation aborts at the libgd-2.1.1.p0 stage with the following error:

libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I/Applications/Sage-6.2.app/Contents/Resources/sage/local/include/libpng12 -I/Users/buildbot/build/sage/bsd-1/sage_git/build/local/include/freetype2 -g -fvisibility=hidden -MT gdft.lo -MD -MP -MF .deps/gdft.Tpo -c gdft.c  -fno-common -DPIC -o .libs/gdft.o
gdft.c:153:22: fatal error: ft2build.h: No such file or directory
 #include <ft2build.h>
                      ^
compilation terminated.
make[5]: *** [gdft.lo] Error 1
make[4]: *** [all] Error 2
make[3]: *** [all-recursive] Error 1
Error building gd.

The ft2build.h file is in the local/include/freetype2 subdirectory.
I think the problem is in the -I/Users/buildbot/build/sage/bsd-1/sage_git/build/local/include/freetype2 part, which seems incorrectly copy-pasted from someone else's computer (I don't have the buildbot directory on my computer!). I'm too much of a newbie to fiddle with it, so could someone tell me what to do? I tried to do

make -I/Applications/Sage-6.2.app/Contents/Resources/sage/local/include/freetype2 start

from the root directory, but that failed too.

Here is the link to the log file if it helps: https://www.dropbox.com/s/hal4fp0p3rb2hrw/libgd-2.1.1.p0?dl=0

Thanks for your help.

PS: Sage-6.7 installed perfectly well on my linux desktop. I'm surprised that other people are not having this issue.

CC: @ppurka @kiwifb @kcrisman @vbraun @jdemeyer

Component: relocation

Reviewer: Jeroen Demeyer, Arvind Ayyer, Karl-Dieter Crisman

Issue created by migration from https://trac.sagemath.org/ticket/18816

[kcrisman]

comment:1

Hi Arvind, I highly suspect your problem is related to this sage-devel discussion. Is this a binary, as I assume it is? It's interesting that you were able to update 6.2 to 6.4 on it!

Volker or Jeroen, I'm wondering whether there is anything else we can do (in your experience with releasing) to avoid this kind of thing, or whether it's just not possible. These things always seem to come up but apparently we haven't gotten the "don't develop with binaries" out enough - and apparently some people do experience success there.

[vbraun]

comment:2

IMHO it is a mistake to ship "binary tarballs" consisting of source code and random build artifacts. Can't work reliably, which is why nobody else does that.

Thats not to say that it isn't the fault of our build system, which it totally is. Others have solved binary packaging before so you don't have to compile everything from scratch all the time. Hashdist is working on it (https://github.com/hashdist/hashdist/wiki/HDEP-1:-Crating-System-for-Redistributing-Packages)

A kind of bandaid would be to have some makefile rule that runs "make distclean" before "make" in a binary build.

[jdemeyer]

comment:3

I'm having trouble upgrading from Sage-6.4 to Sage-6.7 on my Macbook Pro.

Could you please explain more in detail what you mean with this? Which commands did you execute?

There are two things you could try:

1. Force sage-location:

$ ./sage --python local/bin/sage-location
$ make

2. If that doesn't work, a more drastic solution:

$ rm -rf local/var/lib/sage/installed/
$ make

This will force rebuilding of every package. With some luck, it will solve your problem.

[jdemeyer]

comment:4

Replying to @vbraun:

Others have solved binary packaging before

...assuming a fixed installation directory.

[kiwifb]

comment:5

So much stuff already said. I seem to have been added from the start. As Jeroen said not long ago running ./sage --python local/bin/sage-location should help greatly. In fact it should be run when you install a binary distribution no matter which platform, it should be part of the instructions.

The problem here is almost certainly that you get the include directory from freetype2.pc which was created on the build platform. The sage-location script should fix all the .pc files to make them refer to your new SAGE_ROOT directory rather than the one the binary was built on.

[vbraun]

comment:6

Replying to @jdemeyer:

Others have solved binary packaging before

...assuming a fixed installation directory.

That has been solved for a long time, but that is not what I meant. The destination directory in Conda and the Hashdist proposal is not tied to the build directory.

[ayyer]

comment:7

Dear Karl (and others),

Thanks for your prompt replies. I will follow the instructions from your sage-devel link tonight and see if it works.

Maybe I am misunderstanding what "binary" means, but I don't think I am doing that. I am using git and I followed the procedures exactly from the http://wiki.sagemath.org/TentativeConventions#Get_the_latest_official_stable_version_and_development_version_of_Sage document (in response to jdemeyer's comment). I have followed them in the past without troubles.

More updates in 10-12 hours!

[jdemeyer]

comment:8

Replying to @ayyer:

I don't think I am doing that.

I am afraid that you are then doing something different from what you think you are doing.

The path you are seeing refers to the OS X buildbot. The only way how that can appear in your install is if you downloaded a binary build of Sage. It's impossible for that path to appear if you're building from the git sources.

[ayyer]

comment:9

Replying to @jdemeyer:

The path you are seeing refers to the OS X buildbot. The only way how that can appear in your install is if you downloaded a binary build of Sage. It's impossible for that path to appear if you're building from the git sources.

I am sorry, but I don't see how that is possible. This is exactly what I did in this order.

$ git fetch origin
$ git reset --hard master    
$ git clean -d -f            
$ git checkout master        
$ git pull --ff-only         
$ make start

Karl-Dieter helped me set up xcode and git on my Mac last July and I have never downloaded the binaries since. I will run make distclean and make start as suggested in the link.

[vbraun]

comment:10

git fetch accesses an alredy-configured remote, so you had a whole git tree already at that point. Starting from scratch would be git clone as described at http://doc.sagemath.org/html/en/developer/walk_through.html#obtaining-the-sage-source-code

[vbraun]

comment:11

PS: You can use git clean -f -d -x to get a pristine tree (akin to make distclean), but you really need the -x

[jdemeyer]

comment:12

Replying to @ayyer:

This is exactly what I did in this order.

$ git fetch origin
$ git reset --hard master    
$ git clean -d -f            
$ git checkout master        
$ git pull --ff-only         
$ make start

What matters is what you did before the above commands. How did you obtain the sources in the first place?

Also, are you sure that you have only one Sage installation on your system?

[jdemeyer]

comment:13

Replying to @ayyer:

Karl-Dieter helped me set up xcode and git on my Mac last July and I have never downloaded the binaries since.

The "since" seems to imply that you did at one point install a Sage binary.

[ayyer]

comment:14

Replying to @jdemeyer:

What matters is what you did before the above commands. How did you obtain the sources in the first place?

I didn't do anything related to the installation of sage before typing those commands, if that's what you mean. If I understand vbraun 's comments above, then git fetch origin would fetch the latest sources, right?

Also, are you sure that you have only one Sage installation on your system?

Yes.

The "since" seems to imply that you did at one point install a Sage binary.

I was just trying to be precise. I don't remember any more, but I highly doubt it.

[jdemeyer]

comment:15

Replying to @ayyer:

I didn't do anything related to the installation of sage before typing those commands

You certainly must have. The command git fetch origin only works if there is already a git tree in the current working directory (think of it this way: if I buy a new computer out of the box, then surely git fetch origin will not download Sage).

So there is still the question of how you obtained that original git tree. And, despite what you think, the answer is that you downloaded a binary (possibly by mistake).

In any case, did you try the suggestions in [comment:3]?

[ayyer]

comment:16

Replying to @vbraun:

PS: You can use git clean -f -d -x to get a pristine tree (akin to make distclean), but you really need the -x

make distclean followed by make start did the trick. Thanks everyone. How do I mark that this ticket is solved?

[kcrisman]

Reviewer: Jeroen Demeyer, Arvind Ayyer, Karl-Dieter Crisman

[kcrisman]

comment:17

We can do that. Moral of the story is for us to have people avoid developing with binaries...

[jdemeyer]

comment:20

I think there really is a bug here...

[kcrisman]

comment:21

Okay, but we need a new title then. Modify if this isn't accurate.

[ayyer]

comment:23

Replying to @ayyer:

make distclean followed by make start did the trick. Thanks everyone. How do I mark that this ticket is solved?

Oops! Question-mark help does not work within sage anymore. Could you help me fix this? I find this feature extremely useful. Thanks. If you like, I can open this as a separate ticket. search_src() works properly though.

Here is the error I get.

sage: Matrix?
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-80-23e6d2b8ac07> in <module>()
----> 1 get_ipython().magic(u'pinfo Matrix')

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s)
   2305         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2306         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2307         return self.run_line_magic(magic_name, magic_arg_s)
   2308 
   2309     #-------------------------------------------------------------------------

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_line_magic(self, magic_name, line)
   2226                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2227             with self.builtin_trap:
-> 2228                 result = fn(*args,**kwargs)
   2229             return result
   2230 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/magics/namespace.pyc in pinfo(self, parameter_s, namespaces)

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
    191     # but it's overkill for just that one bit of state.
    192     def magic_deco(arg):
--> 193         call = lambda f, *a, **k: f(*a, **k)
    194 
    195         if callable(arg):

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/magics/namespace.pyc in pinfo(self, parameter_s, namespaces)
     59         else:
     60             self.shell._inspect('pinfo', oname, detail_level=detail_level,
---> 61                                 namespaces=namespaces)
     62 
     63     @line_magic

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in _inspect(self, meth, oname, namespaces, **kw)
   1570                 pmethod(info.obj, oname, formatter)
   1571             elif meth == 'pinfo':
-> 1572                 pmethod(info.obj, oname, formatter, info, **kw)
   1573             else:
   1574                 pmethod(info.obj, oname)

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/oinspect.pyc in pinfo(self, obj, oname, formatter, info, detail_level)
    647         - detail_level: if set to 1, more information is given.
    648         """
--> 649         text = self._format_info(obj, oname, formatter, info, detail_level)
    650         if text:
    651             page.page(text)

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/oinspect.pyc in _format_info(self, obj, oname, formatter, info, detail_level)
    560         """Format an info dict as text"""
    561         info = self.info(obj, oname=oname, formatter=formatter,
--> 562                             info=info, detail_level=detail_level)
    563         displayfields = []
    564         def add_fields(fields):

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/IPython/core/oinspect.pyc in info(self, obj, oname, formatter, info, detail_level)
    689                     ds += "\nDocstring:\n" + obj.__doc__
    690         else:
--> 691             ds = getdoc(obj)
    692             if ds is None:
    693                 ds = '<no docstring>'

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/sage/misc/sageinspect.pyc in sage_getdoc(obj, obj_name, embedded_override)
   1758         return ''
   1759     r = sage_getdoc_original(obj)
-> 1760     s = sage.misc.sagedoc.format(r, embedded=(embedded_override or EMBEDDED_MODE))
   1761 
   1762     # Fix object naming

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.pyc in format(s, embedded)
    555             s = process_mathtt(s)
    556         s = process_extlinks(s, embedded=embedded)
--> 557         s = detex(s, embedded=embedded)
    558     return s
    559 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.pyc in detex(s, embedded)
    216         s = _rmcmd(s, 'mathrm')
    217         from sagenb.misc.sphinxify import sphinxify
--> 218         s = sphinxify(s, format='text')
    219         # Do math substitutions. The strings to be replaced should be
    220         # TeX commands like "\\blah". Do a regular expression

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/sagenb-0.11.4-py2.7.egg/sagenb/misc/sphinxify.py in sphinxify(docstring, format)
     86     global Sphinx
     87     if not Sphinx:
---> 88         from sphinx.application import Sphinx
     89 
     90     srcdir = mkdtemp()

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/Sphinx-1.2.2-py2.7.egg/sphinx/application.py in <module>()
     20 
     21 from docutils import nodes
---> 22 from docutils.parsers.rst import convert_directive_function, \
     23      directives, roles
     24 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/docutils/parsers/rst/__init__.py in <module>()
     72 
     73 import docutils.parsers
---> 74 import docutils.statemachine
     75 from docutils.parsers.rst import states
     76 from docutils import frontend, nodes, Component

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/docutils/statemachine.py in <module>()
    111 import types
    112 import unicodedata
--> 113 from docutils import utils
    114 from docutils.utils.error_reporting import ErrorOutput
    115 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/docutils/utils/__init__.py in <module>()
     18 from docutils import ApplicationError, DataError
     19 from docutils import nodes
---> 20 import docutils.io
     21 from docutils.utils.error_reporting import ErrorOutput, SafeString
     22 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/docutils/io.py in <module>()
     16 from docutils import TransformSpec
     17 from docutils._compat import b
---> 18 from docutils.utils.error_reporting import locale_encoding, ErrorString, ErrorOutput
     19 
     20 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python2.7/site-packages/docutils/utils/error_reporting.py in <module>()
     45     locale_encoding = None
     46 else:
---> 47     locale_encoding = locale.getlocale()[1] or locale.getdefaultlocale()[1]
     48     # locale.getpreferredencoding([do_setlocale=True|False])
     49     # has side-effects | might return a wrong guess.

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python/locale.pyc in getdefaultlocale(envvars)
    541     else:
    542         localename = 'C'
--> 543     return _parse_localename(localename)
    544 
    545 

/Applications/Sage-6.2.app/Contents/Resources/sage/local/lib/python/locale.pyc in _parse_localename(localename)
    473     elif code == 'C':
    474         return None, None
--> 475     raise ValueError, 'unknown locale: %s' % localename
    476 
    477 def _build_localename(localetuple):

ValueError: unknown locale: UTF-8

[jdemeyer]

comment:24

Relocation is no longer support anyway.