Upgrade R (r-project.org) #12057

vbraun · 2011-11-19T22:08:30Z

The current version of R (r-project) disables graphics if no X11 is available even though r.capabilities() states that PNG is enabled. But you can't generate PNGs, which is why R graphics is broken in the Sage virtual machine. This is fixed in the newest upstream version, of which I made a new spkg.

There is no other change except for the source code and a rediffed patches/R.sh.in

On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.

Apply:

The upgraded spkg also fixes #12065.

CC: @nexttime @jdemeyer @jasongrout

Component: packages: standard

Keywords: R spkg r-project graphics PNG X11

Author: Volker Braun

Reviewer: Karl-Dieter Crisman, Jeroen Demeyer

Merged: sage-4.8.alpha4

Issue created by migration from https://trac.sagemath.org/ticket/12057

The text was updated successfully, but these errors were encountered:

vbraun · 2011-11-19T22:18:40Z

comment:2

This ticket obsoletes #8868

kcrisman · 2011-11-20T01:25:28Z

comment:3

Hmm, I don't mind this in theory. I'd want to test it out a little. Can you point to one of the R announcements where they end up fixing this? They have very detailed and thorough announcements, which I usually cannot make it all the way through when searching for this sort of thing.

Also, does this mean we don't need libpango or cairo or all that stuff? Or would we still need that?

Some relevant things.

Re: Make R graphics work regardless of X and other things #8868 - One concern is whether it always makes graphics work - OS X, Linux, Solaris. Otherwise I wouldn't want to re-enable the doctests, which are currently optional. If they work, then this ticket should enable them.
Great idea on the r-project. I always put R things that are not about statistics per se in the "packages" component. I'm updating others now.
Leif has put a lot of work in separating rpy from R at Move the RPy package outside of the R package #9906. I'm not sure what should have priority, though, given that we are fairly behind the times and reviewing Move the RPy package outside of the R package #9906 is a boatload of work.

Here are a bunch of other relevant tickets which perhaps (?) should be addressed here, or at least mentioned.

R - WARNING: cannot run mixed C/Fortran code (then exits) #7865 R on 64-bit Solaris C and Fortran mixing
Make sure latex doesn't do weird things with R output #7664 R and Latex
Fix hardcoding of paths in R binary #9668 hardcoding paths in R binary
r-2.10.1.p4 may fail to configure with readline (BLOCKS Sage from building on Ubuntu 11.10) #11970 readline issues on Oneric Ocelot or something
R breaks if SAGE_LOCAL undefined #10967 more issues with the script
Make sure R graphics in notebook live in correct place #11249 R graphics occasionally living in weird places

vbraun · 2011-11-20T11:27:19Z

comment:4

You still need cairo+pango+fontconfig+fonts; R doesn't include fonts or a font renderer. But it works now in the virtual machine without X installed. I don't have a reference for the release announcement where it was fixed.

Given that we get relatively frequent requests for R graphics in the virtual machine I would prefer it if we could get this ticket reviewed asap. Really all there is to do is regression testing, and the easiest way to do so is to run it on the build bot.

nexttime · 2011-11-20T18:57:22Z

Changed keywords from R r-project graphics to R spkg r-project graphics PNG X11

nexttime · 2011-11-20T18:57:22Z

Replying to @vbraun:

On an unrelated note, it would be nice if tickets talking about R would contain "r-project" on trac since it is otherwise hopeless to search for.

You can also search trac for "R" with Google.

I intentionally did not (yet) update R or RPy on #9906 or #9668. #11970 is rather unrelated to R itself; it fixes the readline spkg. #10967 could in principle be closed as invalid, as the R script in $SAGE_LOCAL/bin/ is not intended to be run directly; one should instead use sage --R ... (although adding a sanity check to the script doesn't hurt, which I did or intended to do on #9668, which depends on #9906...).

vbraun · 2011-11-21T09:30:59Z

comment:6

Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.

kcrisman · 2011-11-21T13:55:44Z

comment:7

Replying to @vbraun:

Can we get the updated R spkg into Sage before fixing the laundry list of unrelated issues? None of the tickets leif mentioned is close to resolution. This ticket is ready to be reviewed and included right now, and I think a large group of users will benefit from being able to use graphics in the virtual machine.

I think that given Leif's lukewarmness on this and Volker's adamance, along with my own natural inclination, let's go for it. My guess is that we may not even need to backport everything for the others... ?

Testing on OSX 10.4 PPC now :)

Did you notice whether there were any needed patches in a -t -long situation?

vbraun · 2011-11-21T17:02:45Z

comment:8

Patch attached for some small changes in sage/interfaces/r.py. Highlights are that R now returns the class as ordered list and that it doesn't accept the somewhat spurious 'source(file=file(...))' construct that we feed it.

kcrisman · 2011-11-21T18:46:11Z

comment:9

Questions:

Won't

return 'file=file("%s",open="r")\nsource(file)'%filename

cause problems if someone wants to use the file command again, or is that local in scope?

Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...
Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

Once I'm done testing it installs I'll be more formal in looking at this. What a great thing to do, thank you much!

jdemeyer · 2011-11-21T19:53:54Z

comment:10

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

vbraun · 2011-11-21T20:13:41Z

comment:11

Replying to @kcrisman:

Won't [...] cause problems if someone wants to use the file command again, or is that local in scope?

Functions can't be lvalues in R, so its fine though maybe not beautiful:

> file=123
> file
[1] 123
> file('test.txt')
description       class        mode        text      opened    can read 
 "test.txt"      "file"         "r"      "text"    "closed"       "yes" 
  can write 
      "yes" 
> file
[1] 123

Also, might as well add to reference manual if it's now all formatted correctly? Should just be one line adding it to the toc or something...

I only patched up the sphinx breakage where I fixed things. There are still 2kloc to go, but I didn't want to break each and every patch for sage/interfaces/r.py on trac...

Is it possible to provide a VM doing what you showed works I could download into my VirtualBox, or is that too much work?

You just have to recompile the new R in my virtual machine and it'll basically work. You need to install fonts, though, otherwise all text in the plots will be emtpy squares.

vbraun · 2011-11-21T20:15:44Z

Attachment: trac_12057_fix_doctests.patch.gz

Minor docstring fix.

vbraun · 2011-11-21T20:59:35Z

comment:12

Replying to @jdemeyer:

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though. Did you merge anything in alpha3 that might cause this?

jdemeyer · 2011-11-21T21:05:47Z

comment:13

Replying to @vbraun:

Replying to @jdemeyer:

On sage.math, I get:

make[6]: Entering directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
Makefile:111: *** multiple target patterns.  Stop.
make[6]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[5]: *** [R] Error 2
make[5]: Leaving directory `/mnt/usb1/scratch/jdemeyer/merger/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/src/modules/X11'
make[4]: *** [make.X11] Error 2
make[4]: *** Waiting for unfinished jobs....

I just built on sage.math and it worked for me (20x parallel). I'm using sage-4.8.alpha2, though.

Well, it could be some strange error happening only sporadically. Or, an error which happens only if Sage is built from scratch (as opposed to installing the spkg in an existing Sage build). I will try again later.

Did you merge anything in alpha3 that might cause this?

I don't think so.

jhpalmieri · 2011-11-21T23:49:32Z

comment:14

For what it's worth, I took sage-4.8.alpha2, replaced the R spkg with this one, and merged the Sage library patch. Then I built from scratch (on sage.math) and all tests passed. I'm also trying on OS X 10.6.8: the build has succeeded, and doctests are still going.

kcrisman · 2011-12-03T15:29:48Z

comment:24

Jeroen, what else needs to be done for positive review?

kcrisman · 2011-12-03T15:29:48Z

Reviewer: Karl-Dieter Crisman, Jeroen Demeyer

jdemeyer · 2011-12-03T15:48:43Z

comment:25

Replying to @kcrisman:

Jeroen, what else needs to be done for positive review?

I don't know, I haven't actually reviewed the ticket. I just tried to install it on various configurations.

jdemeyer · 2011-12-08T23:08:23Z

comment:27

I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.

kcrisman · 2011-12-08T23:15:09Z

comment:28

Replying to @jdemeyer:

I haven't checked that it fixes the problems mentioned in the ticket, but apart from that: looks good. The spkg could do with some cleaning up (also, who's idea was it to put a spkg inside a spkg???), but I don't want to force this. So, positive_review.

And there are plenty of other tickets open about those things in any case :)

jdemeyer · 2011-12-10T10:40:58Z

Merged: sage-4.8.alpha4

jhpalmieri · 2011-12-16T17:13:27Z

comment:30

With this new version, this doctest is problematic (line 352):

    sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris

On my Mac OS X 10.6.8 box, doctesting this starts up X Windows. This is new: it didn't do this in Sage 4.8.alpha3.

Any ideas why?
I don't like this behavior: doctesting shouldn't start up external applications. What should be done about it?

vbraun · 2011-12-16T18:06:48Z

comment:31

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")? If yes, does

Sys.unsetenv("DISPLAY")

change anything?

jhpalmieri · 2011-12-16T18:29:15Z

comment:32

Replying to @vbraun:

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")?

Yes. It also reports TRUE for this.

If yes, does

Sys.unsetenv("DISPLAY")

change anything?

Yes, X11 doesn't start, and it reports "False" for png:

> Sys.unsetenv("DISPLAY") 
> capabilities("png")
  png 
FALSE 
> capabilities("aqua")
aqua 
TRUE

(I don't know why there are leading spaces in the returned string " png".)

vbraun · 2011-12-16T18:58:23Z

comment:33

Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?

kcrisman · 2011-12-16T21:05:28Z

comment:34

Replying to @jhpalmieri:

Replying to @vbraun:

I don't have a local OSX machine, so I can't debug it myself. Does it also happen if you start R and type in capabilities("png")?

Yes. It also reports TRUE for this.

Wow, that is really unexpected. Also weird, because I tested the entire library overnight on my local 4.8.alpha4 (for another ticket) and nothing happened like that, or I would have noticed this morning.


new-host-2:sage-4.8.alpha4 karl-dietercrisman$ ./sage -t devel/sage/sage/interfaces/r.py 
sage -t  "devel/sage/sage/interfaces/r.py"                  
	 [13.6 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 13.6 seconds

Of course, I also get

> capabilities("png")
  png 
FALSE

Your problem is that someone Sage got compiled with png support! It's not supposed to on Mac, we try to configure it to use (only) Aqua. Hence the doctest being a little unusual:

            sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris
            sage: t = r.eval('capabilities("aqua")') # Should be on all supported Mac versions
            sage: "TRUE" in s+t                      # optional -- rgraphics
            True

My guess is you have something set in your preferences or elsewhere that caused R to pick up a system version of something. In general, getting R to compile on Mac with png support is interesting and hackish - I can't find the page I'm looking for right now, but it's referenced somewhere on one of the previous "get graphics in R in Sage to work" Trac tickets.

In other words, while not ideal, I wouldn't call this something to unmerge this new spkg for.

jhpalmieri · 2011-12-16T22:16:03Z

comment:35

vbraun:

Does R just start up X11 or does it also show anything in it? Can you strace the R execution and see where it executes X?

X11 just starts up. It doesn't seem to have any windows running. I don't have strace installed on this computer, and I wouldn't know how to use it if I did ;)

By the way, if I log into an OS X box remotely, I don't have this problem. It must correctly detect that I can't run X11, and it also says that I don't have 'png' capabilities.

kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.

I remember seeing something about compiling R with png support on http://ask.sagemath.org/, but I don't remember beyond that. I don't think I have a particularly unusual setup, but who knows what R looks at? In the install log for R, I see this:

  Interfaces supported:      X11, aqua, tcltk
  External libraries:        readline, ICU
  Additional capabilities:   PNG, JPEG, NLS

Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we do the equivalent in Sage of

Sys.unsetenv("DISPLAY") 
capabilities("png")
capabilities("aqua")

without breaking things on Linux or Solaris?

jdemeyer · 2011-12-16T22:21:27Z

comment:36

r.eval('Sys.unsetenv("DISPLAY")')

when starting up the R pexpect interface?

In any case, I prefer that you make a new ticket and move the discussion there.

kcrisman · 2011-12-17T01:14:28Z

comment:37

kcrisman: Do you have X11 installed on your Mac? Maybe if it's not installed, it doesn't go through this check and returns "FALSE" for png capabilities.

Yes, I do.

Anyway, I'm not suggesting unmerging the spkg, but fixing the doctest somehow (in a followup ticket, if that's best). Can we

Be sure to cc: me and Jason on that, then.

vbraun · 2011-12-17T11:32:36Z

comment:38

Followup ticket is #12172

koffie · 2011-12-19T12:28:31Z

comment:39

On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).

Doing

md:sage-4.8.alpha3 maarten$ ./sage -f ../sage-4.8.alpha4/spkg/standard/r-2.14.0.p0.spkg

Hangs as described on:
http://pastebin.com/7K7qLMJh

The guilty command is:
/Users/maarten/Downloads/________________/sage-4.8.alpha3/spkg/build/r-2.14.0.p0/src/bin/exec/R --no-restore --slave --args nextArg--pkglocknextArg--data-compress=xznextArg--byte-compilenextArg-lnextArg../../../librarynextArgMASS.tgz

kcrisman · 2011-12-19T14:46:05Z

comment:40

Replying to @koffie:

On OS X 10.6.8 I cannot install this spkg in sage-4.8.alpha3 (I build the alpha3 from source so the old spkg does install).

Questions:

Do you have any unusual environment variables set?
Do you have any R env. variables, or another R installation separate from this one?
Is there anything unusual you have on your machine (I am currently on the exact same OS version, w/out problems) regarding compiling?

Usually we've found that R problems are related to it picking up system versions or variables.

I'd recommend we open a new ticket for this, though, since "most" OS X 10.6 systems seem to be just fine.

vbraun · 2011-12-19T15:42:44Z

comment:41

On the new ticket it would also be useful to attach dtrace to the hanging R process to see what it is doing.

koffie · 2011-12-20T13:34:54Z

comment:42

Ok, I created #12201 . I try to answer your questions there.

vbraun added this to the sage-4.8 milestone Nov 19, 2011

vbraun added c: statistics labels Nov 19, 2011

vbraun assigned sagetrac-amhou Nov 19, 2011

This comment has been minimized.

Sign in to view

vbraun added t: bug labels Nov 19, 2011

nexttime mannequin added t: enhancement and removed t: bug labels Nov 20, 2011

vbraun added p: critical / 2 and removed p: major / 3 labels Nov 21, 2011

This comment has been minimized.

Sign in to view

jdemeyer added s: needs work and removed s: needs review labels Nov 21, 2011

jdemeyer added c: packages: standard and removed c: statistics labels Dec 8, 2011

jdemeyer removed the s: positive review label Dec 10, 2011

jdemeyer closed this as completed Dec 10, 2011

qed777 mannequin mentioned this issue Sep 10, 2012

Move the RPy package outside of the R package #9906

Closed

sagetrac-flint mannequin mentioned this issue Dec 9, 2011

Failing doctest - R installation broken? #12065

Closed

This was referenced Jan 31, 2012

$SAGE_LOCAL/lib and lib64 #12131

Closed

R doctest starts X11 on OSX #12172

Closed

koffie mentioned this issue Aug 1, 2012

The new R spkg breaks building from source on at least one OS X system. #12201

Closed

Upgrade R (r-project.org) #12057

Upgrade R (r-project.org) #12057

Comments

vbraun commented Nov 19, 2011

This comment has been minimized.

vbraun commented Nov 19, 2011

kcrisman commented Nov 20, 2011

vbraun commented Nov 20, 2011

nexttime mannequin commented Nov 20, 2011

nexttime mannequin commented Nov 20, 2011

vbraun commented Nov 21, 2011

kcrisman commented Nov 21, 2011

This comment has been minimized.

vbraun commented Nov 21, 2011

kcrisman commented Nov 21, 2011

jdemeyer commented Nov 21, 2011

vbraun commented Nov 21, 2011

vbraun commented Nov 21, 2011

vbraun commented Nov 21, 2011

jdemeyer commented Nov 21, 2011

jhpalmieri commented Nov 21, 2011

kcrisman commented Dec 3, 2011

kcrisman commented Dec 3, 2011

jdemeyer commented Dec 3, 2011

jdemeyer commented Dec 8, 2011

kcrisman commented Dec 8, 2011

jdemeyer commented Dec 10, 2011

jhpalmieri commented Dec 16, 2011

vbraun commented Dec 16, 2011

jhpalmieri commented Dec 16, 2011

vbraun commented Dec 16, 2011

kcrisman commented Dec 16, 2011

jhpalmieri commented Dec 16, 2011

jdemeyer commented Dec 16, 2011

kcrisman commented Dec 17, 2011

vbraun commented Dec 17, 2011

koffie commented Dec 19, 2011

kcrisman commented Dec 19, 2011

vbraun commented Dec 19, 2011

koffie commented Dec 20, 2011