When running .sage files, pass the preparsed file through a pipe

[jhpalmieri]

If you run "sage new.sage" on a script "new.sage", it creates a preparsed file "new.py". Then when a file in the twisted package tries to import the "new" Python module, it ends up importing this preparsed file instead, which leads to problems which can be hard to track down.

I can think of several solutions:

1. hope that users will know not to use names like "new.sage". This is the current state of affairs, and it seems overly optimistic to me.
2. give a warning, or fail outright with an error, if the name of the file is the same as that of a Python module.
3. name the preparsed file something which is less likely to cause a clash, for example, turn "file.sage" into "file_preparsed.py" (see Give safer names to preparsed files #16955).
4. don't save the preparsed file at all: write it to stdout, then run "sage-python -c " on it.

The attached patch implements number 4.

---

Apply

• attachment: trac_11821-root.patch to the root repo
• attachment: trac_11821-preparse.v3.patch to the scripts repo
• attachment: trac_11821-doctests.patch to the main Sage repo

Depends on #71

Component: scripts

Author: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/11821

[nexttime]

comment:2

I'd opt for either (2) or (3).

(3) is nicer from a user perspective, but may have other side effects.

(2) should be easy to implement (try: import ...), without potentially breaking other code because of assumptions on the filename, so I tend to prefer this variant.

P.S.: Looks like the patch is not yet based on #9739. (Haven't tried.)

If we change the way .sage files are preparsed when doctesting, we might run into errors due to left-over <file>.py files from attach() or load(). Just a thought.

[jhpalmieri]

comment:3

Yet a fourth option, implemented in the "v2" patch: don't save the preparsed file at all. Here are the changes in this patch:

• sage-preparse is split into two scripts: sage-preparse-one-file, which preparses a single file and writes the output to stdout. Then sage-preparse calls this, and saves the output to the appropriate place. Note that for sage-preparse, the help message in sage-sage says that it should "preparse file.sage and produce corresponding file_sage.py". This patch fixes both the behavior and the documentation: it saves to "file_sage.py", which should be a valid Python module name, instead of to "file.py" or "file.sage.py". This should help to prevent name clashes.

• Then sage-run calls sage-preparse-one-file and feeds the output into "sage-python -c ", rather than feeding the preparsed py file as "sage-python ".

• Independently of this patch, running "sage <file1.sage> <file2.sage> ..." acts just like running "sage <file1.sage>" -- it ignores the later arguments. So the documentation has been altered to match this.

• On the other hand, sage-preparse can handle multiple files as inputs, so change its description (at the top of the file) to match.

To do: add some tests to sage/tests/cmdline.py.

[jhpalmieri]

Dependencies: #9739, #10952, #8708

[jhpalmieri]

comment:4

The new patch adds some doctests (not very sophisticated ones, but we can expand on them later).

[jhpalmieri]

Changed dependencies from #9739, #10952, #8708 to #9739, #10952, #8708, #12069

[jhpalmieri]

comment:5

This needs rebasing relative to #12069. I'll get to it eventually.

[jhpalmieri]

Work Issues: rebase on #12069

[jhpalmieri]

comment:6

Okay, now rebased.

[jhpalmieri]

comment:7

Rebased to Sage 5.0.beta8 and #12069.

[jhpalmieri]

Changed work issues from rebase on #12069 to none

[roed314]

comment:9

This needs to be rebased against #12415 (sage-doctest is no longer relevant).

[jhpalmieri]

comment:10

Now rebased.

[jhpalmieri]

root repo

[jhpalmieri]

Attachment: trac_11821-root.patch.gz

Attachment: trac_11821-doctests.patch.gz

Sage library

[jhpalmieri]

Attachment: trac_11821-preparse.v3.patch.gz

scripts repo

[jdemeyer]

Changed dependencies from #9739, #10952, #8708, #12069 to #71

[jdemeyer]

comment:15

I don't like this patch because of #71. I regularly use the .py file when I get an exception to find out where the exception happened.

[jdemeyer]

comment:18

See also #16955.

[jhpalmieri]

comment:19

Should this be closed as "won't fix" because of #16955? Or is there a reason to preparse to stdout and pipe the results through sage?

[jdemeyer]

comment:20

Replying to @jhpalmieri:

Or is there a reason to preparse to stdout and pipe the results through sage?

Not having preparsed .py files cluttered around.