Use SAGE_DOC

[jdemeyer]

We should use the new doc output location (SAGE_DOC) variable from http://trac.sagemath.org/ticket/20030

[kcrisman]

Time frame? I assume this is required by that ticket, right, else the docs break?

[jdemeyer]

I assume this is required by that ticket, right, else the docs break?

No, nothing breaks without this ticket.

[kiwifb]

Just to make things clear. Jeroen has put links to the old location so sagenb wouldn't break. But it would be nice not to have those links forever.

[kcrisman]

Just to make things clear. Jeroen has put links to the old location so
sagenb wouldn't break. But it would be nice not to have those links
forever.

Ah, you mean that the Sage ticket is still okay, but ideally we change
sagenb so one could remove the symlinks? Thanks for the detail on this,
and frightfully sorry I have very little time for actual checking out of
tickets lately.

[kiwifb]

Yes. Although ideally I would have wanted Jeroen to change the variable names so that what is currently SAGE_DOC_OUTPUT in this PR becomes SAGE_DOC. That would also means that I would stop patching sagenb in sage-on-gentoo since I have moved the documentation to a similar location since at least sage-6.6, may be earlier.

[jdemeyer]

New version for http://trac.sagemath.org/ticket/20030

[kiwifb]

This is now a requirement for http://trac.sagemath.org/ticket/20030 otherwise incremental building of documentation will break. Part of the problem is that sage/misc/sagedoc.py calls on sagenb/misc/sphinxify.py and that generate_configuration is indeed giving a broken configuration.

[kcrisman]

Okay. Listen, if you (@kiwifb) can give a 100% positive review to this change I am happy to merge it and make a new release, though it might not happen for a few days. I know that I won't have time to check the complex interaction between sagenb and Sage on this this week. But all these changes seem pretty straightforward cleanup and then moving. Does the live documentation as well as pdf and html static links all seem to work with this change (as I expect they do)?

[kiwifb]

With the exception to the change to misc.py and sphinxify.py the change match what I have been doing for sage-on-gentoo for several years. misc.py is cleanup, for sphinxify.py the difference is that what is now under SAGE_DOC_SRC I ship under SAGE_DOC in sage-on-gentoo so as far as I can see it is all under control.

I do one extra thing but that go beyond the scope of the change we want (Don't like using SAGE_ROOT when SAGE_LOCAL can be used).

[jdemeyer]

I am happy to merge it and make a new release

Can that also include #361 and #362 please?

[kcrisman]

I am happy to merge it and make a new release

If it's already been working for François that is great, good.

Can that also include #361 #361
and #362 #362 please?

#361 yes, #362 I just want to convince myself that you are right that there
is no other doctest which could be sensible.

I also will need time to make a branch with the downstream change, do
sagenb, make sure all tests pass myself, etc. I'll really do my best to
get it out the door by the weekend.

[kcrisman]

Hi - any sense on what a good test of this and corresponding sage change is? Just make doc-clean && make doc? (Which seems to be working with Trac 20030 and this PR.)

[kiwifb]

From SAGE_ROOT

sage -t --long local/lib/python2.7/site-packages/sage/misc/sagedoc.py

would tell you quite a bit. Also Volker said that incremental build is broken, not sure how to simulate that. But I think that with http://trac.sagemath.org/ticket/20030 without this the sagedoc.py test should break.

[kcrisman]

Thanks, @kiwifb - I'm not going to test that part, because I figure you guys have tested that. But I just want to do my due diligence as sagenb release manager. Everything on the testing side is fine, I will just check that sagenb itself (including doc) works tomorrow and then merge @jdemeyer's other things.

[kcrisman]

Except:

sage -t --long local/lib/python2.7/site-packages/sage/misc/sagedoc.py
**********************************************************************
File "local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 758, in sage.misc.sagedoc.?
Failed example:
    'abvar/homology' in _search_src_or_doc('doc', 'homology', 'variety', interact=False)  # long time (4s on sage.math, 2012)
Exception raised:
    Traceback (most recent call last):
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.sagedoc.?[3]>", line 1, in <module>
        'abvar/homology' in _search_src_or_doc('doc', 'homology', 'variety', interact=False)  # long time (4s on sage.math, 2012)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 816, in _search_src_or_doc
        in os.listdir(os.path.join(doc_path, L))
    OSError: [Errno 2] No such file or directory: '/Users/.../sage/local/share/doc/sage/ca'
**********************************************************************
File "local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 1114, in sage.misc.sagedoc.?
Failed example:
    search_doc('creates a polynomial', path_re='tutorial', interact=False) # random
Exception raised:
    Traceback (most recent call last):
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.sagedoc.?[0]>", line 1, in <module>
        search_doc('creates a polynomial', path_re='tutorial', interact=False) # random
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 1133, in search_doc
        **kwds)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 816, in _search_src_or_doc
        in os.listdir(os.path.join(doc_path, L))
    OSError: [Errno 2] No such file or directory: '/Users/.../sage/local/share/doc/sage/ca'
**********************************************************************
File "local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 1126, in sage.misc.sagedoc.?
Failed example:
    len(search_doc('tree', interact=False).splitlines()) > 4000  # long time
Exception raised:
    Traceback (most recent call last):
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.sagedoc.?[1]>", line 1, in <module>
        len(search_doc('tree', interact=False).splitlines()) > Integer(4000)  # long time
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 1133, in search_doc
        **kwds)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 816, in _search_src_or_doc
        in os.listdir(os.path.join(doc_path, L))
    OSError: [Errno 2] No such file or directory: '/Users/.../sage/local/share/doc/sage/ca'
**********************************************************************
File "local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 1128, in sage.misc.sagedoc.?
Failed example:
    len(search_doc('tree', whole_word=True, interact=False).splitlines()) < 2000  # long time
Exception raised:
    Traceback (most recent call last):
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.sagedoc.?[2]>", line 1, in <module>
        len(search_doc('tree', whole_word=True, interact=False).splitlines()) < Integer(2000)  # long time
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 1133, in search_doc
        **kwds)
      File "/Users/.../sage/local/lib/python2.7/site-packages/sage/misc/sagedoc.py", line 816, in _search_src_or_doc
        in os.listdir(os.path.join(doc_path, L))
    OSError: [Errno 2] No such file or directory: '/Users/.../sage/local/share/doc/sage/ca'

[kcrisman]

What is sage/local/share/doc/sage/ca?

[kcrisman]

Ah, it's a Catalan translation... why is that the problem, and why am I not having that file? It seems to have been dealt with in Trac 20030. I'll check back on this tomorrow (USA) if anyone has any updates.

[kiwifb]

Well that's one of the reason we should need this PR, ca should be catalan IIRC. And it is not under local/share/doc/sage but under src/doc/ before trac#20030 the first path was SAGE_DOC_OUTPUT and the second SAGE_DOC. With trac#20030, the first path is SAGE_DOC and the second one is SAGE_DOC_SRC. Because SAGE_DOC is now pointing to the build documentation rather than the documentation's source.

[kcrisman]

But I already have this PR and the Trac ticket in this branch... ? I do have local/share/doc/sage/output/html/ca/ and local/share/doc/sage/html/ca/, having both is perhaps not so good.

[kiwifb]

Must have forgotten a change somewhere. Thing is sage-on-gentoo has also traditionally put those under SAGE_DOC to avoid this trouble. I am having a look.

[kiwifb]

Found it. I missed one thing that Jeroen should have changed in trac#20030, there is a SAGE_DOC not converted to SAGE_DOC_SRC at line 804 of sagedoc.py.

[kcrisman]

sage -t --long src/sage/misc/sagedoc.py
    [102 tests, 112.55 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

[kcrisman]

And Sage too.

----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 2768.1 seconds
    cpu time: 8607.8 seconds
    cumulative wall time: 10816.1 seconds

And sagenb still can find the docs. Only one problem remains - when you go to http://localhost:8080/help and click on the link for http://localhost:8080/doc/live/index.html and then click on the pdf picture links e.g. http://localhost:8080/pdf/en/tutorial/SageTutorial.pdf I get

Not Found

The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

(I still can't get live documentation to evaluate, but apparently I'm the only one with that problem and I don't think it's related to this.)

[kiwifb]

I need a bit of time to check that. I must say I haven't given too much attention to pdf doc. It is sometimes difficult to build.

[kcrisman]

I can confirm that the pdf issue is not related to this. See #365.

[kcrisman]

Anyway, I'll wait to merge this until Trac 20030 is completely ready, and then I'll deal with the other two PRs in question. Do you want to open a Sage ticket for upgrading sagenb for this and put the dependencies on those tickets?

[kiwifb]

OK will do later today. Earthquake at 5.9 just hit close by so there may be delay if someone I know needs assistance.

[kcrisman]

Ack!

OK will do later today. Earthquake at 5.9 just hit close by so there may be

delay if someone I know needs assistance.

Well, no rush on my end for sure! Take care of those in need.

[kiwifb]

Done on trac.

[kiwifb]

http://trac.sagemath.org/ticket/20047 is for the notebook upgrade.

[kcrisman]

Thanks for the update. I'll get to this soon.