Crash/difficulty while building docs

[yarikoptic]

Summary

It might be the perks of my setup but decided to make a record so if it comes useful for someone else before jumping into trying to figure it out: while building debian package for 1.0.1 (1.0.0+git69-gdb2670326-1 didn't show such problem):

: # TODO: We need to set matplotlib backend to Agg before building...
export PYTHONPATH=$PWD MPLCONFIGDIR=/build/nipype-1.0.1/build; cd doc; LC_ALL=C.UTF-8 /usr/bin/make html
make[1]: Entering directory '/build/nipype-1.0.1/doc'
rm -rf _build/* *~ api/generated interfaces/generated users/examples documentation.zip
mkdir -p users/examples
../tools/make_examples.py --no-exec
examples2rst finished.
rm -rf api/generated
python -u ../tools/build_modref_templates.py
Traceback (most recent call last):
  File "../tools/build_modref_templates.py", line 20, in <module>
    docwriter = ApiDocWriter(package)
  File "/build/nipype-1.0.1/tools/apigen.py", line 76, in __init__
    self.package_name = package_name
  File "/build/nipype-1.0.1/tools/apigen.py", line 98, in set_package_name
    self.root_module = __import__(package_name)
  File "/build/nipype-1.0.1/nipype/__init__.py", line 51, in <module>
    from .pipeline import Node, MapNode, JoinNode, Workflow
  File "/build/nipype-1.0.1/nipype/pipeline/__init__.py", line 11, in <module>
    from .engine import Node, MapNode, JoinNode, Workflow
  File "/build/nipype-1.0.1/nipype/pipeline/engine/__init__.py", line 12, in <module>
    from .workflows import Workflow
  File "/build/nipype-1.0.1/nipype/pipeline/engine/workflows.py", line 36, in <module>
    from ...interfaces.base import (traits, TraitedSpec, TraitDictObject,
  File "/build/nipype-1.0.1/nipype/interfaces/__init__.py", line 13, in <module>
    from .io import DataGrabber, DataSink, SelectFiles, BIDSDataGrabber
  File "/build/nipype-1.0.1/nipype/interfaces/io.py", line 41, in <module>
    from ..utils.filemanip import copyfile, list_to_filename, filename_to_list
  File "/build/nipype-1.0.1/nipype/utils/filemanip.py", line 321, in <module>
    _cifs_table = _generate_cifs_table()
  File "/build/nipype-1.0.1/nipype/utils/filemanip.py", line 318, in _generate_cifs_table
    return _parse_mount_table(exit_code, output)
  File "/build/nipype-1.0.1/nipype/utils/filemanip.py", line 297, in _parse_mount_table
    key=lambda x: len(x[0]), reverse=True)
  File "/build/nipype-1.0.1/nipype/utils/filemanip.py", line 296, in <genexpr>
    for l in output.splitlines()),
AttributeError: 'NoneType' object has no attribute 'groups'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/build/nipype-1.0.1/nipype/utils/config.py", line 360, in free_display
    from .. import config
SystemError: Parent module 'nipype' not loaded, cannot perform relative import
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/build/nipype-1.0.1/nipype/utils/config.py", line 360, in free_display
    from .. import config
SystemError: Parent module 'nipype' not loaded, cannot perform relative import
make[1]: *** [Makefile:38: api] Error 1
make[1]: Leaving directory '/build/nipype-1.0.1/doc'
make: *** [debian/rules:34: doc-stamp] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary subprocess returned exit status 2

what makes it tricky -- upon rerun of make html there it doesn't fail... so there is some "state" which makes it trick(ier) to reproduce

[effigies]

This is related to #2444. Can you give the output of mount on the system you ran this on?

[yarikoptic]

it is in a chroot(like) env

# mount
/proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)

[effigies]

Hmm. That looks fine. Are there any extra blank lines? That might cause the issue.

Though we should handle failures to match better, anyway...

[oesteban]

That is a hook attached to the end of the python session to tear down any possible Xvfb open. I think it is safe to wrap the import within a try ... except SystemError

[effigies]

@yarikoptic It would be good to try to resolve dpkg-build-breaking issues before release. What would be a good process for making sure this happens? Are there tests we can add to Circle that trigger on rel/* branches?

Or perhaps you could set up a deb/prerelease branch that has the most recent neurodebian patches, that we can merge master into periodically?

[yarikoptic]

That would a great thing to accomplish @effigies ... ATM though there is still a bit of manual interventions I would need to resolve but we will get there one way or another.
Typically I am just trying to be proactive and test pre-release states whenever upstream buzzes me "release is coming".

[effigies]

Okay. Just an FYI, 1.0.2 is scheduled for March 26 (we're going for monthly releases). At least the last two releases have had a roughly 1-week sprint of merges and cleanups, followed by about a day of testing for the final release. I will make a point to ping you (@mgxd want to be backup in case I forget?) when we get to the final test phase.

Hopefully after another cycle or two, your manual interventions will get predictable enough to script.