Temporary files in FilesWriter

[jhamrick]

I am not sure if this is a bug or not, but it was unexpected behavior for me, so I figured I'd report it.

When using FilesWriter with nbconvert, if you are converting multiple notebooks at once that rely on the same set of supplementary files, they do get copied to the build directory, but so do temporary files (one version per extra notebook). For example, if I have the following directory/file structure:

$ ls -R
.:
bar.ipynb  files/  foo.ipynb  ipython_nbconvert_config.py

./files:
file1.txt  file2.txt

Where ipython_nbconvert_config.py is:

from glob import glob

c = get_config()

c.NbConvertApp.export_format = 'notebook'
c.NbConvertApp.notebooks = glob('*.ipynb')
c.NbConvertApp.writer_class = 'FilesWriter'

c.FilesWriter.build_directory = 'build'
c.FilesWriter.files = glob('files/*')

Then, if I run nbconvert, the build/files directory contains:

$ ls build/files/
file1.txt  file1.txt-temp-77A5  file2.txt  file2.txt-temp-CE50

I traced the root of the problem to IPython.utils.path.link_or_copy, which says "Because os.link does not overwrite files, a unique temporary file will be used if the target already exists, then that file will be moved into place." That's fine, except it seems like the os.rename(new_dst, dst) line isn't actually removing new_dst (which is the temporary file).

[minrk]

@jhamrick can you reproduce this with master? I can't seem to. If you can, can you post --debug output?

[jhamrick]

Yes, I can still reproduce it:

[jhamrick@octavia]:~/project/tmp
$ rm -r build/
[jhamrick@octavia]:~/project/tmp
$ ipython nbconvert --debug --ipython-dir=/tmp
[NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'ipython_dir': u'/tmp', 'log_level': 10}}
[NbConvertApp] Using existing profile dir: u'/tmp/profile_default'
[NbConvertApp] Searching path [u'/Users/jhamrick/project/tmp', u'/tmp/profile_default', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[NbConvertApp] Attempting to load config file: ipython_config.py
[NbConvertApp] Attempting to load config file: ipython_nbconvert_config.py
[NbConvertApp] Loaded config file: /Users/jhamrick/project/tmp/ipython_nbconvert_config.py
[NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'log_level': 10, 'notebooks': ['bar.ipynb', 'foo.ipynb'], 'ipython_dir': u'/tmp', 'writer_class': 'FilesWriter', 'export_format': 'notebook'}, 'FilesWriter': {'files': ['files/file1.txt', 'files/file2.txt'], 'build_directory': 'build'}}
[NbConvertApp] Converting notebook bar.ipynb to notebook
[NbConvertApp] Support files will be in bar_files/
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file1.txt -> build/files/file1.txt
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file2.txt -> build/files/file2.txt
[NbConvertApp] Writing 556 bytes to build/bar.nbconvert.ipynb
[NbConvertApp] Converting notebook foo.ipynb to notebook
[NbConvertApp] Support files will be in foo_files/
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file1.txt -> build/files/file1.txt
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file2.txt -> build/files/file2.txt
[NbConvertApp] Writing 556 bytes to build/foo.nbconvert.ipynb
[jhamrick@octavia]:~/project/tmp
$ ls build/files/
file1.txt  file1.txt-temp-A1D3  file2.txt  file2.txt-temp-924D

[jhamrick]

I can send you a tar of the example directory I have if that helps, too.

[takluyver]

Looking into this.

[takluyver]

See also Python bug 21876 - os.rename() doesn't remove the source file if it and the destination are hard links to the same file. I believe we're running into this. The comments there mention that "It is arguably a bug in early Unix implementations that got mistakenly codified as part of POSIX."

I think we can work around this by removing the destination file first.

[minrk]

I can reproduce it now, not sure what I did wrong before.

[takluyver]

Should be resolved by #7513.

[jhamrick]

Awesome, thanks! 🍰