link_or_copy_directory() error - Invalid cross-device link #744

aborruso · 2020-04-28T06:26:45Z

Hi,
when I run

datasette  publish heroku -n myapp --template-dir ./template mydb.db

I have this error

Traceback (most recent call last):
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
    shutil.copytree(src, dst, copy_function=os.link)
  File "/usr/lib/python3.7/shutil.py", line 365, in copytree
    raise Error(errors)
shutil.Error: [('/myfolder/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmps9_4mzc4/templates/base.html', "[Errno 18] Invalid cross-device link: '/myfolder/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmps9_4mzc4/templates/base.html'"), ('/myfolder/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmps9_4mzc4/templates/index.html', "[Errno 18] Invalid cross-device link: '/myfolder/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmps9_4mzc4/templates/index.html'")]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aborruso/.local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
    extra_metadata,
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
    os.path.join(tmp.name, "templates"),
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 609, in link_or_copy_directory
    shutil.copytree(src, dst)
  File "/usr/lib/python3.7/shutil.py", line 321, in copytree
    os.makedirs(dst)
  File "/usr/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/tmps9_4mzc4/templates'

I'm attaching my very basic template folder.

Thank you

template.zip

The text was updated successfully, but these errors were encountered:

simonw · 2020-04-29T03:23:31Z

Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

Here's the code in question:

datasette/datasette/utils/__init__.py

Lines 595 to 609 in 89c4ddd

    
           def link_or_copy(src, dst): 
        
               # Intended for use in populating a temp directory. We link if possible, 
        
               # but fall back to copying if the temp directory is on a different device 
        
               # https://github.com/simonw/datasette/issues/141 
        
               try: 
        
                   os.link(src, dst) 
        
               except OSError: 
        
                   shutil.copyfile(src, dst) 
        
           def link_or_copy_directory(src, dst): 
        
               try: 
        
                   shutil.copytree(src, dst, copy_function=os.link) 
        
               except OSError: 
        
                   shutil.copytree(src, dst)

It looks to me like we were expecting an OSError but in your environment you got a shutil.Error instead. So a fix could be to capture that error as well.

simonw · 2020-04-29T03:25:45Z

https://docs.python.org/3/library/shutil.html#shutil.copytree says that shutil.Error should be raised here, so I think this is a bug in the link_or_copy_directory function.

I don't have an environment that can replicate this bug. @aborruso I'm going to ship a fix in a branch that you can test against.

simonw · 2020-04-29T03:27:20Z

This exception handling was introduced in #141

simonw · 2020-04-29T03:32:14Z

@aborruso I think I have a branch with a fix - could you try it out?

Install the new branch like this:

pip install https://github.com/simonw/datasette/archive/issue-744.zip

Then try running datasette publish heroku again.

If this fixes your bug I'll merge that pull request!

aborruso · 2020-04-29T06:05:02Z

Hi @simonw , I have installed it and I have the below errors.

Is it possible that your /tmp directory is on a different volume from the template folder? That could cause a problem with the symlinks.

No, /tmp folder is in the same volume.

Thank you

Traceback (most recent call last):
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
    shutil.copytree(src, dst, copy_function=os.link)
  File "/usr/lib/python3.7/shutil.py", line 365, in copytree
    raise Error(errors)
shutil.Error: [('/var/youtubeComunePalermo/processing/./template/base.html', '/tmp/tmpcqv_1i5d/templates/base.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/base.html' -> '/tmp/tmpcqv_1i5d/templates/base.html'"), ('/var/youtubeComunePalermo/processing/./template/index.html', '/tmp/tmpcqv_1i5d/templates/index.html', "[Errno 18] Invalid cross-device link: '/var/youtubeComunePalermo/processing/./template/index.html' -> '/tmp/tmpcqv_1i5d/templates/index.html'")]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aborruso/.local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke    return callback(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
    extra_metadata,
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
    os.path.join(tmp.name, "templates"),
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 609, in link_or_copy_directory
    shutil.copytree(src, dst)
  File "/usr/lib/python3.7/shutil.py", line 321, in copytree
    os.makedirs(dst)
  File "/usr/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/tmp/tmpcqv_1i5d/templates'

aborruso · 2020-04-29T06:17:26Z

A stupid note: I have no tmpcqv_1i5d folder in in /tmp.

It seems to me that it does not create any /tmp/tmpcqv_1i5d/templates folder (or other name folder, inside /tmp)

simonw · 2020-04-29T07:08:09Z

What operating system are you using? I need to figure out a way to replicate this bug.

aborruso · 2020-04-29T07:16:27Z

Hi @simonw it's debian as Windows Subsystem for Linux

PRETTY_NAME="Pengwin"
NAME="Pengwin"
VERSION_ID="10"
VERSION="10 (buster)"
ID=debian
ID_LIKE=debian
HOME_URL="https://github.com/whitewaterfoundry/Pengwin"
SUPPORT_URL="https://github.com/whitewaterfoundry/Pengwin"
BUG_REPORT_URL="https://github.com/whitewaterfoundry/Pengwin"
VERSION_CODENAME=buster

aborruso · 2020-05-07T06:38:24Z

Hi @simonw probably I could try to do it in Python for windows. I do not like to do these things in win enviroment.

Because probably WSL Linux env (in which I do a lot of great things) is not an environment that will be tested for datasette.

In win I shouldn't have any problems. Am I right?

simonw · 2020-05-07T06:40:09Z

Have a try and let me know what happens! I'd like this stuff to just work on Windows but I need to figure out how to get an environment working.

aborruso · 2020-05-07T06:53:09Z

@simonw another error starting from Windows.

I run

datasette  publish heroku -n comunepa --template-dir template commissioniComunePalermo.db

And I have

Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module>
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 53, in heroku
    line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
  File "c:\python37\lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "c:\python37\lib\subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\python37\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found

files.zip

simonw · 2020-05-07T06:55:04Z

I have a Windows 10 gaming PC in the house: what steps can I take to get an environment in there that matches yours? I've not installed Python on Windows before.

aborruso · 2020-05-07T07:34:18Z

In Windows I'm not very strong. I use debian (inside WSL).

However these are the possible steps:

I have installed Python 3 for win (I have 3.7.3);
I have installed heroku cli for win64 and logged in;
I have installed datasette running python -m pip install --upgrade --user datasette.

It's a very basic Python env that I do not use. This time only to reach my goal: try to publish using custom template

aborruso · 2020-05-07T07:51:25Z

I have installed heroku plugins:install heroku-builds, but I have the same error.

Then I have removed from datasette\publish\heroku.py

        # Check for heroku-builds plugin
        plugins = [
            line.split()[0] for line in check_output(["heroku", "plugins"]).splitlines()
        ]
        if b"heroku-builds" not in plugins:
            click.echo(
                "Publishing to Heroku requires the heroku-builds plugin to be installed."
            )
            click.confirm(
                "Install it? (this will run `heroku plugins:install heroku-builds`)",
                abort=True,
            )
            call(["heroku", "plugins:install", "heroku-builds"])

And now I have

Traceback (most recent call last):
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 210, in temporary_heroku_directory
    yield
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 96, in heroku
    list_output = check_output(["heroku", "apps:list", "--json"]).decode(
  File "c:\python37\lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "c:\python37\lib\subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "c:\python37\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\python37\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The specified file could not be found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\Scripts\datasette.exe\__main__.py", line 9, in <module>
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 120, in heroku
    call(["heroku", "builds:create", "-a", app_name, "--include-vcs-ignore"])
  File "c:\python37\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\aborr\AppData\Roaming\Python\Python37\site-packages\datasette\publish\heroku.py", line 213, in temporary_heroku_directory
    tmp.cleanup()
  File "c:\python37\lib\tempfile.py", line 809, in cleanup
    _shutil.rmtree(self.name)
  File "c:\python37\lib\shutil.py", line 513, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\python37\lib\shutil.py", line 401, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "c:\python37\lib\shutil.py", line 399, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] Unable to access file. The file is being used by another process: 'C:\\Users\\aborr\\AppData\\Local\\Temp\\tmpkcxy8i_q'

simonw · 2020-05-21T17:16:15Z

I just hit this bug myself, or at least a variant of it!

https://github.com/simonw/museums/runs/697063068?check_suite_focus=true

2020-05-21T17:14:02.2596471Z Traceback (most recent call last):
2020-05-21T17:14:02.2599146Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory
2020-05-21T17:14:02.2599861Z     shutil.copytree(src, dst, copy_function=os.link)
2020-05-21T17:14:02.2600377Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree
2020-05-21T17:14:02.2600786Z     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
2020-05-21T17:14:02.2601196Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 510, in _copytree
2020-05-21T17:14:02.2601580Z     raise Error(errors)
2020-05-21T17:14:02.2604565Z shutil.Error: [('/home/runner/work/museums/museums/templates/row-browse-museums.html', '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html',
"[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/row-browse-museums.html' -> '/tmp/tmpq47xi96y/datasette/templates/row-browse-museums.html'"), ('/home/runner/work/museums/museums/templates/index.html', '/tmp/tmpq47xi96y/datasette/templates/index.html',
"[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/index.html' -> '/tmp/tmpq47xi96y/datasette/templates/index.html'"), ('/home/runner/work/museums/museums/templates/query-browse-search.html', '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html',
"[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/query-browse-search.html' -> '/tmp/tmpq47xi96y/datasette/templates/query-browse-search.html'"), ('/home/runner/work/museums/museums/templates/_analytics.html', '/tmp/tmpq47xi96y/datasette/templates/_analytics.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_analytics.html' -> '/tmp/tmpq47xi96y/datasette/templates/_analytics.html'"), ('/home/runner/work/museums/museums/templates/_museum_card.html', '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/_museum_card.html' -> '/tmp/tmpq47xi96y/datasette/templates/_museum_card.html'"), ('/home/runner/work/museums/museums/templates/pages/about.html', '/tmp/tmpq47xi96y/datasette/templates/pages/about.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/about.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/about.html'"), ('/home/runner/work/museums/museums/templates/pages/map.html', '/tmp/tmpq47xi96y/datasette/templates/pages/map.html', "[Errno 18] Invalid cross-device link: '/home/runner/work/museums/museums/templates/pages/map.html' -> '/tmp/tmpq47xi96y/datasette/templates/pages/map.html'")]
2020-05-21T17:14:02.2605437Z 
2020-05-21T17:14:02.2605797Z During handling of the above exception, another exception occurred:
2020-05-21T17:14:02.2606102Z 
2020-05-21T17:14:02.2606423Z Traceback (most recent call last):
2020-05-21T17:14:02.2606817Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/datasette", line 8, in <module>
2020-05-21T17:14:02.2607189Z     sys.exit(cli())
2020-05-21T17:14:02.2607907Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
2020-05-21T17:14:02.2608347Z     return self.main(*args, **kwargs)
2020-05-21T17:14:02.2609024Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
2020-05-21T17:14:02.2609451Z     rv = self.invoke(ctx)
2020-05-21T17:14:02.2610116Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
2020-05-21T17:14:02.2610550Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-21T17:14:02.2611451Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
2020-05-21T17:14:02.2611989Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-21T17:14:02.2612682Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
2020-05-21T17:14:02.2613117Z     return ctx.invoke(self.callback, **ctx.params)
2020-05-21T17:14:02.2613801Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
2020-05-21T17:14:02.2614235Z     return callback(*args, **kwargs)
2020-05-21T17:14:02.2614946Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/publish/cloudrun.py", line 111, in cloudrun
2020-05-21T17:14:02.2615371Z     with temporary_docker_directory(
2020-05-21T17:14:02.2615769Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/contextlib.py", line 113, in __enter__
2020-05-21T17:14:02.2616135Z     return next(self.gen)
2020-05-21T17:14:02.2616957Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 392, in temporary_docker_directory
2020-05-21T17:14:02.2617403Z     link_or_copy_directory(
2020-05-21T17:14:02.2618298Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/datasette/utils/__init__.py", line 607, in link_or_copy_directory
2020-05-21T17:14:02.2619270Z     shutil.copytree(src, dst)
2020-05-21T17:14:02.2619872Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 554, in copytree
2020-05-21T17:14:02.2622179Z     return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
2020-05-21T17:14:02.2622609Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/shutil.py", line 455, in _copytree
2020-05-21T17:14:02.2622878Z     os.makedirs(dst, exist_ok=dirs_exist_ok)
2020-05-21T17:14:02.2623157Z   File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/os.py", line 223, in makedirs
2020-05-21T17:14:02.2623405Z     mkdir(name, mode)
2020-05-21T17:14:02.2623978Z FileExistsError: [Errno 17] File exists: '/tmp/tmpq47xi96y/datasette/templates'

simonw · 2020-05-21T17:23:39Z

Actually maybe the answer here is to use dirs_exist_ok=True when calling shutil.copytree.

aborruso · 2020-05-21T17:47:40Z

@simonw can I test it know? What I must do to update it?

Thank you

simonw · 2020-05-21T17:54:55Z

I think this is the fix! I just landed it in master.

@aborruso you can install Datasette master to test it out like this:

pip install https://github.com/simonw/datasette/archive/cee671a58f417f827d1735b1abaa40716534ea67.zip

aborruso · 2020-05-21T17:58:51Z

Thank you very much!!

I will try and I write you here

aborruso · 2020-05-21T19:43:23Z

@simonw now I have

Traceback (most recent call last):
  File "/home/aborruso/.local/bin/datasette", line 8, in <module>
    sys.exit(cli())
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aborruso/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 103, in heroku
    extra_metadata,
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/publish/heroku.py", line 191, in temporary_heroku_directory
    os.path.join(tmp.name, "templates"),
  File "/home/aborruso/.local/lib/python3.7/site-packages/datasette/utils/__init__.py", line 605, in link_or_copy_directory
    shutil.copytree(src, dst, copy_function=os.link, dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'

Do I must open a new issue?

Thank you

simonw · 2020-05-26T20:14:35Z

What version of Python are you running?

simonw · 2020-05-26T20:15:11Z

Oh no! It looks like dirs_exist_ok is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

aborruso · 2020-05-26T21:15:34Z

Oh no! It looks like dirs_exist_ok is Python 3.8 only. This is a bad fix, it needs to work on older Python's too. Re-opening.

Thank you very much

simonw · 2020-05-27T02:49:26Z

OK, here's a new branch you can try. Install it like this:

pip install https://github.com/simonw/datasette/archive/shutil-backport.zip

If it works for you I'll merge that branch into master.

aborruso · 2020-05-27T06:01:28Z

Dear @simonw thank you for your time, now IT WORKS!!!

I hope that this edit to datasette code is not for an exceptional case (my PC configuration) and that it will be useful to other users.

Thank you again!!

simonw · 2020-05-27T18:18:01Z

Thanks for helping test this @aborruso!

Refs #581, #770, #729, #706, #751, #706, #744, #771, #773

simonw · 2020-05-28T14:28:58Z

This is now released. pip install datasette==0.43 should get the fix.

aborruso · 2020-05-28T14:32:53Z

Wow, I'm in some way very proud!

simonw changed the title ~~datasette heroku publish: errore using template-dir option~~ link_or_copy_directory() error - catches the wrong exception Apr 29, 2020

simonw mentioned this issue Apr 29, 2020

datasette publish can fail if /tmp is on a different device #141

Closed

simonw added a commit that referenced this issue Apr 29, 2020

shutil.Error, not OSError - refs #744

af3a5b9

simonw mentioned this issue Apr 29, 2020

shutil.Error, not OSError #746

Closed

simonw added bug datasette-publish labels Apr 29, 2020

simonw changed the title ~~link_or_copy_directory() error - catches the wrong exception~~ link_or_copy_directory() error - Invalid cross-device link May 21, 2020

simonw added a commit that referenced this issue May 21, 2020

Use dirs_exist_ok=True, refs #744

f2e0a55

simonw closed this as completed in simonw/museums@8d2aaa1 May 21, 2020

simonw mentioned this issue May 21, 2020

Use dirs_exist_ok=True #768

Merged

simonw added a commit that referenced this issue May 21, 2020

Use dirs_exist_ok=True, refs #744 (#768)

cee671a

simonw reopened this May 26, 2020

simonw added a commit that referenced this issue May 27, 2020

Backport of Python 3.8 shutil.copytree, refs #744

6f80757

aborruso closed this as completed May 27, 2020

simonw mentioned this issue May 27, 2020

Backport of Python 3.8 shutil.copytree #769

Merged

simonw added a commit that referenced this issue May 27, 2020

Backport of Python 3.8 shutil.copytree, refs #744 (#769)

2d099ad

simonw added a commit that referenced this issue May 28, 2020

Release notes for 0.43

d56f402

Refs #581, #770, #729, #706, #751, #706, #744, #771, #773

soobrosa mentioned this issue Sep 29, 2021

Win32 "used by another process" error with datasette publish #1479

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

link_or_copy_directory() error - Invalid cross-device link #744

link_or_copy_directory() error - Invalid cross-device link #744

aborruso commented Apr 28, 2020 •

edited

Loading

simonw commented Apr 29, 2020

simonw commented Apr 29, 2020

simonw commented Apr 29, 2020

simonw commented Apr 29, 2020

aborruso commented Apr 29, 2020

aborruso commented Apr 29, 2020

simonw commented Apr 29, 2020

aborruso commented Apr 29, 2020

aborruso commented May 7, 2020

simonw commented May 7, 2020

aborruso commented May 7, 2020

simonw commented May 7, 2020

aborruso commented May 7, 2020

aborruso commented May 7, 2020

simonw commented May 21, 2020 •

edited

Loading

simonw commented May 21, 2020 •

edited

Loading

aborruso commented May 21, 2020

simonw commented May 21, 2020

aborruso commented May 21, 2020

aborruso commented May 21, 2020

simonw commented May 26, 2020

simonw commented May 26, 2020

aborruso commented May 26, 2020

simonw commented May 27, 2020

aborruso commented May 27, 2020

simonw commented May 27, 2020

simonw commented May 28, 2020

aborruso commented May 28, 2020

link_or_copy_directory() error - Invalid cross-device link #744

link_or_copy_directory() error - Invalid cross-device link #744

Comments

aborruso commented Apr 28, 2020 • edited Loading

simonw commented Apr 29, 2020

simonw commented Apr 29, 2020

simonw commented Apr 29, 2020

simonw commented Apr 29, 2020

aborruso commented Apr 29, 2020

aborruso commented Apr 29, 2020

simonw commented Apr 29, 2020

aborruso commented Apr 29, 2020

aborruso commented May 7, 2020

simonw commented May 7, 2020

aborruso commented May 7, 2020

simonw commented May 7, 2020

aborruso commented May 7, 2020

aborruso commented May 7, 2020

simonw commented May 21, 2020 • edited Loading

simonw commented May 21, 2020 • edited Loading

aborruso commented May 21, 2020

simonw commented May 21, 2020

aborruso commented May 21, 2020

aborruso commented May 21, 2020

simonw commented May 26, 2020

simonw commented May 26, 2020

aborruso commented May 26, 2020

simonw commented May 27, 2020

aborruso commented May 27, 2020

simonw commented May 27, 2020

simonw commented May 28, 2020

aborruso commented May 28, 2020

aborruso commented Apr 28, 2020 •

edited

Loading

simonw commented May 21, 2020 •

edited

Loading

simonw commented May 21, 2020 •

edited

Loading