SOQLDataset.shuffle generates an error when used in a macro

[dcinzona]

If you use the SOQLDataset plugin in a macro, the system attempts to delete the file ( maybe here cleanup()) when the context is closed using TempDirectory().cleanup() - but I'm not sure why it is sending the filename rather than the folder.

If I move this same shuffle declaration into a recipe, not a macro called by the recipe, I no longer get the error.

Recipe
Account.yml

- plugin: snowfakery.standard_plugins.Salesforce.SOQLDataset
- plugin: snowfakery.standard_plugins.Salesforce.SalesforceQuery
- include_file: account.macro.yml
- object: Account
  include: account
  fields:
    __user:
      SOQLDataset.shuffle:
      #SalesforceQuery.random_record:
        fields: Id
        from: User
        where: IsActive = true  LIMIT 10
    OwnerId: ${{__user.Id}}

Macro
account.macro.yml

- plugin: snowfakery.standard_plugins.Salesforce.SOQLDataset
- plugin: snowfakery.standard_plugins.Salesforce.SalesforceQuery
- macro: account
  fields:
    __user:
      SOQLDataset.shuffle:
      #SalesforceQuery.random_record:
        fields: Id, LastName
        from: User
        where: IsActive = true
    OwnerId: ${{__user.Id}}
    Name: ${{__user.LastName}}

I'm guessing that once the macro is loaded, it gets pulled out of memory and the context is closed, and so it's trying to clean up everything - but not sure why it is trying to call cleanup() on the actual queryresults.db rather than the enclosing directory.

Here is the stack trace:

Traceback (most recent call last):
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 820, in _cleanup
    cls._rmtree(name)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 808, in onerror
    cls._rmtree(path)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 608, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 605, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\GUSTAV~1\\AppData\\Local\\Temp\\tmp3tsiuqwe\\queryresults.db'

[dcinzona]

Looks like this might definitely be windows specific and perhaps related to this
https://bugs.python.org/issue42796

But the cleanup routine could still fail due to a sharing violation from some other process (likely a child process) setting the directory or a child directory as the working directory. Also, a sharing violation when trying to delete a regular file in the tree causes the cleanup routine to fail with NotADirectoryError because the onerror function calls rmtree() if unlink() fails with a PermissionError.