windows support redux

[barundel]

Hello

Can anyone provide some advice on the below error when trying to package a simple policy for lambda.

PS C:\Users\USER\Custodian> custodian run --region=eu-west-1 --output-dir=. --config=custod
2017-01-04 10:58:51,770: custodian.policy:INFO Provisioning policy lambda ec2-tag-complianc
2017-01-04 10:58:52,921: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\c7n\policy.py", line 300, in provision
    role=self.policy.options.assume_role)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 221, in publish
    func, role, s3_uri, qualifier=alias)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 319, in _create_or_update
    code_ref = {'ZipFile': archive.get_bytes()}
  File "c:\python27\lib\site-packages\c7n\mu.py", line 165, in get_bytes
    return open(self._temp_archive_file.name, 'rb').read()
IOError: [Errno 13] Permission denied: 'c:\\users\\USER\\appdata\\local\\temp\\tmpvah3za'
2017-01-04 10:58:52,934: custodian.commands:ERROR Error while executing policy ec2-tag-comp
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\c7n\commands.py", line 114, in run
    policy()
  File "c:\python27\lib\site-packages\c7n\policy.py", line 461, in __call__
    return mode.provision()
  File "c:\python27\lib\site-packages\c7n\policy.py", line 300, in provision
    role=self.policy.options.assume_role)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 221, in publish
    func, role, s3_uri, qualifier=alias)
  File "c:\python27\lib\site-packages\c7n\mu.py", line 319, in _create_or_update
    code_ref = {'ZipFile': archive.get_bytes()}
  File "c:\python27\lib\site-packages\c7n\mu.py", line 165, in get_bytes
    return open(self._temp_archive_file.name, 'rb').read()
IOError: [Errno 13] Permission denied: 'c:\\users\\USER\\appdata\\local\\temp\\tmpvah3za'

I have checked permissions/ UAC etc

[kapilt]

its a permission issue for ops on a closed file, which works on posix but not windows. we had fixed this a while back, but the fix ended up causing a regression that would leak temp dirs, and was reverted. we should revisit proper support for windows.

[kapilt]

References #547 and #118

[jdubs]

Would it be easier to shove CC into docker and execute it that way?

[kapilt]

probably ;-) we publish per commit into docker hub, or linux compatibility stuff on windows, but we're pretty close to portable outside of that one semantic around the zip file packaging tempfiles, a context manager at the right level would probably resolve it, tbd.

[barundel]

Got it running on an EC2 instance and getting much further :)

[chadwhitacre]

we should revisit proper support for windows.

I've had good luck with Appveyor for Windows CI. You willing to bite the bullet on complicating our CI story further, @kapilt?

[stefangordon]

In the Python bug https://bugs.python.org/issue14243 I noticed this recommendation, updating the issue here until I have a chance to play with it...

After all, you can currently get deterministic cleanup (with a __del__ fallback) via:

  @contextmanager
  def named_temp(name):
    f = NamedTemporaryFile(name, delete=False)
    try:
        yield f
    finally:
        try:
            os.unlink(name)
        except OSError:
            pass

You need to be careful to make sure you keep the CM alive (or it will delete the file behind your back), but the idiom RDM described in the other issues handles that for you:

  with named_temp(fname) as f:
     data = "Data\n"
     f.write(data)
     f.close() # Windows compatibility
     with open(fname) as f:
         self.assertEqual(f.read(), data)

[jakauppila]

Running into this issue while attempting to test with Windows.

[stefangordon]

This is still an open issue for running some tests on Windows, most of us developing on Windows are using Windows Subsystem for Linux currently so it hasn't been a blocker - but we should get to it at some point for sure. It is basically functionality that does not exist in Windows so it will require refactoring a bit.

[jakauppila]

Sure, I did go down the route of using WSL. If we get to the point where we're deploying automatically we'll have to spin up some Linux hosts since we're largely a Windows shop.

[stefangordon]

This was resolved with #3842 and you should be able to do all your tests on Windows now if you'd like. Please open a new issue if you run into any problems.