boto3 unable to read __file__ attribute when loaded from an egg

[MBloom]

We are running boto3 by appending all required eggs to PYTHONPATH. However, when we call session.get_available_resources() it returns []. Through a bit of debug, we realized that the JSON service files were not being read in from the <BOTOCORE_ROOT>/data directory. We actually were getting a NotADirectoryError when directly calling os.listdir(<BOTOCORE_ROOT>/data).

This is because BOTOCORE_ROOT is initialized by reading in __file__ and according to this site,

[W]hen code like this is packed inside a zipfile, it can no longer assume that __file__ or __path__ contain filenames or directory names, and so it will fail.

In order to support compressed installation, it is suggested to use pkg_resources.resource_filename() instead of __file__.

This seems to be an issue that other people have also run into ( #332 #308) and was hoping it could be patched in a future version. Or, I would love to know some alternative solution to getting an s3 resource from a boto3 session.

[JordonPhillips]

I don't think this will be a simple change. I would really rather not pull in any dependencies if at all possible, but it seems like the relevant pkgutil method get_data opens the file too. Reasonable considering it's opening from a zip, but the way we currently load data doesn't depend on being in a package while that would. @dstufft what do you think?

[dstufft]

The way we currently load data depends on things like checking isdir, listdir, etc. That's not going to work with pkgutil.get_data(). I think the only options here are:

1. Rework loading so that it can rely entirely on pkgutil.get_data().
2. Use pkg_resources.
3. Don't support zipped installs.

The only way to do (1) in a backwards compatible way is to support both the current way and the hypothetical new way. If we're going to make a change this is likely to be the most robust (but also the most invasive).

Doing (2) should be pretty straight forward I believe, but depending on pkg_resources is hard. It has import time side effects and it almost makes things a lot slower.

(3) is where we're at now, although add a zip_safe=False to the setup kwargs might help?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

[tfmorris]

#2191 sounds like another variation of this.