Allow readline.__file__ to be missing

[jayvdb]

See indygreg/PyOxidizer#69 for more info about why, but the tl;dr version is __file__ is an optional attribute and should not be relied upon.

Currently cmd2 fails under PyOxidizer with:

  File "cmd2", line 15, in <module>
  File "cmd2.cmd2", line 56, in <module>
  File "cmd2.rl_utils", line 120, in <module>
AttributeError: module 'readline' has no attribute '__file__'

If sys.oxidized is set, and readline is imported (and isnt libedit), then any symbols needed are in the binary, if that is helpful.

[tleonhardt]

@jayvdb Thanks for brining this to our attention. I at least wasn't aware of PyOxidizer.

Getting cmd2 to interoperate properly with PyOxidizer would likely not be a trivial thing to do due to some of the interesting design choices made by PyOxidizer. It seems like it would be possible, but honestly I don't see a way to do it reliably without moving to Python 3.7 or 3.8+ and even then it might be problematic.

Changes which would appear to be required include the following:

• Replace usage of pkg_resources in __init__.py since this module doesn't appear to be available within PyOxidizer applications (found by my testing)
  • We use this in combination with setuptools_scm for managing the cmd2 package version
  • Python 3.8 appears to have an importlib.metadata module which could probably fill the need, but that doesn't exist prior to 3.8
  • Another option would be to re-design how cmd2 versions are done, but this would be a significant refactoring which I don't believe we would like to undertake
• Replace usage of __file__ in rl_utils.py as you mention
  • Here we just need to get the path to a C code dynamic library so we can load it with ctypes, we do NOT want to open the file, nor do we want to create it
  • It looks like importlib.resources is the recommended way to load "resource" files when using PyOxidizer, but I don't see any suitable APIs there for this
  • It does look like importlib.util.find_spec() would do what we want for finding the underlying dynamic library, but it is unclear to me if this would be properly supported within PyOxidizer

@kmvanbrunt @kotfu What are your thoughts? Do you have any smart ideas on how we can support interoperability with PyOxidizer?

For developers wishing to package a cmd2 application, we can recommend all of the following based on personal and professional experience:

• Deploy your Python app using a Docker container
  • Powerful and flexible - allows you to control entire user space and setup other applications like databases
• PyInstaller
  • Quick and easy - it "just works" and everything you need is installable via pip
• Nuitka
  • Converts your Python to C and compiles it to a native binary
• Conda Constructor
  • Allows you to create a custom Python distro based on miniconda

NOTE: I personally have packaged cmd2 applications with all of the above and it worked without any issues and we didn't have to change any code to make it work. So from my perspective, this is primarily a design/implementation flaw in PyOxidizer. That being said, I would like cmd2 to be as compatible as possible with other tools in the Python ecosystem (though PyOxidizer appears to properly be part of the Rust ecosystem much more so than the Python one). It would be nice if we could support PyOxidizer, but it might be a distant future thing once we abandon support for earlier versions of Python prior to 3.7 or 3.8.

[jayvdb]

The key difference is that PyOxidizer pulls in libs and compiles them into the static binary. Nuitka also can do static binaries, but that isnt as integrated yet. (btw, you might be interested in trying staticx)
I guess cmd2 needs some readline symbols. Is there documentation about what it needs? Or does it really need a shared lib readline? Can it degrade gracefully without readline?

importlib.metadata has a backport importlib_metadata; likewise importlib.resources has backport importlib-resources, but pkgutil.get_data can also be used to be backwards compatible without the backport. I expect that pkg_resources will be solved soon in PyOxidizer (I have a hacky fix for it)

[kmvanbrunt]

@jayvdb cmd2 uses readline functions which are not exposed by the Python readline wrapper to enable some of our more advanced tab-completion features as well as asynchronous alert functions. The only way to access these functions is to call directly into readline's shared library.

How does PyOxidizer support ctypes.CDLL() and ctypes.LoadLibrary() functions which most likely call LoadLibrary() on Windows and dl_open() on Linux? Both of these OS functions load shared libraries from disk.

As @tleonhardt suggested, I can change the __file__ usage to importlib.util.find_spec('readline').origin, but will PyOxidizer support that?

[jayvdb]

In PyOxidizer, there is no shared readline lib. It is is statically linked into the binary sys.executable. The features cmd2 uses are in the binary, somewhere ;)

Could you give me an example symbol which is critical for cmd2, and I'll check how to get it.

[kmvanbrunt]

@jayvdb
Here are the symbols we use from the readline.so file.

• rl_basic_quote_characters
• rl_display_match_list
• rl_forced_update_display
• rl_display_fixed
• rl_point
• rl_set_prompt

[kmvanbrunt]

I'll add exception handling so cmd2 won't crash in a PyOxidizer environment. The CLI will be missing a lot of features, but it will work.