When MacOS is updated from Homebrew any Python2 YDK application fails

[ygorelik]

Current Behavior

When MacOS is updated from Homebrew, which is normal and expected behavior, any run of YDK Python2 based application fails with error:

Fatal Python error: PyThreadState_Get: no current thread

This is known and well documented issue, which has few suggested workaround processes. These solution must be tested on MacOS platform and well documented in YDK documentation along with suggested workaround.

The symptoms and workaround are also documented here.

Steps to Reproduce

Update MacOS from Homebrew using command:

$ brew update

Install ydk and IOS XR bundle:

$ pip install ydk
$ pip install ydk-models-cisco-ios-xr

Copy hello-ydk.py from below and try to run it:

$ python  hello-ydk.py

Observe the error.

Your script

# import providers, services and models
from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider
from ydk.models.cisco_ios_xr import Cisco_IOS_XR_shellutil_oper as xr_shellutil_oper
from datetime import timedelta

def enable_logging(level):
    import logging
    log = logging.getLogger('ydk')
    log.setLevel(level)
    handler = logging.StreamHandler()
    formatter = logging.Formatter(("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
    handler.setFormatter(formatter)
    log.addHandler(handler)

if __name__ == "__main__":
    """Main execution path"""

    enable_logging(logging.INFO)
    
    # create NETCONF session
    provider = NetconfServiceProvider(address="192.168.122.169",
                                      port=830,
                                      username="admin",
                                      password="admin",
                                      protocol="ssh")
    # create CRUD service
    crud = CRUDService()

    # create system time object
    system_time = xr_shellutil_oper.SystemTime()

    # read system time from device
    system_time = crud.read(provider, system_time)

    # print system uptime
    print("System uptime is " +
          str(timedelta(seconds=system_time.uptime.uptime)))

    exit()

Logs

ANTHONYP-M-K0ZN:ydk-py-samples anthonyp$ ./hello-ydk.py 
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

System Information

YDK-0.7.3 + cisco-ios-xr bundle

[ghost]

To fix this run the below commands:

pip uninstall ydk
export CMAKE_LIBRARY_PATH=/Library/Frameworks/Python.framework/Versions/2.7/lib/
pip install ydk

The issue is caused because macOS comes with a default python 2.7 library installed under /System. We need to link to the python 2.7 library under /System when building ydk

 cd /
/ > locate libpython2.7.dylib
/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib # Library we want to link to
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib # Default python installation we don't want to link to
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
/usr/lib/libpython2.7.dylib # Links to default python installation

/> ls -l /usr/lib/libpython2.7.dylib
lrwxr-xr-x  1 root  wheel  68 Nov 20 18:43 /usr/lib/libpython2.7.dylib@ -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/Python

[ghost]

Need to add update the documentation and add it to Travis run

[ygorelik]

The approach above did not work on MacOS virtual machine.
Another approach assumes use of Python2 virtual environment. To install and create virtual environment::

sudo pip install virtualenv virtualenvwrapper
source /usr/local/bin/virtualenvwrapper.sh
mkvirtualenv -p python2 ydk_venv
cd ydk-py
pip install -r requirements.txt
deactivate

To activate virtual environment::

source ~/.virtualenvs/ydk_venv/bin/activate

This approach is tested and proven to work in Mac VM. But it fails in Travis tests with intermittent segmentation fault.

[ygorelik]

The following section was added to ydk-gen and ydk-py documentation:

Mac OS
======

The developers of Python2 on Mac OS might face an issue ([#837](https://github.com/CiscoDevNet/ydk-gen/issues/837)). 
This is well known and documented issue. Each developer might have different approaches for its resolution. 
One of them is to use Python2 virtual environment. See section below for details.

In addition it is required properly set CMAKE_LIBRARY_PATH environment variable to assure that `cmake` uses correct Python library. 
Follow these steps to find and set correct library path.

1. Find installations of `libpython2.7` library:

  locate libpython2.7.dylib
  
Example:

  $ locate libpython2.7.dylib
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
  /usr/lib/libpython2.7.dylib
  /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
  /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
  
2. Choose non-system Python library installation and set CMAKE_LIBRARY_PATH before any YDK component installation. Example:

  export CMAKE_LIBRARY_PATH=/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib

3. Run YDK core package installation with '-v' flag to check that `PythonLibs` points to correct library path. Example:

  $ ./generate.py --core
  $ pip install -v gen-api/python/ydk/dist/ydk*.tar.gz
  
  # In 'cmake' log look for 'PythonLibs' and 'found version' setting
  -- Found PythonLibs: /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib (found version "2.7.15")

4. Finally test you YDK core library installation from CLI, making sure there are no errors:

  $ python -c "import ydk.types"