Python 2.7 exploration

Exploration of using Python 2.7 instead of Python 3.5

This wiki contains the notes from my exploration of using Python 2.7 instead of the dependency on Python 3.5

Test Status

Currently running ant target test.application.api in test/python -- 23 tests with 2 failures. The two failures are both related to "import" differences.

ERROR: test_TopologyImportCommonNamespacePackage (main.TestTopologyMethods)

 [exec] Traceback (most recent call last):
 [exec]   File "test1.py", line 199, in test_TopologyImportCommonNamespacePackage
 [exec]     import common_namespace.module1
 [exec] ImportError: No module named common_namespace.module1

ERROR: test_TopologyImportNamespacePackage (main.TestTopologyMethods)

 [exec] Traceback (most recent call last):
 [exec]   File "test1.py", line 185, in test_TopologyImportNamespacePackage
 [exec]     from test_namespace_package.test_subpackage import test_module as test_ns_module
 [exec] ImportError: No module named test_namespace_package.test_subpackage

Discovery/Changes so far

Python differences

• encoding parameter on tempfile.NamedTemporaryFile() is not valid com.ibm.streamsx.topology/opt/python/packages/streamsx/topology/context.py tf = tempfile.NamedTemporaryFile(mode="w+t", suffix=".json", prefix="splpytmp", delete=False)

• print with flush parameter not supported com.ibm.streamsx.topology/opt/python/packages/streamsx/topology/functions.py

  • print(v, flush=True)
  • #2.7 print(v, flush=True)
  • print(v)
  • sys.stdout.flush()

• enum not supported com.ibm.streamsx.topology/opt/python/packages/streamsx/topology/schema.py @@ -1,4 +1,6 @@ -import enum +#2.7 import enum +def enum(**enums):

  • return type('Enum', (), enums)

  class StreamSchema(object) : """SPL stream schema.""" @@ -26,8 +28,8 @@ class StreamSchema(object) :

  XML = StreamSchema("tuple")

  -@enum.unique -class CommonSchema(enum.Enum): +#2.7 @enum.unique +class CommonSchema(enum()):

• Note: Python 3.4 enums in 2.7: https://pypi.python.org/pypi/enum34 and http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python

• queue package is Queue com.ibm.streamsx.topology/opt/python/packages/streamsx/topology/topology.py @@ -7,9 +7,9 @@ from streamsx.topology import schema import streamsx.topology.functions import json import threading -import queue +import Queue import time -from enum import Enum +#2.7 from enum import Enum

  class Topology(object):
  

  @@ -370,12 +370,6 @@ class Stream(object): oport = op.addOutputPort() return Stream(self.topology, oport)

  • def print(self):

  •    """
    

  •    Prints each tuple to stdout flushing after each tuple.
    

  •    :returns: None
    

  •    """
    

  •    self.sink(streamsx.topology.functions.print_flush)
    

    def publish(self, topic, schema=schema.CommonSchema.Python): """ @@ -412,7 +406,8 @@ class Stream(object): op = self.topology.graph.addOperator("com.ibm.streamsx.topology.topic::Publish", params=publishParams) op.addInputPort(outputPort=self.oport)

  -class Routing(Enum):

  • #2.7 +class Routing(schema.enum()): ROUND_ROBIN=1 KEY_PARTITIONED=2 HASH_PARTITIONED=3 @@ -425,7 +420,7 @@ class View(threading.Thread): def init(self, name, port, buffer_time, sample_size): super(View, self).init() self._stop = threading.Event()

  •    self.items = queue.Queue()
    

  •    self.items = Queue.Queue()
    

• python vs python3 test/python/build.xml changed to use python instead of python3

  com.ibm.streamsx.topology/opt/python/templates/common/pyversion.sh changed to use python-config instead of python3-config

  com.ibm.streamsx.topology/opt/python/include/splpy.h @@ -53,12 +53,14 @@ namespace streamsx { */ inline void pyAttributeFromPyObject(SPL::rstring & attr, PyObject * value) { Py_ssize_t size = 0;

  •  char * bytes = PyUnicode_AsUTF8AndSize(value, &size);          
    

  •  // 2.7 char * bytes = PyUnicode_AsUTF8AndSize(value, &size);          
    

  •  PyObject *utf8String = PyUnicode_AsUTF8String(value); 
    

  •  char *bytes = PyString_AsString(utf8String); 
     if (bytes == 0) {
        SPLAPPTRC(L_ERROR, "Python can't convert to UTF-8!", "python");
        throw;
     }
    

  •  attr.assign((const char *)bytes, (size_t) size);
    

  •  attr.assign((const char *)bytes, strlen((const char *)bytes)); 
    

    }

    /**************************************************************/ @@ -162,7 +164,8 @@ namespace streamsx { }

  •    std::string pyLib("libpython3.5m.so");
    

  •    //2.7 std::string pyLib("libpython3.5m.so");
    

  •    std::string pyLib("libpython2.7.so");
       char * pyHome = getenv("PYTHONHOME");
       if (pyHome != NULL) {
           std::string wk(pyHome);