Omniosce merge, step 1

[pyhalov]

Well, this brings us python 3.5 IPS modules and automatic BE naming.

[citrus-it]

@pyhalov - you will need omniosorg@f724e9f
too (modified for the time: prefix changes)

[pyhalov]

Thanks

[pyhalov]

Python 2.7 test results:

# Ran 1070 tests in 12223.606s - skipped 9 tests.                  
Skipped tests:                                                          
cli.t_pkgdep.py TestPkgdepBasics.test_bug_12697: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                  
cli.t_pkgdep.py TestPkgdepBasics.test_bug_15843: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                  
cli.t_pkgdep.py TestPkgdepBasics.test_bug_17808: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                  
cli.t_pkgdep.py TestPkgdepBasics.test_bypass_full_paths: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                          
cli.t_pkgdep.py TestPkgdepBasics.test_elf_dependency_tags: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                        
cli.t_pkgdep.py TestPkgdepBasics.test_elf_warning: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                
cli.t_pkgdep.py TestPkgdepBasics.test_output: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                     
cli.t_pkgdep.py TestPkgdepBasics.test_relative_run_path: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                          
cli.t_pkg_help.py TestPkgHelp.test_help_character_encoding: Test Skipped: The test system must have the ja_JP.eucJP locale installed to run this test.                                                                                      
                                                                        
FAILED (successes=1068, failures=2, errors=0, mismatches=2)        
                                                                        
======================================================================
BASELINE MISMATCH: The following results didn't match the baseline.
----------------------------------------------------------------------
cli.t_pkgdep.py TestPkgdepBasics.test_bug_13059: fail                   
cli.t_pkg_linked.py TestPkgLinkedScale.test_li_scale: fail         
----------------------------------------------------------------------  

test_li_scale uses 32-bit python and runs out of file descriptors. test_bug_13059 is a bit trickier.
I don't understand how it's supposed to work. It resolves

#!/usr/bin/python

import pkg.indexer as indexer
import pkg.search_storage as ss
from pkg.misc import EmptyI

against image which doesn't contain any python packages (as well as proto area)
and fails with
unresolved dependency for search_storage.py (which exists on host, but not in image):

$ /usr/bin/python2.7 \
        /export/home/alp/srcs/oi-userland/components/openindiana/pkg/pkg/proto/root_i386/usr/bin/pkgdepend \
        -R /tmp/ips.test.17851/0/image0 resolve -v \
        /tmp/ips.test.17851/0/manifests/mfst.CDBR5v \
        /tmp/ips.test.17851/0/manifests/mfst.6B7UEW
/tmp/ips.test.17851/0/manifests/mfst.CDBR5v has unresolved dependency '
    depend type=require fmri=__TBD \
        pkg.debug.depend.reason=usr/lib/python3.5/vendor-packages/pkg/client/indexer.py \
        pkg.debug.depend.type=python \
        pkg.debug.depend.file=64/search_storage.abi3.so \
        pkg.debug.depend.file=64/search_storage.cpython-35m.so \
        pkg.debug.depend.file=64/search_storage.so \
        pkg.debug.depend.file=search_storage.abi3.so \
        pkg.debug.depend.file=search_storage.cpython-35m.so \
        pkg.debug.depend.file=search_storage.py \
        pkg.debug.depend.file=search_storage.pyc \
        pkg.debug.depend.file=search_storage.pyo \
        pkg.debug.depend.file=search_storage.so \
        pkg.debug.depend.file=search_storage/__init__.py \
        pkg.debug.depend.path=export/home/alp/.local/lib/python3.5/site-packages/pkg \
        pkg.debug.depend.path=export/home/alp/srcs/oi-userland/components/openindiana/pkg/pkg/src/tests/pkg \
        pkg.debug.depend.path=usr/lib/python3.5/lib-dynload/pkg \
        pkg.debug.depend.path=usr/lib/python3.5/pkg \
        pkg.debug.depend.path=usr/lib/python3.5/plat-sunos5/pkg \
        pkg.debug.depend.path=usr/lib/python3.5/site-packages/pkg \
        pkg.debug.depend.path=usr/lib/python3.5/vendor-packages/pkg \
        pkg.debug.depend.path=usr/lib/python3.5/vendor-packages/pkg/client/pkg \
        pkg.debug.depend.path=usr/lib/python35.zip/pkg'.

[pyhalov]

We have some nasty issue with python 3.5 modules:

>>> import pkg.actions as action
>>> action.fromstr("file path=usr/bin")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.5/vendor-packages/pkg/actions/file.py", line 867, in __init__
    _common._file_init(self, data, **attrs)
KeyError: 'pkg.actions'

The same works with python2.7

[pyhalov]

It seems I know a cure (which is likely a trouble by itself):
Instead of

#if PY_MAJOR_VERSION >= 3
#ifdef Py_LIMITED_API
        #define PyBytes_AS_STRING PyUnicode_AsUTF8
        #define PyBytes_AsString PyUnicode_AsUTF8
        #define PyBytes_GET_SIZE PyUnicode_GET_LENGTH
        #define PyBytes_FromStringAndSize PyUnicode_FromStringAndSize
#endif
#endif

we can use:

#if PY_MAJOR_VERSION >= 3
        #undef PyBytes_AS_STRING
        #undef PyBytes_AsString
        #undef PyBytes_GET_SIZE
        #undef PyBytes_FromStringAndSize
        #define PyBytes_AS_STRING PyUnicode_AsUTF8
        #define PyBytes_AsString PyUnicode_AsUTF8
        #define PyBytes_GET_SIZE PyUnicode_GET_LENGTH
        #define PyBytes_FromStringAndSize PyUnicode_FromStringAndSize
#endif

As using PyBytes_AsString gives
"TypeError: expected bytes, str found"
in the following code:

if ((path = PyBytes_AsString(path_attr)) != NULL) {
                if (path[0] == '/') {
                        PyObject *stripped = PyObject_CallMethod(
                            path_attr, "lstrip", "(s)", "/");
                        if (stripped == NULL) {
                                Py_DECREF(key_aname);
                                return (NULL);
                        }
                        if (PyDict_SetItemString(attrs, "path",
                            stripped) == -1) {
                                Py_DECREF(key_aname);
                                Py_DECREF(stripped);
                                return (NULL);
                        }
                        if (PyBytes_GET_SIZE(stripped) == 0)
                                invalid_path = 1;
                        Py_DECREF(stripped);
                } else {
                        if (PyBytes_GET_SIZE(path_attr) == 0)
                                invalid_path = 1;
                }
        } else {
                /* path attribute is not a string. */
                invalid_path = 1;                    /**************** ISSUE IS HERE ****/
        }

[pyhalov]

Python3 test results:

# Ran 1070 tests in 13988.990s - skipped 9 tests.
Skipped tests:
cli.t_pkgdep.py TestPkgdepBasics.test_bug_12697: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                  
cli.t_pkgdep.py TestPkgdepBasics.test_bug_15843: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                  
cli.t_pkgdep.py TestPkgdepBasics.test_bug_17808: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                  
cli.t_pkgdep.py TestPkgdepBasics.test_bypass_full_paths: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                          
cli.t_pkgdep.py TestPkgdepBasics.test_elf_dependency_tags: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                        
cli.t_pkgdep.py TestPkgdepBasics.test_elf_warning: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                
cli.t_pkgdep.py TestPkgdepBasics.test_output: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                                     
cli.t_pkgdep.py TestPkgdepBasics.test_relative_run_path: Test Skipped: No suitable GCC compiler found. Tried: ['$CC', '/usr/bin/gcc', 'gcc'].  Try setting $CC to a validcompiler.                                                          
cli.t_pkg_help.py TestPkgHelp.test_help_character_encoding: Test Skipped: The test system must have the ja_JP.eucJP locale installed to run this test.                                                                                      

FAILED (successes=1067, failures=3, errors=0, mismatches=10)

======================================================================
BASELINE MISMATCH: The following results didn't match the baseline.
----------------------------------------------------------------------
cli.t_pkgrepo.py TestPkgRepo.test_03_info: pass
cli.t_depot_config.py TestHttpDepot.test_12_htpkgclient: fail
cli.t_pkg_depotd.py TestDepotOutput.test_0_depot_bui_output: pass
cli.t_pkg_install.py TestPkgInstallApache.test_corrupt_web_cache: pass
cli.t_pkg_install.py TestPkgInstallApache.test_granular_proxy: pass
cli.t_pkg_install.py TestPkgInstallLicense.test_01_install_update: fail
cli.t_https.py TestDepotHTTPS.test_01_basics: pass
cli.t_pkgrecv.py TestPkgrecvHTTPS.test_01_basics: pass
cli.t_pkgrepo.py TestPkgrepoHTTPS.test_01_basics: pass
cli.t_pkgsend.py TestPkgsendHTTPS.test_01_basics: pass
----------------------------------------------------------------------

[pyhalov]

The suspicious failures are

cli.t_depot_config.py TestHttpDepot.test_12_htpkgclient: fail
cli.t_pkg_install.py TestPkgInstallLicense.test_01_install_update: fail

First one fails with

 /usr/bin/python3.5 /tmp/ips.test.13326/fakeroot/pkg -R \
        /tmp/ips.test.13326/0/image0 -D plandesc_validate=1 -D \
        manifest_validate=Always -D \
        smf_cmds_dir=/tmp/ips.test.13326/0/smf_cmds set-publisher -g \
        http://localhost:12004/default/carrots carrots
pkg set-publisher: The requested operation requires that one or more package repositories are
configured for publisher 'test2'.

Second - with

Traceback (most recent call last):
  File "./pkg5unittest.py", line 758, in run
    testMethod()
  File "./api/../cli/t_pkg_install.py", line 8677, in test_01_install_update
    "copyright.licensed", False, True]]])
  File "./pkg5unittest.py", line 1069, in assertEqualParsable
    image_name, k, ev, outd[k]))
AssertionError: Lists differ: [['pk[45 chars]0Z', None, ['pkg://bobcat/licensed@1.2,5.11-0:[349 chars]ue]]] != [['pk[45 chars]0Z', ['pkg://bobcat/licensed@1.0,5.11-0:201903[349 chars]se]]]

First differing element 0:
['pkg[44 chars]0Z', None, ['pkg://bobcat/licensed@1.2,5.11-0:[68 chars]lse]]
['pkg[44 chars]0Z', ['pkg://bobcat/licensed@1.0,5.11-0:201903[178 chars]rue]]

  [['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z',
-   None,
-   ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z',
-    'license.licensed',
-    'license.licensed',
-    False,
-    False]],
-  ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z',
    ['pkg://bobcat/licensed@1.0,5.11-0:20190307T075959Z',
     'copyright.licensed',
     'copyright.licensed',
     False,
     False],
    ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z',
     'copyright.licensed',
     'copyright.licensed',
     False,
-    True]]]
?          ^

+    True]],
?          ^

+  ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z',
+   None,
+   ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z',
+    'license.licensed',
+    'license.licensed',
+    False,
+    False]]] : In image None, the value of licenses was expected to be
[['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', None, ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', 'license.licensed', 'license.licensed', False, False]], ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', ['pkg://bobcat/licensed@1.0,5.11-0:20190307T075959Z', 'copyright.licensed', 'copyright.licensed', False, False], ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', 'copyright.licensed', 'copyright.licensed', False, True]]] but was
[['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', ['pkg://bobcat/licensed@1.0,5.11-0:20190307T075959Z', 'copyright.licensed', 'copyright.licensed', False, False], ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', 'copyright.licensed', 'copyright.licensed', False, True]], ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', None, ['pkg://bobcat/licensed@1.2,5.11-0:20190307T080000Z', 'license.licensed', 'license.licensed', False, False]]]

I don't see any evident error here. As this concerns only Python3 modules, which currently are not used by pkg tools explicitly, I'm going to merge this.