Unvendor pyobj + fix py2->py3 compatability fixes. #3574

Code0x58 · 2020-07-19T20:52:51Z

Re. https://heronstreaming.slack.com/archives/C0VKQLKV0/p1595162763201800

This also uses the longer DataWatcher handler signature to avoid a TypeError, which it relies on to indicate it should retry with the shorter signature.

Also sorts #3575

joshfischer1108 · 2020-07-20T02:53:55Z

heron/statemgrs/src/python/zkstatemanager.py

+    def watch_topology(data, stats, event):
      """ watch topology """
      if data:
        topology = Topology()


I think I see the problem (mismatch number of passed arguments) you are trying to address.

"/Users/someUser/.pex/installed_wheels/7b3448a0635a4c114f98f11df06af06d250c297a/kazoo-2.7.0-py2.py3-none-any.whl/kazoo/recipe/watchers.py", line 162, in _log_func_exception result = self._func(data, stat, event) TypeError: watch_execution_state() takes 2 positional arguments but 3 were given

I'm not sure I follow the reasoning of adding event to each of the method signatures. If it is never used, why do we need to pass it at all?

I think it helps by avoiding the questionable pattern here which can obscure the exceptions when they occur within the handler - at least we certainly won't see the TypeError now even if the parameter is junk (no more "exception occurred while handling exception", so that much easier to read)

Ok. Thanks for clarifying. Makes sense to me.

thinker0 · 2020-07-20T12:35:06Z

@Code0x58

TypeError: 'float' object cannot be interpreted as an integer

[2020-07-20 21:33:24 +0900] [INFO]: Setting topology info for topology: heron-lass_win_loss_notice_log-to-ladp-log-es
[2020-07-20 21:33:24 +0900] [ERROR]: ==Oops state dump=============
[2020-07-20 21:33:24 +0900] [ERROR]: References: [[java.util.HashMap:0x507DAC1], <javaobj:java.util.HashMap>, 'NJ_INSTANCE_PHASE', 'BETA', 'LD_LIBRARY_PATH', '${LD_LIBRARY_PATH}:./lib64:/usr/lib/hadoop/lib/native', 'SENTRY_DSN', 'https://55d2bf90467e4814908d94c10d154229@lad-sentry.line-apps.com/3?sample.rate=0.10', 'SERVER_ENV', 'beta', 'NJ_INSTANCE_ID', 'heron-lass_win_loss_notice_log-to-ladp-log-es', 'SENTRY_ENVIRONMENT']
[2020-07-20 21:33:24 +0900] [ERROR]: Stream seeking back at -16 byte (2nd line is an actual position!):
[2020-07-20 21:33:24 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 221, in readObject
    _, res = self._read_and_exec_opcode(ident=0)
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 256, in _read_and_exec_opcode
    return (opid, handler(ident=ident))
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 428, in do_object
    tmp_object = transformer.transform(java_object)
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 625, in transform
    for i in range((len(obj.annotations)-1)/2):
TypeError: 'float' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/tracker.py", line 62, in _convert_java_value
    pobj = javaobj.loads(kv.serialized_value)
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 65, in loads
    return marshaller.readObject()
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 235, in readObject
    self._oops_dump_state()
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/javaobj.py", line 545, in _oops_dump_state
    self.object_stream.seek(-16, mode=1)
TypeError: seek() takes no keyword arguments
[2020-07-20 21:33:24 +0900] [ERROR]: Caught exception while triggering callback: 'in <string>' requires string as left operand, not int
[2020-07-20 21:33:24 +0900] [INFO]: Unregister a watch with uid: 0ce59fcd-9c28-473e-94d7-f09aebc29085
[2020-07-20 21:33:24 +0900] [INFO]: Adding data watch for path: /heron/tmasters/heron-lass_win_loss_notice_log-to-ladp-log-es

Code0x58 · 2020-07-20T12:45:48Z

thanks @thinker0, I just pushed a quick amendment that should resolve the /→// issue. There is still be the separate seek issue which I think is because the underlying (C) stream type doesn't implement seek, but that hopefully only shows itself when when things blow up.

thinker0 · 2020-07-20T12:55:46Z

@Code0x58 another

[2020-07-20 21:33:24 +0900] [INFO]: Setting topology info for topology: heron-ladp_ladm_perf_adgroup_breakdown_report-kafka-to-ladp-log-es-eco
[2020-07-20 21:33:24 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/tracker.py", line 77, in _convert_java_value
    'raw' : utils.hex_escape(kv.serialized_value)}
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/utils.py", line 44, in hex_escape
    return ''.join(ch if ch in printable else r'0x{0:02x}'.format(ord(ch)) for ch in bin_str)
  File "/Users/thinker0/.pex/code/20c8103cc4fd30b436b4787554e90a405988f63f/heron/tools/tracker/src/python/utils.py", line 44, in <genexpr>
    return ''.join(ch if ch in printable else r'0x{0:02x}'.format(ord(ch)) for ch in bin_str)
TypeError: 'in <string>' requires string as left operand, not int
[2020-07-20 21:33:24 +0900] [ERROR]: Caught exception while triggering callback: 'in <string>' requires string as left operand, not int
[2020-07-20 21:33:24 +0900] [INFO]: Unregister a watch with uid: e56c8661-0da9-4793-a2a8-60d733e74528
[2020-07-20 21:33:24 +0900] [INFO]: Adding data watch for path: /heron/tmasters/heron-ladp_ladm_perf_adgroup_breakdown_report-kafka-to-ladp-log-es-eco

Code0x58 · 2020-07-20T13:13:39Z

thanks again @thinker0, I pushed another amendment. The amendment isn't 1-to-1 with the python2 implementation, but it's probably saner. I don't know if it will have a knock on effect down the line 🤞

thinker0 · 2020-07-20T13:40:42Z

@Code0x58 Another

[2020-07-20 22:40:03 +0900] [INFO]: Setting topology info for topology: heron-ladp_oa_message_id_report-to-ladp-log-es
[2020-07-20 22:40:03 +0900] [ERROR]: ==Oops state dump=============
[2020-07-20 22:40:03 +0900] [ERROR]: References: [[com.typesafe.config.impl.SerializedConfigValue:0x0], <javaobj:com.typesafe.config.impl.SerializedConfigValue>]
[2020-07-20 22:40:03 +0900] [ERROR]: Stream seeking back at -16 byte (2nd line is an actual position!):
[2020-07-20 22:40:03 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 221, in readObject
    _, res = self._read_and_exec_opcode(ident=0)
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 256, in _read_and_exec_opcode
    return (opid, handler(ident=ident))
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 420, in do_object
    opcode, obj = self._read_and_exec_opcode(ident=ident+1)
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 255, in _read_and_exec_opcode
    raise RuntimeError("Unknown OpCode in the stream: 0x%x" % opid)
RuntimeError: Unknown OpCode in the stream: 0x7a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/tracker.py", line 62, in _convert_java_value
    pobj = javaobj.loads(kv.serialized_value)
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 65, in loads
    return marshaller.readObject()
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 235, in readObject
    self._oops_dump_state()
  File "/Users/thinker0/.pex/code/89ddc931e94debf0336dd56177721e9f35b441e3/heron/tools/tracker/src/python/javaobj.py", line 545, in _oops_dump_state
    self.object_stream.seek(-16, mode=1)
TypeError: seek() takes no keyword arguments
[2020-07-20 22:40:03 +0900] [INFO]: Adding new topology: heron-system-access-es-a-line-me, state_manager: lad

Code0x58 · 2020-07-20T13:47:43Z

humm, that error looks like it could be a deeper pyobj issue, so I should probably stop trying to patch it, and just upgrade pyobj for #3575 - I'll be able to give it more attention tonight

Code0x58 · 2020-07-20T20:46:48Z

CI passed on my machine, so hopefully the switch to the maintained version of pyobj sorts things out!

thinker0 · 2020-07-20T23:24:17Z

@Code0x58 sorry. The last one goes like this.

Other features seem to work fine.

[2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaMap' object has no attribute 'is_primitive'
[2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaMap' object has no attribute 'is_primitive'
[2020-07-21 08:21:52 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaObject' object has no attribute 'is_primitive'


[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaBool' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaBool' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaBool' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'
[2020-07-21 09:29:25 +0900] [ERROR]: Failed to parse data as java object
Traceback (most recent call last):
  File "/Users/thinker0/.pex/code/33e1a410e8efde46fe6074757a0631f8eb7c7d3b/heron/tools/tracker/src/python/tracker.py", line 65, in _convert_java_value
    if pobj.is_primitive():
AttributeError: 'JavaInt' object has no attribute 'is_primitive'

Code0x58 · 2020-07-21T00:06:48Z

Thanks again @thinker0! So much for hopes of a dumb upgrade. I'll have to spend some time getting to understand the old and new javaobj modules to do a proper migration for convert_pb_kvs(...), which I'll aim to get on tomorrow evening

This also uses the longer DataWatcher handler signature to avoid a TypeError, which it relies on to indicate it should retry with the shorter signature. Without this change, exceptions can be reported messily due to a raise in the except path being reasonably expected.

Code0x58 · 2020-07-21T22:13:48Z

an initial look into convert_pb_kvs(...) makes it me think it's a pretty hairy thing, so I didn't try to do anything smart and pushed what might get things running. I'll keep having a look through the file to see if I can work out how to clean it up, but I'm not optimistic

thinker0 · 2020-07-21T23:30:19Z

+1
really good work.
Approved

This also uses the longer DataWatcher handler signature to avoid a TypeError, which it relies on to indicate it should retry with the shorter signature. Without this change, exceptions can be reported messily due to a raise in the except path being reasonably expected.

Code0x58 mentioned this pull request Jul 19, 2020

Unvendor javaobj #3575

Closed

Code0x58 changed the title ~~Decode string when deserializing Jaba object in python~~ Decode string when deserializing Java object in python Jul 19, 2020

Code0x58 force-pushed the py3-upgrade-issue branch from 89e112e to f483c67 Compare July 20, 2020 00:45

nicknezis requested a review from joshfischer1108 July 20, 2020 01:02

joshfischer1108 reviewed Jul 20, 2020

View reviewed changes

huijunwu approved these changes Jul 20, 2020

View reviewed changes

Code0x58 force-pushed the py3-upgrade-issue branch from f483c67 to 763c82a Compare July 20, 2020 12:44

Code0x58 force-pushed the py3-upgrade-issue branch from 763c82a to cf3a3e6 Compare July 20, 2020 13:12

Code0x58 force-pushed the py3-upgrade-issue branch from cf3a3e6 to f23265f Compare July 20, 2020 13:45

Code0x58 force-pushed the py3-upgrade-issue branch from f23265f to 548635a Compare July 20, 2020 18:52

Code0x58 changed the title ~~Decode string when deserializing Java object in python~~ Unvendor pyobj + fix py2->py3 compatability fixes. Jul 20, 2020

Code0x58 force-pushed the py3-upgrade-issue branch 2 times, most recently from 1ec15dd to e74fa85 Compare July 20, 2020 19:33

Code0x58 force-pushed the py3-upgrade-issue branch from e74fa85 to 0cf3781 Compare July 21, 2020 21:55

nicknezis self-requested a review July 22, 2020 00:06

nicknezis approved these changes Jul 22, 2020

View reviewed changes

thinker0 approved these changes Jul 22, 2020

View reviewed changes

nicknezis merged commit 82ee354 into apache:master Jul 22, 2020

Code0x58 deleted the py3-upgrade-issue branch July 22, 2020 17:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unvendor pyobj + fix py2->py3 compatability fixes. #3574

Unvendor pyobj + fix py2->py3 compatability fixes. #3574

Code0x58 commented Jul 19, 2020 •

edited

joshfischer1108 Jul 20, 2020

Code0x58 Jul 20, 2020 •

edited

joshfischer1108 Jul 20, 2020

thinker0 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020 •

edited

thinker0 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020

thinker0 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020

thinker0 commented Jul 20, 2020 •

edited

Code0x58 commented Jul 21, 2020

Code0x58 commented Jul 21, 2020

thinker0 commented Jul 21, 2020 •

edited

Unvendor pyobj + fix py2->py3 compatability fixes. #3574

Unvendor pyobj + fix py2->py3 compatability fixes. #3574

Conversation

Code0x58 commented Jul 19, 2020 • edited

joshfischer1108 Jul 20, 2020

Choose a reason for hiding this comment

Code0x58 Jul 20, 2020 • edited

Choose a reason for hiding this comment

joshfischer1108 Jul 20, 2020

Choose a reason for hiding this comment

thinker0 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020 • edited

thinker0 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020

thinker0 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020

Code0x58 commented Jul 20, 2020

thinker0 commented Jul 20, 2020 • edited

Code0x58 commented Jul 21, 2020

Code0x58 commented Jul 21, 2020

thinker0 commented Jul 21, 2020 • edited

Code0x58 commented Jul 19, 2020 •

edited

Code0x58 Jul 20, 2020 •

edited

Code0x58 commented Jul 20, 2020 •

edited

thinker0 commented Jul 20, 2020 •

edited

thinker0 commented Jul 21, 2020 •

edited