Mleap serializeToBundle error for Pyspark custom Transformer

[femibyte]

I have a Pyspark custom Transformer that I am trying to serialize to an mLeap bundle object for later model scoring but I’m getting the following error:

---> 42         self._java_obj.serializeToBundle(transformer._to_java(), path, dataset._jdf)
     43
     44     def deserializeFromBundle(self, path):

/usr/lib/spark/python/pyspark/ml/pipeline.py in _to_java(self)
    314         java_stages = gateway.new_array(cls, len(self.stages))
    315         for idx, stage in enumerate(self.stages):
--> 316             java_stages[idx] = stage._to_java()
    317
    318         _java_obj =\

AttributeError: 'MyCustomTransformer' object has no attribute '_to_java'

My custom transformer inherits from pyspark.ml.Transformer but it seems as if there needs to be a _to_java() implementation for the mleap serialization to work ?

What are my options ?

[koldkoder]

@femibyte did you find the solution to the issue?

[afsalthaj]

@femibyte Any solutions/hacks you found on this issue?

[ancasarb]

@femibyte @koldkoder @afsalthaj As we're starting to add support for spark-extensions transformers in Pyspark, I've created #570 to track this development, could you please follow that issue and add comments or questions to it regarding custom transformers in Pyspark? Thank you!