Core ML: Predicted feature named '___' was not output by pipeline

[johnlev]

I successfully exported a Boosted Trees Classifier model to a Core ML model and imported it into a Xcode project. All seemed well until I tried using it in Swift. Whenever I run the predict function (even with valid inputs) I receive an error: "Predicted feature named 'room' was not output by pipeline" where room is my target name. Is this a Core ML issue or did I export it wrong from Turi Create?

sf = tc.SFrame.read_csv('all.csv')
model = tc.boosted_trees_classifier.create(sf, target='room')
model.export_coreml("Model.mlmodel")

For context, the top 10 lines of the SFrame:

+-------------+-------------+-------------+-------------+-------------+------+
| b1 | b2 | b3 | b4 | b5 | room |
+-------------+-------------+-------------+-------------+-------------+------+
| 15.40925007 | 2.448436747 | 0.875747109 | 3.116232715 | 0.55931885 | 3 |
| 12.34282797 | 2.448436745 | 1.050529028 | 2.49309418 | 0.547421866 | 3 |
| 11.35115228 | 2.593769273 | 1.314364103 | 2.327012418 | 0.56737106 | 3 |
| 10.90912658 | 2.76689536 | 1.639642453 | 2.284626409 | 0.691871329 | 3 |
| 10.96834609 | 3.053793429 | 1.953247127 | 2.252586412 | 0.722658468 | 3 |
| 10.62518757 | 3.562255003 | 2.129856143 | 1.912806725 | 0.668820263 | 3 |
| 10.34918951 | 3.569631802 | 2.27094845 | 1.708808372 | 0.798663933 | 3 |
| 10.12294684 | 3.359425223 | 2.446464643 | 1.805624742 | 0.995895032 | 3 |
| 10.65090275 | 3.216177055 | 2.655107978 | 1.88591092 | 1.264108161 | 3 |
| 10.88905096 | 3.566873863 | 2.684348503 | 1.888478765 | 1.325077843 | 3 |
+-------------+-------------+-------------+-------------+-------------+------+

[znation]

@johnlev Could you please provide detailed steps to reproduce the issue? I tried with the following steps, with synthetic data based on what you pasted above: repro.txt

# Create the model
import turicreate as tc
sf = tc.SFrame.read_csv('repro.txt')
model = tc.boosted_trees_classifier.create(sf, target='room')
model.export_coreml("Model.mlmodel")

# Make predictions
import coremltools
model = coremltools.models.MLModel('Model.mlmodel')
model.predict({'b1': 1, 'b2': 2, 'b3': 3, 'b4': 4, 'b5': 5})

Gives output with no error:

{u'room': 3L, u'roomProbability': {3L: 0.5, 4L: 0.5}}

[johnlev]

Oh, Python is not the problem. Running it with coremltools works fine, but importing into an Xcode project is where it fails to predict

[znation]

Interesting. Let me see if I can repro the issue there.

[znation]

@johnlev I'm still unable to repro. In Swift in Xcode, I tried the following:

let model = Model()
let output = try! model.prediction(b1: 1, b2: 3, b3: 43, b4: 2, b5: 10)
dump(output)

And it gives me:

▿ repro.ModelOutput #0
  - room: 3
  ▿ roomProbability: 2 key/value pairs
    ▿ (2 elements)
      - key: 3
      - value: 0.5
    ▿ (2 elements)
      - key: 4
      - value: 0.5

What version of Xcode are you using, and what version of macOS are you running? It's possible this bug is specific to a particular version.

[johnlev]

I am on 10.13.2 using Xcode 9.2. I tried using your repro.txt and it worked just fine, but using the same code but with the expanded dataset I again encountered the error. Any reason you rounded the numbers in repro.txt?

[znation]

@johnlev Interesting, I see! Not sure when or how the numbers got rounded (somewhere in copy/paste the way I made a CSV) but I doubt that would affect it. Can you by any chance share your full training data (or a subset of it that also results in the same issue), in CSV format, as a .txt file here so I can reproduce the issue? Thanks!

[johnlev]

Sure:
all2.txt

[znation]

Thanks @johnlev, I'm able to reproduce the issue with the model trained on all2.txt. I'm not sure yet what the cause is but I'm able to see that this model gives predictions in Python but gives errors when called from Swift. I'll investigate further.

[johnlev]

I might be confused, but after upgrading to 4.1, regenerating the mlmodel, and replacing it in my project, I am still experiencing the same error:

[znation]

Thanks @johnlev, looks like this did not in fact get fixed in 4.1. Sorry for the miscommunication. I'll reopen and we'll make sure the fix gets into 4.2.

[johnlev]

Thanks @znation. Looking forward to using Turicreate in my project once this is done

[srikris]

@johnlev We have traced down the bug to the presence of a byte-ordered mark in your file. This is corrupting the read_csv of SFrame resulting in columns that contain non-ascii characters

sf.column_names()
['\xef\xbb\xbfb1', 'b2', 'b3', 'b4', 'b5', 'room']

The simplest work around is for you to remove the marker from the file which you can do with a simple sed script.

awk 'NR==1{sub(/^\xef\xbb\xbf/,"")}1' all2.txt > all2_byte_ordered.txt

We will be fixing the underlying issue in #227. Hopefully that should unblock you to use Turi Create in your application.