Support for ktrain v0.26.x with tf_model and tf_preproc

[NicoleJaneway]

Hey @amaiya, hoping you can help me refactor my NER app so it works with ktrain v0.26.x.

Background: in February 2021, I had a Streamlit app that was using a model.h5, model.json, and preproc.sav file from training an NER model based on Bio_ClinicalBERT. Unfortunately, when I run the model this month, I get the following error: 404 Client Error: Not Found for url: https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT/resolve/main/tf_model.h5.

Approach: in response, I thought retraining with the upgraded strain v0.26.x might be required to get the app to work. I created a tf_model.h5 and tf_model.preproc file, and updated the app.

Current issue: now when I run the app, I get the following error: invalid load key, '\xef'. Based on the line: features = pickle.load(open('tf_model.preproc', 'rb'))

Thoughts for how to proceed?

[amaiya]

Hi @NicoleJaneway

As mentioned in this post, the 404 error is simply saying that there is no TensorFlow version (tf_model.h5) for this model in which case a PyTorch version will automatically be downloaded and loaded into a TensorFlow model (which requires PyTorch to be installed). In the next version of ktrain, there will be additional warning messages with more information to help avoid confusion in these situations.

Also, the 404 error is completely unrelated to your invalid load key, '\xef' error. I was not able to reproduce your loading problem using the latest version of ktrain.

Are you able to load the Predictor successfully in a Jupyter notebook outside of your app? Does the following code work for you (using the CoNLL 2003 dataset):

import ktrain
from ktrain import text
TDATA = 'data/conll2003/train.txt'
VDATA = 'data/conll2003/valid.txt'
(trn, val, preproc) = text.entities_from_conll2003(TDATA, val_filepath=VDATA)
model = text.sequence_tagger('bilstm-bert', preproc, bert_model='emilyalsentzer/Bio_ClinicalBERT')
learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=128)
learner.fit(0.01, 1, cycle_len=1)
predictor = ktrain.get_predictor(learner.model, preproc)
predictor.predict('As of 2019, Donald Trump was still the President of the United States.')
predictor.save('/tmp/mypred')
reloaded_predictor = ktrain.load_predictor('/tmp/mypred')
reloaded_predictor.predict('Paul Newman is my favorite actor.')

Although you will see a 404 client error (from transformers) due to the fact that the Pytorch version of Bio_ClinicalBERT is downloaded/loaded (because no TF version is available), all of the above should work using the latest version of ktrain.

[NicoleJaneway]

Thanks for the quick response - much appreciated!!

The code you shared works. It does throw a 404 error at the model selection and prediction steps.

So the issue with my app (the one that worked in February and is now not loading) is something else. In addition to the 404, I'm also seeing "ValueError('model must be of instance Model')" on the line 'predictor = ktrain.get_predictor(loaded_model, features).'

def load_model():
    # Load model json file
    json_file = open('model.json', 'r')

    # Load ktrain preproc file
    features = pickle.load(open('preproc.sav', 'rb'))

    loaded_model_json = json_file.read()
    json_file.close()
    loaded_model = model_from_json(loaded_model_json)

    loaded_model.load_weights("model.h5")
    print("Model Loaded from disk")

    # compile and evaluate loaded model
    loaded_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return loaded_model, features

def get_predictions(loaded_model,features,text):
    predictor = ktrain.get_predictor(loaded_model, features)
    return predictor.predict(text, return_proba=True)

For clarity, the above code uses the original [February 2021] model.json, preproc.sav, and model.h5 objects, per this article.

[amaiya]

There are probably several things happening in your code.

Unable to Load Old Predictor in New Version of ktrain

The newest version of ktrain (v0.26.x), uses transformers>=4.0. There was a complete reorganization of the module structure in transformers>=4.0. As described in detail at the top of the README, if you need to load a Predictor trained in an older version of ktrain in ktrain>=0.26.0, you may need need to either manually edit the Preproc file or regenerate it. The steps to do this are described in News & Announcements in the README. In your case, it's probably easiest to just retrain your model from scratch using ktrain>=0.26.0.

How to Deploy

The blog article you're following is quite old and was written before TF2 was released. Also, it is not necessary to manually save and load the model and .preproc file. As illustrated in this FAQ entry, you can use ktrain.load_predictor to load a Predictor instance (no need for manually calling picke.load, etc.). I would recommend using load_predictor in your app, as it's been proven to work in many applications using different frameworks (e.g., Flask , FastAPI, etc.) by many users.

invalid load key, '\xef' Error

I was not able to reproduce this error, and it may be something on your end. As mentioned above, I would recommend using ktrain.load_predictor in your app, which will automatically load the model and preproc files for you. You've already tested this out above using the code I provided and it works for you.

ValueError('model must be of instance Model') Error

If you're seeing this error, it means the model you're supplying to get_predictor is not an instance of tensorflow.keras.Model. You can do something like type(model) or model.summary() to shed more light on what's happening on your end.

404 client error

As mentioned, you can ignore this, as it's normal.

If you have further problems/questions as you investigate what's happening, feel free to reply in this thread.

[NicoleJaneway]

Following the approach of the FAQ entry, I'm still getting an error.

predictor = ktrain.load_predictor('./model')

throws:

Exception: Failed to load .preproc file in either the post v0.16.x loction (./model/tf_model.preproc) or pre v0.16.x location (./model.preproc)

The tf_model.preproc and tf_model.h5 are in the model folder, which is in the root directory of the streamlit app where the call to load_predictor is made. I've tried supplying the direct file path as well, and that approach similarly didn't work.

The .preproc file was generated last night after upgrading ktrain.

But just in case, I tried the suggestion here to check the tf_model.preproc file to make sure it contains the line 'transformers.models.xlm.configuration_xlm.' When I open the .preproc file in a text editor it looks like gibberish.

Sorry if I'm asking extremely obvious questions, but is there another way I should be doing this?

[amaiya]

Hi @NicoleJaneway

I just tested it with StreamLit and everything worked for me.

Step 1

I trained an NER predictor using emilyalsentzer/Bio_ClinicalBERT (with the code above that you said you were able to run successfully). Saved the predictor in /tmp/mypred folder.

Step 2

Create the StreamLit app with:

# file: main.py
import streamlit as st
import ktrain

st.title('Using ktrain with streamlit')
st.subheader('Make a prediction from the supplied sentence:')

@st.cache(allow_output_mutation=True)
def load_model():
    model = ktrain.load_predictor('/tmp/mypred')
    return model
with st.spinner('Loading Model Into Memory...'):
    model = load_model()
input = st.text_area('Enter your text', 'Type here')
if input != 'Type here':
    with st.spinner('Doing AI things...'):
        output = model.predict(input)
    st.success("Raw output of predictor: %s"% (output))

Step 3

Run the app with:

streamlit run main.py

Use the app at URL printed out from above command.

If you follow the same steps and experience problems, then it may be some sort of issue on your end.

• Is the error only occurring in your streamlit app? You can load the files correctly from the Python prompt, correct?

• If the answer to the above question is "yes", then, in your streamlit app, what is the entire traceback when you try to load the pickle file directly:

with open('/tmp/mypred/tf_model.preproc', 'rb') as f: preproc = pickle.load(f)

[NicoleJaneway]

I'm mystified at why this isn't working. It's not an issue with Streamlit - the .load_predictor() function is throwing the "Failed to load .preproc file" from the terminal as well. From the terminal, trying the last line you suggested trying throws "_pickle.UnpicklingError: invalid load key, '\xef'."

Here's the exact code in the app.py file (with the tf_model.preproc and tf_model.h5 files in the same directory as the app):

def get_predictions(text):
    ktrain.load_predictor('')
    return predictor.predict(text, return_proba=True)

user_input = st.text_area("Add your clinical notes below (up to 250 characters):", value="", height=600)
get_predictions(user_input)

which results in "Failed to load .preproc file in either the post v0.16.x loction (tf_model.preproc) or pre v0.16.x location (.preproc)"

I tried unpickling my Feb 2021 preproc and model files. Seems to work for preproc, but for model.h5, I'm getting "_pickle.UnpicklingError: invalid load key, 'H'."

[NicoleJaneway]

I feel like the last thing to do I just go and retrain the model to generate new (since Sunday) tf_model.preproc and tf_model.h5 files. On Sunday, I did the training in Colab, so I'm not sure if the predictor files got corrupted somehow as I was downloading them from that environment.

[amaiya]

You said earlier that, using the code I provided above to train an emilyalsentzer/Bio_ClinicalBERT model on CoNNL 2003, you were able to successfully reload the Predictor . Does this mean that it only doesn't work when you use the same/similar code to train the model on your dataset on Google Colab?

If that's the case, the problem sounds like something on your end related to 1) corrupted files downloaded from Google Colab or 2) something weird with your environment.

I would do the following:

1. Retrain the model on Google Colab
2. Compute MD5 of both model and preproc file on Google Colab. After you download the files to your local machine, re-compute MD5s and make sure they're the same to ensure there is no corruption. Example: md5sum tf_model.h5.
3. On your local machine, create a brand new virtual environment and re-install TensorFlow and ktrain from scratch.
4. Then, try re-loading Predictor.

The version of TensorFlow shouldn't matter, but, as an extra measure, you might want to use the same version of TensorFlow both on Google Colab and on your local machine.

[NicoleJaneway]

Or is there some way I can make the code here back-compatible?

This throws "model must be of instance Model" on the line
predictor = ktrain.get_predictor(loaded_model, features)

loaded_model is class 'keras.engine.functional.Functional'

[NicoleJaneway]

And thank you for your suggestions for further troubleshooting with Colab. Sorry I didn't see your post before my last comment.

[amaiya]

@NicoleJaneway

I looked at your GitHub repository. I think your .preproc file was corrupted when downloading from Google Colab. But, your old .preproc still works (preproc.sav) and your new model works (tf_model.h5). So, if you go to your GitHub repo folder and do the following, you can load your predictor correctly in ktrain v0.26.x:

1. Make a new predictor folder: mkdir fixed_predictor
2. Copy tf_model.h5 to folder: cp tf_model.h5 ./fixed_predictor
3. Copy preproc.sav to folder and rename it to tf_model.preproc: cp preproc.sav ./fixed_predictor/tf_model.preproc

After that things seem to work:

import ktrain
p = ktrain.load_predictor('./fixed_predictor')
p.predict('Paul Newman is a great actor.')
# output
[('paul', 'B-person'),
 ('newman', 'I-person'),
 ('is', 'O'),
 ('a', 'O'),
 ('great', 'O'),
 ('actor', 'O'),
 ('.', 'O')]

Also, you said earlier:

I tried unpickling my Feb 2021 preproc and model files. Seems to work for preproc, but for model.h5, I'm getting "_pickle.UnpicklingError: invalid load key, 'H'."

The reason for this is that model.h5 is just saved weights file, not a saved model. So, using the new model with the old preproc works.

I will close this issue, but feel free to reply if you have further issues.