get_result_from_job_id AssertionError while initializing redeployed LS/ML NER backend

[wojnarabc]

I am trying to deploy NER example model trained on my local machine along with the Label Studio project to another machine. I've gone through following steps:

1. Recreated Label Studio and ML Backend environments similarly as on a target machine
2. Copied folder with the model itself (folder named with just integers) to target machine ML Backend folder.
3. Extracted content (data, annotations and predictions) of the project through Label Studio API into json format (using ...export?exportType=JSON&download_all_tasks=true command)
4. Imported project json file into the newly created Label Studio project.
   When trying to initialize and pair LS and ML Backend on a new machine, i am getting :
   [2022-05-30 10:18:56,133] [ERROR] [label_studio_ml.model::get_result_from_last_job::128] 1647350146 job returns exception: Traceback (most recent call last): File "/Users/user/Projects/label-studio-ml-backend/label_studio_ml/model.py", line 126, in get_result_from_last_job result = self.get_result_from_job_id(job_id) File "/Users/user/Projects/label-studio-ml-backend/label_studio_ml/model.py", line 108, in get_result_from_job_id assert isinstance(result, dict) AssertionError
   and it keeps repeating for each job

Should any additional steps be performed during deploy of the project/model to other environments ?

I've tried with following LS versions (1.1.1 - my initial one, 1.4.1post1 - most recent one) and the most current code base of ML backend. Using Python 3.8 and MacOS for both source and target environments.

[KonstantinKorotaev]

Hi @wojnarabc
This error indicates that training of your model ended without any contrete result.
ner example doesn't have fit method, but you can add it like that:

def fit(self, completions, workdir=None, **kwargs):
    import random
    return {'random': random.randint(1, 10)}

and the error should disappear.
I will add this stub to the ner example in future release.

If you are using TransformersBasedTagger - please check that fit method ends with appropriate result.

[wojnarabc]

Hello @KonstantinKorotaev, there is a fit method in example in ...examples/ner/ner.py folder starting in line 461. TransformersBasedTagger is part of it.

[jrdalenberg]

I run into the exact same problem with my custom backend.

I am in the process of upgrading my system to the latest LS and backend. Everything was working fine with LS 1.1.1 and the backend from a year ago.

After training, another job is sent for some reason, and then train_output is being cleared causing the backend to lose the knowledge about the last trained model.

I already set LABEL_STUDIO_ML_BACKEND_V2_DEFAULT = True

 'train_output': {'model_path': '././my_backend/5.1655881660/1658156596'},
 'value': 'image'}
[2022-07-18 17:03:37,760] [INFO] [werkzeug::_log::225] 192.168.123.133 - - [18/Jul/2022 17:03:37] "POST /train HTTP/1.1" 201 -
[2022-07-18 17:03:37,781] [INFO] [werkzeug::_log::225] 192.168.123.133 - - [18/Jul/2022 17:03:37] "GET /health HTTP/1.1" 200 -
[2022-07-18 17:03:37,787] [ERROR] [label_studio_ml.model::get_result_from_last_job::130] 1658156583 job returns exception: 
Traceback (most recent call last):
  File "/home/USER/.virtualenvs/ls-1.5/lib/python3.8/site-packages/label_studio_ml/model.py", line 128, in get_result_from_last_job
    result = self.get_result_from_job_id(job_id)
  File "/home/USER/.virtualenvs/ls-1.5/lib/python3.8/site-packages/label_studio_ml/model.py", line 110, in get_result_from_job_id
    assert isinstance(result, dict)
AssertionError

[jrdalenberg]

So, to be more clear, I based my custom backend on pytorch_transfer_learning.py. I had to set LABEL_STUDIO_ML_BACKEND_V2_DEFAULT = True in model.py, because otherwise I run into the issue described in #118.

Now, when I monitor train_output after training, label studio initializes the backend with the correct train_output. But immediately after, another initialization event follows with an empty train_output.

I made an ugly workaround for this issue using an OS environment variable that saves the location of last trained model locally. Note that this causes issues if users switch between projects and it won't redeploy an existing model.

# ...

from label_studio_ml import model
model.LABEL_STUDIO_ML_BACKEND_V2_DEFAULT = True

# ...

class ImageClassifierAPI(model.LabelStudioMLBase):
   def __init__(self, label_config=None, train_output=None,**kwargs):
       super(ImageClassifierAPI, self).__init__(**kwargs)
       
      # ...
       
       if self.train_output:
           print(f"trying to use {self.train_output['model_path']} as model path")
           self.model = ImageClassifier(self.classes, self.boxType)
           self.model.load(self.train_output['model_path'])
       elif os.environ.get("LAST_TRAINED_MODEL") is not None:
           model_path = os.environ.get("LAST_TRAINED_MODEL")
           print(f"trying to use {model_path} as model path")
           self.model = ImageClassifier(self.classes, self.boxType)
           self.model.load(model_path)
       else: 
           self.model = ImageClassifier(self.classes, self.boxType)

   # ...

   def fit(self, annotations, workdir=None, batch_size=12, num_epochs=100, **kwargs):
       self.model.train(annotations, workdir, self.boxType, batch_size=batch_size, num_epochs=num_epochs)

       # Save workdir in environment
       os.environ["LAST_TRAINED_MODEL"] = workdir

       return {'model_path': workdir}

Having to manually change LABEL_STUDIO_ML_BACKEND_V2_DEFAULT = True and Label studio loosing the returned dict from the fit function is a bit awkward to work with though.

[KonstantinKorotaev]

@jrdalenberg
LABEL_STUDIO_ML_BACKEND_V2_DEFAULT this variable tells your ML backend that you are using active learning cycle. Starting from version 1.4.1 of Label Studio training is invoked with webhooks.
Please check this documentation.

def fit(self, annotations, workdir=None, batch_size=12, num_epochs=100, **kwargs):
self.model.train(annotations, workdir, self.boxType, batch_size=batch_size, num_epochs=num_epochs)

Your fit method seems to be expecting annotations, but from version 1.4.1 there will be an event instead of annotations. Maybe this is leading to empty train_output.

If you want to switch this logic off - disable webhook in LS.
Check this example if you want to use active learning cycle.

[jrdalenberg]

Your fit method seems to be expecting annotations, but from version 1.4.1 there will be an event instead of annotations. Maybe this is leading to empty train_output.

I see. It’s still odd that train_output is returned once and then cleared right after, though.

Check this example if you want to use active learning cycle.

Thanks! I did not see this in the object recognition examples. I’ll test it out after my vacation 😄

[idreamerhx]

same bug

/opt/label-studio-ml-backend/coco-detector/mmdetection.py(137)fit()
-> logger.info(f'tasks={tasks}, workdir={workdir}, kwargs={kwargs}')
(Pdb) bt
/usr/lib/python3.10/threading.py(966)_bootstrap()
-> self._bootstrap_inner()
/usr/lib/python3.10/threading.py(1009)_bootstrap_inner()
-> self.run()
/usr/lib/python3.10/threading.py(946)run()
-> self._target(*self._args, **self._kwargs)
/usr/lib/python3.10/socketserver.py(683)process_request_thread()
-> self.finish_request(request, client_address)
/usr/lib/python3.10/socketserver.py(360)finish_request()
-> self.RequestHandlerClass(request, client_address, self)
/usr/lib/python3.10/socketserver.py(747)init()
-> self.handle()
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/werkzeug/serving.py(342)handle()
-> BaseHTTPRequestHandler.handle(self)
/usr/lib/python3.10/http/server.py(425)handle()
-> self.handle_one_request()
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/werkzeug/serving.py(374)handle_one_request()
-> self.run_wsgi()
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/werkzeug/serving.py(319)run_wsgi()
-> execute(self.server.app)
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/werkzeug/serving.py(308)execute()
-> application_iter = app(environ, start_response)
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/flask/app.py(2464)call()
-> return self.wsgi_app(environ, start_response)
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/flask/app.py(2447)wsgi_app()
-> response = self.full_dispatch_request()
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/flask/app.py(1950)full_dispatch_request()
-> rv = self.dispatch_request()
/opt/pyvenv-labelstudio-ml-backend/lib/python3.10/site-packages/flask/app.py(1936)dispatch_request()
-> return self.view_functionsrule.endpoint
/opt/label-studio-ml-backend/label_studio_ml/exceptions.py(39)exception_f()
-> return f(*args, **kwargs)
/opt/label-studio-ml-backend/label_studio_ml/api.py(93)_train()
-> job = _manager.train(annotations, project, label_config, **params)
/opt/label-studio-ml-backend/label_studio_ml/model.py(711)train()
-> job_result = cls.train_script_wrapper(
/opt/label-studio-ml-backend/label_studio_ml/model.py(667)train_script_wrapper()
-> train_output = m.model.fit(data_stream, workdir, **train_kwargs)
/opt/label-studio-ml-backend/coco-detector/mmdetection.py(137)fit()
-> logger.info(f'tasks={tasks}, workdir={workdir}, kwargs={kwargs}')

022-08-19 03:17:21,961] [ERROR] [label_studio_ml.model::get_result_from_last_job::131] 1660820149 job returns exception:
Traceback (most recent call last):
File "/opt/label-studio-ml-backend/label_studio_ml/model.py", line 129, in get_result_from_last_job
result = self.get_result_from_job_id(job_id)
File "/opt/label-studio-ml-backend/label_studio_ml/model.py", line 111, in get_result_from_job_id
assert isinstance(result, dict)
AssertionError

[idreamerhx]

# train, see https://labelstud.io/guide/ml_create.html
def fit(self, tasks, workdir=None, **kwargs):
    # Retrieve the annotation ID from the payload of the webhook event
    # Use the ID to retrieve annotation data using the SDK or the API
    # Do some computations and get your model
    #import pdb; pdb.set_trace()
    logger.info(f'tasks={tasks}, workdir={workdir}, kwargs={kwargs}')
    return {'checkpoints': '3.1660708258/1660885087/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth', 'model_file': "3.1660708258/1660885087/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth", "model_version":"3.1660708258/1660885087", 'classes': 80}
    ## JSON dictionary with trained model artifacts that you can use later in code with self.train_output

Load new model from: /opt/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py /opt/mmdetection/checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
load checkpoint from local path: /opt/mmdetection/checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
Load new model from: /opt/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py /opt/mmdetection/checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
load checkpoint from local path: /opt/mmdetection/checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[2022-08-19 05:06:31,709] [INFO] [werkzeug::_log::225] 192.168.1.7 - - [19/Aug/2022 05:06:31] "POST /train HTTP/1.1" 201 -
[2022-08-19 05:06:32,102] [INFO] [werkzeug::_log::225] 192.168.1.7 - - [19/Aug/2022 05:06:32] "GET /health HTTP/1.1" 200 -
Load new model from: /opt/mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py /opt/mmdetection/checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
load checkpoint from local path: /opt/mmdetection/checkpoints/faster_rcnn/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
[2022-08-19 05:06:33,168] [INFO] [werkzeug::_log::225] 192.168.1.7 - - [19/Aug/2022 05:06:33] "POST /setup HTTP/1.1" 200 -

seems return values not work

[KonstantinKorotaev]

Hi @idreamerhx

seems return values not work

I don't see any error in your log, what do you mean by not work?

assert isinstance(result, dict)
AssertionError

This means that your training procedure didn't return results, could you please check what your fit method returns?

[sjtuytc]

This should be a bug somehow. Because the results should be a list of dict but it tells me it is wrong.

[KonstantinKorotaev]

@sjtuytc could you please clarify what do you mean?

[sjtuytc]

I mean this assertion is too restrictive. A dummy result would tricker the assertion error. This should not be desired.

[seanswyi]

I'm also getting the same assertion error problem. Does the model always have to return a dictionary? I'm following the examples provided in Label Studio and it seems like lists of dicts are also allowed: https://github.com/heartexlabs/label-studio-ml-backend/blob/db6d1d6d3efde1db503532f1f77a6977a7f100d2/label_studio_ml/examples/simple_text_classifier/simple_text_classifier.py#L93