CT Error

[glenn-jocher]

Search before asking

• I have searched the HUB-SDK issues and found no similar bug report.

HUB-SDK Component

Other

Bug

@yogendrasinghx 3 failing CI tests here introduced in #116

See https://github.com/ultralytics/hub-sdk/actions/runs/8650461977/job/23719232082

=================================== FAILURES ===================================
___________________________ TestModel.test_model_001 ___________________________

self = <functional.test_model.TestModel object at 0x7fe415d86120>

    @pytest.mark.smoke
    def test_model_001(self):
        """Verify successful retrieval of a model by ID."""
    
        model_id = TestData().get_models_data()["valid_model_ID"]
        object_manager = ObjectManager(self.client)
        model_obj = object_manager.get_model()
    
        log = self.getLogger()
        log.info(f"Attempting to retrieve model with ID: {model_id}")
    
        model = model_obj.get_model_by_id(model_id)
    
        log.info(f"Model retrieved successfully. Model data: {model.data}")
    
>       assert "config" in model.data, "Config information not found in the model data"
E       AssertionError: Config information not found in the model data
E       assert 'config' in {}
E        +  where {} = <hub_sdk.modules.models.Models object at 0x7fe413bfc770>.data

tests/functional/test_model.py:23: AssertionError
----------------------------- Captured stderr call -----------------------------
2024-04-11 17:05:01,098 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
2024-04-11 17:05:01,099 - hub_sdk.helpers.logger - ERROR - Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
------------------------------ Captured log call -------------------------------
INFO     test_model_001:test_model.py:17 Attempting to retrieve model with ID: pmcIKukVlTQfnkk3FxgJ
ERROR    hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
ERROR    hub_sdk.helpers.logger:models.py:87 Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
INFO     test_model_001:test_model.py:21 Model retrieved successfully. Model data: {}
___________________________ TestModel.test_model_008 ___________________________

self = <functional.test_model.TestModel object at 0x7fe415d872c0>
clear_export_model = None

    @pytest.mark.smoke
    def test_model_008(self, clear_export_model):
        """Verify successful export of a model."""
    
        log = self.getLogger()
    
        # Retrieve necessary data
        model_id = TestData().get_models_data()["valid_model_ID"]
        object_manager = ObjectManager(self.client)
        model_obj = object_manager.get_model()
        desired_format = TestData().get_models_data()["desired_model_format"]
    
        # Export the model
        log.info(f"Exporting model {model_id} in {desired_format} format")
        model_obj.export_model(model_id, format_name=desired_format)
    
        # Check if the model is successfully exported
        export_status = model_obj.is_model_exported(model_id, format_name=desired_format)
        log.info(f"Model export status: {'Success' if export_status else 'Failure'}")
    
>       assert export_status
E       assert False

tests/functional/test_model.py:207: AssertionError
----------------------------- Captured stderr call -----------------------------
2024-04-11 17:05:34,109 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
2024-04-11 17:05:34,110 - hub_sdk.helpers.logger - ERROR - Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
2024-04-11 17:05:35,269 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
------------------------------ Captured log call -------------------------------
INFO     test_model_008:test_model.py:200 Exporting model pmcIKukVlTQfnkk3FxgJ in ncnn format
ERROR    hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
ERROR    hub_sdk.helpers.logger:models.py:87 Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
ERROR    hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
INFO     test_model_008:test_model.py:205 Model export status: Failure
___________________________ TestModel.test_model_009 ___________________________

self = <functional.test_model.TestModel object at 0x7fe415d874a0>

    @pytest.mark.smoke
    def test_model_009(self):
        """Verify successful retrieval of model storage URL."""
    
        log = self.getLogger()
    
        model_id = TestData().get_models_data()["valid_model_ID"]
    
        log.info(f"Attempting to retrieve the storage URL for model with ID: {model_id}")
    
        object_manager = ObjectManager(self.client)
        model_obj = object_manager.get_model()
    
        # Get the model storage URL
        link = model_obj.get_model_download_link(model_id)
    
        log.info(f"Storage URL retrieved successfully. URL: {link}")
    
>       assert f"{model_id}/best.pt" in link, f"Model ID not found in the storage URL: {link}"
E       TypeError: argument of type 'NoneType' is not iterable

tests/functional/test_model.py:227: TypeError
----------------------------- Captured stderr call -----------------------------
2024-04-11 17:06:37,[62](https://github.com/ultralytics/hub-sdk/actions/runs/8650461977/job/23719232082#step:6:63)8 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
2024-04-11 17:06:37,629 - hub_sdk.helpers.logger - ERROR - Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
------------------------------ Captured log call -------------------------------
INFO     test_model_009:test_model.py:217 Attempting to retrieve the storage URL for model with ID: pmcIKukVlTQfnkk3FxgJ
ERROR    hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
ERROR    hub_sdk.helpers.logger:models.py:87 Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
INFO     test_model_009:test_model.py:225 Storage URL retrieved successfully. URL: None
=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/functional/test_model.py::TestModel::test_model_001 - AssertionError: Config information not found in the model data
assert 'config' in {}
 +  where {} = <hub_sdk.modules.models.Models object at 0x7fe413bfc7[70](https://github.com/ultralytics/hub-sdk/actions/runs/8650461977/job/23719232082#step:6:71)>.data
FAILED tests/functional/test_model.py::TestModel::test_model_008 - assert False
FAILED tests/functional/test_model.py::TestModel::test_model_009 - TypeError: argument of type 'NoneType' is not iterable
======= 3 failed, 21 passed, 1 skipped, 2 warnings in 162.00s (0:02:42) ========

Environment

No response

Minimal Reproducible Example

No response

Additional

Also a few deprecation warnings here, but lower priority.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[yogendrasinghx]

@glenn-jocher The issue with the model ID in the test data was causing the test cases to fail. I've updated the test data, so if you rerun the CT, it should work now.

[glenn-jocher]

@yogendrasinghx yes, I see the latest scheduled CI from today is now passing. Nice work!!!

[glenn-jocher]

@

[glenn-jocher]

@yogendrasinghx ah nevermind, I was looking at the HUB-SDK CI action, I thought you had placed the tests in there.

Ok so what we want to do is take your new YAML that you created and place it in the HUB SDK CI, which is the reason this yaml exists in the first place.

[glenn-jocher]

@yogendrasinghx the HUB SDK CI is the offical repo CI and follows exactly the template from Ultralytics, with all the correct run criteria like schedule, manual run capability etc, so the other yml that you have there is duplicating code and effort and adding confusion.

[yogendrasinghx]

@glenn-jocher Sure, I've added continuous testing to our existing CI pipeline and removed the ct.yml file. This will make things simpler. I've created a PR #120 for this; please check it out.

[glenn-jocher]

@yogendrasinghx awesome, nice work on #120, closing this issue as completed :)