Home Assistant REST API data retrieval "hangs" in 0.15.1

[scruysberghs]

I do a daily model fit using 9 days of history.
After switching to 0.15.1 the model_fit api call is stuck in "Retrieve hass get data method initiated..."

2025-12-11 14:50:44,497] INFO in web_server:  >> Obtaining params: 
[2025-12-11 14:50:44,499] INFO in web_server: Passed runtime parameters: {'historic_days_to_retrieve': 9, 'model_type': 'load_forecast', 'var_model': 'sensor.sec_pac_non_deferable', 'sklearn_model': 'KNeighborsRegressor', 'num_lags': 96, 'split_date_delta': '48h', 'perform_backtest': False}
[2025-12-11 14:50:44,499] INFO in web_server:  >> Setting input data dict
[2025-12-11 14:50:44,499] INFO in command_line: Setting up needed data
[2025-12-11 14:50:44,501] INFO in retrieve_hass: get HA config from rest api.
[2025-12-11 14:50:44,507] INFO in retrieve_hass: Using REST API for data retrieval
[2025-12-11 14:50:44,507] INFO in retrieve_hass: Retrieve hass get data method initiated...

(this is where it ends, it never finishes)

If I switch on @gieljnssns new websocket connection in the config the same model fit api call runs fine and nearly instant:

[2025-12-11 14:52:49,242] INFO in web_server:  >> Obtaining params: 
[2025-12-11 14:52:49,244] INFO in web_server: Passed runtime parameters: {'historic_days_to_retrieve': 9, 'model_type': 'load_forecast', 'var_model': 'sensor.sec_pac_non_deferable', 'sklearn_model': 'KNeighborsRegressor', 'num_lags': 96, 'split_date_delta': '48h', 'perform_backtest': False}
[2025-12-11 14:52:49,245] INFO in web_server:  >> Setting input data dict
[2025-12-11 14:52:49,245] INFO in command_line: Setting up needed data
[2025-12-11 14:52:49,308] INFO in websocket_client: Connected and authenticated
[2025-12-11 14:52:49,308] INFO in connection_manager: WebSocket client started successfully
[2025-12-11 14:52:50,406] INFO in retrieve_hass: Statistics data retrieval took 1.09 seconds
[2025-12-11 14:52:50,406] INFO in web_server:  >> Performing a machine learning forecast model fit...
[2025-12-11 14:52:50,407] INFO in machine_learning_forecaster: Performing a forecast model fit for load_forecast
/app/src/emhass/machine_learning_forecaster.py:222: FutureWarning:
The `regressor` argument is deprecated and will be removed in a future version. Please use `estimator` instead.
[2025-12-11 14:52:50,411] INFO in machine_learning_forecaster: Training a KNeighborsRegressor model
[2025-12-11 14:52:50,422] INFO in machine_learning_forecaster: Elapsed time for model fit: 0.010062932968139648
[2025-12-11 14:52:50,507] INFO in machine_learning_forecaster: Prediction R2 score of fitted model on test data: 0.2558068156218185

[davidusb-geek]

A mystery for now...

[scruysberghs]

correction, if run the model fit api call, with HA rest api for data retrieval, trough curl or the emhass ui it gives me me a result after about 4-5 minutes. If I make the rest call trough home assistant it now times out before it finishes.
I have timeout set to 90 seconds and that was enough until version 0.14.1

  emhass_ml_model_fit:
    url: "http://homeassistant:5000/action/forecast-model-fit"
    method: post
    content_type: "application/json"
    timeout: "90"
    payload: "{{ payload }}"
    headers:
      Content-Type: application/json

Using websocket it is 0.2 seconds

And using influxdb also 0.2 seconds
2025-12-11 15:44:57,405] INFO in command_line: Setting up needed data
[2025-12-11 15:44:57,408] INFO in retrieve_hass: InfluxDB integration enabled: homeassistant.local:8086/homeassistant
[2025-12-11 15:44:57,408] INFO in retrieve_hass: get HA config from rest api.
[2025-12-11 15:44:57,413] INFO in retrieve_hass: Retrieve InfluxDB get data method initiated...
[2025-12-11 15:44:57,417] INFO in retrieve_hass: Retrieving 1 sensors over 9 days from InfluxDB
[2025-12-11 15:44:57,591] INFO in retrieve_hass: Retrieved 923 data points for sensor.sec_pac_non_deferable
[2025-12-11 15:44:57,599] INFO in retrieve_hass: InfluxDB data retrieval completed: (923, 1)
[2025-12-11 15:44:57,599] INFO in web_server: >> Performing a machine learning forecast model fit...
[2025-12-11 15:44:57,600] INFO in machine_learning_forecaster: Performing a forecast model fit for load_forecast
/app/src/emhass/machine_learning_forecaster.py:222: FutureWarning:
The regressor argument is deprecated and will be removed in a future version. Please use estimator instead.
[2025-12-11 15:44:57,604] INFO in machine_learning_forecaster: Training a KNeighborsRegressor model
[2025-12-11 15:44:57,613] INFO in machine_learning_forecaster: Elapsed time for model fit: 0.009634733200073242

[davidusb-geek]

It seems that you are retrieving data using InfluxDB. So check connection to your InfluxDB instance.

No problem on my side using the HA API:

[scruysberghs]

@davidusb-geek , I only tried switching to websocket and influx data retrieval because rest API was so/too slow. So:

API rest is slow

API websocket is fast

InfluxDB is fast

---

I'm no expert @gieljnssns but below is what Claude code thinks is happening :

v0.14.1 (Fast):
response = requests.get(url, headers=headers)  # Single persistent session
v0.15.1 (Slow):
async with aiohttp.ClientSession() as session:  # NEW SESSION EVERY REQUEST!
    async with session.get(url, headers=headers) as response:
The problem: In v0.15.1, they're creating a new aiohttp.ClientSession() for EVERY SINGLE request inside the loop! This is a massive performance anti-pattern.
Let me verify this is inside the loop:
→ Read src/emhass/retrieve_hass.py [offset=230, limit=80]
Confirmed! Line 305 shows the bug. For 9 days × 2 sensors = 18 requests, it's creating 18 separate ClientSession objects, each one:
1. Creating new TCP connection
2. TLS handshake
3. HTTP connection overhead
4. Closing connection
Instead of reusing a single persistent session.
The Performance Impact
v0.14.1: 1 session → 18 requests → ~fast (under 90 seconds)
v0.15.1: 18 sessions → 18 requests → ~5+ seconds per request = 90+ seconds total

[davidusb-geek]

Ok so we might need to revert back to synchronous just for this part of the code?
What do you think @gieljnssns ?

[davidusb-geek]

That would mean reinstating request and:

                try:
                    response = requests.get(url, headers=headers)
                    response.raise_for_status()
                    data_list = orjson.loads(response.content)
                except Exception:
                    self.logger.error("Unable to access Home Assistant instance, check URL")
                    self.logger.error("If using addon, try setting url and token to 'empty'")
                    return False

[scruysberghs]

That would mean reinstating request and:

Should also be possible with aiohttp . imho the session should be created outside the loop and reused.

something like

async def _get_data_rest_api(self, ...):
    # ... setup code ...
    
    async with aiohttp.ClientSession() as session:  # ONCE, outside loop
        for day in days_list:
            for i, var in enumerate(var_list):
                # ... build url ...
                async with session.get(url, headers=headers) as response:  # Reuse session
                    # ... handle response ..

Why this probably didn't show in testing, my load sensor is updated once a second. So the influxdb or websocket call will actually also return a lot less data

[gieljnssns]

I think this should be the fix
use one ClientSession

    async def _get_data_rest_api(
        self,
        days_list: pd.date_range,
        var_list: list,
        minimal_response: bool | None = False,
        significant_changes_only: bool | None = False,
        test_url: str | None = "empty",
    ) -> None:
        """Internal method to handle REST API data retrieval."""
        self.logger.info("Retrieve hass get data method initiated...")
        headers = {
            "Authorization": "Bearer " + self.long_lived_token,
            "content-type": "application/json",
        }
        # Remove empty strings from var_list
        var_list = [var for var in var_list if var != ""]
        # Looping on each day from days list
        self.df_final = pd.DataFrame()

        async with aiohttp.ClientSession() as session:
            x = 0  # iterate based on days

            for day in days_list:
                for i, var in enumerate(var_list):
                    if test_url == "empty":
                        if (
                            self.hass_url == "http://supervisor/core/api"
                        ):  # If we are using the supervisor API
                            url = (
                                self.hass_url
                                + "/history/period/"
                                + day.isoformat()
                                + "?filter_entity_id="
                                + var
                            )
                        else:  # Otherwise the Home Assistant Core API it is
                            if self.hass_url[-1] != "/":
                                self.logger.warning(
                                    "Missing slash </> at the end of the defined URL, appending a slash but please fix your URL"
                                )
                                self.hass_url = self.hass_url + "/"
                            url = (
                                self.hass_url
                                + "api/history/period/"
                                + day.isoformat()
                                + "?filter_entity_id="
                                + var
                            )
                        if minimal_response:  # A support for minimal response
                            url = url + "?minimal_response"
                        if significant_changes_only:  # And for signicant changes only (check the HASS restful API for more info)
                            url = url + "?significant_changes_only"
                    else:
                        url = test_url

                    try:
                        async with session.get(url, headers=headers) as response:
                            response.raise_for_status()
                            data = await response.read()
                            data_list = orjson.loads(data)
                    except Exception:
                        self.logger.error("Unable to access Home Assistant instance, check URL")
                        self.logger.error("If using addon, try setting url and token to 'empty'")
                        return False
                    else:
                        if response.status == 401:
                            self.logger.error("Unable to access Home Assistant instance, TOKEN/KEY")
                            self.logger.error(
                                "If using addon, try setting url and token to 'empty'"
                            )
                            return False
                        if response.status > 299:
                            self.logger.error(
                                f"Home assistant request GET error: {response.status} for var {var}"
                            )
                            return False
                    """import bz2 # Uncomment to save a serialized data for tests
                    import _pickle as cPickle
                    with bz2.BZ2File("data/test_response_get_data_get_method.pbz2", "w") as f:
                        cPickle.dump(response, f)"""

                    try:  # Sometimes when there are connection problems we need to catch empty retrieved json
                        data = data_list[0]
                    except IndexError:
                        if x == 0:
                            self.logger.error(
                                "The retrieved JSON is empty, A sensor:"
                                + var
                                + " may have 0 days of history, passed sensor may not be correct, or days to retrieve is set too high. Check your Logger configuration, ensuring the sensors are in the include list."
                            )
                        else:
                            self.logger.error(
                                "The retrieved JSON is empty for day:"
                                + str(day)
                                + ", days_to_retrieve may be larger than the recorded history of sensor:"
                                + var
                                + " (check your recorder settings)"
                            )
                        return False

                    df_raw = pd.DataFrame.from_dict(data)
                    if len(df_raw) == 0:
                        if x == 0:
                            self.logger.error(
                                "The retrieved Dataframe is empty, A sensor:"
                                + var
                                + " may have 0 days of history or passed sensor may not be correct"
                            )
                        else:
                            self.logger.error(
                                "Retrieved empty Dataframe for day:"
                                + str(day)
                                + ", days_to_retrieve may be larger than the recorded history of sensor:"
                                + var
                                + " (check your recorder settings)"
                            )
                        return False

                    if (
                        len(df_raw) < ((60 / (self.freq.seconds / 60)) * 24)
                        and x != len(days_list) - 1
                    ):  # check if there is enough Dataframes for passed frequency per day (not inc current day)
                        self.logger.debug(
                            "sensor:"
                            + var
                            + " retrieved Dataframe count: "
                            + str(len(df_raw))
                            + ", on day: "
                            + str(day)
                            + ". This is less than freq value passed: "
                            + str(self.freq)
                        )

                    if i == 0:  # Defining the DataFrame container
                        from_date = pd.to_datetime(df_raw["last_changed"], format="ISO8601").min()
                        to_date = pd.to_datetime(df_raw["last_changed"], format="ISO8601").max()
                        ts = pd.to_datetime(
                            pd.date_range(start=from_date, end=to_date, freq=self.freq),
                            format="%Y-%d-%m %H:%M",
                        ).round(self.freq, ambiguous="infer", nonexistent="shift_forward")
                        df_day = pd.DataFrame(index=ts)

                    # Caution with undefined string data: unknown, unavailable, etc.
                    df_tp = (
                        df_raw.copy()[["state"]]
                        .replace(["unknown", "unavailable", ""], np.nan)
                        .astype(float)
                        .rename(columns={"state": var})
                    )
                    # Setting index, resampling and concatenation
                    df_tp.set_index(
                        pd.to_datetime(df_raw["last_changed"], format="ISO8601"),
                        inplace=True,
                    )
                    df_tp = df_tp.resample(self.freq).mean()
                    df_day = pd.concat([df_day, df_tp], axis=1)
                self.df_final = pd.concat([self.df_final, df_day], axis=0)
                x += 1

        self.df_final = set_df_index_freq(self.df_final)
        if self.df_final.index.freq != self.freq:
            self.logger.error(
                "The inferred freq:"
                + str(self.df_final.index.freq)
                + " from data is not equal to the defined freq in passed:"
                + str(self.freq)
            )
            return False
        self.var_list = var_list
        return True

[davidusb-geek]

I think this should be the fix use one ClientSession

Could you open a patch PR with this? Thanks