### Search before asking - [X] I have searched the Inference [issues](https://github.com/roboflow/inference/issues) and found no similar bug report. ### Bug When someone runs ```python from inference_sdk import InferenceHTTPClient client = InferenceHTTPClient( api_url="http://XXX:9001", api_key="XXX" ) client.start_inference_pipeline_with_workflow( video_reference=["INVALID"], workspace_name="XXX", workflow_id="XXX", ) ``` and the InferencePipeline cannot connect to source, for some cases one cannot run: ```python client.terminate_inference_pipeline("<PIPELINE-ID>") ``` as the video source is not in state letting for termination and will never be - we are getting `StreamOperationNotAllowedError` inside server ### Environment _No response_ ### Minimal Reproducible Example _No response_ ### Additional _No response_ ### Are you willing to submit a PR? - [ ] Yes I'd like to help by submitting a PR!