Fix external weights loading during runtime in OpenVINO EP #7525
Conversation
| cwd = os.getcwd() | ||
| if model.mode == "vision": | ||
| os.chdir(Path(model.visual_path).parent) | ||
| elif model.mode == "text": | ||
| os.chdir(Path(model.textual_path).parent) | ||
| try: | ||
| outputs = await run(model.predict, inputs) | ||
| finally: | ||
| os.chdir(cwd) |
There was a problem hiding this comment.
This doesn't make sense with multiple requests in flight.
|
FWIW I think it's possible to solve the external data not being loaded eagerly by calling |
Yeah, that is something we can do. I am not a fan of hacky stuff either it's just that I don't think intel cares enough that we would see newer packages anytime soon, so maybe we shouldn't rely on them. I would try this out and update this PR if it is viable else let's just drop this approach. |
|
I looked at their release cadence and they seem to like doing April releases. They probably scheduled it to be some time next month. Granted if we have a different issue with that release, it'll be half a year before they release the next one lol. I guess we can just see if this patch works and wait for April, then maybe look at building it ourselves if it comes to that. |
Yes, I building can be a better option in the future. They provide a ready to use DockerFile for OpenVINO here, so I don't think we should be too worried about building it ourselves. |
Some models that are large e.g. ViT-H-14-378-quickgelu__dfn5b and have their attributes split into multiple files are loaded at runtime instead of the entire model being loaded during session creation.
This PR attempts to fix that behaviour.