Skip to content
This repository has been archived by the owner on Sep 13, 2023. It is now read-only.

Expose GTO's model version in FastAPI's interface.json #665

Open
aguschin opened this issue Apr 27, 2023 · 2 comments · May be fixed by #681
Open

Expose GTO's model version in FastAPI's interface.json #665

aguschin opened this issue Apr 27, 2023 · 2 comments · May be fixed by #681
Assignees
Labels
customer Request from customer serialization Dumping and loading Python objects serve Serving models

Comments

@aguschin
Copy link
Contributor

Follow-up for #664. Sometimes it's desired to store predictions along with the specific model version that returned those predictions. There are at least two ways to support that in MLEM:

  1. Return model version in prediction - then what's returned will be a json like {"prediction": [0.4, 0.6], "version": 0.1.3}. I've seen some generic ML frameworks doing this IIRC.
  2. Return it in interface.json - we already have MLEM version there, so adding model version looks logical

image

Regarding how we get this info into the service. Again, there are two approaches:

  1. Add it at mlem.api.save
  2. Allow to specify it when building server

First seems more reasonable to me. Since this will require some under-the-hood integration with GTO, I'd do this after #664 - which have the same decision to make.

fyi @omesser

@aguschin aguschin added serialization Dumping and loading Python objects serve Serving models customer Request from customer labels Apr 27, 2023
@aguschin aguschin self-assigned this May 24, 2023
@aguschin
Copy link
Contributor Author

aguschin commented May 25, 2023

For the record, I'm going to take 2nd option (Return version in interface.json).

And it looks like I need iterative/gto#335 completed first to support this in MLEM.

@aguschin
Copy link
Contributor Author

UPD: getting the version at mlem.api.save doesn't work since at this moment the commit doesn't exist, not mentioning the right GTO git tag. So it should be read once you build a server.

@aguschin aguschin linked a pull request May 30, 2023 that will close this issue
3 tasks
aguschin added a commit to iterative/gto that referenced this issue May 31, 2023
Bare minimum needed for iterative/mlem#665

```[tasklist]
- [ ] Allow passing `refs`, getting right version
- [ ] Add tests
```

Overall, feels like GTO codebase needs a refactoring... While working on
this I started with #361, and it's tough to add anything like that
without rewriting big chunks of GTO now. There are many things each
stepping on other's toes... Plus many implementation decisions were made
taking annotations into account, which is no longer a thing in GTO,
except for some API for Studio to enable backward compatibility while
making updates easier. They should be thrown away to make it easier to
contribute new features 🪣
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
customer Request from customer serialization Dumping and loading Python objects serve Serving models
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant