Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support GenerativeModel.count_tokens[_async] for fully-featured generation requests #3631

Open
adubovik opened this issue Apr 17, 2024 · 1 comment
Assignees
Labels
api: vertex-ai Issues related to the googleapis/python-aiplatform API.

Comments

@adubovik
Copy link

It should be possible to count number of prompt tokens via GenerativeModel.count_tokens[_async] method before running an actual generation request.

However, the method takes into account only the contents component of a request.

Other components of a request (such as tools, tool_config, and system_instruction) presumably affect prompt token count, so it would be great to support them in the count-token methods too.

@product-auto-label product-auto-label bot added the api: vertex-ai Issues related to the googleapis/python-aiplatform API. label Apr 17, 2024
@Ark-kun
Copy link
Contributor

Ark-kun commented May 8, 2024

Thank you for the feedback. This makes sense.
However this is currently not supported by the service API:

class CountTokensRequest(proto.Message):

@Ark-kun Ark-kun self-assigned this May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: vertex-ai Issues related to the googleapis/python-aiplatform API.
Projects
None yet
Development

No branches or pull requests

2 participants