Token Count Difference

[abhatia-07]

Thanks for implementing token count logic in Java, this is very helpful!

I found a few cases in which the count calculated was different from the actual OpenAI count. While I know this is an estimation, it would be good if you could check the logic.

1. When an integer enum is passed as a function parameter, 1 less token is counted.
2. When the message contains role assistant with tool_calls and a subsequent message with role tool, 6 extra tokens are counted.

If you want, I can provide examples for the above.
Thanks!

[forestwanglin]

Thanks for your testing，I will try to cover the cases you mentioned. Please show me the examples.

[abhatia-07]

Sure, here are the examples.
Note: I have ported your code and haven't directly tried running your library, so please consider running these examples once with your code.

Tool Call and Role "Tool" token count
Request Body:

{
	"model": "gpt-4",
	"messages": [
		{
			"role": "assistant",
			"content": null,
			"tool_calls": [
				{
					"id": "call_Id8ycVMsW8gdsf7kSXfgAcf1",
					"type": "function",
					"function": {
						"name": "get_current_weather",
						"arguments": "{\n  \"location\": \"Boston, MA\"\n}"
					}
				}
			]
		},
		{
			"role": "tool",
			"tool_call_id": "call_Id8ycVMsW8gdsf7kSXfgAcf1",
			"name": "get_current_weather",
			"content": "29 degree celcius"
		}
	]
}

My prompt token: 41
OpenAI Count: 35

Integer enum case

{
	"model": "gpt-4",
	"messages": [
		{
			"role": "system",
			"content": "You are a helpful assistant"
		},
		{
			"role": "user",
			"content": "Sample input for data demonstaration?"
		}
	],
	"tools": [
		{
			"type": "function",
			"function": {
				"name": "data_demonstration",
				"description": "This is the main function description",
				"parameters": {
					"type": "object",
					"properties": {
						"integer_enum": {
							"type": "integer",
							"enum": [
								-1,
								1
							]
						}
					}
				}
			}
		}
	],
	"tool_choice": "none"
}

My prompt token: 68
OpenAI Count: 69

[forestwanglin]

Thanks a lot.

I've fixed the problems above. You can find the latest in version 3.7.20240115

[abhatia-07]

The token count for Tools is still having a difference of 1.
Request Body:

{
	"model": "gpt-4",
	"messages": [
		{
			"role": "assistant",
			"content": null,
			"tool_calls": [
				{
					"id": "call_Id8ycVMsW8gdsf7kSXfgAcf1",
					"type": "function",
					"function": {
						"name": "get_current_weather",
						"arguments": "{\n  \"location\": \"Boston, MA\"\n}"
					}
				}
			]
		},
		{
			"role": "tool",
			"tool_call_id": "call_Id8ycVMsW8gdsf7kSXfgAcf1",
			"name": "get_current_weather",
			"content": "29 degree celcius"
		}
	]
}

My count: 34
OpenAI Count: 35

[forestwanglin]

I've investigated and confirmed the issue. It is cased by the model gpt-4. It goes well when you use gpt-4-1106-preview which is the same count with model gpt-3.5. I haven't found any doc in which it showed the difference between gpt-4 and gpt-3.5(or gpt-4-1106-preview).

I'm trying to get the answer in openai community. Please let me know if anyone know it.

Thanks.

[abhatia-07]

Oh okay, thanks for checking.
I tried this too and looks like with the newer 1106 models (Both GPT-4 and 3.5), the count is 34. And with all the older models, the count is 35.

Let me know if you get a response on this.
Thanks

[SoulBlaze4j]

您好，请问这个issue有进展吗？ 由于目前使用流式请求接口，返回没有usage字段，所以在流式调用function call（或tools call）时无法准确计算token用量

Hello, I was wondering if there has been any progress on this issue? Currently, when using the streaming request interface, there is no "usage" field in the response, making it difficult to accurately calculate token consumption when making a streaming call to the function or tools call.

[forestwanglin]

@q27810365 你好，这个issue里面最开始的关于下面两个bug已经修复。
1、When an integer enum is passed as a function parameter, 1 less token is counted.
2、When the message contains role assistant with tool_calls and a subsequent message with role tool, 6 extra tokens are counted.

后面提到的是当使用GPT-4（包括GPT-4和GPT-4-32K）的时候，会少计算一个TOKEN，这个还没有找到任何修复方案，因为现在还不确定是什么原因少计算了一个TOKEN。2023年1106发布的GPT-4-Turbo计算是准确的。

[forestwanglin]

@q27810365 @abhatia-07 I've just uploaded new version 3.7.20240129 to fix the issue.

You can find the solution via page Difference token count between gpt-4 and gpt-3.5 (also gpt-4-1106-preview) , does anyone know the reason?