Skip to content

Commit 091130d

Browse files
sabrennersabrenner
authored
test(llmobs): re-work deepEqualWithMockValues to use node:assert (#6709)
This changes the LLM Observability deepEqualWithMockValues to use node:assert instead of mocha expect. Also refactor the logic to do assertions. Now all properties will be validated, not only particular entries. That caught a couple of issues that will be handled separately.
1 parent 75c2809 commit 091130d

File tree

16 files changed

+1409
-1637
lines changed

16 files changed

+1409
-1637
lines changed

‎packages/datadog-plugin-aws-sdk/test/fixtures/bedrockruntime.js‎

Lines changed: 45 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -154,50 +154,51 @@ bedrockruntime.models = [
154154
text: 'The capital of France is Paris.'
155155
}
156156
},
157-
{
158-
provider: PROVIDER.COHERE,
159-
modelId: 'cohere.command-r-v1:0',
160-
userPrompt: prompt,
161-
requestBody: {
162-
message: prompt,
163-
temperature,
164-
max_tokens: maxTokens
165-
},
166-
response: {
167-
inputTokens: 7,
168-
outputTokens: 335,
169-
cacheReadTokens: 0,
170-
cacheWriteTokens: 0,
171-
text: 'The current capital of France is Paris. It has been the capital since 1958 and' +
172-
' is also the most populous city in the country. Paris has a rich history and' +
173-
' is known for its iconic landmarks and cultural significance.\n\nThe history' +
174-
' of the capital of France is somewhat complex, with the city of Paris itself' +
175-
' having a long and fascinating past. There was a shift in the capital\'s location' +
176-
' over the centuries, with various cities and towns fulfilling the role. The' +
177-
' earliest French capital based on historical records is thought to be the city' +
178-
' of Tours. The capital moved to various locations, often due to political and' +
179-
' dynastic reasons, including cities like Reims and Orleans. Paris initially' +
180-
' became the capital during the era of the Louvre in the 14th century, under' +
181-
' the rule of King Philip IV.\n\nThe status of Paris as the capital of France' +
182-
' has been reaffirmed many times, even during the French Revolution and the' +
183-
' establishment of the First French Empire by Napoleon Bonaparte. The city\'s' +
184-
' significance grew further with its designation as the centre of the Department' +
185-
' of Seine. Paris remained the capital through the changes in regime, including' +
186-
' the restoration of the monarchy, the July Monarchy, the Second Empire, and' +
187-
' the establishment of the French Third Republic.\n\nModern France\'s political' +
188-
' system, following the end of the Second World War, saw the capital remain' +
189-
' in Paris. The city continues to be a cultural hub, attracting artists, writers,' +
190-
' and musicians from around the world. Paris remains a prominent global city,' +
191-
' influencing art, fashion, gastronomy, and culture.\n\nIf you would like to' +
192-
' know more about the history of France or the city of Paris, please let me' +
193-
' know!'
194-
},
195-
streamedResponse: {
196-
inputTokens: 7,
197-
outputTokens: 7,
198-
text: 'The capital of France is Paris.'
199-
}
200-
},
157+
// TODO(sabrenner): input messages are undefined?
158+
// {
159+
// provider: PROVIDER.COHERE,
160+
// modelId: 'cohere.command-r-v1:0',
161+
// userPrompt: prompt,
162+
// requestBody: {
163+
// message: prompt,
164+
// temperature,
165+
// max_tokens: maxTokens
166+
// },
167+
// response: {
168+
// inputTokens: 7,
169+
// outputTokens: 335,
170+
// cacheReadTokens: 0,
171+
// cacheWriteTokens: 0,
172+
// text: 'The current capital of France is Paris. It has been the capital since 1958 and' +
173+
// ' is also the most populous city in the country. Paris has a rich history and' +
174+
// ' is known for its iconic landmarks and cultural significance.\n\nThe history' +
175+
// ' of the capital of France is somewhat complex, with the city of Paris itself' +
176+
// ' having a long and fascinating past. There was a shift in the capital\'s location' +
177+
// ' over the centuries, with various cities and towns fulfilling the role. The' +
178+
// ' earliest French capital based on historical records is thought to be the city' +
179+
// ' of Tours. The capital moved to various locations, often due to political and' +
180+
// ' dynastic reasons, including cities like Reims and Orleans. Paris initially' +
181+
// ' became the capital during the era of the Louvre in the 14th century, under' +
182+
// ' the rule of King Philip IV.\n\nThe status of Paris as the capital of France' +
183+
// ' has been reaffirmed many times, even during the French Revolution and the' +
184+
// ' establishment of the First French Empire by Napoleon Bonaparte. The city\'s' +
185+
// ' significance grew further with its designation as the centre of the Department' +
186+
// ' of Seine. Paris remained the capital through the changes in regime, including' +
187+
// ' the restoration of the monarchy, the July Monarchy, the Second Empire, and' +
188+
// ' the establishment of the French Third Republic.\n\nModern France\'s political' +
189+
// ' system, following the end of the Second World War, saw the capital remain' +
190+
// ' in Paris. The city continues to be a cultural hub, attracting artists, writers,' +
191+
// ' and musicians from around the world. Paris remains a prominent global city,' +
192+
// ' influencing art, fashion, gastronomy, and culture.\n\nIf you would like to' +
193+
// ' know more about the history of France or the city of Paris, please let me' +
194+
// ' know!'
195+
// },
196+
// streamedResponse: {
197+
// inputTokens: 7,
198+
// outputTokens: 7,
199+
// text: 'The capital of France is Paris.'
200+
// }
201+
// },
201202
{
202203
provider: PROVIDER.META,
203204
modelId: 'meta.llama3-8b-instruct-v1:0',

‎packages/dd-trace/test/llmobs/cassettes/openai/openai_chat_completions_post_219658cc.yaml‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
interactions:
2+
- request:
3+
body: '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"What is
4+
the weather in New York City?"}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get
5+
the weather in a given city","parameters":{"type":"object","properties":{"city":{"type":"string","description":"The
6+
city to get the weather for"}}}}}],"tool_choice":"auto","stream":true,"stream_options":{"include_usage":true}}'
7+
headers:
8+
? !!python/object/apply:multidict._multidict.istr
9+
- Accept
10+
: - application/json
11+
? !!python/object/apply:multidict._multidict.istr
12+
- Accept-Encoding
13+
: - gzip, deflate
14+
? !!python/object/apply:multidict._multidict.istr
15+
- Accept-Language
16+
: - '*'
17+
? !!python/object/apply:multidict._multidict.istr
18+
- Connection
19+
: - keep-alive
20+
Content-Length:
21+
- '410'
22+
? !!python/object/apply:multidict._multidict.istr
23+
- Content-Type
24+
: - application/json
25+
? !!python/object/apply:multidict._multidict.istr
26+
- User-Agent
27+
: - OpenAI/JS 6.4.0
28+
? !!python/object/apply:multidict._multidict.istr
29+
- X-Stainless-Arch
30+
: - arm64
31+
? !!python/object/apply:multidict._multidict.istr
32+
- X-Stainless-Lang
33+
: - js
34+
? !!python/object/apply:multidict._multidict.istr
35+
- X-Stainless-OS
36+
: - MacOS
37+
? !!python/object/apply:multidict._multidict.istr
38+
- X-Stainless-Package-Version
39+
: - 6.4.0
40+
? !!python/object/apply:multidict._multidict.istr
41+
- X-Stainless-Retry-Count
42+
: - '0'
43+
? !!python/object/apply:multidict._multidict.istr
44+
- X-Stainless-Runtime
45+
: - node
46+
? !!python/object/apply:multidict._multidict.istr
47+
- X-Stainless-Runtime-Version
48+
: - v22.17.0
49+
? !!python/object/apply:multidict._multidict.istr
50+
- sec-fetch-mode
51+
: - cors
52+
method: POST
53+
uri: https://api.openai.com/v1/chat/completions
54+
response:
55+
body:
56+
string: 'data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_FOfwGtELG2od6UEZKIOg9c3T","type":"function","function":{"name":"get_weather","arguments":""}}],"refusal":null},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"zGW"}
57+
58+
59+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\""}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"34TgNLxdmdCRK"}
60+
61+
62+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"city"}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"covhD9t0pUjb"}
63+
64+
65+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\""}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"lUWK1fqiRgy"}
66+
67+
68+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"New"}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"SBm0l8w1hkARw"}
69+
70+
71+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"
72+
York"}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"raK6arPwTlI"}
73+
74+
75+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"
76+
City"}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"EN7HgoklOww"}
77+
78+
79+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"}}]},"logprobs":null,"finish_reason":null}],"usage":null,"obfuscation":"wd8s2xSmM3Xyc"}
80+
81+
82+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}],"usage":null,"obfuscation":"d0tZAczcCMwrZm"}
83+
84+
85+
data: {"id":"chatcmpl-CSw4x8Te053upyjQ9iUXktFGwbe3b","object":"chat.completion.chunk","created":1761012475,"model":"gpt-3.5-turbo-0125","service_tier":"default","system_fingerprint":null,"choices":[],"usage":{"prompt_tokens":65,"completion_tokens":16,"total_tokens":81,"prompt_tokens_details":{"cached_tokens":0,"audio_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"audio_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"obfuscation":"LKGhb8GvL"}
86+
87+
88+
data: [DONE]
89+
90+
91+
'
92+
headers:
93+
CF-RAY:
94+
- 991d34444adb7d18-EWR
95+
Connection:
96+
- keep-alive
97+
Content-Type:
98+
- text/event-stream; charset=utf-8
99+
Date:
100+
- Tue, 21 Oct 2025 02:07:56 GMT
101+
Server:
102+
- cloudflare
103+
Set-Cookie:
104+
- __cf_bm=IdNyfBOHfWBj_mBsHAlYh8nrMzoC7J8MxQqtcQHNgeE-1761012476-1.0.1.1-cpOs8CUoL4HF0cm9NmhB2T1Zj_ZPZQr.99BnM5b3trMjWVA.e9OmvLm6iwUvzbPm8DIHRNa24zpoOqp749wy.MoslcHCZHrAQY1FUrGKG5A;
105+
path=/; expires=Tue, 21-Oct-25 02:37:56 GMT; domain=.api.openai.com; HttpOnly;
106+
Secure; SameSite=None
107+
- _cfuvid=qIJBolCy7BMyQbSPPS9nL2cD9fA3UOJ2HIi7Xmcc.qM-1761012476034-0.0.1.1-604800000;
108+
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
109+
Strict-Transport-Security:
110+
- max-age=31536000; includeSubDomains; preload
111+
Transfer-Encoding:
112+
- chunked
113+
X-Content-Type-Options:
114+
- nosniff
115+
access-control-expose-headers:
116+
- X-Request-ID
117+
alt-svc:
118+
- h3=":443"; ma=86400
119+
cf-cache-status:
120+
- DYNAMIC
121+
openai-organization:
122+
- datadog-staging
123+
openai-processing-ms:
124+
- '362'
125+
openai-project:
126+
- proj_gt6TQZPRbZfoY2J9AQlEJMpd
127+
openai-version:
128+
- '2020-10-01'
129+
x-envoy-upstream-service-time:
130+
- '407'
131+
x-openai-proxy-wasm:
132+
- v0.1
133+
x-ratelimit-limit-requests:
134+
- '10000'
135+
x-ratelimit-limit-tokens:
136+
- '50000000'
137+
x-ratelimit-remaining-requests:
138+
- '9999'
139+
x-ratelimit-remaining-tokens:
140+
- '49999987'
141+
x-ratelimit-reset-requests:
142+
- 6ms
143+
x-ratelimit-reset-tokens:
144+
- 0s
145+
x-request-id:
146+
- req_ee0a1796ca9a48fbabc207b1a2b7e925
147+
status:
148+
code: 200
149+
message: OK
150+
version: 1

0 commit comments

Comments
 (0)