feat(evals): add Expo quickstart eval - #164
Conversation
📝 WalkthroughWalkthroughAdds an Expo quickstart eval for Auth0 login using ChangesExpo quickstart
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint install failed: dependency version conflict. Check your lock file or package.json. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
subhankarmaiti
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
apps/auth0-evals/src/evals/quickstarts/expo/scaffold/package.json (1)
5-10: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove the obsolete Android-only build script.
The evaluation declares
npm run typecheckas its compile command, so retaining a Gradle build path reintroduces the native toolchain this change is intended to avoid and may mislead callers.[details]
[summary]Suggested change[/summary]"scripts": { "start": "expo start", "android": "expo run:android", "ios": "expo run:ios", "typecheck": "tsc --noEmit", - "build": "expo prebuild --platform android --no-install && cd android && ./gradlew assembleDebug" },[/details]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/auth0-evals/src/evals/quickstarts/expo/scaffold/package.json` around lines 5 - 10, Remove the obsolete "build" entry from the scripts object in package.json, leaving the existing start, android, ios, and typecheck commands unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/auth0-evals/src/evals/quickstarts/expo/graders.ts`:
- Around line 23-27: Replace the fixture-specific notContainsInSource check in
the Expo graders with a source-aware AST or regex validation that rejects
literal clientId values in source files while permitting environment-variable
and configuration references. Keep the existing allowance for app.json and
preserve the grader’s failure behavior and level.
- Around line 3-64: Add Vitest coverage for the grader definitions in
apps/auth0-evals/src/evals/quickstarts/expo/graders.ts (lines 3-64), covering
positive, negative, structural, security, compilation, and judge cases. Also add
Vitest tests for apps/auth0-evals/src/evals/quickstarts/expo/scaffold/App.tsx
(lines 5-26) covering both unauthenticated and authenticated rendering branches,
placing tests in the corresponding tests/ directory.
- Around line 31-36: Update the Expo graders’ matches checks to be file- and
config-node-aware rather than searching the entire workspace text. Require the
react-native-auth0 plugin and customScheme in the relevant source/config, parse
app.json, and verify the plugin object declares both domain and customScheme.
Restrict the Auth0Provider domain check to its opening tag, and add a regression
test under the package’s tests directory covering unrelated mentions.
In `@apps/auth0-evals/src/evals/quickstarts/expo/PROMPT.md`:
- Around line 9-13: Update the Expo task prompt to explicitly require the grader
contract: use react-native-auth0 with Auth0Provider and useAuth0, implement
authorize() and clearSession(), define a customScheme, avoid hard-coding the
client ID, gate rendering on loading state, register the customScheme plugin in
app.json, follow v5 hook/plugin patterns, and display the authenticated profile.
---
Nitpick comments:
In `@apps/auth0-evals/src/evals/quickstarts/expo/scaffold/package.json`:
- Around line 5-10: Remove the obsolete "build" entry from the scripts object in
package.json, leaving the existing start, android, ios, and typecheck commands
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f6099d72-426c-4c05-8bde-18bcce3b7b35
📒 Files selected for processing (7)
apps/auth0-evals/README.mdapps/auth0-evals/src/evals/quickstarts/expo/PROMPT.mdapps/auth0-evals/src/evals/quickstarts/expo/graders.tsapps/auth0-evals/src/evals/quickstarts/expo/scaffold/App.tsxapps/auth0-evals/src/evals/quickstarts/expo/scaffold/app.jsonapps/auth0-evals/src/evals/quickstarts/expo/scaffold/package.jsonapps/auth0-evals/src/evals/quickstarts/expo/scaffold/tsconfig.json
| export function defineGraders() { | ||
| return [ | ||
| // ── L1: Positive presence (correct SDK and patterns) ────────────────────── | ||
| contains('react-native-auth0', 'Uses the react-native-auth0 SDK', GraderLevel.L1), | ||
| contains('Auth0Provider', 'Wraps app with Auth0Provider', GraderLevel.L1), | ||
| contains('useAuth0', 'Uses the useAuth0 hook', GraderLevel.L1), | ||
| contains('authorize', 'Implements login via authorize()', GraderLevel.L1), | ||
| contains('clearSession', 'Implements logout via clearSession()', GraderLevel.L1), | ||
| contains('customScheme', 'Configures a customScheme for the callback URL', GraderLevel.L1), | ||
|
|
||
| // ── L2: Negative / anti-pattern detection ───────────────────────────────── | ||
| notContains('@auth0/auth0-react', 'No web React SDK (correct package is react-native-auth0)', GraderLevel.L2), | ||
| notContains('@auth0/auth0-spa-js', 'No browser SPA SDK in a native Expo app', GraderLevel.L2), | ||
| notContains( | ||
| 'expo-auth-session', | ||
| 'Does not fall back to expo-auth-session instead of the Auth0 SDK', | ||
| GraderLevel.L2, | ||
| ), | ||
|
|
||
| // ── L3: Security checks ────────────────────────────────────────────────── | ||
| notContainsInSource( | ||
| 'barkbook_client_abc123xyz', | ||
| 'No hardcoded client ID in source files (ok in app.json)', | ||
| GraderLevel.L3, | ||
| ), | ||
|
|
||
| // ── L4: Structural / behavioral correctness ─────────────────────────────── | ||
| compiles('TypeScript typechecks (tsc --noEmit)', GraderLevel.L4), | ||
| matches( | ||
| String.raw`react-native-auth0[\s\S]*?customScheme`, | ||
| 'Registers the react-native-auth0 Expo config plugin with a customScheme in app.json', | ||
| GraderLevel.L4, | ||
| ), | ||
| matches(String.raw`<Auth0Provider[\s\S]*?domain`, 'Auth0Provider configured with a domain prop', GraderLevel.L4), | ||
| judge( | ||
| 'Does the code handle the loading state (isLoading from useAuth0) before rendering ' + | ||
| 'auth-dependent UI? A correct implementation should not render login/logout UI while isLoading is true.', | ||
| GraderLevel.L4, | ||
| ), | ||
| judge( | ||
| 'Does the code wire login and logout correctly for react-native-auth0 on Expo? ' + | ||
| 'Specifically: authorize() for login and clearSession() for logout, each passing ' + | ||
| '{ customScheme: ... } as the second argument, matching the scheme configured in the app.json plugin.', | ||
| GraderLevel.L4, | ||
| ), | ||
|
|
||
| // ── L5: Version-specific API correctness ────────────────────────────────── | ||
| judge( | ||
| 'Does the code use the current react-native-auth0 v5 Expo patterns? ' + | ||
| 'Specifically: the Auth0Provider + useAuth0 hook API (not the legacy imperative Auth0 client class), ' + | ||
| 'and the react-native-auth0 Expo config plugin in app.json (not manual native iOS/Android edits)?', | ||
| GraderLevel.L5, | ||
| ), | ||
|
|
||
| // ── Holistic judge ─────────────────────────────────────────────────────── | ||
| judge( | ||
| 'Does the solution correctly integrate Auth0 into an Expo app with the react-native-auth0 SDK: ' + | ||
| 'Auth0Provider wrapping the app, the useAuth0 hook, authorize()/clearSession() login and logout with ' + | ||
| 'customScheme, the Expo config plugin registered in app.json, and user profile display guarded by isLoading?', | ||
| ), | ||
| ]; | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
Add the required Vitest coverage for both new functions.
apps/auth0-evals/src/evals/quickstarts/expo/graders.ts#L3-L64: test grader definitions across positive, negative, structural, security, compilation, and judge cases.apps/auth0-evals/src/evals/quickstarts/expo/scaffold/App.tsx#L5-L26: test unauthenticated and authenticated rendering branches.
As per coding guidelines, every new function and logic change must include or update Vitest tests in the corresponding tests/ directory.
📍 Affects 2 files
apps/auth0-evals/src/evals/quickstarts/expo/graders.ts#L3-L64(this comment)apps/auth0-evals/src/evals/quickstarts/expo/scaffold/App.tsx#L5-L26
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/quickstarts/expo/graders.ts` around lines 3 - 64,
Add Vitest coverage for the grader definitions in
apps/auth0-evals/src/evals/quickstarts/expo/graders.ts (lines 3-64), covering
positive, negative, structural, security, compilation, and judge cases. Also add
Vitest tests for apps/auth0-evals/src/evals/quickstarts/expo/scaffold/App.tsx
(lines 5-26) covering both unauthenticated and authenticated rendering branches,
placing tests in the corresponding tests/ directory.
Source: Coding guidelines
| notContainsInSource( | ||
| 'barkbook_client_abc123xyz', | ||
| 'No hardcoded client ID in source files (ok in app.json)', | ||
| GraderLevel.L3, | ||
| ), |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Reject hardcoded client IDs generically, not just the fixture value.
This check only forbids barkbook_client_abc123xyz; a solution can hardcode any other client ID and still pass. Add a source-aware AST or regex check that rejects literal clientId values while allowing environment/config references.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/quickstarts/expo/graders.ts` around lines 23 - 27,
Replace the fixture-specific notContainsInSource check in the Expo graders with
a source-aware AST or regex validation that rejects literal clientId values in
source files while permitting environment-variable and configuration references.
Keep the existing allowance for app.json and preserve the grader’s failure
behavior and level.
| matches( | ||
| String.raw`react-native-auth0[\s\S]*?customScheme`, | ||
| 'Registers the react-native-auth0 Expo config plugin with a customScheme in app.json', | ||
| GraderLevel.L4, | ||
| ), | ||
| matches(String.raw`<Auth0Provider[\s\S]*?domain`, 'Auth0Provider configured with a domain prop', GraderLevel.L4), |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg 'apps/auth0-evals/src/evals/quickstarts/expo/graders\.ts|apps/auth0-evals/src/evals|grader' || true
echo "== target file outline/contents =="
if [ -f apps/auth0-evals/src/evals/quickstarts/expo/graders.ts ]; then
wc -l apps/auth0-evals/src/evals/quickstarts/expo/graders.ts
cat -n apps/auth0-evals/src/evals/quickstarts/expo/graders.ts
fi
echo "== matches implementation =="
rg -n "function matches|const matches|matches\\(" apps/auth0-evals/src -S || true
echo "== quickstart candidates files =="
find apps/auth0-evals -type f \( -name '*.ts' -o -name '*.tsx' -o -name '*.json' \) | sed -n '1,200p'Repository: auth0/auth0-evals
Length of output: 22963
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package test files =="
find apps/auth0-evals -path '*tests*' -o -path '*test*' | sed -n '1,200p' || true
find apps/auth0-evals -iname '*grader*test*' -o -iname '*grader*spec*' | sed -n '1,200p' || true
echo "== all quickstarts =="
find apps/auth0-evals/src/evals -maxdepth 5 -type f | sed -n '1,250p'Repository: auth0/auth0-evals
Length of output: 6568
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== matches implementation =="
cat -n packages/evals-core/src/graders/executors/matches.ts
echo "== text-search-utils =="
cat -n packages/evals-core/src/graders/executors/text-search-utils.ts
echo "== scaffold files =="
cat -n apps/auth0-evals/src/evals/quickstarts/expo/scaffold/app.json
printf '\n--- App.tsx relevant ---\n'
cat -n apps/auth0-evals/src/evals/quickstarts/expo/scaffold/App.tsx | sed -n '1,220p'
echo "== tests mentioning expo/graders/matches =="
rg -n "expo|graders\\.ts|matches\\(|auth0-evals|quickstarts" packages tests apps --glob '*.{ts,tsx,js,mts}' --glob '!apps/auth0-evals/dist/**' -S | sed -n '1,240p'
echo "== package config tests =="
for f in apps/auth0-evals/package.json package.json； do
if [ -f "$f" ]; then
echo "--- $f ---"
jq '.scripts, .devDependencies, .dependencies' "$f"
fi
doneRepository: auth0/auth0-evals
Length of output: 28077
🌐 Web query:
auth0 react-native-auth0 expo customScheme app.json
💡 Result:
To integrate Auth0 with an Expo application using the react-native-auth0 SDK, you must configure the Auth0 config plugin within your app.json (or app.config.js) file [1][2]. Configuration in app.json Include the react-native-auth0 plugin in your expo.plugins array, providing your Auth0 domain and a customScheme [1][3]. The customScheme must be a unique, all-lowercase string containing no special characters [1][4]. { "expo": { "ios": { "bundleIdentifier": "com.yourcompany.yourapp" }, "android": { "package": "com.yourcompany.yourapp" }, "plugins": [ [ "react-native-auth0", { "domain": "your-tenant.auth0.com", "customScheme": "yourcustomscheme" } ] ] } } Important Requirements 1. Native Identifiers: You must explicitly define bundleIdentifier (iOS) and package (Android) in your app.json [1][5]. These are used by the SDK to generate the necessary callback URLs [5]. 2. Code Usage: The customScheme string configured in app.json must be passed as an option when calling the authorize and clearSession methods in your application code [6][5]. // Example usage await authorize( { scope: 'openid profile email' }, { customScheme: 'yourcustomscheme' }); If a customScheme is not provided in app.json, the SDK defaults to using the bundle identifier or package name suffixed with.auth0 as the URL scheme [2][5]. If you choose to use Android App Links, you may set the customScheme value to "https" [2][3]. Ensure that the customScheme and the callback URLs are correctly registered in your Auth0 Application settings [1][5].
Citations:
- 1: https://auth0.com/docs/quickstart/native/react-native-expo
- 2: https://github.com/auth0/react-native-auth0
- 3: https://www.npmjs.com/package/react-native-auth0
- 4: https://github.com/auth0/agent-skills/blob/main/plugins/auth0/skills/auth0-expo/references/setup.md
- 5: https://auth0.mintlify.app/docs/quickstart/native/react-native-expo
- 6: https://developer.auth0.com/resources/guides/mobile/react-native/expo-authentication
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package manifests =="
for f in package.json apps/auth0-evals/package.json packages/evals-graders/package.json packages/evals-core/package.json packages/evals/package.json; do
if [ -f "$f" ]; then
echo "--- $f ---"
jq '.type, .scripts, .dependencies, .devDependencies, .peerDependencies' "$f"
fi
done
echo "== read-only regex behavior probe ====="
node - <<'JS'
const patterns = [
[String.raw`react-native-auth0[\s\S]*?customScheme`, L4_EXPO_APP_TSX_MERGED],
[String.raw`<Auth0Provider[\s\S]*?domain`, L5_AUTH0_PROVIDER_BAD_WITH_FOLLOWING_DOMAIN],
];
for (const [pat, text] of patterns) {
const flags = 'im';
console.log({ passed: new RegExp(pat, flags).test(text), pattern: pat.substring(0, 80) });
}
function L4_EXPO_APP_TSX_MERGED() {
const app = `
import { react-native-auth0 from '`@auth0/react-native`'; }Repository: auth0/auth0-evals
Length of output: 3203
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package manifests =="
for f in package.json apps/auth0-evals/package.json packages/evals-graders/package.json packages/evals-core/package.json packages/evals/package.json; do
if [ -f "$f" ]; then
echo "--- $f ---"
jq '.type, .scripts, .dependencies, .devDependencies, .peerDependencies' "$f"
fi
done
echo "== read-only regex behavior probe ====="
python3 - <<'PY'
import re
text = 'import { reactNativeAuth0 } from "`@auth0/react-native`";\nconsole.log("customScheme not configured");\nAppJson={plugins:[]}'
pat = r'react-native-auth0[\s\S]*?customScheme'
print('plugin pattern passed:', bool(re.search(pat, text, flags=re.I)))
provider_text = '''<Auth0Provider domain="old.example.com">{/* nothing */}</Auth0Provider>\n<Auth0Provider domain="should-not-count.example.com"/>'''
pat2 = r'<Auth0Provider[\s\S]*?domain'
print('provider pattern passed:', bool(re.search(pat2, provider_text, flags=re.I | re.M)))
print('matches same source false positives:', any(
re.search(r'react-native-auth0[\s\S]{0,200}?customScheme', text, flags=re.I)
or re.search(r'<Auth0Provider[\s\S]{0,200}?domain', text, flags=re.I | re.M)
for text in [
'import { reactNativeAuth0 } from "`@auth0/react-native`";\nconsole.log("customScheme not configured");\nAppJson={plugins:[]}',
'<Auth0Provider domain="old.example.com">{/* nothing */}</Auth0Provider>\n<Auth0Provider domain="should-not-count.example.com"/>',
]
))
PYRepository: auth0/auth0-evals
Length of output: 2653
Make the structural checks file- and config-node-aware.
matches tests the entire workspace text, so the current Expo checks can pass from unrelated mentions of react-native-auth0, customScheme, and any later domain. Add explicit source/config checks for react-native-auth0 + customScheme occurrences before relying on this pattern, and parse app.json so the react-native-auth0 plugin object actually declares both domain and customScheme; match the provider’s domain prop on the opening <Auth0Provider> tag instead of the next domain in the source. This also needs a regression test in the package’s tests/ directory.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/quickstarts/expo/graders.ts` around lines 31 - 36,
Update the Expo graders’ matches checks to be file- and config-node-aware rather
than searching the entire workspace text. Require the react-native-auth0 plugin
and customScheme in the relevant source/config, parse app.json, and verify the
plugin object declares both domain and customScheme. Restrict the Auth0Provider
domain check to its opening tag, and add a regression test under the package’s
tests directory covering unrelated mentions.
Source: MCP tools
| ## Task | ||
| Add Auth0 login to my Expo app. | ||
|
|
||
| Domain: dev-barkbook.us.auth0.com | ||
| Client ID: barkbook_client_abc123xyz |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg 'apps/auth0-evals/src/evals/quickstarts/expo/(PROMPT\.md|graders\.ts|.*\.md)' || true
echo "== prompt =="
if [ -f apps/auth0-evals/src/evals/quickstarts/expo/PROMPT.md ]; then
cat -n apps/auth0-evals/src/evals/quickstarts/expo/PROMPT.md
fi
echo "== graders outline/text =="
if [ -f apps/auth0-evals/src/evals/quickstarts/expo/graders.ts ]; then
wc -l apps/auth0-evals/src/evals/quickstarts/expo/graders.ts
cat -n apps/auth0-evals/src/evals/quickstarts/expo/graders.ts
fi
echo "== search related quickstarts/graders =="
rg -n "logout|profile|isLoading|customScheme|CustomScheme|useAuth0|react-native-auth0|v5|Quickstart|Domain|Client ID|expo" apps/auth0-evals/src/evals -S || trueRepository: auth0/auth0-evals
Length of output: 31736
Align the Expo prompt with the grader contract.
graders.ts checks for react-native-auth0, Auth0Provider, useAuth0, authorize(), clearSession(), a customScheme, hard-coded client-ID absence, loading-state gating, plugin registration for customScheme in app.json, v5 hook/plugin patterns, and profile display. Update PROMPT.md to explicitly require these so submissions are not task-compliant but fail the grader.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/auth0-evals/src/evals/quickstarts/expo/PROMPT.md` around lines 9 - 13,
Update the Expo task prompt to explicitly require the grader contract: use
react-native-auth0 with Auth0Provider and useAuth0, implement authorize() and
clearSession(), define a customScheme, avoid hard-coding the client ID, gate
rendering on loading state, register the customScheme plugin in app.json, follow
v5 hook/plugin patterns, and display the authenticated profile.
Source: MCP tools
2 participants
Adds an Expo / React Native quickstart eval that measures how well agents integrate Auth0 into an Expo app using the
react-native-auth0SDK.The eval seeds a bare Expo SDK 57 app (co-located
scaffold/) and asks the agent to add Auth0 login. Graders cover the correct SDK and Expo config-plugin wiring (Auth0Provider/useAuth0,authorize/clearSession,customScheme), reject web/SPA SDKs andexpo-auth-session, check no client ID is hardcoded in source, and finish with a holistic judge.Correctness is validated with a type-only check (
compile_command: npm run typecheck→tsc --noEmit) so it runs anywhere without a native Android/iOS toolchain. Also adds a configurableworkspace.compileCommandTimeoutMsso apps can raise the compile timeout when needed.Summary by CodeRabbit
New Features
Documentation