feat(evals): add bare React Native quickstart eval and fix judge corpus limits - #166
Merged
Conversation
|
Warning Review limit reached
Next review available in: 34 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR adds a React Native Barkbook quickstart, its Auth0 evaluation graders, native Android and iOS scaffolding, and judge updates for native-file exclusions and oversized corpus truncation. ChangesReact Native quickstart and judge support
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant QuickstartPrompt
participant defineGraders
participant BarkbookScaffold
participant LLMJudge
QuickstartPrompt->>defineGraders: provide Auth0 and callback requirements
defineGraders->>BarkbookScaffold: inspect generated React Native project
defineGraders->>LLMJudge: evaluate loading, API, and integration criteria
LLMJudge-->>defineGraders: return grader result
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The Expo and bare RN scaffolds now ship an AGENTS.md stating that no native toolchain is available and that npm run typecheck is the only verification needed. The framework renames it to whichever context file the active runner reads, so all four runners get it. Environment facts only — SDK API guidance stays in the skills so the no-skills configs keep measuring what the model knows on its own.
The workspace walker reads every file as UTF-8, so the bare React Native scaffold fed the judge gradle-wrapper.jar and the launcher PNGs as mojibake — about 115KB of a 172KB corpus, 5x over maxCodeChars. Every judge grader in that eval errored on size without the model ever being asked. Exclude binaries and generated native project files. .plist, .xml and .gradle stay in: judges assert on Info.plist, AndroidManifest.xml and app/build.gradle. RN is now ~24K chars. Separately, an oversized corpus threw, turning an infrastructure limit into a grader failure. Truncate with a visible notice instead.
subhankarmaiti
force-pushed
the
feat/bare-rn-quickstart-eval
branch
from
August 4, 2026 10:56
4033690 to
d61686b
Compare
subhankarmaiti
changed the title
subhankarmaiti
marked this pull request as ready for review
August 5, 2026 08:01
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/auth0-evals/src/evals/quickstarts/react-native/graders.ts`:
- Around line 14-96: Add Vitest coverage for defineGraders in the package’s test
directory. Verify the expected grader count and GraderLevel assignments, and
assert that the Android manifest-placeholder matchers are anchored to
android/app/build.gradle and validate both auth0Domain and auth0Scheme
configuration.
In
`@apps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/build.gradle`:
- Around line 100-105: Update the release build configuration in the release
block so it no longer uses signingConfigs.debug or the bundled debug.keystore.
Configure a protected release signing key sourced from CI secrets, and make
packaging fail when that key is unavailable rather than falling back to debug
credentials.
In `@apps/auth0-evals/src/evals/quickstarts/react-native/scaffold/App.tsx`:
- Around line 8-28: Add Vitest tests in the package tests/ directory for the App
component’s unauthenticated and authenticated UI branches, including the
conditional login/logout controls and profile rendering. Once the login handler
is wired, cover the empty userProfile behavior as well, using the existing
component test conventions and avoiding changes to unrelated App logic.
In
`@apps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold.xcodeproj/xcshareddata/xcschemes/scaffold.xcscheme`:
- Around line 30-41: Remove the dangling TestableReference for scaffoldTests and
blueprint 00E356ED1AD99517003FC87E from the Testables section of the scaffold
scheme, unless the corresponding test target is added to the project. Keep the
scheme valid for the targets currently defined in scaffold.xcodeproj.
In
`@apps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/LaunchScreen.storyboard`:
- Line 19: Update the LaunchScreen storyboard label identified by id GJd-Yh-RWb,
replacing its placeholder text “scaffold” with “Barkbook”; preserve all other
label attributes.
In `@packages/evals-core/src/graders/llm-judge.ts`:
- Around line 69-70: Update the truncation logic around judgeCode so the final
value never exceeds judgeMaxCodeChars, including when that limit is shorter than
notice; truncate the notice itself as needed while preserving the existing
full-notice behavior for larger limits. Add a regression test covering a limit
shorter than the notice.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 83e0b1e0-cb0d-4bf2-b356-f14af6339a4a
⛔ Files ignored due to path filters (11)
apps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-hdpi/ic_launcher.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-hdpi/ic_launcher_round.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-mdpi/ic_launcher.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-mdpi/ic_launcher_round.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-xhdpi/ic_launcher.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-xhdpi/ic_launcher_round.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-xxhdpi/ic_launcher.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-xxhdpi/ic_launcher_round.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-xxxhdpi/ic_launcher.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/mipmap-xxxhdpi/ic_launcher_round.pngis excluded by!**/*.pngapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/gradle/wrapper/gradle-wrapper.jaris excluded by!**/*.jar
📒 Files selected for processing (44)
AGENTS.mdapps/auth0-evals/src/evals/quickstarts/expo/scaffold/AGENTS.mdapps/auth0-evals/src/evals/quickstarts/react-native/PROMPT.mdapps/auth0-evals/src/evals/quickstarts/react-native/graders.tsapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/.bundle/configapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/.watchmanconfigapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/AGENTS.mdapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/App.tsxapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/Gemfileapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/build.gradleapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/debug.keystoreapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/proguard-rules.proapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/AndroidManifest.xmlapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/java/com/barkbook/app/MainActivity.ktapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/java/com/barkbook/app/MainApplication.ktapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/drawable/rn_edit_text_material.xmlapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/values/strings.xmlapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/app/src/main/res/values/styles.xmlapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/build.gradleapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/gradle.propertiesapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/gradle/wrapper/gradle-wrapper.propertiesapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/gradlewapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/gradlew.batapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/android/settings.gradleapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/app.jsonapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/babel.config.jsapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/index.jsapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/.xcode.envapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/Podfileapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold.xcodeproj/project.pbxprojapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold.xcodeproj/xcshareddata/xcschemes/scaffold.xcschemeapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/AppDelegate.swiftapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/Images.xcassets/AppIcon.appiconset/Contents.jsonapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/Images.xcassets/Contents.jsonapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/Info.plistapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/LaunchScreen.storyboardapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/ios/scaffold/PrivacyInfo.xcprivacyapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/metro.config.jsapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/package.jsonapps/auth0-evals/src/evals/quickstarts/react-native/scaffold/tsconfig.jsonpackages/evals-core/src/graders/executors/llm-judge.tspackages/evals-core/src/graders/llm-judge.tspackages/evals-core/tests/graders/engine.test.tspackages/evals-core/tests/graders/executors.test.ts
With a limit shorter than the notice itself, slice(0, 0) plus the whole notice produced a string longer than the budget being enforced. Clamp the notice too. Unreachable at the configured 32,768, but the function should not exceed a limit it exists to apply.
Contributor
Author
|
yes we can remove the assets for now, but later if we decide to try the complete build it will fail. So As we are planning to make it build as well, we should keep all assets |
sanchitmehtagit
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
react_native_quickstarteval covering Auth0 integration in a bare (non-Expo) React Native app, complementing the existing Expo eval. The scaffold is a minimal RN 0.86.2 project — Barkbook login/logout stub, fullandroid/andios/projects, no test or lint tooling — verified to typecheck and bundle.What it exposes: 21 graders across L1–L5, including L4 checks that the agent wired the native callback config for both platforms —
auth0Domain/auth0SchememanifestPlaceholders inandroid/app/build.gradleandCFBundleURLTypesinInfo.plist— plus L2 checks that it didn't reach for the Expo config plugin orreact-native-app-auth.The Android graders are file-anchored rather than plain substring matches.
matchesruns against every workspace file concatenated into// FILE: <path>blocks, so an unanchoredmanifestPlaceholders[\s\S]*?auth0Domaincan straddle two files and pass on code that never touchedbuild.gradle:Building this eval surfaced a judge bug that made it unscoreable. The grader engine reads every workspace file as UTF-8 with no binary filter, so this scaffold fed the judge
gradle-wrapper.jarand the launcher PNGs as mojibake — about 115KB of a 172KB corpus, 5x overmaxCodeChars. Every judge grader in the eval errored on size without the model ever being asked, capping the pass rate around 0.85 no matter how good the output was. The judge now skips binaries and generated native project files, which brings the corpus to ~24K;.plist,.xmland.gradlestay in, since the judges assert onInfo.plist,AndroidManifest.xmlandapp/build.gradle. With that,agent-skillsscores 100 at a 1.00 pass rate.An oversized corpus also threw rather than truncating, which turned an infrastructure limit into a grader failure. It now truncates with a visible notice.
Finally, the Expo and RN scaffolds ship an
AGENTS.mdtelling the agent there is no native toolchain and thatnpm run typecheckis the only verification needed. The framework renames that file to whichever context file the active runner reads, so all four runners get it. It carries environment facts only — SDK guidance stays in the skills, so the no-skills configs keep measuring what the model knows on its own.Summary by CodeRabbit
New Features
Bug Fixes
Tests