feat: derive all upload limits from single APP_MEMORY_MB config#92
feat: derive all upload limits from single APP_MEMORY_MB config#92NotYuSheng merged 11 commits intomainfrom
Conversation
Replace the scattered MAX_UPLOAD_SIZE_BYTES knob with APP_MEMORY_MB — a single value in .env that drives everything automatically: APP_MEMORY_MB JVM heap (-Xmx) Max upload Nginx body Timeout ───────────── ─────────────── ────────── ────────── ─────── 2048 MB 1536 MB 512 MB 562 MB 900 s 4096 MB 3072 MB 1024 MB 1074 MB 900 s 8192 MB 6144 MB 2048 MB 2098 MB 900 s Changes: - backend/docker-entrypoint.sh: computes JVM -Xmx, MAX_UPLOAD_SIZE_BYTES and ANALYSIS_TIMEOUT_SECONDS from APP_MEMORY_MB at container start - backend/Dockerfile: uses entrypoint script instead of bare java -jar - nginx/docker-entrypoint.sh: computes NGINX_MAX_BODY_SIZE (+50 MB multipart buffer) and NGINX_PROXY_TIMEOUT from APP_MEMORY_MB - nginx/nginx.conf.template: timeouts now templated via envsubst - application.yml: timeout-seconds reads ANALYSIS_TIMEOUT_SECONDS env - SystemController: GET /api/system/limits returns maxUploadBytes at runtime so the frontend does not need a build-time baked value - UploadPage: fetches /api/system/limits on mount, passes live maxUploadBytes to FileUploadZone (which already displays "max NMB") Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request centralizes system configuration by deriving JVM heap size, upload limits, and timeouts from a single APP_MEMORY_MB environment variable. It introduces entrypoint scripts for the backend and Nginx to perform these calculations and adds a new API endpoint for the frontend to retrieve upload limits dynamically. Review feedback suggests setting the initial JVM heap size equal to the maximum for performance, adding error logging to the frontend's limit-fetching logic, and deduplicating the timeout calculation logic shared between the backend and Nginx entrypoint scripts.
chmod on /docker-entrypoint.sh failed because it ran after USER spring:spring was set. Move the COPY + chmod before the USER instruction so they run as root, then copy the JAR after switching to the non-root user. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The frontend was hardcoded to give up after 60s regardless of backend settings, causing 'Analysis timeout' errors for any non-trivial file. - SystemController: add analysisTimeoutMs to /api/system/limits response - useAnalysisData: fetch analysisTimeoutMs from API before starting the poll loop; use it for the setTimeout duration instead of 60000 (falls back to 5 min if the endpoint is unreachable) With APP_MEMORY_MB=2048 the timeout is now 900s on both sides. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Build and save PacketEntity objects in chunks of 1000 per conversation instead of materialising the full list first - Clear the PacketInfo list after each conversation's packets are saved so the parser heap shrinks progressively during the save phase - Flush + clear the JPA EntityManager every 50 conversations to prevent Hibernate's first-level cache from accumulating unbounded - Bump hibernate.jdbc.batch_size 20 → 1000 and enable order_inserts / order_updates so saveAll() batches at the JDBC level Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Shows file name, size, estimated duration, elapsed timer, and an animated progress bar while the backend analysis is running. Estimation uses ~0.75s/MB heuristic; the bar is capped at 95% so it never completes prematurely. Includes a hint that analysis continues in the background if the user navigates away. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pcap4J throws ArrayIndexOutOfBoundsException when it encounters a truncated GTPv1 tunnel packet (payload length in header > actual bytes). The exception was propagating out of the while loop and aborting the whole analysis with 'Failed to analyze file'. Catch RuntimeException per-packet inside the read loop; log the skipped packet number at DEBUG level and continue with the rest of the file. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Protocol and Category pie charts now compute ResponsiveContainer height from the number of legend items (max(300, 240 + ceil(n/3)*24)) so the legend is never clipped on files with many protocols/categories - Analysis loading view: show static estimated duration instead of a countdown; remove unused 'remaining' variable Closes #93 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a centralized configuration model where system limits, such as JVM heap size, max upload size, and timeouts, are dynamically derived from a single APP_MEMORY_MB environment variable. Key improvements include enhanced memory management during PCAP analysis through JPA session flushing and batching, the addition of a system limits API to inform the frontend, and a more informative analysis loading UI with progress estimation. A suggestion was made to refactor the packet counting logic in PcapParserService to improve readability and maintainability.
…ory tab - LlmConfig: wire configured LLM_TIMEOUT into SimpleClientHttpRequestFactory so connect + read both time out after the configured seconds (default 60s) instead of hanging indefinitely on a silent host - StoryPage: remove useEffect that auto-fired generateStory on every visit; user now clicks Generate Story explicitly, avoiding a doomed request and 60 s spinner when no LLM server is configured Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
For large captures (e.g. bigFlows.pcap: 28 686 conversations, 17 736
at-risk) the previous code loaded every ConversationEntity into memory
at once and then dumped all 17 736 at-risk rows into the LLM prompt,
producing a ~1.4 MB prompt that would blow any model's context window.
Changes:
- Add targeted ConversationRepository queries:
findTopByFileIdOrderByTotalBytesDesc (Pageable) — top-N only
countByFileId — total conversation count
findAtRiskByFileIdLimited (LIMIT) — capped at-risk sample
countAtRiskByFileId — total at-risk count for summary line
findCategoryDistributionByFileId — GROUP BY aggregation, no entity load
- Remove conversationRepository.findByFileId(fileId) from StoryService
- buildUserPrompt now issues 5 small targeted queries instead of
materialising the full conversation list
- Security Alerts section capped at STORY_MAX_CONVERSATIONS rows;
total count still reported so the LLM knows the full scope
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Clarify that file metadata, protocol breakdown, and category distribution are always included - Note TLS certificate details are included in conversation entries - Document that security alerts are capped at STORY_MAX_CONVERSATIONS but the LLM is still told the total at-risk count - Reword limitations line to cover all excluded data in one place Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- backend/docker-entrypoint.sh: set -Xms to JVM_HEAP_MB (same as -Xmx) so the JVM pre-allocates the full heap at startup, avoiding growth pauses on a server that will quickly use its allocation - UploadPage.tsx: log console.error when /api/system/limits is unreachable instead of silently swallowing the error Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NotYuSheng
left a comment
There was a problem hiding this comment.
Addressed all 4 review comments: fixed -Xms/-Xmx parity and added console.error for the limits fetch error. Declined the DRY entrypoint suggestion (separate containers, no shared filesystem) and the packetNumber refactor (pre-increment + decrement on EOF is less readable than the current two-path approach).
Summary
MAX_UPLOAD_SIZE_BYTESwith a singleAPP_MEMORY_MBknob in.envthat automatically derives every related limit at container startupHow it works
APP_MEMORY_MB-Xmx)Formulas (all computed in the entrypoint scripts):
APP_MEMORY_MBAPP_MEMORY_MBAPP_MEMORY_MB, clamped to 300–900 sChanges
backend/docker-entrypoint.sh(new) — shell script that computes and exports all values, thenexec java -Xmx... -jar app.jarbackend/Dockerfile— switched from barejava -jarto the entrypoint scriptnginx/docker-entrypoint.sh— rewritten to derive body size and timeouts fromAPP_MEMORY_MBinstead of raw bytesnginx/nginx.conf.template— proxy timeouts now templated (${NGINX_PROXY_TIMEOUT})application.yml—timeout-secondsreadsANALYSIS_TIMEOUT_SECONDSenv var set by the entrypointSystemController(new) —GET /api/system/limitsreturns{ maxUploadBytes, maxUploadMb }at runtimeUploadPage— fetches/api/system/limitson mount; upload zone displays the live limit instead of a build-time baked valueTest plan
APP_MEMORY_MB=4096in.env, rundocker compose build && docker compose up -dMax upload size = 1024 MB,JVM heap = 3072 MBNginx body limit = 1074M,Proxy timeout = 900 sAPP_MEMORY_MB=2048) still shows "max 512MB"🤖 Generated with Claude Code