Dev#32
Conversation
Replace in-memory JSONArray readers with streaming BufferedReader-based readers for movies and series to avoid loading entire payloads into memory. Each reader now consumes an InputStream from MediaApi, parses one JSON object per line with JSONParser, tracks currentLine for checkpointing in ExecutionContext, skips already-processed lines on open, and closes the reader in close(). Added @slf4j logging and improved error logs. Also changed DelayChunkListener log level from INFO to DEBUG for the post-chunk delay message.
Add stream-based TMDB fetching and update readers: MediaApi gained updateTitlesStream(), MovieAPIImpl and TvSeriesApiImpl now return a GZIP InputStream with retry semantics. DataJumpUtils was refactored to expose getDataJumpTMDBStream(...) (returning an uncompressed GZIP InputStream) and to rebuild getDataJumpTMDBArray(...) by reading/parsing the stream line-by-line into a JSONArray with improved error handling. MovieJsonReader and SerieJsonReader now consume the JSONArray in-memory with index-based checkpointing instead of line-based BufferedReader parsing; update() persists nextIndex. Also changed DelayChunkListener logging from debug to info and expanded .gitignore entries. These changes centralize HTTP/stream handling, add better retry/error handling, and simplify JSON reader logic.
Replace bitwise & with logical && in MediaController input checks to correctly evaluate null/empty name conditions. In MediaServiceImpl, avoid casting Optional to Page when an id is provided: assemble a single-element list, use PageImpl and a safe pageable fallback so single-result queries behave like paged results. Add @SuppressWarnings where needed. Change artwork refresh loop to call MediaUtils.updateMediaWhenLastTimeUpdateMoreThanOneDay(...) before switching and add break statements to prevent fall-through. Make MediaUtils.updateMediaWhenLastTimeUpdateMoreThanOneDay public so it can be reused. Minor cleanup of an extraneous comment.
There was a problem hiding this comment.
Pull request overview
This PR appears to focus on improving persistence throughput (Hibernate batching + switching several entities from IDENTITY to SEQUENCE), reducing unnecessary media artwork refreshes, and adding a streaming option for TMDB “datajump” exports.
Changes:
- Add Hibernate batching settings and introduce DB sequences for multiple entities (plus index drops via Flyway).
- Refactor TMDB daily export fetching to support streaming (
InputStream) and reuse it to build theJSONArray. - Fix paging behavior for “find by id” flows and avoid repeated artwork updates unless the media is stale.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/main/resources/db/migration/V16__drop_indexes_create_seq.sql |
Creates sequences, restarts some sequences, and drops indexes as part of the move away from IDENTITY + improving write performance. |
src/main/resources/application.properties |
Enables Hibernate JDBC batching + ordered inserts/updates. |
src/main/java/com/espacogeek/geek/utils/MediaUtils.java |
Makes staleness-check helper public so services can gate updates. |
src/main/java/com/espacogeek/geek/utils/DataJumpUtils.java |
Adds getDataJumpTMDBStream and refactors array-building to read from a stream. |
src/main/java/com/espacogeek/geek/services/impl/MediaServiceImpl.java |
Fixes “find by id” paging and gates artwork updates behind the staleness check. |
src/main/java/com/espacogeek/geek/models/PeopleModel.java |
Switches ID generation to a sequence generator for batching compatibility. |
src/main/java/com/espacogeek/geek/models/MediaModel.java |
Switches ID generation to a sequence generator for batching compatibility. |
src/main/java/com/espacogeek/geek/models/ExternalReferenceModel.java |
Switches ID generation to a sequence generator for batching compatibility. |
src/main/java/com/espacogeek/geek/models/AlternativeTitleModel.java |
Switches ID generation to a sequence generator for batching compatibility. |
src/main/java/com/espacogeek/geek/data/api/impl/TvSeriesApiImpl.java |
Exposes the new streaming titles export method. |
src/main/java/com/espacogeek/geek/data/api/impl/MovieAPIImpl.java |
Exposes the new streaming titles export method. |
src/main/java/com/espacogeek/geek/data/api/MediaApi.java |
Adds a default updateTitlesStream() method to the API. |
src/main/java/com/espacogeek/geek/controllers/MediaController.java |
Replaces & with && in input validation guards. |
src/main/java/com/espacogeek/geek/controllers/BatchController.java |
Fixes the required role for stopping jobs. |
src/main/java/com/espacogeek/geek/batch/DelayChunkListener.java |
Lowers log level for per-chunk delay messages. |
.gitignore |
Expands ignored env/secrets/dependency/log patterns. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | ||
|
|
||
| ALTER TABLE medias DROP INDEX idx_title; | ||
| ALTER TABLE medias DROP INDEX idx_name; No newline at end of file |
There was a problem hiding this comment.
This migration drops idx_name/idx_title, but the corresponding JPA entities still declare these indexes via @Table(indexes=...) (MediaModel has idx_name, AlternativeTitleModel has idx_title). With spring.jpa.hibernate.ddl-auto=update, Hibernate may recreate the indexes on startup, effectively undoing this migration. If the goal is to permanently remove these indexes, also remove/adjust the @Index annotations (or set ddl-auto to validate).
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | |
| ALTER TABLE medias DROP INDEX idx_title; | |
| ALTER TABLE medias DROP INDEX idx_name; | |
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; |
|
|
||
| ALTER SEQUENCE seq_medias RESTART WITH 458830; | ||
| ALTER SEQUENCE seq_externals_references RESTART WITH 201324; | ||
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; |
There was a problem hiding this comment.
seq_people is created but never restarted to a value above the current max people.id_person. If the table already contains rows, the next insert using GenerationType.SEQUENCE can collide with existing IDs. Add an ALTER SEQUENCE seq_people RESTART WITH ... (ideally derived from current MAX(id_person)+1).
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | |
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | |
| ALTER SEQUENCE seq_people RESTART WITH 1000000; |
| @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "gen_people") | ||
| @SequenceGenerator(name = "gen_people", sequenceName = "seq_people", allocationSize = 50)) |
There was a problem hiding this comment.
PeopleModel now uses @SequenceGenerator but the class doesn't import jakarta.persistence.SequenceGenerator, so this won't compile. Add the missing import (or switch to a wildcard jakarta.persistence.* import consistent with other models).
| @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "gen_alternative_titles") | ||
| @SequenceGenerator(name = "gen_alternative_titles", sequenceName = "seq_alternative_titles", allocationSize = 50) |
There was a problem hiding this comment.
AlternativeTitleModel now uses @SequenceGenerator but this file doesn't import jakarta.persistence.SequenceGenerator, so it won't compile. Add the missing import (or consolidate JPA imports).
| ALTER SEQUENCE seq_externals_references RESTART WITH 201324; | ||
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | ||
|
|
||
| ALTER TABLE medias DROP INDEX idx_title; |
There was a problem hiding this comment.
In this migration, idx_title is dropped from medias, but the previous migration creates idx_title on alternative_titles (see V15). Dropping it from the wrong table will fail at runtime; drop idx_title from alternative_titles instead (and keep idx_name drop on medias if intended).
| ALTER TABLE medias DROP INDEX idx_title; | |
| ALTER TABLE alternative_titles DROP INDEX idx_title; |
| ALTER SEQUENCE seq_medias RESTART WITH 458830; | ||
| ALTER SEQUENCE seq_externals_references RESTART WITH 201324; | ||
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | ||
|
|
There was a problem hiding this comment.
The ALTER SEQUENCE ... RESTART WITH <hard-coded> values make this migration environment/data dependent. If IDs differ between environments (or future restores), this can either cause collisions or waste large ID ranges. Prefer computing MAX(id)+1 at migration time (e.g., via dynamic SQL) to make the migration deterministic.
| ALTER SEQUENCE seq_medias RESTART WITH 458830; | |
| ALTER SEQUENCE seq_externals_references RESTART WITH 201324; | |
| ALTER SEQUENCE seq_alternative_titles RESTART WITH 122752; | |
| DO $$ | |
| DECLARE | |
| v_next_value bigint; | |
| BEGIN | |
| -- Set seq_medias based on current data | |
| SELECT COALESCE(MAX(id), 0) + 1 INTO v_next_value FROM medias; | |
| PERFORM setval('seq_medias', v_next_value, false); | |
| -- Set seq_externals_references based on current data | |
| SELECT COALESCE(MAX(id), 0) + 1 INTO v_next_value FROM externals_references; | |
| PERFORM setval('seq_externals_references', v_next_value, false); | |
| -- Set seq_alternative_titles based on current data | |
| SELECT COALESCE(MAX(id), 0) + 1 INTO v_next_value FROM alternative_titles; | |
| PERFORM setval('seq_alternative_titles', v_next_value, false); | |
| END; | |
| $$; |
| @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "gen_medias") | ||
| @SequenceGenerator(name = "gen_medias", sequenceName = "seq_medias", allocationSize = 50) |
There was a problem hiding this comment.
MediaModel now uses @SequenceGenerator but there is no jakarta.persistence.SequenceGenerator import in this file, so it won't compile. Add the missing import (or use jakarta.persistence.*).
Qodana for JVM42 new problems were found
View the detailed Qodana reportTo be able to view the detailed Qodana report, you can either:
To get - name: 'Qodana Scan'
uses: JetBrains/qodana-action@v2025.3.1
with:
upload-result: trueContact Qodana teamContact us at qodana-support@jetbrains.com
|
vitorhugo-java
left a comment
There was a problem hiding this comment.
Fix imports and indexes
No description provided.