Skip to content

Latest commit

 

History

History
536 lines (461 loc) · 44.6 KB

NEWS.md

File metadata and controls

536 lines (461 loc) · 44.6 KB
Raw

2024-xx-xx 5.8.0-SNAPSHOT

  • MODSOURCE-506 Remove rawRecord field from source record
  • MODSOURCE-709 MARC authority record is not created when use Job profile with match profile by absent subfield/field
  • MODSOURCE-677 Import is completed with errors when control field that differs from 001 is used for marc-to-marc matching
  • MODSOURCE-722 deleteMarcIndexersOldVersions: relation "marc_records_tracking" does not exist

2023-10-13 v5.7.0

  • MODSOURCE-648 Upgrade mod-source-record-storage to Java 17
  • MODSOURCE-601 Optimize Insert & Update of marc_records_lb table
  • MODSOURCE-635 Delete marc_indexers records associated with "OLD" source records
  • MODSOURCE-636 Implement async migration service
  • MODSOURCE-674 Ensure only one background job can be triggered to clean up outdated marc indexers
  • MODSOURCE-530 Fix duplicate records in incoming file causes problems after overlay process with no error reported
  • MODSOURCE-690 Make changes in SRS post processing handler to update MARC for shared Instance
  • MODSOURCE-646 Make changes to perform MARC To MARC Matching in Local Tenant & Central Tenant
  • MODSOURCE-667 Upgrade folio-kafka-wrapper to 3.0.0 version

Asynchronous migration job API

METHOD URL DESCRIPTION
POST /source-storage/migrations/jobs Initialize asynchronous migration job
GET /source-storage/migrations/jobs/{jobId} Get asynchronous migration job entity by its id

2023-03-18 v5.6.2

  • MODSOURCE-585 Data import matching takes incorrect SRS records into consideration
  • MODDATAIMP-786 Update data-import-util library to v1.11.0

2023-02-17 v5.6.0

2022-10-19 v5.5.0

2022-09-02 v5.4.2

  • MODDICORE-248 MARC field protections apply to MARC modifications of incoming records when they should not

2022-08-25 v5.4.1

  • MODSOURCE-528 Data Import Updates should add 035 field from 001/003, if it's not HRID or already exists
  • MODSOURCE-531 Can't update "MARC" record, which was created by stopped import job.
  • MODSOURCE-535 Upgrade folio-di-support, folio-liquibase-util, PostgreSQL, Vert.x
  • MODSOURCE-534 Allow matching on created 035

2022-06-27 v5.4.0

  • MODSOURCE-470 Fix permission definition for updating records
  • MODSOURCE-459 Determine how to handle deleted authority records
  • MODSOURMAN-724 SRM does not process and save error records
  • MODSOURCE-477 Configure job to delete authority records
  • MODSOURCE-447 035 created from 001/003 is not working in SRS record when using a MARC Modification action in Data Import Job Profile
  • MODSOURCE-349 Function set_id_in_jsonb is present in migrated Juniper/Kiwi environment
  • MODSOURCE-496 Single record imports show the incorrect record number in the summary log 1st column
  • MODSOURMAN-779 Add "CANCELLED" status for Import jobs that are stopped by users.
  • MODSOURCE-499 Alleviate Eventloop Blocking During Batch Save of Records
  • MODSOURMAN-801 Inventory Single Record Import: Overlays for Source=MARC Instances retain 003 when they shouldn't
  • MODSOURCE-495 Logs show incorrectly formatted request id
  • MODSOURCE-509 Data Import Updates should add 035 field from 001/003, if it's not HRID or already exists
  • MODSOURCE-531 Can't update "MARC" record, which was created by stopped import job.

2022-06-02 v5.3.3

  • MODSOURCE-508 Inventory Single Record Import: Overlays for Source=MARC Instances retain 003 when they shouldn't

2022-04-04 v5.3.2

  • MODDATAIMP-645 Fixed update a MARC authority record multiple times
  • MODSOURCE-489 Records that are overlaid multiple times via Inventory Single Record Import can't be overlaid after the second time
  • MODSOURCE-447 035 created from 001/003 is not working in SRS record when using a MARC Modification action in Data Import Job Profile
  • MODSOURCE-349 Remove unused function set_id_in_jsonb

2022-03-25 v5.3.1

2021-02-22 v5.3.0

  • MODSOURCE-461 Upgrade RMB and Vertx versions that contain fixes for the connection pool
  • MODSOURCE-420 The support for 'maxPoolSize' (DB_MAXPOOLSIZE) from RMB was added
  • MODSOURCE-419 Upgrade to RAML Module Builder 33.2.x
  • MODDATAIMP-419 Fix "'idx_records_matched_id_gen', duplicate key value violates unique constraint"
  • MODSOURCE-261 Cover kafka handlers with tests and fix ignored
  • MODSOURCE-387 Optimistic locking: mod-source-record-storage modifications
  • MODSOURCE-290 Implement ProcessRecordErrorHandler for Kafka Consumers
  • MODSOURCE-401 Left anchored srs api queries don't complete on kiwi bugfest
  • MODSOURCE-402 Properly handle DB failures during events processing
  • MODDATAIMP-491 Improve logging to be able to trace the path of each record and file_chunks
  • MODSOURCE-429 Authority update: Implement match handler
  • MODSOURCE-434 Remove Kafka cache from QuickMarc handlers
  • MODDATAIMP-623 Remove Kafka cache initialization and Maven dependency
  • MODSOURCE-446 Extend MARC-MARC matching to 9xx and 0xx fields
  • MODSOURCE-450 Increase folio-liquibase-util and liquibase schema version
  • MODSOURCE-438 Fix SQL exceptions regarding connection termination when running CREATE import
  • MODSOURCE-457 Delete Authority: Create processor for Delete MARC Authority record

2022-02-09 v5.2.8

  • MODSOURCE-438 Fix SQL exceptions regarding connection termination when running CREATE import
  • MODSOURCE-452 Updated dependencies on liquibase-util, data-import-processing-core and folio-kafka-wrapper

2022-01-13 v5.2.7

2021-12-15 v5.2.6

2021-11-22 v5.2.5

2021-11-19 v5.2.4

  • MODSOURCE-413 Update data-import-processing-core to v3.2.5 to enable fix of duplicating values of repeatable control fields on MARC update

2021-11-16 v5.2.3

2021-11-11 v5.2.2

  • MODSOURCE-401 Left anchored srs api queries don't complete on kiwi bugfest

2021-10-29 v5.2.1

  • MODSOURCE-399 Fix "'idx_records_matched_id_gen', duplicate key value violates unique constraint"
  • MODSOURCE-390 Fix the effect of DI_ERROR messages when trying to duplicate records on the import job progress bar
  • MODSOURCE-393 Enable fix of duplicate control fields on MARC update (update data-import-processing-core to v3.2.2)

2021-09-30 v5.2.0

2021-09-08 v5.1.6

  • MODSOURCE-368 Fix "fill-instance-hrid" script for envs where it was applied before
  • MODSOURCE-360 The DB type MARC did not get updated to MARC_BIB

2021-09-02 v5.1.5

  • MODSOURCE-357 Improve "fill-instance-hrid" script for avoid failing if invalid data exists.

2021-08-04 v5.1.4

  • MODSOURCE-345 Error log for unknown event type
  • MODPUBSUB-187 Add support for max.request.size configuration for Kafka messages
  • Update data-import-processing-core dependency to v3.1.4
  • Update folio-kafka-wrapper dependency to v2.3.3

2021-07-21 v5.1.3

  • MODSOURCE-329 Create script to clean up Snapshot statuses in mod-source-record-storage
  • MODSOURMAN-451 Log details for Inventory single record imports for Overlays
  • MODSOURMAN-508 Log details for Inventory single record imports for Overlays - Part 2
  • Update data-import-processing-core dependency to v3.1.3

2021-06-25 v5.1.2

  • MODSOURCE-323 Change dataType to have common type for MARC related subtypes
  • Update data-import-processing-core dependency to v3.1.2

2021-06-22 v5.1.1

2021-06-11 v5.1.0

2021-xx-xx v5.0.6

  • MODSOURCE-329 Create script to clean up Snapshot statuses in mod-source-record-storage

2021-06-17 v5.0.5

  • MODSOURCE-318 Set processingStartedDate for snapshot when updating via QM
  • MODSOURCE-310 Handling 001/003/035 handling in SRS for MARC bib records broken
  • MODSOURCE-301 Cannot import GOBI EDIFACT invoice
  • MODSOURCE-265 Implement presence or absence searches
  • Update folio-kafka-wrapper to v2.0.8
  • Update data-import-processing-core to v3.0.3

2021-05-22 v5.0.4

  • MODSOURCE-280 Issue with database migration for Iris release
  • MODSOURCE-278 Take into account saved records from failed data import on generation calculation. Prevent import hanging if records saving failed
  • MODSOURCE-285 Order of execution keeps marc updates from occurring
  • MODSOURCE-295 Multiple records are found by match criteria after subsequent updates

2021-04-23 v5.0.3

2021-04-22 v5.0.2

2021-04-09 v5.0.1

  • MODSOURCE-222 Design API & implement data structures and end-points for search functionality in mod-source-record-storage
  • MODSOURCE-223 Design and implement service layer for MARC search functionality
  • MODSOURCE-255 Return HTTP response according to the schema
  • MODSOURCE-257 Implement the search by position
  • MODSOURCE-256 Implement the search by the date range
  • MODSOURCE-258 Implement NOT searches
  • MODSOURCE-260 Position count should begin at 0 when searching control fields
  • MODSOURCE-263 SRS MARC Bib column indicates the MARC record was "Created" for "Update" operation

2021-03-12 v5.0.0

  • MODSOURMAN-385 Enable OCLC update processing.
  • MODSOURCE-217 If incoming MARC bib lacks 001, sometimes it is added and sometimes not.[BUGFIX].
  • MODSOURCE-218 Performance issues with SRS requests when instance ID is not found
  • MODSOURCE-232 Making Instance Records Suppress from Discovery with the Batch Import is not reflected in the SRS
  • MODSOURCE-220 Migration script between Goldenrod-hotfix-5 and Honeysuckle.
  • MODSOURCE-233 Support for storing EDIFACT records in the database
  • MODSOURCE-216 Update MARC 005 field when MARC record has changes
  • MODSOURCE-204 Join query for get records. Add records_lb index for order column.
  • MODSOURCE-203 Ensure queries with offset include order. Add records_lb order index to liquibase change log.
  • MODSOURCE-202 Records stream API using reactivex.
  • MODSOURCE-201 Source Records stream API using reactivex.
  • MODSOURCE-238 Use docker-maven-plugin to support build with newer versions of postgres.
  • MODSOURCE-242 Use Testcontainers for tests.
  • MODSOURCE-234 Add recordType query parameter to Records and Source Records APIs. Update join queries to condition on record_type. Add database index for record_type column of records_lb table.
  • MODSOURCE-205 Batch API implementation redesign.
  • MODSOURCE-177 Use kafka to receive chunks of parsed records for saving
  • MODSOURCE-225 Add personal data disclosure.
  • MODSOURCE-225 Add personal data disclosure.
  • MODSOURCE-239 Upgrade to RAML Module Builder 32.x.
  • MODSOURCE-250 Make tenant API asynchronous.
  • MODSOURCE-248 Incoming MARC Bib with 003, but no 001 should not create an 035[BUGFIX].

Stream Records API

METHOD URL DESCRIPTION
GET /source-storage/stream/records Stream collection of records; including raw, parsed, and error record
GET /source-storage/stream/source-records Stream collection of source records; latest generation with parsed record

2020-11-20 v4.1.3

2020-10-22 v4.1.2

  • MODSOURCE-200 Post to /source-storage/batch/records returns "null value in column "description" violates not-null constraint" [BUGFIX]
  • MODSOURCE-174 Disable CQL2PgJSON & CQLWrapper extra logging in mod-source-record-storage.
  • MODSOURCE-207 Upgrade to RMB v31.1.5
  • MODSOURCE-199 Store ParsedRecord content as json instead of string

2020-10-09 v4.1.1

2020-10-09 v4.1.0

2020-06-25 v4.0.0

Snapshots API

METHOD URL DESCRIPTION
GET /source-storage/snapshots Get list of snapshots
POST /source-storage/snapshots Create new snapshot
PUT /source-storage/snapshots/{jobExecutionId} Update snapshot
GET /source-storage/snapshots/{jobExecutionId} Get snapshot by JobExecution id
DELETE /source-storage/snapshots/{jobExecutionId} Delete snapshot and all records by JobExecution id

Records API

METHOD URL DESCRIPTION
GET /source-storage/records Get list of records
POST /source-storage/records Create new record
PUT /source-storage/records/{id} Update record
GET /source-storage/records/{id} Get record by id
GET /source-storage/records/{id}/formatted Get formatted record by sourceRecordId or by instanceId
PUT /source-storage/records/{id}/suppress-from-discovery Change suppress from discovery flag for record
DELETE /source-storage/records/{id} Set record state to DELETED

Source Records API

METHOD URL DESCRIPTION
GET /source-storage/source-records Get list of source records
POST /source-storage/source-records Get list of source records from list of ids
GET /source-storage/source-records/{id} Get source record by sourceRecordId or by instanceId

Batch Records API

METHOD URL DESCRIPTION
POST /source-storage/batch/records Create new records
PUT /source-storage/batch/parsed-records Update a collection of ParsedRecords

Event Handlers API

METHOD URL DESCRIPTION
POST /source-storage/handlers/inventory-instance Update record instance id from inventory instance event
POST /source-storage/handlers/updated-record Crete updated record with generation under new snapshot

Test Records API

METHOD URL DESCRIPTION
POST /source-storage/populate-test-marc-records Fill db with test marc records

2020-06-10 v3.2.0

2020-04-23 v3.1.4

2020-04-07 v3.1.3

  • Fixed processing of DataImportEventPayload
  • MODSOURCE-117 Added migration script to remove deprecated indexes

2020-03-27 v3.1.2

  • Added Instance id values to sample data
  • Fixed GET source-storage/sourceRecords endpoint to allow filtering by recordId
  • Updated reference on ramls/raml-storage

2020-03-13 v3.1.1

  • Updated reference on ramls/raml-storage

2020-03-06 v3.1.0

  • Updated RMB to version 29.1.5
  • Added module registration as publisher/subscriber to mod-pubsub
  • Added condition for generation calculate
  • Implemented event handler that updates instance id for MARC bib records

2020-01-03 v3.0.2

  • Added records metadata filling for batch api

2019-12-17 v3.0.1

  • Fixed timeout on GET /source-storage/sourceRecords
  • Changed default container memory allocation

2019-12-04 v3.0.0

  • Added migration script to fill externalIdHolder.instanceId fields in records table
  • Applied new JVM features to manage container memory

2019-11-04 v2.7.0

  • Changed payload of endpoint for updating parsed records from ParsedRecordCollection to RecordCollection
  • Added order number to the record
  • Fixed sorting records by order number
  • Broken down source-record-storage interface into smaller ones: source-storage-snapshots, source-storage-records, source-storage-test-records, source-storage-suppress-discovery.

2019-09-09 v2.6.0

  • Fixed security vulnerability with jackson databind
  • Changed response status on partial success of batch save/update - return 201/200 if at least one of the records was saved/updated, 500 if none of the records was saved/updated
  • Refactored DAOs to return saved/updated entities
  • Added suppress from discovery endpoint for change records value
  • Added delete records by jobExecution id endpoint
  • Added delete method to the handlers for _tenant interface
  • Filled in "fromModuleVersion" value for each "tables" and "scripts" section in schema.json
METHOD URL DESCRIPTION
PUT /source-storage/record/suppressFromDiscovery Change suppress from discovery flag for record
DELETE /source-storage/snapshots/{jobExecutionId}/records Delete records and snapshot by JobExecution id

2019-07-23 v2.5.0

  • Added endpoint for updating parsed records
  • Moved endpoint for updating parsed records into the new API interface for batch operations (URL is changed from '/parsedRecordsCollection' to '/batch/parsed-records')
  • Added error message list to ParsedRecords DTO
METHOD URL DESCRIPTION
GET /source-storage/snapshots Get list of snapshots
POST /source-storage/snapshots Create new snapshot
PUT /source-storage/snapshots/{jobExecutionId} Update snapshot
GET /source-storage/snapshots/{jobExecutionId} Get snapshot by JobExecution id
DELETE /source-storage/snapshots/{jobExecutionId} Delete snapshot by JobExecution id
GET /source-storage/records Get list of records
POST /source-storage/records Create new record
PUT /source-storage/records/{id} Update record
GET /source-storage/records/{id} Get record by id
DELETE /source-storage/records/{id} Mark record deleted
GET /source-storage/sourceRecords Get list of source records
POST /source-storage/populate-test-marc-records Fill db with test marc records
POST /source-storage/batch/records Create new records
PUT /source-storage/batch/parsed-records Update a collection of ParsedRecords
GET /source-storage/formattedRecords/{id} Get Formatted Record by sourceRecordId or by instanceId

2019-06-12 v2.4.0

  • Removed ParsedRecord and ErrorRecord id overriding on save.
  • Added endpoint to update a collection of ParsedRecords.
  • Added support for ParsedRecord.content represented in json.
  • Added endpoint for getting a formatted Record either by sourceRecordId or instanceId
  • Moved endpoint for saved record collection from /source-storage/recordsCollection to /source-storage/batch/records
  • Changed response. List of error messages was added that is filled if some record was not saved
METHOD URL DESCRIPTION
GET /source-storage/snapshots Get list of snapshots
POST /source-storage/snapshots Create new snapshot
PUT /source-storage/snapshots/{jobExecutionId} Update snapshot
GET /source-storage/snapshots/{jobExecutionId} Get snapshot by JobExecution id
DELETE /source-storage/snapshots/{jobExecutionId} Delete snapshot by JobExecution id
GET /source-storage/records Get list of records
POST /source-storage/records Create new record
PUT /source-storage/records/{id} Update record
GET /source-storage/records/{id} Get record by id
DELETE /source-storage/records/{id} Mark record deleted
GET /source-storage/sourceRecords Get list of source records
POST /source-storage/populate-test-marc-records Fill db with test marc records
POST /source-storage/batch/records Create new records
PUT /source-storage/parsedRecordsCollection Update a collection of ParsedRecords
GET /source-storage/formattedRecords/{id} Get Formatted Record by sourceRecordId or by instanceId

2019-05-17 v2.3.2

  • Added generation of rawRecord id only if it is null.

2019-05-16 v2.3.1

  • Fixed sorting on GET /source-storage/sourceRecords.

2019-05-10 v2.3.0

  • Fixed indexes creation for Source Records.
  • Optimized performance for a records search.

2019-05-03 v2.2.0

  • Updated parsedRecord.content and errorRecord.content to contain an object instead of String.
  • Updated saving and updating a Record using PreparedStatement.
  • Fixed issues with sample data population.
  • Added "suppressDiscovery" field for Records.
  • Changed delete logic for Records. Records are being marked as deleted and not deleted permanently from storage.
  • Added endpoint to create new records from records collection.
  • Improved performance for querying sourceRecords on GET /source-storage/sourceRecords.
METHOD URL DESCRIPTION
GET /source-storage/snapshots Get list of snapshots
POST /source-storage/snapshots Create new snapshot
PUT /source-storage/snapshots/{jobExecutionId} Update snapshot
GET /source-storage/snapshots/{jobExecutionId} Get snapshot by JobExecution id
DELETE /source-storage/snapshots/{jobExecutionId} Delete snapshot by JobExecution id
GET /source-storage/records Get list of records
POST /source-storage/records Create new record
PUT /source-storage/records/{id} Update record
GET /source-storage/records/{id} Get record by id
DELETE /source-storage/records/{id} Mark record deleted
GET /source-storage/sourceRecords Get list of source records
POST /source-storage/populate-test-marc-records Fill db with test marc records
POST /source-storage/recordsCollection Create new records

2019-03-25 2.1.1

  • Updated tenant API version.
  • Removed IMPORT_IN_PROGRESS and IMPORT_FINISHED statuses for Snapshot entity.

2019-03-20 v2.1.0

  • Implemented calculation of generation numbers for Records.

2019-02-26 v2.0.0

  • Changed project structure to contain server and client parts. Client builds as a lightweight java library.
  • Used folio-di-support library for Spring Dependency Injection.
  • Added a non-production endpoint to populate MARC records for testing purposes, which is available only in case "loadSample" parameter of TenantAttributes is set to true.
  • Renamed entities sourceRecord -> rawRecord, result -> sourceRecord.
  • Updated sourceRecord to contain rawRecord as well as parsedRecord.
  • Added sample sourceRecords. Sample data is populated during module initialization only in case "loadSample" parameter of TenantAttributes is set to true.
  • Changed logging configuration to slf4j.
  • Renamed endpoints
METHOD URL DESCRIPTION
GET /source-storage/snapshots Get list of snapshots
POST /source-storage/snapshots Create new snapshot
PUT /source-storage/snapshots/{jobExecutionId} Update snapshot
GET /source-storage/snapshots/{jobExecutionId} Get snapshot by JobExecution id
DELETE /source-storage/snapshots/{jobExecutionId} Delete snapshot by JobExecution id
GET /source-storage/records Get list of records
POST /source-storage/records Create new record
PUT /source-storage/records/{id} Update record
GET /source-storage/records/{id} Get record by id
DELETE /source-storage/records/{id} Delete record by id
GET /source-storage/sourceRecords Get list of source records
POST /source-storage/populate-test-marc-records Fill db with test marc records

2018-11-29 v1.0.0

  • Created API for managing Snapshot, Record and Result entities
METHOD URL DESCRIPTION
GET /source-storage/snapshot Get list of snapshots
POST /source-storage/snapshot Create new snapshot
PUT /source-storage/snapshot/{jobExecutionId} Update snapshot
GET /source-storage/snapshot/{jobExecutionId} Get snapshot by JobExecution id
DELETE /source-storage/snapshot/{jobExecutionId} Delete snapshot by JobExecution id
GET /source-storage/record Get list of records
POST /source-storage/record Create new record
PUT /source-storage/record/{id} Update record
GET /source-storage/record/{id} Get record by id
DELETE /source-storage/record/{id} Delete record by id
GET /source-storage/result Get list of results