Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Why these changes are being introduced: Ingesting records is an essential feature of TIM and is how all records will be indexed, whether manually or via automation. The ingest command needs to create or update records in the correct index while enforcing our business logic around index settings, naming, promotion, record identifiers, etc. How this addresses that need: * Adds config and helper functions to: * Generate index settings from a file * Generate an index name from a source name using our naming convention * Get the source name from an index name * Iteratively parse records from a json file without loading the entire file into memory, as these files may be quite large * Generate an appropriate json request body using the syntax required by the OpenSearch bulk indexing API. * Adds functions in the opensearch module to: * Create an index using our index settings and mappings from config * Get the primary index for a source * Return either the primary index for a source or a new index for the source depending on supplied parameters and whether a primary index for the source already exists * Bulk index records from a json file to an index, logging indexing errors as they arise and returning summary result information when complete * Updates the cli ingest command to call the functions above as needed to index records, and log summary results when complete. * Adds tests and fixtures for all new functionality. * Updates README to include all optional ENV variables. Side effects of this change: None Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TIMX-76
- Loading branch information