Skip to content

Commit

Permalink
Minor updates to documentation, logging, and variable names for clarity
Browse files Browse the repository at this point in the history
  • Loading branch information
hakbailey committed Oct 20, 2022
1 parent 798a995 commit 8f91881
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 8 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ Takes input JSON (usually from EventBridge although it can be passed to a manual
- `source`: Short name for the source repository, must match one of the source names configured for use in transform and load apps. The provided source is passed to the transform and load app CLI commands, and is also used in the input/output file naming scheme for all steps of the pipeline.
- *Note*: if provided source is "aspace", a method option is passed to the harvest command (if starting at the extract step) to ensure that we use the "get" harvest method instead of the default "list" method used for all other sources. This is required because ArchivesSpace inexplicably provides incomplete oai-pmh responses using the "list" method.

#### Required harvest fields
#### Required OAI-PMH harvest fields

- `oai-pmh-host`: *required if next-step is extract*, not needed otherwise. Should be the base OAI-PMH URL for the source repository.
- `oai-metadata-format`: *required if next-step is extract*, optional otherwise. The metadata prefix to use for the OAI-PMH harvest command, must match an available metadata prefix provided by the `oai-pmh-host` (see source repository OAI-PMH documentation for details).
- `oai-pmh-host`: *required if next-step is extract via OAI-PMH harvest*, not needed otherwise. Should be the base OAI-PMH URL for the source repository.
- `oai-metadata-format`: *required if next-step is extract via OAI-PMH harvest*, optional otherwise. The metadata prefix to use for the OAI-PMH harvest command, must match an available metadata prefix provided by the `oai-pmh-host` (see source repository OAI-PMH documentation for details).

#### Optional fields

- `oai-set-spec`: optional, only used when limiting the record harvest to a single set from the source repository.
- `oai-set-spec`: optional, only used when limiting the OAI-PMH record harvest to a single set from the source repository.
- `verbose`: optional, if provided with value `"true"` (case-insensitive) will pass the `--verbose` option (debug level logging) to all pipeline task run commands.

### Example format input with all fields
Expand Down
4 changes: 3 additions & 1 deletion lambdas/alma_prep.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,10 @@ def extract_file_from_source_bucket_to_target_bucket(
file_contents = next(extract_tarfile(tar_file))
s3_client.upload_fileobj(file_contents, target_bucket, target_file_key)
logger.debug(
"File '%s' extracted and uploaded to bucket '%s' with new file name %s",
"File '%s' extracted from bucket '%s' and uploaded to bucket '%s' with new "
"file name %s",
source_file_key,
source_bucket,
target_bucket,
target_file_key,
)
Expand Down
6 changes: 3 additions & 3 deletions lambdas/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,10 +123,10 @@ def generate_load_commands(
files_to_index.append({"load-command": load_command})

promote_index_command = ["promote", "--index", new_index_name]
for key, value in config.INDEX_ALIASES.items():
if source in value:
for alias, sources in config.INDEX_ALIASES.items():
if source in sources:
promote_index_command.append("--alias")
promote_index_command.append(key)
promote_index_command.append(alias)

return {
"create-index-command": ["create", "--index", new_index_name],
Expand Down

0 comments on commit 8f91881

Please sign in to comment.