Elasticsearch 7.x.0 management instruction

1. Index management

Display all indices and aliases
cf run-task api --command "python cli.py display_index_alias" -m 2G --name display_index_alias
Create index 'xxxx_index' and two aliases ('xxxx_alias' and 'search_alias')
ex1: cf run-task api --command "python cli.py create_index case_index" -m 2G --name create_index_case

ex2: cf run-task api --command "python cli.py create_index ao_index" -m 2G --name create_index_ao

ex3: cf run-task api --command "python cli.py create_index arch_mur_index" -m 2G --name create_index_arch_mur
Delete an index
cf run-task api --command "python cli.py delete_index <index_name>" -m 2G --name delete_index

ex1: cf run-task api --command "python cli.py delete_index case_index" -m 2G --name delete_index_case

ex2: cf run-task api --command "python cli.py delete_index ao_index" -m 2G --name delete_index_ao

ex3: cf run-task api --command "python cli.py delete_index arch_mur_index" -m 2G --name delete_index_arch_mur
Display mapping by one index
ex1: cf run-task api --command "python cli.py display_mapping case_index" -m 2G --name display_mapping_case

ex2: cf run-task api --command "python cli.py display_mapping ao_index" -m 2G --name display_mapping_ao

ex3: cf run-task api --command "python cli.py display_mapping arch_mur_index" -m 2G --name display_mapping_arch_mur

2. Initialize legal data by one index when first time or mapping change

Initialize case_index legal data with downtime < 1.5 hours
ex1: cf run-task api --command "python cli.py initialize_legal_data case_index" -m 4G --name initialize_legal_data_case
Initialize ao_index legal data with downtime < 15 mins.
ex1: cf run-task api --command "python cli.py initialize_legal_data ao_index" -m 4G --name initialize_legal_data_ao
Initialize arch_mur_index legal data with downtime < 40 mins.
ex1: cf run-task api --command "python cli.py initialize_legal_data arch_mur_index" -m 4G --name initialize_legal_data_arch_mur

3. Update mapping and reload legal data by one index when mapping change

This command only works on ao_index with short downtime < 1 min.
(if on 'dev' and 'stage' spaces, run increase instance command first: cf scale api -i 2 -m 2G)
ex1: cf run-task api --command "python cli.py update_mapping_and_reload_legal_data ao_index" -m 4G --name update_mapping_reload_data_ao

4. Reload legal data by one index without downtime

Reload all case_index data without downtime
ex1: `cf run-task api --command "python cli.py reload_all_data_by_index case_index" -m 4G --name reload_all_data_case
Reload all ao_index data without downtime
ex1: `cf run-task api --command "python cli.py reload_all_data_by_index ao_index" -m 4G --name reload_all_data_ao
Reload all arch_mur_index data without downtime
ex1: `cf run-task api --command "python cli.py reload_all_data_by_index arch_mur_index" -m 4G --name reload_all_data_arch_mur

5. Reload legal data by one document type without downtime

Advisory opinions(type=advisory_opinions)
a) Reload all AO data
cf run-task api --command "python cli.py load_advisory_opinions" -m 2G --name load_all_advisory_opinions

b) Reload AO data from FROM_AO_NO to the newest AO
ex1: cf run-task api --command "python cli.py load_advisory_opinions 2023-01" -m 2G --name load_one_advisory_opinions
Current murs(type=murs and mur_type=current)
a) Rpload all current mur data
cf run-task api --command "python cli.py load_current_murs" -m 2G --name load_all_current_murs

b) Reload single current mur
ex1: cf run-task api --command "python cli.py load_current_murs 8003" -m 2G --name load_one_current_mur
Alternative Dispute Resolution(type=adrs)
a) Reload all adr data
cf run-task api --command "python cli.py load_adrs" -m 2G --name load_all_adrs

b) Reload single adr
ex1: cf run-task api --command "python cli.py load_adrs 008" -m 2G --name load_one_adrs
Admin fines (type=admin_fines)
a) Reload all af data
cf run-task api --command "python cli.py load_admin_fines" -m 2G --name load_all_admin_fines

b) Reload single af
ex: cf run-task api --command "python cli.py load_admin_fines 3314" -m 2G --name load_single_admin_fines
Regulations (type=regulations)
a) Reload all regulations data
cf run-task api --command "python cli.py load_regulations" -m 2G --name load_all_regulations
Statutes (type=statutes)
a) Reload all statutes data
cf run-task api --command "python cli.py load_statutes" -m 2G --name load_all_statutes
Archived murs(type=murs and mur_type=archived)
a) Reload all archived murs data
cf run-task api --command "python cli.py load_archived_murs" -m 2G --name upload_all_arch_mur

b) Reload single archived mur
ex1: cf run-task api --command "python cli.py load_archived_murs 400" -m 2G --name upload_one_arch_mur

Open log terminal cf logs api |grep "<task_name>"

6. Delete individual legal data by index and document type

Advisory opinions(ao_index, type/DOC_TYPE=advisory_opinions)
a) Delete all AO data
ex1: cf run-task api --command "python cli.py delete_doctype_from_es ao_index advisory_opinions" -m 2G --name delete_all_advisory_opinions

b) Delete single AO
ex1: cf run-task api --command "python cli.py delete_single_doctype_from_es ao_index advisory_opinions advisory_opinions_2021-08" -m 2G --name delete_one_advisory_opinions
Current murs(case_index, type/DOC_TYPE=murs and mur_type=current)
a) Delete all current mur data
ex1: cf run-task api --command "python cli.py delete_doctype_from_es case_index murs" -m 2G --name delete_all_current_murs

b) Delete single current mur
ex1: cf run-task api --command "python cli.py delete_single_doctype_from_es case_index murs mur_7212" -m 2G --name delete_one_current_mur
Alternative Dispute Resolution(case_index, type=adrs)
a) Delete all adr data
ex1: cf run-task api --command "python cli.py delete_doctype_from_es case_index adrs" -m 2G --name delete_all_adrs

b) Delete single adr
ex1: cf run-task api --command "python cli.py delete_single_doctype_from_es case_index adrs adr_008" -m 2G --name delete_one_adr
Admin fines (case_index, type=admin_fines)
a) Delete all af data
ex1: cf run-task api --command "python cli.py delete_doctype_from_es case_index admin_fines" -m 2G --name delete_all_admin_fines

b) Delete single af
ex1: cf run-task api --command "python cli.py delete_single_doctype_from_es case_index admin_fines af_35" -m 2G --name delete_single_admin_fines
Regulations (ao_index, type=regulations)
a) Delete all regulations data
ex: cf run-task api --command "python cli.py delete_doctype_from_es ao_index regulations" -m 2G --name delete_all_regulations
Statutes (ao_index, type=statutes)
No delete statutes feature
Archived murs(arch_mur_index, type=murs and mur_type=archived)
a) Delete all archived murs data
ex1: cf run-task api --command "python cli.py delete_doctype_from_es arch_mur_index murs" -m 2G --name delete_all_archived_murs

b) Delete single archived mur
ex1: cf run-task api --command "python cli.py delete_single_doctype_from_es arch_mur_index murs mur_400" -m 2G --name delete_one_archived_mur

Open log terminal cf logs api |grep "<task_name>"

7. Repository management

Display repository
cf run-task api --command "python cli.py display_repositories" -m 2G --name display_repositories
Register or Configure a repository
ex1: cf run-task api --command "python cli.py configure_snapshot_repository case_repo" -m 4G --name configure_snapshot_repository_case

ex2: cf run-task api --command "python cli.py configure_snapshot_repository ao_repo" -m 4G --name configure_snapshot_repository_ao

ex3: cf run-task api --command "python cli.py configure_snapshot_repository arch_mur_repo" -m 4G --name configure_snapshot_repository_arch_mur
Delete a repository
ex1: cf run-task api --command "python cli.py delete_repository repo_name" -m 4G --name delete_repository

8. Snapshot and restore management

Display snapshot
- Display all the snapshots in the repository.
ex1:cf run-task api --command "python cli.py display_snapshots case_repo" -m 2G --name display_snapshots_case

ex2: cf run-task api --command "python cli.py display_snapshots ao_repo" -m 2G --name display_snapshot_ao

ex3: cf run-task api --command "python cli.py display_snapshots arch_mur_repo" -m 2G --name display_snapshot_arch_mur
Display snapshot detail
- Display all the snapshot details (include uuid) in the repository.
cf run-task api --command "python cli.py display_snapshot_detail <repo_name> <snapshot pattern>" -m 2G --name display_snapshot_detail

ex1: cf run-task api --command "python cli.py display_snapshot_detail case_repo case_snapshot_2023*" -m 2G --name display_snapshot_detail_case

ex2: cf run-task api --command "python cli.py display_snapshot_detail ao_repo ao_snapshot_2023*" -m 2G --name display_snapshot_detail_ao

ex3: cf run-task api --command "python cli.py display_snapshot_detail arch_mur_repo arch_mur*" -m 2G --name display_snapshot_detail_arch_mur
Create snapshot
cf run-task api --command "python cli.py create_es_snapshot <index_name>" -m 4G --name snapshot_docs

ex1: cf run-task api --command "python cli.py create_es_snapshot case_index" -m 2G --name create_snapshot_case

ex2: cf run-task api --command "python cli.py create_es_snapshot ao_index" -m 2G --name create_snapshot_ao

ex3: cf run-task api --command "python cli.py create_es_snapshot arch_mur_index" -m 2G --name create_snapshot_arch_mur
Delete a snapshot
cf run-task api --command "python cli.py delete_snapshot repository_docs <snapshot_name>" -m 2G --name delete_snapshot

ex1: cf run-task api --command "python cli.py delete_snapshot case_repo case_snapshot_202010272134" -m 2G --name delete_snapshot_case

ex2: cf run-task api --command "python cli.py delete_snapshot ao_repo ao_snapshot_202010272132" -m 2G --name delete_snapshot_ao

ex3: cf run-task api --command "python cli.py delete_snapshot arch_mur_repo arch_mur_snapshot_202010272132" -m 2G --name delete_snapshot_arch_mur
Restore snapshot without downtime by index name and snapshot name.
cf run-task api --command "python cli.py restore_es_snapshot <repo_name> <snapshot_name> <index_name>" -m 4G --name restore_es_snapshot
ex1: cf run-task api --command "python cli.py restore_es_snapshot case_repo case_snapshot_202010272132 case_index" -m 2G --name restore_es_snapshot_case -m 2G --name restore_es_snapshot_case

ex2: cf run-task api --command "python cli.py restore_es_snapshot ao_repo ao_snapshot_202010272132 ao_index" -m 2G --name restore_es_snapshot_ao -m 2G --name restore_es_snapshot_ao

ex3: cf run-task api --command "python cli.py restore_es_snapshot arch_mur_repo arch_mur_snapshot_202010272132 arch_mur_index" -m 2G --name restore_es_snapshot_arch_mur
Restore snapshot with downtime (takes approximately < a second to restore)
cf run-task api --command "python cli.py restore_es_snapshot_downtime repository_docs <snapshot_name <index_name>" -m 4G --name restore_es_snapshot_downtime

ex1: cf run-task api --command "python cli.py restore_es_snapshot_downtime case_repo case_snapshot_202010272130 case_index" -m 2G --name restore_es_snapshot_downtime_case

ex2: cf run-task api --command "python cli.py restore_es_snapshot_downtime ao_repo ao_snapshot_202010272130 ao_index" -m 2G --name restore_es_snapshot_downtime_ao

ex3: cf run-task api --command "python cli.py restore_es_snapshot_downtime arch_mur_repo arch_mur_snapshot_202010272130 arch_mur_index" -m 2G --name restore_es_snapshot_downtime_arch_mur

9. Access public and private s3 buckets

=======on dev space======

Public s3 bucket (fec-s3-api): store all legal pdf files
aws s3 ls s3://cg-cc8e3d72-34c9-411d-b369-96c0ce6572fd
Private s3 bucket (fec-s3-snapshot): store elasticsearch snapshot files
aws s3 ls s3://cg-55a0b774-46d0-41ca-bdbd-39c931854856/es-backups/

=======on stage space======

Public s3 bucket (fec-s3-api): store all legal pdf files
aws s3 ls s3://cg-d3527d7d-0344-45b5-aab6-5a39d2aa409f
Private s3 bucket (fec-s3-snapshot): store elasticsearch snapshot files
aws s3 ls s3://cg-52c16ed3-879b-4163-951a-50b1c8b79f8b/es-backups/

=======on prod space======

Public s3 bucket (fec-s3-api): store all legal pdf files
aws s3 ls s3://cg-519a459a-0ea3-42c2-b7bc-fa1143481f74
Private s3 bucket (fec-s3-snapshot): store elasticsearch snapshot files
aws s3 ls s3://cg-a779d3fe-b4a5-4f36-ac11-704d858e0032/es-backups/

8. Test endpoints

9. Pre-load legal docs using headless app

Make a copy manifest_task_<space>.yml.template to manifest_task_<space>.yml
Modify manifest_task_<space>.yml
name: pre-load-legal-doc-kill-12182020
memory: 4G
disk_quota: 4G
command: "(python cli.py initialize_legal_data case_index && echo SUCCESS || echo FAIL) && sleep infinity"
...
fec-api-elasticsearch-<space>
Push headless app to space
cf push -f manifests/manifest_task_<space>.yml
check headless app: cf apps
Open log terminal: cf logs <task_name> wait 2 hours to load legal docs
Unbind headless app from services
cf us pre-load-legal-doc-kill-12222020 fec-api-elasticsearch
cf us pre-load-legal-doc-kill-12222020 fec-elasticache-redis
cf us pre-load-legal-doc-kill-12222020 fec-s3-api
cf us pre-load-legal-doc-kill-12182020 fec-s3-snapshot
cf us pre-load-legal-doc-kill-12182020 fec-creds-<space>
Re-create ARN and update env variable Follow wiki: Elasticsearch 7.x.0 service and private s3 bucket setup on 'space'
Rebuild the least api from circleci
Run the management commands to create snapshot/backup repositories.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elasticsearch 7.x.0 management instruction

1. Index management

2. Initialize legal data by one index when first time or mapping change

3. Update mapping and reload legal data by one index when mapping change

4. Reload legal data by one index without downtime

5. Reload legal data by one document type without downtime

6. Delete individual legal data by index and document type

7. Repository management

8. Snapshot and restore management

9. Access public and private s3 buckets

8. Test endpoints

9. Pre-load legal docs using headless app

Clone this wiki locally