import_vendors.py

About

This should just be executed with Python directly. It will then ensure that the index vendors properly exists in ESand then import the vendor data to it.

It will print the return from ES for each query.

Example:

(venv) [jbanasik@jump scraper]$ python import_vendors.py
{'acknowledged': True, 'shards_acknowledged': True, 'index': 'vendors'}
{'_index': 'vendors', '_type': 'vendor', '_id': '12749cee-2d5e-56da-88f0-63676eff788c', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '55bf3088-c0d4-52db-aad8-f58386ee53ac', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '6ce59b40-b8ce-53fb-ab29-10166391d60b', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '4e57ad18-d2b0-5499-82d2-58917e29e4ab', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 1, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'fda97bcb-979b-5a95-9ebf-84f3160ad63d', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 1, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '7049ac8e-9423-5817-89a9-af3e29c6ec5d', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'd6ff9799-a256-5ad4-a95b-0829cfb898bb', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '02777c62-fff7-5e18-be6d-ce66ca2ea8aa', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 1, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '173e67af-8ef6-5e58-aaee-b3f689199ce2', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 1, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'd994ae75-abdf-56ab-98b9-8365b3081591', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 2, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '4ecd54b0-1e38-5610-9199-ec249d69c305', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 2, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'e6c3a1db-182d-5725-a625-cf1818650490', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 1, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '825f8cc2-fc16-54f5-ad86-96869f73d46c', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 3, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '3fd1a868-d281-5209-b20a-88bdac20c8ad', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 2, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'b1e03f29-7e42-5f47-9056-6663569a8e28', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 3, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '11c89f4f-956c-5baa-b3bb-23ea60ef4e07', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 2, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '378fc2b5-930e-5f75-8896-d68310c7395b', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 4, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'a4a1af5f-e4f3-5085-b598-1844fcb57c27', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 5, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'b1517456-5c22-5028-8724-eb4df38ac7a0', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '99ce7503-c3c9-5719-8dbc-eefd79e2b9dc', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 3, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'a7d9be20-0546-5d16-aa35-a5806c5ed43c', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 7, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'd4dbc221-de62-5da5-ad59-9f5c743d837e', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 8, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '5c4ccd05-e0d3-5ee6-b536-7189b2aaacfe', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 4, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'f7c20fb1-a3ea-593a-bf7f-d2ce6b1e59d0', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 2, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'a99ceb5a-ee06-537b-a245-26b9acc2738a', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 3, '_primary_term': 1}

If ran again, ES updates the entries and fails on creation of the index, which is fine.

(venv) [jbanasik@jump scraper]$ python import_vendors.py
{'error': {'root_cause': [{'type': 'resource_already_exists_exception', 'reason': 'index [vendors/INdvvQ55SKixcNz25gJ1sw] already exists', 'index_uuid': 'INdvvQ55SKixcNz25gJ1sw', 'index': 'vendors'}], 'type': 'resource_already_exists_exception', 'reason': 'index [vendors/INdvvQ55SKixcNz25gJ1sw] already exists', 'index_uuid': 'INdvvQ55SKixcNz25gJ1sw', 'index': 'vendors'}, 'status': 400}
{'_index': 'vendors', '_type': 'vendor', '_id': '12749cee-2d5e-56da-88f0-63676eff788c', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 9, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '55bf3088-c0d4-52db-aad8-f58386ee53ac', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 5, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '6ce59b40-b8ce-53fb-ab29-10166391d60b', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 4, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '4e57ad18-d2b0-5499-82d2-58917e29e4ab', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'fda97bcb-979b-5a95-9ebf-84f3160ad63d', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 5, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '7049ac8e-9423-5817-89a9-af3e29c6ec5d', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 3, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'd6ff9799-a256-5ad4-a95b-0829cfb898bb', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 4, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '02777c62-fff7-5e18-be6d-ce66ca2ea8aa', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 5, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '173e67af-8ef6-5e58-aaee-b3f689199ce2', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 10, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'd994ae75-abdf-56ab-98b9-8365b3081591', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 11, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '4ecd54b0-1e38-5610-9199-ec249d69c305', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 7, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'e6c3a1db-182d-5725-a625-cf1818650490', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 4, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '825f8cc2-fc16-54f5-ad86-96869f73d46c', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 12, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '3fd1a868-d281-5209-b20a-88bdac20c8ad', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'b1e03f29-7e42-5f47-9056-6663569a8e28', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 8, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '11c89f4f-956c-5baa-b3bb-23ea60ef4e07', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '378fc2b5-930e-5f75-8896-d68310c7395b', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 13, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'a4a1af5f-e4f3-5085-b598-1844fcb57c27', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 14, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'b1517456-5c22-5028-8724-eb4df38ac7a0', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 15, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '99ce7503-c3c9-5719-8dbc-eefd79e2b9dc', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 7, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'a7d9be20-0546-5d16-aa35-a5806c5ed43c', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 16, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'd4dbc221-de62-5da5-ad59-9f5c743d837e', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 17, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': '5c4ccd05-e0d3-5ee6-b536-7189b2aaacfe', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 9, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'f7c20fb1-a3ea-593a-bf7f-d2ce6b1e59d0', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 5, '_primary_term': 1}
{'_index': 'vendors', '_type': 'vendor', '_id': 'a99ceb5a-ee06-537b-a245-26b9acc2738a', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 7, '_primary_term': 1}