Multiprocessing indexing not working on M1 mac

[quadrismegistus]

No description provided.

[rlskoeser]

@quadrismegistus did you test this in qa? I think we should confirm it works there before closing

[quadrismegistus]

Testing in QA now, looks like it's working but will check in when it's completed.

(ppa-3.9.0-dev-e433f8) conan@cdh-test-prosody1:/srv/www/ppa/current$ python manage.py index_pages -p 4
Indexing with 4 processes
INFO:parasolr.django.solrclient:Connecting to default Solr http://lib-solr8-staging.princeton.edu:8983/solr/cdh_ppa
ERROR:ppa.archive.models:Pairtree data for mdp.39015003633594 not found but status is Public
DEBUG:ppa.archive.gale:get https://api.gale.com/api/tools/generate_key 200: 0.216647 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/tools/generate_key 200: 0.213967 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/tools/generate_key 200: 0.545899 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCB0127495592 200: 0.384538 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0115631505 200: 0.366330 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0110723387 200: 0.534640 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0111689573 200: 0.489130 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0113146825 200: 0.554811 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0112087160 200: 1.053743 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCB0127909832 200: 0.492310 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCB0127088549 200: 2.150538 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCB0126881769 200: 0.473428 sec
INFO:parasolr.django.solrclient:Connecting to default Solr http://lib-solr8-staging.princeton.edu:8983/solr/cdh_ppa
  0% (68 of 1885529) |                                                                                                                                                       | Elapsed Time: 0:00:00 ETA:   0:54:10DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0115427928 200: 1.185924 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0112440735 200: 1.006570 sec
  0% (1694 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:01 ETA:   0:26:36DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0112440877 200: 0.782623 sec
DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0116757677 200: 0.866124 sec
  0% (2880 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:02 ETA:   0:22:47DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0111435722 200: 0.515090 sec
  0% (3146 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:02 ETA:   0:21:44DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0111140533 200: 0.617114 sec
  0% (5166 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:03 ETA:   0:18:55DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0116864444 200: 1.470558 sec
  0% (5587 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:03 ETA:   0:18:55DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0114589903 200: 1.474867 sec
  0% (7276 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:05 ETA:   0:22:15DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0116618490 200: 0.780807 sec
  0% (7633 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:05 ETA:   0:22:32DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCB0127785659 200: 1.423939 sec
  0% (8575 of 1885529) |                                                                                                                                                     | Elapsed Time: 0:00:07 ETA:   0:42:21DEBUG:ppa.archive.gale:get https://api.gale.com/api/v1/item/GALE%7CCW0115714112 200: 1.907809 sec
  0% (9355 of 1885529) | 

[quadrismegistus]

Finished!

Elapsed Time: 0:50:28 Time:  0:50:28
INFO:parasolr.django.solrclient:Connecting to default Solr http://lib-solr8-staging.princeton.edu:8983/solr/cdh_ppa

Items in Solr by item type: 1898219 pages, 6460 works

Should we close?

[rlskoeser]

yes! thanks for confirming & documenting