local dev experience

how-to/run-locally.md

@@ -1,6 +1,10 @@
 # SHARE Quickstart or: How I Learned to Stop Worrying and Love the Dock
 
-this guide guides you through setting up SHARE locally using Docker.
+this guide guides you through setting up SHARE locally using Docker
+for development and manual testing.
+
+this guide does NOT guide you to anything appropriate for the open Internet.
+
 
 ## pre-requisites
 - [git](https://git-scm.com/)
@@ -55,57 +59,58 @@ sharectl search setup --initial
 ```
 
 ### 3. start 'em up
-all other services can now be started (from the host machine):
+all other services can now be started from the host machine (upping `worker` ups all)
 ```
-docker-compose up -d rabbitmq worker web indexer frontend
+docker-compose up -d worker
 ```
 
 ## handy commands
 
-### start a docker shell
+### start a shell in a container
+there are several ways to open a shell with SHARE's python environment (including
+django's `manage.py` and SHARE's own `sharectl` utility, defined in `share/bin/`)
 
-this is the same command you ran in step 2:
+if `worker` is already up, can open a shell within that container:
+```
+docker-compose exec worker bash
+```
 
+if no services are up, can open a shell within a new, temporary `worker` container:
 ```
 docker-compose run --rm --no-deps worker bash
 ```
+(remove `--no-deps` if you'd like the other services started automatically)
 
 ### start a django shell
-
-this should be run inside the docker shell (see previous):
+this should be run inside a container (see previous):
 
 ```
 python manage.py shell_plus
 ```
 
 ## admin interface
-http://localhost:8003/admin -- (admin/password)
-
-## harvesting data
-> TODO: once share.osf.io/oaipmh is reliable, make it easy to init a local deployment by harvesting data from there
+http://localhost:8003/admin (username: "admin", password: "password")
 
-> also TODO: put some thought into unconvoluting the whole harvest-scheduling, ingest-disabling system
-
-for now, maybe grab a day of data from arxiv.org? at the moment, the `Source` needs to be marked
-`canonical` for the system to ingest its data -- either:
-  - update it in the admin interface: http://localhost:8003/admin/share/source/
-  - update it from the django shell:
+## using with local [osf.io](https://github.com/CenterForOpenScience/osf.io)
+0. [set up your local osf with docker](https://github.com/CenterForOpenScience/osf.io/blob/HEAD/README-docker-compose.md), if you haven't already
+1. in a SHARE container, run `python manage.py add_local_osf_user` and copy the access token from the output.
+    ```
+    # python manage.py add_local_osf_user
+    added user "my-local-osf" for local osf
+    access-token: THISISMYACCESSTOKENITISLONGANDINSCRUTABLEANDSECRET
     ```
-    Source.objects.filter(name='org.arxiv').update(canonical=True)
+2. add settings to your local osf's `website/settings/local.py`, including the access token from step 1:
     ```
+    SHARE_ENABLED = True
+    SHARE_PROVIDER_PREPEND = 'local'
+    SHARE_URL = 'http://192.168.168.167:8003'
+    SHARE_API_TOKEN = 'THISISMYACCESSTOKENITISLONGANDINSCRUTABLEANDSECRET'
+    ```
+    (you may need to restart osf services that use these settings)
+3. make things "public" on your local osf to start populating indexes
 
-next, choose a recent date, and start a harvest task for it from the docker shell:
 
-```
-sharectl schedule -t org.arxiv YYYY-MM-DD
-```
-
-you could watch its progress several different ways:
-  - looking at task queues in the rabbitmq management interface at http://localhost:15673/ (guest/guest)
-  - following the `worker` container's logs: `docker-compose logs -f worker`
-  - checking the result count as you refresh the search interface at http://localhost:8003/share/discover
-  - watching `IngestJob` statuses update in the admin at http://localhost:8003/admin/share/ingestjob/ (admin/password)
-    - useful for debugging! if ingest fails, the `IngestJob` will contain the error type, message, and stack trace
+> TODO: once share.osf.io/oaipmh is reliable, make it easy to init a local deployment by harvesting data from there
 
 ## troubleshooting
 - my containers keep mysteriously dying!

share/management/commands/add_local_osf_user.py

@@ -0,0 +1,26 @@
+from django.contrib.auth.models import Permission
+from django.contrib.contenttypes.models import ContentType
+
+from share.models import ShareUser, Source
+from share.management.commands import BaseShareCommand
+
+
+class Command(BaseShareCommand):
+    def add_arguments(self, parser):
+        parser.add_argument('--name', default='my-local-osf')
+
+    def handle(self, *args, **options):
+        user = ShareUser.objects.create_robot_user(
+            username=options['name'],
+            robot=options['name'],
+            is_trusted=True,
+        )
+        content_type = ContentType.objects.get_for_model(Source)
+        add_source_permission = Permission.objects.get(
+            content_type=content_type,
+            codename='add_source',
+        )
+        user.user_permissions.add(add_source_permission)
+        access_token = user.oauth2_provider_accesstoken.first().token
+        self.stdout.write(self.style.SUCCESS(f'added user "{user.username}" for local osf'))
+        self.stdout.write(f'access-token: {access_token}')

share/search/index_strategy/_base.py

@@ -270,5 +270,5 @@ def pls_stop_keeping_live(self):
         #     raise NotImplementedError
 
         # optional for subclasses
-        def pls_handle_query__sharev2_backcompat(self, request_body, request_queryparams=None):
+        def pls_handle_query__sharev2_backcompat(self, request_body, request_queryparams=None) -> dict:
             raise NotImplementedError(f'{self.__class__.__name__} does not implement pls_handle_query__sharev2_backcompat (either implement it or don\'t use this strategy for backcompat)')

share/search/index_strategy/sharev2_elastic5.py

@@ -402,7 +402,7 @@ def pls_get_status(self) -> IndexStatus:
             )
 
         # optional method from IndexStrategy.SpecificIndex
-        def pls_handle_query__sharev2_backcompat(self, request_body, request_queryparams=None):
+        def pls_handle_query__sharev2_backcompat(self, request_body, request_queryparams=None) -> dict:
             '''the definitive sharev2-search api: passthru to elasticsearch version 5
             '''
             try:

share/search/index_strategy/sharev2_elastic8.py

@@ -151,13 +151,22 @@ def _build_delete_action(self, target_id):
 
     class SpecificIndex(Elastic8IndexStrategy.SpecificIndex):
         # optional method from IndexStrategy.SpecificIndex
-        def pls_handle_query__sharev2_backcompat(self, request_body, request_queryparams=None):
-            # TODO: mangle request/response for limited backcompat with elasticsearch5
+        def pls_handle_query__sharev2_backcompat(self, request_body, request_queryparams=None) -> dict:
+            es8_request_body = {
+                **request_body,
+                'track_total_hits': True,
+            }
             try:
-                return self.index_strategy.es8_client.search(
+                json_response = self.index_strategy.es8_client.search(
                     index=self.indexname,
-                    body=request_body,
+                    body=es8_request_body,
                     params=request_queryparams,
                 )
             except elasticsearch8.TransportError as error:
                 raise exceptions.IndexStrategyError() from error  # TODO: error messaging
+            # mangle response for some limited backcompat with elasticsearch5
+            try:
+                json_response['hits']['total'] = json_response['hits']['total']['value']
+            except KeyError:
+                pass
+            return json_response