Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSDI action handler and server for import and people, questions, answers, messages #1166

Open
wants to merge 38 commits into
base: main
from

Conversation

@joshco
Copy link

joshco commented Jul 1, 2019

Fixes

#597 OSDI record_canvass_helper action handler
#949 API to push contacts into a campaign

Description

This PR contains an OSDI action handler to submit canvass responses like Question Answers, Tags (aka in VAN terms Survey Question Responses, Activist Codes). The handler will automatically download the available questions/SQs and tags/ACs and show them in the interaction script editing form as actions.
This can be used to submit data to EveryAction/VAN, Action Network, CiviCRM (in-progress) and other systems that support the OSDI Simple Organizing Profile. ( opensupporter/osdi-docs#337 )

This also contains OSDI Server functionality that lets other OSDI clients read data from Spoke in OSDI format, as well as import contacts with batch mode.
opensupporter/osdi-docs#335

Checklist:

  • I have manually tested my changes on desktop and mobile
  • The test suite passes locally with my changes
  • If my change is a UI change, I have attached a screenshot to the description section of this pull request
  • My change is 300 lines of code or less, or has a documented reason in the description why it’s longer
  • I have made any necessary changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • My PR is labeled [WIP] if it is in progress

This pull request is more than 300 lines of code. This is due to the functionality added, as well as incorporating several rounds of feedback, and customer requests.


@joemcl @lperson I ended up creating a more expansive OSDI implementation. In addition to supporting people import and collection browsing, I mapped spoke interaction steps, question_response into OSDI Questions and Answers. I also expose the messages collection, assignments, and users. as well as the relationship links between them.

In addition to automating import, this can be used to simplify integrations moving Q&A data into other systems like VAN et al. A script in a nightly cron job could suck down new answers, associated people, and do the mapping between the spoke Q&A and the VAN survey questions, or other logic and push them into VAN et al.

I set up a test server with some sample data and a public API for people to play with here:

Browse to http://spoke.dev.joshco.org/osdi

While looking at the HAL browser, try the following walkthrough

  1. Click the link for the organization you want to browse
  2. See the list of campaigns
  3. Choose the campaign you want to browse
  4. You should now be at the API Entry Point for the Campaign
  5. Click the link for 'osdi:people' and see the first page of people results
  6. Choose a person, and look at it's links. You can navigate to that person's Answers (question responses), or messages
  7. Go back to the AEP. (or Click "Go To Entry Point" in the top navbar to start over)
  8. Click the link for 'osdi:answers' and see the recently added answers. From each answer you can navigate to the person (campaign contact) or the question itself (derived from interaction steps)

You can also start navigating users -> assignments -> messages

More docs here: https://github.com/joshco/Spoke/blob/osdi_key/docs/OSDI.md

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Jul 4, 2019

I created a free script that automates moving recent question_responses into VAN with a customization mapping between Spoke and VAN Q&A. You can put it in a nightly cron job on your Spoke server to update VAN on the day's work.
https://github.com/opensupporter/queersync

Copy link
Collaborator

schuyler1d left a comment

I haven't reviewed the entire API surface yet, but stopped with some early feedback.

This PR creates a fantastic external API that will help other programs interface with Spoke -- it's breadth is excellent, but also means we'll have to be very careful about the additional security surface it exposes.

.gitmodules Outdated
@@ -0,0 +1,3 @@
[submodule "hal"]

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

This belongs in package.json in dev-dependencies. Let's not introduce git submodules to a project that needs to be accessible to beginners.

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

I dont think this is a dev dependency. The HAL browser is essentially the API console for humans to experiment, learn, and guide them in their scripting.
Ideally I'd like to avoid hard coding it into Spoke, as I make periodic updates to it in my github fork.
What alternatives can you think of?

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

I could clone it as part of the yarn install. what do you think about that?

This comment has been minimized.

Copy link
@joshco
@@ -1152,6 +1152,17 @@ const rootMutations = {
)

return await reassignConversations(campaignIdContactIdsMap, campaignIdMessagesIdsMap, newTexterUserId)
},
updateApiKey: async (_, { organizationId, apiKey }, { user }) => {
await accessRequired(user, organizationId, 'ADMIN', /* superadmin*/ true)

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator
  1. This should be OWNER -- not just ADMIN
  2. If this is going to be updated, then I think we should make it a separate field rather than inside the JSON features field.....in which case, maybe it can be our first knex migration (post-init) after #1154 lands (note to @ibrand )
@@ -173,6 +178,9 @@ app.get('/graphiql', graphiqlExpress({
endpointURL: '/graphql'
}))


osdiInitialize(app)

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

Please put this behind a default-off environment variable setting (and then document in docs/REFERENCE-environment_variables.md -- maybe OSDI_SERVER_ACCESS or something? (also, obviously updating the API key interface should also be hidden in the OFF case)

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

I put a toggle to turn OSDI off and on, like enforce texter hours.
image

and
image

and

image

textingHoursEnd: (organization) => organization.texting_hours_end
textingHoursEnd: (organization) => organization.texting_hours_end,
apiKey: (organization) => {
if (!organization.features) {

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

This also needs a await accessRequired(user, organizationId, 'OWNER', /* superadmin*/ true) (note you'll need to update the function sig to be more like uuid above to get the user obj)

}

function publicApi(features) {
if (features.publicApi === true || process.env.PUBLIC_API === 'true') {

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

Is this a new environment variable? Let's 1. rename it to OSDI_PUBLIC_API, and 2. I guess this could be the same var in server/index.js

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

I've taken this out in the new version. Now the authentication will allow a logged in user to browse the API with HAL browser, without futzing with an API key. Right now the role required is ADMIN. My thought is that the owner might want to give tech supervols access to do some scripting.

}

export async function authShortCircuit(req, res, orgId) {

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

I know we mostly don't document functions in this project (too bad), but with a name like this, let's describe what it does please.

}
}
} else if (req.method === 'DELETE') {
const deleted = await r

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

This needs a check on whether the campaign has started. If so, then this isn't allowed.

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

see next...


resp = {contacts: {deleted}}
} else if (req.method === 'POST') {
const osdi_body = req.body;

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

same with any of this content that adds contacts -- contacts are locked after a campaign starts.

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

Both of these already have it. See line 56 - that's the other short-circuit :)

if (campaignStatusShortCircuit(campaign, res)) {

src/server/api/osdi/contactsApi.js#L55

}
)

const contactsToSave = await Promise.all(contactsToSavePromises)

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

if there is any way at all, it would be better to re-use the contact upload code in workers/jobs.js -- it's ok to tweak that code a bit -- that way we can ensure that the same thing happens no matter what the channel.

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

looking into that

if (osdi_api_token) {
apiKeyInHeader = osdi_api_token;
}
const bypass=(features.bypass==apiKeyInHeader);

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

This should be === and I'm nervous about this bypass feature -- if this is for testing, maybe we can connect it to the global.TEST_ENVIRONMENT === '1'

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

bypass is gone now too. triple equality added


if (!(('authorization' in req.headers)
|| ('osdi-api-token' in req.headers))) {
sendUnauthorizedResponse(res)

This comment has been minimized.

Copy link
@schuyler1d

schuyler1d Jul 16, 2019

Collaborator

Can we also bail early if apiKey isn't present?

Also, I worry a bit about the difference in response if we enable this feature or not -- maybe we should be returning 404 in these cases? Thoughts?

This comment has been minimized.

Copy link
@joshco

joshco Aug 24, 2019

Author

this changed too. for an auth failure we just return 403 Forbidden. If OSDI is disabled, it will return 510. Both include a json body with hints on why. eg:
Access Denied - 403 Forbidden

{
  "osdi:error": {
    "response_code": 403,
    "error_description": "You need to use an OSDI-API-Token HTTP header containing your token, or you need to be logged in with sufficient privileges.",
    "auth_status": {
      "token_status": "INVALID_OSDI_TOKEN",
      "user_status": "LOGGED_OUT",
      "authenticated": false,
      "osdi_enabled": true
    }
  }
}

OSDI Disabled - 510 Not Extended

{
  "osdi:error": {
    "response_code": 510,
    "error_description": "OSDI is not enabled for this resource"
  }
}
@joshco

This comment has been minimized.

Copy link
Author

joshco commented Jul 30, 2019

@schuyler1d Thanks for the kind remark. I've reviewed your feedback and agree. I've also discussed the issue with other OSDI stakeholders, and have a proposal for next steps.

The current pull request includes some functionality for vision's sake. It also includes the bypass and public API switches which were meant to reduce friction for those who wanted to experience it for themselves using the demo server. I'll remove those, or refactor it as skyler suggested.

The current PR also goes beyond the settled resources in OSDI based on my own improvisations of assignments and messages. I'll remove these resources and produce an updated, more constrained PR, incorporating Sky and others' feedback. This way we could have working code this cycle.

In parallel, settling on the definition of what these new resources should be a democratic consensus among OSDI stakeholders, with leadership from those who build products that implement these features.

Augustus Franklin @augfrank , from CallHub (which also implements similar features) has offered to serve as a convener of an OSDI a breakout group on these resources. It would be great if someone from Spoke would participate as well as Hustle, Relay and any other implementers.

@augfrank

This comment has been minimized.

Copy link

augfrank commented Jul 30, 2019

@joshco @schuyler1d Yes, I'd love to help. It would be good to setup a common way to represent messages and its related resources.

@ibrand

This comment has been minimized.

Copy link
Collaborator

ibrand commented Aug 6, 2019

@joshco @schuyler1d @augfrank What's the status on this work?

Josh Cohen added 2 commits Aug 23, 2019
# Conflicts:
#	src/api/schema.js
#	src/server/api/schema.js
@joshco

This comment has been minimized.

Copy link
Author

joshco commented Aug 24, 2019

@ibrand This is almost complete. I've made most of the changes Skyler requested. I'll push a branch today.

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Sep 25, 2019

Latest update. @schuyler1d @ibrand I've added the global ENV var, improved the settings dialog. I've also removed the git submodule for the osdi browser, and instead snapshotted the browser code and am using webpack copy plugin. This will put it in the webpack, and also drop it in the assets directory which will be picked up by the S3 upload script.

https://github.com/joshco/Spoke/blob/osdi_key/webpack/config.js#L21

@joemcl

This comment has been minimized.

Copy link
Collaborator

joemcl commented Sep 25, 2019

thanks @joshco also worth looking at https://github.com/move-coop/parsons_public in relation to this. cc: @schuyler1d @ibrand @shakalee14

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Sep 26, 2019

@joemcl Thanks for passing this along; I doubt my response will surprise you. Approaches like this that encourage the use of proprietary APIs disproportionately harm the most marginalized communities in the progressive movement. The result is vendor lock-in where vendors will win sales based on avoiding the higher cost of integration with other products, rather than the merits of the product itself. This helps the big get bigger due to network effect. Larger customer organizations, or larger campaigns (like a presidential or $12m statewide gay marriage campaign) can overcome these costs with money or volunteer brute-force. However, smaller and more marginalized campaigns and organisations will not be able to do this as easily, and will stick with the larger "suite" type vendors. This further disadvantages products from marginalized innovators, because the additional cost of integrating those products will dis-incentivize customers from choosing one of these products based on merit and solidarity.

Given the recent consolidation in our marketplace, IMHO this will make our problems worse not better.

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Sep 29, 2019

I've updated the branch with 2 OSDI test cases.

  1. get people with apitoken succeeds, and without fails
  2. try to get people with osdi disabled and it fails

One weird thing I'm seeing in the tests is that intermittently, I'm getting an error in test_helpers. It seems to happen when I run the tests for the first time after time has passed. But then it works.

TypeError: Cannot read property 'data' of undefined
at: const organizationId = organization.data.createOrganization.id 

@schuyler1d Any ideas why? Who's the most familiar with the tests?

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Oct 16, 2019

@schuyler1d @ibrand @joemcl I've added outbound OSDI (like action handlers) for interaction steps to submit survey question answers or apply tags/activist codes. By default, it works with VAN / EveryAction unless a different AEP is configured.

I'm looking for dogfooders who would like to give it a spin.

From the updated documentation:

Outbound OSDI Actions

Configuration

Outbound OSDI is configured with these environment variables

  • OSDI_OUTBOUND_API_KEY Your OSDI API TOKEN
  • OSDI_OUTBOUND_AEP [Optional] The URL of your OSDI API Entry Point

By default, if you configured an API TOKEN but no AEP, Spoke will push to VAN / EveryAction.

Usage

Once configured, when you are editing your interaction steps, Spoke will download the available survey questions and tags (aka Activist Codes in VAN)

In the drop down for actions, you'll see options to apply tags/activist codes or submit responses to survey questions.

Dropdown of Choices

osdi_outbound_choices

Choice Chosen

osdi_outbound_choice

Matching Logic

By default, when pushing via OSDI, Spoke will invoke the osdi:person_signup_helper endpoint on the OSDI server to match or create a corresponding OSDI person. It will do based on fields in spoke for mobile phone number, zip, name etc.

Once matched, Spoke will invoke osdi:record_canvass_helper to apply tags/activist codes and submit responses to survey questions.

Spoke will also save the remote OSDI Identifier in a contact custom field named osdi_identifier to avoid matching on subsequent actions for teh same contact.

To improve matching, when you import into spoke, include email as a custom field.

If you are importing contacts that you got from the OSDI system itself (eg VAN), include the OSDI Identifier itself (eg VANID) in a custom field named osdi_identifier

If osdi_identifier is present, then Spoke will skip the osdi:person_signup_helper and just to the osdi:record_canvass_helper

@joemcl

This comment has been minimized.

Copy link
Collaborator

joemcl commented Oct 17, 2019

hey @joshco @schuyler1d @ibrand and adding in @lperson - sorry I am just now jumping in on this again. Can y'all possibly collaborate and fast track getting this merged? It's great stuff, and I'd really love to use it for an effort that I'm working on - it's for an election happening this November.

# Conflicts:
#	__test__/test_helpers.js
#	app.json
#	jest.config.js
#	src/api/schema.js
#	src/components/CampaignInteractionStepsForm.jsx
#	src/containers/Settings.jsx
#	src/lib/index.js
#	src/lib/zip-format.js
#	src/server/api/organization.js
#	src/server/api/schema.js
#	src/server/index.js
#	src/server/middleware/render-index.js
#	src/workers/jobs.js
#	webpack/config.js
@joshco

This comment has been minimized.

Copy link
Author

joshco commented Oct 18, 2019

@joemcl I've merged in main, as of today.

@joemcl

This comment has been minimized.

Copy link
Collaborator

joemcl commented Oct 18, 2019

@joshco thanks so much and I'll test that out today/tonight on a Spoke instance.

@joemcl

This comment has been minimized.

Copy link
Collaborator

joemcl commented Oct 19, 2019

So I deployed @joshco 's osdi_key branch tonight on to a production site and will test in the a.m., after finishing set up. Just a note that the OSDI_CONFIGURATION.md doc doesn't say anything about OSDI_MASTER_ENABLE env var, however I added it and set it to true which works, so now I see the OSDI API Access toggle which I'm leaving off for now. Also, the doc should be called something like HOWTO_OSDI_CONFIGURATION to conform to the rest of the how-to docs naming convention in /docs.

Josh Cohen
@joshco

This comment has been minimized.

Copy link
Author

joshco commented Oct 19, 2019

@joemcl I did some testing with the other CRM and made an update which pertains to you. You'll want to pull this.
I've added an optional environment variable OSDI_SERVER_CONTENT_TYPE which sets the content type of responses. The default is application/json, but some CRMs need application/hal+json. So you should set that to application/hal+json in your environment.
I've updated the documentation to include both variables.

@joemcl

This comment has been minimized.

Copy link
Collaborator

joemcl commented Oct 19, 2019

thanks @joshco can't deploy that latest update til late tonight probably. But I can attempt to set up a VAN connection with what I have now, correct?

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Oct 19, 2019

@joemcl yes, that's correct. You can set the outbound push to VAN.

@joshco

This comment has been minimized.

Copy link
Author

joshco commented Oct 19, 2019

@joemcl added odata date filtering, so queersync can be more efficient. I've also updated the civi extension to improve the AEP and PSH endpoint.

https://lab.civicrm.org/joshalot/CiviCRM-OSDI/tree/webhook_updates

Updated queersync to run as signup only (when survey questions aren't needed in other systems)

You should now be able to have it autopush to VAN via interaction steps/scripts, and sync nightly with Civi via queersync.

…stomer feedback and testing.
@joshco

This comment has been minimized.

Copy link
Author

joshco commented Oct 23, 2019

Improvements based on customer testing and feedback to make it easier to use and set up. Thanks to @joemcl

  • Added yarn osdi-info script to check config, view available information from remote OSDI Server
  • Added additional environment variables to option in logging of OSDI wire level traces
  • UI improvements

@lperson Can we update the google doc script importer so the customer can put a token for an answer_action in the google doc, and your script will stick it in the DB answer_action column?
Customers can now use the osdi-info CLI to discover available actions (activist codes, survey questions, tags) on their remote OSDI server. It will give them a token they can use in the google doc.

@ibrand @augfrank @schuyler1d

@joshco joshco changed the title OSDI Support for import and people, questions, answers, messages OSDI action handler and server for import and people, questions, answers, messages Nov 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.