Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

management command to import state information field types and data from prod #46

Merged
merged 3 commits into from Mar 3, 2020

Conversation

@jlev
Copy link
Member

jlev commented Mar 2, 2020

[ch402]
adds serializer field to check format. will not work until that's deployed

…rom prod api

adds serializer field to check format. will not work until that's deployed
@jlev jlev requested a review from nickcatal Mar 2, 2020
Copy link
Member

nickcatal left a comment

Few changes.

This is great!!!

app/election/management/commands/importfromprod.py Outdated Show resolved Hide resolved
app/election/serializers.py Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
app/election/management/commands/importfromprod.py Outdated Show resolved Hide resolved
info, created = StateInformation.objects.get_or_create(
state=state,
field_type__slug=state_information.get('field_type')
)

This comment has been minimized.

Copy link
@nickcatal

nickcatal Mar 3, 2020

Member

This could be sped up significantly if you didn't pull fields individually. Also, there is no need to get_or_create StateInformation

I haven't tested this, but this should be a good path

data = requests.get(f'{API_URL}/states').json()
for state in data:
  data_dictionary = dict([(x["field_type"], x["text"]) for x in state["state_information"]])
  # Our custom manager for StateInformation will ensure a select_related() to field_type is done here
  existing_information = StateInformation.objects.filter(state__code=state.get("code"), field_type__slug__in=data_dictionary.keys()) 
  for info in existing_information:
    updated = False
    specific_data = data_dictionary[info.field_type.slug]
    if info.text != specific_data:
      info.text = specific_data
      info.save()
      updated = True
    # output all the useful notifications here

This comment has been minimized.

Copy link
@jlev

jlev Mar 3, 2020

Author Member

Fair. I was trying to be explicit, but this is indeed much faster. Forgot that the StateInformation manager handles select_related and StateInformationFieldType pre-created all the objects. No more get_or_create.

@jlev

This comment has been minimized.

Copy link
Member Author

jlev commented Mar 3, 2020

Need to push to staging or dev to test.

@nickcatal

This comment has been minimized.

Copy link
Member

nickcatal commented Mar 3, 2020

@jlev make shelldev should get you there

@jlev

This comment has been minimized.

Copy link
Member Author

jlev commented Mar 3, 2020

tested as working with python manage.py importfromprod --api-url=https://api-staging.voteamerica.us/v1

@jlev jlev merged commit e16b7dd into master Mar 3, 2020
2 checks passed
2 checks passed
Travis CI - Branch Build Passed
Details
Travis CI - Pull Request Build Passed
Details
@jlev jlev deleted the jlev/importfromprod branch Mar 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.