New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port Django management command from Kitsune and Kuma. #168

Closed
wants to merge 13 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@jezdez
Member

jezdez commented Sep 10, 2013

This is work in progress since I need to port over some tests, too ;)

@willkg

This comment has been minimized.

Show comment
Hide comment
@willkg

willkg Sep 10, 2013

Member

Wow! I'm pretty psyched about this. I skimmed it and it looks good so far. I'm pretty hard up for free time, but after you land some tests and docs, I'll make a point of making time to look through it more carefully.

Member

willkg commented Sep 10, 2013

Wow! I'm pretty psyched about this. I skimmed it and it looks good so far. I'm pretty hard up for free time, but after you land some tests and docs, I'll make a point of making time to look through it more carefully.

@robhudson

View changes

Show outdated Hide outdated elasticutils/contrib/django/management/commands/index.py
@robhudson

View changes

Show outdated Hide outdated elasticutils/contrib/django/management/commands/index.py
@robhudson

View changes

Show outdated Hide outdated elasticutils/contrib/django/management/commands/index.py
try:
cls.registry[cls.get_mapping_type_name()] = cls
except (NotImplementedError, NoModelError):
pass

This comment has been minimized.

@robhudson

robhudson Sep 10, 2013

Member

This is pretty slick.

The words in the comment "branch" and "mount point" confuse me a little but it's still understandable.

@robhudson

robhudson Sep 10, 2013

Member

This is pretty slick.

The words in the comment "branch" and "mount point" confuse me a little but it's still understandable.

This comment has been minimized.

@jezdez

jezdez Sep 11, 2013

Member

Right, this is just me copy/pasting things around 💃

@jezdez

jezdez Sep 11, 2013

Member

Right, this is just me copy/pasting things around 💃

@willkg

View changes

Show outdated Hide outdated docs/django.rst
@willkg

View changes

Show outdated Hide outdated elasticutils/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/__init__.py
@@ -7,14 +7,13 @@
from django.shortcuts import render
from django.utils.decorators import decorator_from_middleware_with_args
from elasticutils import F, InvalidFieldActionError, MLT, NoModelError # noqa

This comment has been minimized.

@willkg

willkg Sep 20, 2013

Member

These were here so when someone is writing ElasticUtils code in Django, they could import everything from this module rather than have to remember what's overridden in the django stuff and what should get pulled from the regular stuff.

@willkg

willkg Sep 20, 2013

Member

These were here so when someone is writing ElasticUtils code in Django, they could import everything from this module rather than have to remember what's overridden in the django stuff and what should get pulled from the regular stuff.

This comment has been minimized.

@jezdez

jezdez Sep 25, 2013

Member

I find that pretty distasteful and against best practices, to be honest. But you're the boss.

@jezdez

jezdez Sep 25, 2013

Member

I find that pretty distasteful and against best practices, to be honest. But you're the boss.

This comment has been minimized.

@willkg

willkg Sep 25, 2013

Member

That's a pretty strong opinion. You do understand the ease of use issues, right? I'm not sure this case is against best practices because what we really have here is ElasticUtils the framework agnostic library and the Django layer on top of it, but we package both in the same repository. If I had more time to spend, I'd split them into two separate libraries. But that's a pain in the ass maintenance-wise. Regardless, that's how I'm treating them.

@willkg

willkg Sep 25, 2013

Member

That's a pretty strong opinion. You do understand the ease of use issues, right? I'm not sure this case is against best practices because what we really have here is ElasticUtils the framework agnostic library and the Django layer on top of it, but we package both in the same repository. If I had more time to spend, I'd split them into two separate libraries. But that's a pain in the ass maintenance-wise. Regardless, that's how I'm treating them.

log = logging.getLogger('elasticutils')
log = logging.getLogger('elasticutils.contrib.django')

This comment has been minimized.

@willkg

willkg Sep 20, 2013

Member

This is fine, but it means that a developer who wants to watch the logs now has to set up two loggers. I'm not entirely sure it's important to know whether something came from elasticutils or elasticutils.contrib.django. If you think this is a good idea, then we should at least document it in the section on debugging.

@willkg

willkg Sep 20, 2013

Member

This is fine, but it means that a developer who wants to watch the logs now has to set up two loggers. I'm not entirely sure it's important to know whether something came from elasticutils or elasticutils.contrib.django. If you think this is a good idea, then we should at least document it in the section on debugging.

This comment has been minimized.

@jezdez

jezdez Sep 25, 2013

Member

Well, I understand that the logger 'elasticutils' would automatically inherit the calls from the 'elasticutils.contrib.django' logger if no handlers for the latter are setup. No?

@jezdez

jezdez Sep 25, 2013

Member

Well, I understand that the logger 'elasticutils' would automatically inherit the calls from the 'elasticutils.contrib.django' logger if no handlers for the latter are setup. No?

This comment has been minimized.

@mythmon

mythmon Sep 25, 2013

Member

@jezdez you've got it right here. The events will bubble up by default, so I think this change is a good one.

@mythmon

mythmon Sep 25, 2013

Member

@jezdez you've got it right here. The events will bubble up by default, so I think this change is a good one.

@willkg

View changes

Show outdated Hide outdated elasticutils/contrib/django/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/contrib/django/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/contrib/django/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/contrib/django/__init__.py
es = get_es()
delete_index(index, es=es)
es.create_index(index, settings={'mappings': get_mappings()})

This comment has been minimized.

@willkg

willkg Sep 20, 2013

Member

We've got a bunch of MappingTypes registered and they may be in different indexes. I think this get_mappings() should only be returning the mappings for the index being created. I think that means you need to:

  1. figure out which mapping types should be in this index
  2. figure out the mappings for those mapping types

Maybe go through ES_WRITE_INDEXES.items() (or ES_INDEXES.items()) looking for the mapping type names for the index being recreated. Then figure out which mapping type names aren't in ES_WRITE_INDEXES (or ES_INDEXES) and thus use whatever default points to.

@willkg

willkg Sep 20, 2013

Member

We've got a bunch of MappingTypes registered and they may be in different indexes. I think this get_mappings() should only be returning the mappings for the index being created. I think that means you need to:

  1. figure out which mapping types should be in this index
  2. figure out the mappings for those mapping types

Maybe go through ES_WRITE_INDEXES.items() (or ES_INDEXES.items()) looking for the mapping type names for the index being recreated. Then figure out which mapping type names aren't in ES_WRITE_INDEXES (or ES_INDEXES) and thus use whatever default points to.

This comment has been minimized.

@jezdez

jezdez Sep 25, 2013

Member

I assumed that ES is capable of ignoring the unneeded mappings when creating an index. This blows this feature completely out of proportion, frankly.

@jezdez

jezdez Sep 25, 2013

Member

I assumed that ES is capable of ignoring the unneeded mappings when creating an index. This blows this feature completely out of proportion, frankly.

@willkg

View changes

Show outdated Hide outdated elasticutils/contrib/django/__init__.py
@willkg

View changes

Show outdated Hide outdated elasticutils/contrib/django/management/commands/index.py
def __len__(self):
return len(_model_cache)

This comment has been minimized.

@willkg

willkg Sep 20, 2013

Member

I'm glad you cleaned this up. This whole thing was so icky. Thank you!

@willkg

willkg Sep 20, 2013

Member

I'm glad you cleaned this up. This whole thing was so icky. Thank you!

@@ -1,6 +1,7 @@
from nose.tools import eq_
from elasticutils.contrib.django import S, F, InvalidFieldActionError
from elasticutils import F, InvalidFieldActionError
from elasticutils.contrib.django import S

This comment has been minimized.

@willkg

willkg Sep 20, 2013

Member

This is the sort of thing I was getting at earlier. It's easier on a dev to not have to remember which things come from where. Plus if we ever change it and Django-ify F (for example), then it's less likely to create subtle errors if the devs are importing everything from elasticutils.contrib.django.

@willkg

willkg Sep 20, 2013

Member

This is the sort of thing I was getting at earlier. It's easier on a dev to not have to remember which things come from where. Plus if we ever change it and Django-ify F (for example), then it's less likely to create subtle errors if the devs are importing everything from elasticutils.contrib.django.

This comment has been minimized.

@jezdez

jezdez Sep 25, 2013

Member

I disagree, if a developer can't remember what module to use, well, maybe he shouldn't be developing. If we're truly willing to provide a Django-like API, how can we assume that it matches the behavior of a maybe to be developed Django specific implementation? Imagine the horrible situation of having to explain why the elasticutils.contrib.django.F has changed its behavior?

The only solution to explain to the users that there is an F class, it lives in elasticutils. End of story.

@jezdez

jezdez Sep 25, 2013

Member

I disagree, if a developer can't remember what module to use, well, maybe he shouldn't be developing. If we're truly willing to provide a Django-like API, how can we assume that it matches the behavior of a maybe to be developed Django specific implementation? Imagine the horrible situation of having to explain why the elasticutils.contrib.django.F has changed its behavior?

The only solution to explain to the users that there is an F class, it lives in elasticutils. End of story.

@willkg

This comment has been minimized.

Show comment
Hide comment
@willkg

willkg Sep 20, 2013

Member

I'm really really excited you're taking this on. I'm really really sorry it took me a couple of weeks to get really work through this.

I think there's a lot of good stuff in here. Clearly it needs some documentation changes and probably some tests, too. There are some API decisions in here that I want to play with in an example project (or two or three) before I know how I feel about them.

I think it's worth doing another round of fixes in this PR and if that's good, then we'll land it. We have a couple of options:

  1. throw caution to the wind and land it in master and fix issues as they come up
  2. toss it in a new branch and continue to flesh it out

Option 2 lets us do a 0.8.2 release if we need to. Plus it lets us tinker with things in a more leisurely pace. I think I'm inclined to go with that.

@jezdez What's your timeline for this? Is this something you need to land asap or is this something we can take the next month or so to work on?

Member

willkg commented Sep 20, 2013

I'm really really excited you're taking this on. I'm really really sorry it took me a couple of weeks to get really work through this.

I think there's a lot of good stuff in here. Clearly it needs some documentation changes and probably some tests, too. There are some API decisions in here that I want to play with in an example project (or two or three) before I know how I feel about them.

I think it's worth doing another round of fixes in this PR and if that's good, then we'll land it. We have a couple of options:

  1. throw caution to the wind and land it in master and fix issues as they come up
  2. toss it in a new branch and continue to flesh it out

Option 2 lets us do a 0.8.2 release if we need to. Plus it lets us tinker with things in a more leisurely pace. I think I'm inclined to go with that.

@jezdez What's your timeline for this? Is this something you need to land asap or is this something we can take the next month or so to work on?

@jezdez

This comment has been minimized.

Show comment
Hide comment
@jezdez

jezdez Sep 25, 2013

Member

@willkg My timeline is basically ASAP. I'm a bit tired of discussing basic Python library design though, so don't expect me to spend (or wait) weeks for this to land. I thought I could make a quick leap for the project but the feedback you've given me so far shows that that was a mistake.

Member

jezdez commented Sep 25, 2013

@willkg My timeline is basically ASAP. I'm a bit tired of discussing basic Python library design though, so don't expect me to spend (or wait) weeks for this to land. I thought I could make a quick leap for the project but the feedback you've given me so far shows that that was a mistake.

@willkg

This comment has been minimized.

Show comment
Hide comment
@willkg

willkg Sep 25, 2013

Member

I'm just trying to help make sure the code is good. I didn't mean to piss you off.

@robhudson Can you take over reviewing from here? It's clear I'm sucking at it.

Member

willkg commented Sep 25, 2013

I'm just trying to help make sure the code is good. I didn't mean to piss you off.

@robhudson Can you take over reviewing from here? It's clear I'm sucking at it.

@kevinastone

This comment has been minimized.

Show comment
Hide comment
@kevinastone

kevinastone Nov 10, 2013

Contributor

What's remaining to get the management command merged? I think have a out-of-the-box indexing management is a essential feature for django integration. ElasticUtils has been great with everything else (celery tasks, etc), but this was an obvious omission.

Contributor

kevinastone commented Nov 10, 2013

What's remaining to get the management command merged? I think have a out-of-the-box indexing management is a essential feature for django integration. ElasticUtils has been great with everything else (celery tasks, etc), but this was an obvious omission.

@willkg willkg added this to the 0.10 milestone Mar 1, 2014

@willkg

This comment has been minimized.

Show comment
Hide comment
@willkg

willkg May 19, 2014

Member

After talking with @jezdez and @robhudson at PyCon, I'm going to close this out. We're going to go in a different direction.

Member

willkg commented May 19, 2014

After talking with @jezdez and @robhudson at PyCon, I'm going to close this out. We're going to go in a different direction.

@willkg willkg closed this May 19, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment