Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inefficient Gini Coefficient calculation? #855

Closed
oliviaguest opened this issue Jul 29, 2016 · 3 comments
Closed

Inefficient Gini Coefficient calculation? #855

oliviaguest opened this issue Jul 29, 2016 · 3 comments
Assignees

Comments

@oliviaguest
Copy link

@oliviaguest oliviaguest commented Jul 29, 2016

When I run:

gini_coef = pysal.inequality.gini.Gini(a).g

where a is any huge numpy array, e.g., with 300000 elements, it gets stuck, irreparably crashing my whole PC, trying to run this line of code (see here):

d = np.abs(np.array([x - xi for xi in x]))

This is not the case when I run it using my own code which does not make use of list comprehensions, and instead calculates Gini like this.
I wrote that just before I discovered you had a Gini function too, so I do not fully understand your code, although I am willing to help. The two functions produce very very similar output (indistinguishable before about 6 decimal places).

Anyway, I thought I would mention that it be worth considering using numpy to calculate the difference.

@sjsrey sjsrey self-assigned this Jul 29, 2016
@sjsrey
Copy link
Member

@sjsrey sjsrey commented Jul 30, 2016

Thanks for raising this issue. I am just about to leave for a week of travel and will definitely be looking into this. At first glance the efficient approach is based on a ranking of the values, which assumes no ties. I originally ruled that out as we had to both handle ties (in an upstream application) and also needed to keep tabs on geographical positions of each observation. So the memory inefficient implementation is what we used.

That said, once I have a little time after vacation, I think it might be possible to refactor things with an eye towards a more efficient approach (as you point to) that also addresses our upstream needs.

@oliviaguest
Copy link
Author

@oliviaguest oliviaguest commented Jul 30, 2016

Have a nice time during your travels! I'm curious to see how you solve it.
😄

@sjsrey sjsrey mentioned this issue Aug 31, 2016
0 of 1 task complete
@sjsrey
Copy link
Member

@sjsrey sjsrey commented Sep 21, 2016

Closeed with #862

@sjsrey sjsrey closed this Sep 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.