Repo & Maintainer Status & Availability #151
Comments
tannerlinsley
changed the title
|
Fair question, thanks for asking! True: nobody at Square is actively maintaining this library right now, and we don't use it in the Square Dashboard where it was originally deployed (we moved all computation server side again). That doesn't mean nobody is using it. I'll ask around and see if someone is willing to help triage issues and merge down good PRs. If not, we'll look for a different solution (perhaps a couple more external maintainers to help Jason out!) |
|
Does crossfilter have any bugs, besides people shooting themselves in the foot and getting stack overflows with NaNs? I think in a sense it is complete. I can't contribute due to the copyright assignment agreement. That's something that creates a sea of red tape in corporations. |
|
Not that it has bugs, or that it is incomplete. But there are some features
|
|
At least to triage issues and give the repo a sense of activity.
|
|
I'd love to see some more activity. There are a few PRs that have been submitted that I'm very interested in, like adding support for more dimensions #75 and one that I can't find by @esjewett that allowed you to remove rows that don't match the current filter. @esjewett also has https://github.com/esjewett/reductio that is a good place for a number of things that might be useful but deemed out of the purview of core crossfilter. Maybe moving crossfilter to an official organization so that there could be additional plugins brought into the official crossfilter ecosystem. |
|
I second that
|
|
I wonder if some people have moved onto other solutions for some reason. Lodash keeps getting faster and more functions. There are now lazy evaluated libraries (lazy.js and immutable.js) which are really performant for large data sets (I don't know if lazy evaluation would make a difference here). transducer.js is another approach that could be fast. Is there less activity in this library because it's pretty solid and mostly feature complete? Or is it because people are starting to use other options? Does anyone have a sense of that? (btw, It would be awesome to see some benchmarks between crossfilter and the equivalent operations in lazy-eval libs and lodash for small, medium and large datasets) |
An organization would be sweet, with copyright staying with the contributors, as we have done with dc.js. The only reason for copyright assignment is to make it easier to change the copyright later, and that is unlikely to be needed with crossfilter. |
|
@jefffriesen, has anyone built up the same cross filtering functionality on top of lodash? That would not be trivial but would indeed be interesting. |
|
@jefffriesen - I feel like dc.js and other libraries are making crossfilter more accessible and hence more popular. The core functionality is awesome, so there aren't major core changes that need to be implemented. Mostly just some tweaks and a few feature additions. |
|
@gordonwoodhull I don't know if anyone has built up something like a _.crossfilter() or some combinations of _.reduce() and other things. I would be interested to know too. @derekperkins I agree with you that dc.js has made crossfilter more popular - that was when I first used it. I could imagine someone taking a step further in functionality like reductio. Or I could imagine performance gains for very large datasets or many dimensions, by offloading the computations in a web worker or doing lazy evaluation. But I don't know what's possible or what's out there. I am very fond of this library though... |
|
I love crossfilter and reductio and even the idea of having a crossfilter Who is technically at the reigns right now?
|
|
I'm the Square maintainer for crossfilter, and @jasondavies is also a maintainer who independently merges PRs and new releases. In 99% of cases over the last 2+ years I have just deferred to Jason. Still asking around internally at Square for volunteers (not opposed to more outside maintainers but it's more work to manage, so starting locally). Give us a day or two to figure it out. @gordonwoodhull I will also ask about the agreement in case there's room for an update there. Is any agreement an impediment, or something about our specific one? Understand either way, just curious if it can be fixed. |
|
@RandomEtc Thanks for being so responsive. FYI, crossfilter was available as an organization name, which I just registered on a whim the other day, when @tannerlinsley and @esjewett and I were talking about some extensions we'd like to make. It's available if there is interest in moving it outside of square. |
|
@RandomEtc I would say in many organizations it is only moderately difficult to get internal approval for contributing to a project under some well known open source license. But it is near impossible to get approval to sign some other agreement that lawyers would have to read. (Spoken from my limited understanding of IP and organizations.) |
|
Hi @RandomEtc and all. Thanks for discussing this. I'd be happy to help maintain if that would be helpful. I'm definitely on-board with the general philosophy of keeping Crossfilter as lean as possible and keeping complexity in helper libraries, but there are some things that it would be extremely helpful to address internally that have not been getting attention. I'm thinking here of issues like #109, #92, and #75. It would be great to have a small team of maintainers and major users of Crossfilter in place who could work together to review, improve, and merge these sorts of changes. As far as I've seen, there's nothing else out there like Crossfilter. I'd like to see it thrive as I think it's a very important piece of any type of interactive visualization of more than a small amount of data. |
|
On #109 it seems like there was some support from Jason already, but no code. I would create a branch and try it, and get buy-in from other contributors. Forgiveness not permission. It would mean cutting a major release number if it breaks the API, but that shouldn't be an issue if the pay-off is worth it. #92 certainly looks like cause for concern since I've ignored it for 18 months. Apologies for that. Folks who are following along, it would be helpful to review and test this PR if it's a strong candidate for merging. I have no opinion on it except what I already shared :) #75 needs reviewing, verifying and benchmarking again following @sciyoshi's fixes. Please help. |
|
#109 has code (see the linked branch), but since Jason wanted to take another approach I was trying to get buy-in before creating a pull request. I don't want to waste effort on this. It doesn't break the API, just extends it. If you (@RandomEtc) think you'd be OK with merging that code, I'll submit a pull request ASAP. On the other two, it's a bit of a horse/cart problem. If there is not active maintenance of the project, it is difficult to justify putting significant effort into review that may never be looked at. I look forward to hearing what Square decides on that front. Sorry to be difficult on that, but I think that's the key issue to be resolved. |
|
I freely admit there is no active maintenance from Square on the project at this time. I'm able to drop-in to discussions like this quickly, and can provide stylistic and process guidance, but I can't commit to properly reviewing PRs if that's what it takes to unblock them. Still asking around internally, that will give us the info we need to decide what happens next. I'm open to "blessing" a fork as well, which is what we did with Cube when it became clear we weren't going to actively maintain that. |
|
@RandomEtc No worries, we all know how it is with finding the time for these things! And I didn't mean to cast blame as far as ongoing maintenance, just to make clear that the issues of getting proper review and involvement from the maintainers is related in my mind. I'm wasn't familiar with what happened with Cube. Good to see there is some precedence for dealing with these types of projects. |
|
@RandomEtc Out of curiosity - can you share some details as to why you moved dashboard aggregations to the backend? The added interactivity of frontend calculations was not worth it after all? |
|
Many reasons. Some including: data volume, mobile performance, parity between results on different reporting platforms (web, app, email, etc). |
|
Hi @RandomEtc, just checking in to see if there is any update from the internal discussion @ Square? Cheers, Ethan |
|
No volunteers inside of Square. I'm not sure of the best steps forward, I'll find a time to swap notes with @jasondavies and see if we can identify one or two people to assist. |
|
Aside: it sounds like a lot of contributors are finding crossfilter via dc.js? We've also had several issues filed here that turned out to be dc.js issues. Would the dc.js maintainers consider forking crossfilter for their purposes and accepting patches directly there? |
|
Thanks for the update. I think @gordonwoodhull would be the key person to ask regarding the dc.js angle. I'd be up for helping maintain a dc.js organization fork or another fork - whatever is agreed. |
|
@RandomEtc - I don't love the idea of forking crossfilter into dc.js or into that organization. Not that dc.js isn't great, but there are still plenty of people using crossfilter outside of that context, myself included. I'd love to have @gordonwoodhull and the dc.js team heavily involved, but I think crossfilter is big enough to warrant separate maintenance. My vote would be to officially transfer this repo to a crossfilter organization. If square doesn't want to do that or there are other issues preventing that, then next best would be a blessed fork to that same organization. Whatever happens, I think a key thing would be to change the licensing to something more open so that we don't have any other contribution issues. |
|
Thanks for the update, @RandomEtc. Yes, dc-js could host crossfilter, but it might be more appropriate to put it in its own organization as @derekperkins suggested. Since the dependency is really the other way and lots of people use crossfilter without dc.js. Either way I'll be glad to help maintain it, as long as there are few other maintainers and no contributor agreement. |
|
@gordonwoodhull - Great minds think alike :) |
|
@RandomEtc Thanks for following up! We'll keep our ears up in case something changes on the legal front, but we definitely understand the power of inertia at a big company. :) Our thanks go out to you, @mbostock, @jasondavies and everyone else involved with getting crossfilter to this point! |
|
Don't thank me for sitting on it for 3 years followed by a flurry of copy-paste issue closing 😬 |
|
Thanks @RandomEtc! There is a wealth of information in the issues on this repo, so let's link back here when we continue these conversations. Looking forward to continuing the amazing work done here! |
Hello maintainers!
I hope you're all doing well. Many of us feel that we haven't heard from you in very long (aside from the few issue comments here and there), and are curious concerning the future of crossfilter. We've heard that Square is no longer using it internally, or that it has become less of a priority to them, and that some of the maintainers have moved on to other ventures. Can anyone give us any details in this area? There are some great PR's and people that want to contribute to crossfilter and (despite it's continuous growth), I worry that it gets the attention it deserves.
Cheers!
The text was updated successfully, but these errors were encountered: