Sunspot 2.0 Discussion (29 June 2011)

20:04 <@outoftime> hello all!
20:04 < alindeman> Hi!
20:06 <@outoftime> so!
20:06 < alindeman> I haven't gotten anywhere yet :(
20:06 < alindeman> We're deploying a big thing at work tomorrow
20:07 <@outoftime> oh no worries
20:07 <@outoftime> until I actually produce something I'm certainly not going to fault others for the same : )
20:10 <@outoftime> OK so while we're waiting for nick (HEY nz) I guess I'll mention some of the things that I've been thinking about
20:11 <@outoftime> first, the lower level -- I'm pretty on board with making solrskates (formerly soolr) a pretty lightweight layer that doesn't try to expose the query as an object representation
20:11 < alindeman> Haha, love it
20:11 <@outoftime> but instead just expose methods to build boolean queries, embedded queries, and generally serialize literals correctly
20:12 <@outoftime> and then just expose a method_missing API that maps directly to Solr params
20:12 <@outoftime> so like
20:14 <@outoftime> Solrskates.q('delicious pizza').fq('category_id:5').facet(true, :limit => 10, :field => [:blog_id, :author_id])
20:14 <@outoftime> would return
20:14 <@outoftime> { :q => 'delicious pizza', :fq => 'category_id: 5', :facet => 'true', :"facet.limit" => 10, :"facet.field" => ['blog_id, 'author_id'] }
20:14 < alindeman> Sweet!
20:15 <@outoftime> when thinking about merging params, there are some things I'm not sure it would be smart at -- an example that's come to mind is dismax
20:15 <@outoftime> so if I have a bunch of dismax params that I want to merge with another bunch of dismax params, it might be too much to ask that it merges them into dismax subqueries in a single :q
20:16 <@outoftime> does that make sense? I'm thinking Sunspot can probably solve that problem simply by always using the embedded-query form of dismax
20:16 <@outoftime> but if say I had
20:16 <@outoftime> Solrskates.q('delicious pizza').q('extra pepperoni') it would know to concat them
20:16 <@outoftime> { :q => 'delicious pizza extra pepperoni' }
20:17 <@outoftime> whereas it would know to merge
20:17 <@outoftime> Solrskates.fq('category_id:5').fq('author_id:1') into { :fq => ['category_id: 5', 'author_id: 1'] }
20:17 <@outoftime> basically just setting up different merge rules for different param names/patterns
20:18 <@outoftime> does that seem reasonable?
20:19 < alindeman> Yep
20:19 <@outoftime> awesome
20:20 <@outoftime> ok so now for building queries
20:20 <@outoftime> Solrskates::Query(:category_id => 5)
20:20 <@outoftime> would return "category_id:5"
20:20 <@outoftime> Solrskates::Query(:published_at => { :less_than => Time.now })
20:21 <@outoftime> would return "published_at:(* TO 2011-06-29T20:18:00Z)"
20:21 <@outoftime> except properly escaped : )
20:21 < alindeman> Not sure how you feel about this, but mongo_mapper (or one of those other ones) allows stuff like :published_at.lt(Time.now)
20:21 <@outoftime> well sir
20:21 <@outoftime> I would be glad to tell you how I felt about that
20:21 <@outoftime> in no uncertain terms
20:21 < alindeman> Hahaha
20:22 <@outoftime> it's a horrible antipattern
20:22 <@outoftime> because it's defining methods on Symbol that are only used by MongoMapper
20:22 <@outoftime> and it means that any library that decides to do that clobbers any other library that decides to do that
20:23 < alindeman> True, and we wouldn't want to clobber mongo_mapper either
20:23 <@outoftime> exactly
20:23 < alindeman> Excellent, I like it
20:23 <@outoftime> I mean, I'd really like to use that nice library that extends AR for building complex conditions without string programming
20:23 <@outoftime> except I also use Mongoid and you can't use both for that exact reason
20:23 <@outoftime> so yeah. no overriding Symbol.
20:24 <@outoftime> but that doesn't mean I'm in love with what I proposed above. it's not especially pretty.
20:24 <@outoftime> so if you can think of anything better...
20:25 < alindeman> Not without more classes and such
20:25 <@outoftime> yeah
20:25 <@outoftime> my thoughts too
20:25 <@outoftime> I mean in Sunspot itself I'm thinking
20:26 <@outoftime> Post.search.where(:category_id => 1).lt(:published_at => Time.now) etc.
20:26 <@outoftime> which is the approach Mongoid takes with some scope types, but for the simpler ones it does the dirty symbol override
20:27 <@outoftime> OK how about connectives...
20:27 < alindeman> Yep, I think chaining is the way to go
20:28 <@outoftime> Solrskates::Query(:any_of => [{:category_id => 1}, { :author_id => 2}]) ?
20:28 <@outoftime> problem there is that :any_of goes where a field name would otherwise
20:29 <@outoftime> maybe Solrskates::Query needs some methods
20:29 <@outoftime> Solrskates::Query.where(:category_id => 1)
20:29 <@outoftime> Solrskates::Query.any_of({ :category_id => 1 }, { :author_id => 2 })
20:29 <@outoftime> Solrskates::Query.lt(:published_at => Time.now)
20:30 <@outoftime> so basically the same thing I proposed for Sunspot, except without the chaining at that levle
20:32 <@outoftime> huh but that still gets ugly when you're trying to nest...
20:32 <@outoftime> (published_at < Time.now || evergreen == true)
20:32 <@outoftime> for instance
20:33 < alindeman> Right, I was trying to hink of how to phrase that
20:33 <@outoftime> Solrskates::Query(:any_of => [{ :published_at => { :less_than => Time.now }}, { :evergreen => true }])
20:35 <@outoftime> pretty hairy, but at least allows indefinite complexity through recursion...
20:35 <@outoftime> then the problem just becomes one of what if there is a field called 'any_of'
20:35 <@outoftime> one possible solution would be that symbols are connectives and strings are field names...?
20:36 <@outoftime> Solrskates::Query(:any_of => [{ 'published_at' => { :less_than => Time.now }}, { 'evergreen' => true }])
20:36 <@outoftime> anyone else in the room have thoughts on this, or curious about what the hell we're talking about?
20:39 <@outoftime> Solrskates::Query(:published_at => { :less_than => Time.now }).or(:evergreen => true)
20:40 <@outoftime> prettier but probably falls apart when nesting gets compl4ex
20:40 < alindeman> Yep, much easier to build one big hash I'd think
20:40 <@outoftime> what do you think of the symbol/string distinction?
20:40 <@outoftime> Rails has taught us not to think too much about that
20:40 <@outoftime> which I don't think is really a good thing
20:41 < alindeman> Part of me does this it's a bit dirty, but since it's a lower level library, I think it's OK
20:41 < alindeman> Er, think*
20:41 <@outoftime> or actually, how about -- you can always use symbols, but if you have a field name that is also a reserved key, just use a string to indicate it's a field name
20:41 < alindeman> But then Sunspot has to deal with that
20:42 <@outoftime> well Sunspot could just always pass field names as strings
20:42 < alindeman> Ahhh, true
20:42 < alindeman> I like that
20:42 <@outoftime> this seems like a not so bad solution.
20:42 <@outoftime> okay, awesome.
20:43 <@outoftime> so I think from the standpoint of setting up classes for indexing, what we've got is fantastic. just use the ::index method, do type inference with fallback to text and the ability to override
20:43 <@outoftime> agree?
20:44 < squeegy> outoftime: I'm reading / catching up, no questions yet, still digesting it all
20:44 <@outoftime> squeegy: cool
20:45 <@outoftime> squeegy: to give you some context, one of my goals with Sunspot 2 is to build it on top of a lower-level library that just makes building Solr queries easier without making any assumptions about your schema etc. My first crack at that was called Soolr and I constructed an entire Ruby object model to represent Solr queries
20:45 < alindeman> Will there be some sort of adapter that each knows how to do inference based on ActiveRecord, Mongoid, etc.. ?
20:45 <@outoftime> alindeman: absolutely
20:46 <@outoftime> alindeman: I'm expecting the adapters in SS2 to be *much* more robust
20:46 < squeegy> outoftime: yeah I've read the README and the soolr code you committed back in may
20:46 <@outoftime> squeegy: but I realized I was basically just re-typing the Solr query API in Ruby, without really adding a lot of value. so take 2 on that is "Solrskates", which just maintains the query as a params hash and only aims to do useful things -- build boolean queries, merge repeated params, serialize different types properly, etc.
20:47 <@outoftime> squeegy: Solrskates doesn't actually exist yet though : )
20:48 < squeegy> outoftime: sounds good
20:48 <@outoftime> squeegy: aws.
20:48 <@outoftime> so from the standpoint of Sunspot's query API, it seems like everyone would really like to go the chaining route, myself included
20:49 < squeegy> yeah, that works for me, I didn't like it in AR at first, but it has grown on me
20:49 <@outoftime> squeegy: yeah, me too -- I actually helped build a Sunspot-esque database query DSL for ActiveRecord 2 and was disappointed by the chaining in AR3
20:50 <@outoftime> squeegy: but it has its advantages, for sure.
20:50 < squeegy> outoftime: in my experience sunspot has done everything i could want it to do.  I'm mostly interested in understanding your thought process for designing the library.  Sunspot is by far one of the most robust and well designed libraries i have used
20:50 <@outoftime> squeegy: well that's really nice to hear!
20:50 <@outoftime> basically my big goal for Sunspot 2 is to make it drop-dead easy to use
20:51 < squeegy> definitely, although I've never had any issues
20:51 < squeegy> I've supported a few friends, but their biggest problem is the query syntax with blocks
20:51 <@outoftime> it's gotten a lot better, mostly thanks to the efforts of others. my initial releases of Sunspot::Rails were kind of a pain to set up
20:52 <@outoftime> squeegy: yeah, that's my big argument for switching to chaining. it will look familiar to people who have never seen Sunspot before.
20:52 < squeegy> yeah, most of their issues were understanding why something worked in irb and not in production
20:52 <@outoftime> heh, gotcha
20:52 < squeegy> once they pass the block an argument, and reference the methods that way, it all works :)
20:53 < alindeman> Also there are some edge cases where the block syntax creates some WTF moments .. like ivars
20:53 <@outoftime> totally
20:53 <@outoftime> so I think the new query-chaining DSL will make things a lot easier because of its familiarity
20:53 <@outoftime> and the new index setup DSL will make things a lot easier because it's less work -- let the adapter figure most everything out
20:54 < squeegy> you guys prefer the bare index syntax over a searchable block?
20:55 <@outoftime> another big goal is just making sure bootstrapping is painless -- one generator, good defaults, etc. we're largely there with Sunspot 1 but I think we can do better by starting with that in mind from the ground up
20:55 < squeegy> I definitely like the idea of the adapter inferring the types tho
20:55 <@outoftime> squeegy: yeah, mostly because the adapter should be able to figure out everything it needs to know through introspection. and if you don't need to manually specify types in most cases, there isn't much point in a block
20:55 <@outoftime> s/everything/much of
20:55 < squeegy> k, makes sense
20:56 <@outoftime> and again, runtime class configuration with a block isn't common in the Rails world -- it's not familiar
20:56 < squeegy> yeah, i assume you'll use the same technique as sunspot 1 and define solr_index and index if it is not already defined?
20:57 <@outoftime> squeegy: jury's out on that one : )
20:57 <@outoftime> squeegy: I think Sunspot 2 will have an explicit module include
20:57 < squeegy> I thought I remember someone mentioning that mongoid or mongomapper uses index as a class method already?
20:57 <@outoftime> squeegy: e.g. include Sunspot::Searchable
20:57 <@outoftime> squeegy: yes it does
20:57 < tomcocca> sorry i'm late, i was wondering if there was plans to support :referrences options on Polymorphic associations?
20:58 < squeegy> outoftime: I like the idea of being explicit with the include
20:58 < squeegy> outoftime: less "pollution" the better
20:58 < tomcocca> i know we had discussed this on the mailing list a long time ago
20:58 <@outoftime> so I'm wondering if maybe just providing a Sunspot::Searchable::NoConflict module for cases where the method names overlap would be best...
20:58 <@outoftime> tomcocca: my intention is to have :reference be a type, just like String or Integer
20:58 < squeegy> ah, now that's an interesting approach and makes sense
20:58 < alindeman> Might be too surprising for Mongoid users, though
20:59 < alindeman> Part of me now wants to reconsider the name .. make it solr_index :foo for everyone .. or similar?
20:59 < tomcocca> i like the idea of the whole: belongs_to :blog ; index :blog thing
20:59 <@outoftime> alindeman: and solr_search ?
20:59 < tomcocca> if that could also work with polymorphic it would be awesome
20:59 <@outoftime> tomcocca: yes, exactly. the adapter should be able to infer references just as it infers scalar types.
20:59 < tomcocca> awesome
21:00 <@outoftime> tomcocca: yeah, I don't think there's any need for reference fields to have a statically-defined class. just index each value with the class name and ID.
21:00 < alindeman> I suppose, yah .. it doesn't look as pretty .. but I think it is better than jumping through hoops when conflicts occur
21:00 <@outoftime> alindeman: there's certainly an argument for that. as I said, I think it's best we let that issue simmer and see if inspiration strikes
21:00 < alindeman> Sounds good
21:00 < nz> hello gentlemen!
21:01 <@outoftime> I certainly don't like the way SS1 does it. and I certainly don't like the idea of just clobbering other libraries' methods.
21:01 <@outoftime> nz: sir!
21:01 <@outoftime> nz: get yourself caught up : )
21:01 < nz> yeah no kidding
21:01 < nz> looks like some fun stuff going on in here
21:01 < tomcocca> are you going to be able to do the following:    index :name, :title, :location
21:01 < squeegy> outoftime: what do you think about including some of the test helper projects (or adaptions) in Sunspot 2?  Coming from the world of Rails I've grown accustomed to testing niceness being included.  Might be overkill, just a thought...
21:01 <@outoftime> tomcocca: I don't see why not
21:02 < tomcocca> or is each index going to need to be a specific definition
21:02 <@outoftime> tomcocca: well I assume that would be shorthand for index :name; index :title; index :location
21:02 < tomcocca> right exactly
21:02 < squeegy> outoftime: I'm specifically thinking about sunspot_matchers from pivotal labs (although i think it might be rspec specific)
21:02 <@outoftime> squeegy: I'm glad you mentioned that -- it is a big priority for me to have SS2 have really good support for testing
21:02 <@outoftime> tomcocca: yeah seems like an easy win
21:03 <@outoftime> squeegy: the core of that will be exposing introspection on Sunspot searches and setups
21:03 <@outoftime> squeegy: so like search.has_scope?(:category_id => 1)
21:03 < tomcocca> what about if you wanted each of those as a string (not fulltext) like so:    index :name, :title, :location, :as => :string
21:03 < tomcocca> asking too much ?
21:03 <@outoftime> squeegy: or Post.has_solr_field?(:title)
21:03 < squeegy> outoftime: awesome, that's exactly the type of introspection that would be helpful
21:04 <@outoftime> tomcocca: presumably any options you passed in would apply to all fields -- similarly to how AR3's validates() works
21:04 < tomcocca> yes! awesome
21:04 < squeegy> outoftime: I'm mostly interested in introspecting the search objects, but the models themselves could be useful
21:04 <@outoftime> squeegy: yep I'm sure that'd be the main use case, but might as well just do it across the board
21:05 <@outoftime> from a design standpoint, Sunspot 1 has a massive non-public API. I'd like to minimize the non-public API for Sunspot 2.
21:05 < alindeman> An easy way to mock out Sunspot would be nice
21:05 < alindeman> And then retrieve the last search
21:05 <@outoftime> alindeman: agreed.
21:06 <@outoftime> alindeman: probably a session proxy that ships with the library would do the job there
21:06 < alindeman> Yah! Cool
21:06 < squeegy> outoftime: makes sense to me, move those private methods into other classes
21:06 <@outoftime> (I happen to like the session proxy pattern a lot; I see that carrying over pretty much as-is to Sunspot 2)
21:06 < squeegy> yeah, session proxy is awesome
21:07 < squeegy> it was foreign to me at first, but I've grown to love the flexibility
21:07 <@outoftime> squeegy: well more than Sunspot 1 has entire classes that aren't considered part of the public API. it makes a lot of things confusing -- documentation, introspection, etc.
21:07 < squeegy> gotcha
21:07 < alindeman> Maybe some class level method to enable it .. (similar to how OmniAuth does it) .. Sunspot.test_mode = true or something
21:07 <@outoftime> alindeman: absolutely
21:07 < alindeman> Even if that actually just shuffles some session proxies around
21:07 <@outoftime> alindeman: absolutely!
21:08 < alindeman> Sweet
21:08 <@outoftime> like I should be able to call Post.solr_field(:title).type
21:08 <@outoftime> as part of the public API
21:08 <@outoftime> in SS1 you can do Sunspot::Setup.for(Post).type but none of that is part of the public API
21:09 <@outoftime> which is bologna (my bad)
21:10 < nz> IMO, Post.solr_fields # => [ #<SolrField:0xdeadbeef>, ... ]
21:11 < nz> or just a hash: Post.solr_fields # => { :title => { :type => :text, :proc => #<Proc:...> }, :content => { ... }}
21:11 < nz> and a #solr_hash method that can introspect on that configuration to build up the final Solr document
21:11 < nz> which #solr_index can send via RSolr over the wire
21:11 <@outoftime> nz: dig it
21:12 <@outoftime> nz: but also with a ::solr_field(name) convenience method
21:12 < nz> Or Post.solr_fields[name]...?
21:12 <@outoftime> nz: well yes that would be the implementation : )
21:12 < nz> I guess solr_field(name) maps easily enough to that :)
21:12 < nz> hah
21:13 <@outoftime> nz: have you had a chance to read over our Solrskates discussion?
21:13 < nz> yeah, bits
21:13 < nz> I have a crazy idea
21:13 <@outoftime> nz: oh snap
21:13 < nz> > Solrskates::Query(:published_at => { :less_than => Time.now })
21:13 < nz> why not do some light compilation...?
21:13 < nz> and try to match AR's SQL-esque syntax
21:14 < nz> Post.search.where(['published_at < ?', Time.now])
21:14 < nz> where we turn "foo < bar" into "foo:[bar TO *]"
21:14 <@outoftime> nz: hmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm
21:14 < alindeman> OTOH, AR is simply exposing SQL.  The underlying thing here is decidedly not SQL.
21:14 <@outoftime> yeah I'm not loving that idea
21:15 <@outoftime> I see the motivation
21:15 < nz> well, who says "less than" is SQL? I like to think it's a higher level concept :)
21:15 <@outoftime> but we're still kind of turning it into string programming
21:15 <@outoftime> i mean not really because of the substitutions
21:15 <@outoftime> but still
21:15 < nz> yes, slippery slope for sure
21:16 < nz> just injecting a crazy idea -- *I'm* certainly not excited enough about it to go and write it
21:16 < nz> yet ;)
21:16 < nz> something I *am* excited about...
21:16 < nz> a Sunspot 1.3 release
21:16 <@outoftime> oh good! because I'm not excited about that
21:16 <@outoftime> although in my new spirit of not procrastinating, I have been merging pull requests like a champ
21:17 < nz> how about lardawge on GitHub, eh?
21:17 < nz> that man deserves a medal
21:17 <@outoftime> yeah! Sunspot 1.3 Employee of the Month
21:17 < nz> Sunspot 1.3 VIP
21:17 <@outoftime> yes
21:17 <@outoftime> MVP?
21:17 < nz> MVP there you go
21:17 <@outoftime> it's settled, then.
21:18 < alindeman> Any thoughts on if we want/how to do syntax like "searchable :if => :foo?"
21:18 < alindeman> (aka https://github.com/outoftime/sunspot/pull/45 )
21:18 < nz> what was the consensus on the "index" method?
21:18 < nz> and its interference with mongoid? (mongomapper?)
21:18 < squeegy> I'm moving to my home office, be back in 10 if you all are still here
21:19 < nz> indexable :if => :foo
21:19 < nz> I like the idea
21:19 < nz> although there is some room for surprise
21:19 < alindeman> index conflicts was left to simmer
21:19 <@outoftime> right -- see if anything brilliant comes to mind
21:20 <@outoftime> if not, probably just have clunky namespaced methods -- solr_index, solr_search
21:20 < nz> with respect to syntax...
21:20 < nz> I'm sort of coming to the opinion that we should do a back-end overhaul that aims to be 100% syntax backwards compatible
21:20 < nz> as in, Sunspot's value is largely in its DSL
21:21 < nz> people love it. there are a lot of lines of code out there that use it
21:21 < nz> so what if Sunspot 2.0 is some hypothetical light DSL wrapper that implements some other lower-level cutting-edge awesome client?
21:21 < nz> that's a little bit what we're discussing, I think
21:22 < nz> since in any case I'm sure that a Sunspot 1.x syntax compatibility module is going to be necessary
21:23 <@outoftime> so wait
21:23 <@outoftime> hmm. this largely sounds like a semantic distinction
21:24 < nz> yes, definitely
21:24 <@outoftime> like either we build an awesome Solr library with all our new ideas and new API and call it solr_derby
21:24 <@outoftime> and then build a library called Sunspot that just implements the Sunspot 1.0 API on top of solr_derby
21:24 <@outoftime> or
21:24 < nz> yes, precisely
21:24 <@outoftime> we build an awesome Solr library with all our new ideas and new API and call it Sunspot 2.0
21:25 < nz> exactly right
21:25 < nz> what I'm thinking here...
21:25 <@outoftime> and then ship it with an optional module that exposes Sunspot 1.0's API
21:25 < nz> is RSpec 1.x vs 2.x
21:25 <@outoftime> go on
21:25 < nz> lots of backwards-incompatible changes
21:25 <@outoftime> were there?
21:25 < nz> yeah
21:25 < nz> well
21:25 < alindeman> I think new folks would respond a lot better to an API that looks like AR than Sunspot currently
21:25 < nz> in random stuff
21:26 <@outoftime> well regardless of the exact history of rspec
21:26 <@outoftime> go on
21:26 < nz> recently I was googling for some rspec information recently
21:26 < nz> and kept getting articles that used 1.x class names
21:26 <@outoftime> ah, yes.
21:26 < nz> that weren't compatible with 2.x
21:26 < nz> so we have this "Sunspot" namespace
21:27 < nz> and it has, for better or worse, come to represent largely the API in place now
21:27 <@outoftime> this is an intriguing line of thought.
21:27 < nz> so, this isn't a definitive argument yet for either approach
21:27 < nz> I could just as easily be talking about a compatibility module here
21:27 <@outoftime> i mean we need that at a minimum, no doubt
21:28 < nz> but I think it could make sense if we separate the concerns of "maintain the Sunspot DSL's conceptual consistency"
21:28 <@outoftime> more weight is lent to your idea, I would say, by the fact that all the big API changes were your ideas : )
21:28 < nz> with the concerns of "build a better Solr client"
21:28 < nz> haha
21:28 < nz> quite ;)
21:28 < nz> so it's like, "Sunspot user? Great, keep using it, you're done."
21:29 < nz> "Want to dig deeper? drop down to solr_derby under the hood."
21:29 <@outoftime> hmm
21:29 <@outoftime> so I guess that's what I'm having trouble being on board with
21:29 <@outoftime> I want our main product to be the new library with the sweet new API. I wouldn't want that to be "under the hood".
21:30 <@outoftime> I want a new version of Sunspot with a better, more intuitive API, but with the same basic premise.
21:30 <@outoftime> whether it's called Sunspot is not necessarily important
21:30 < alindeman> I feel like most folks won't have incredibly complicated search code, eh?  It certainly would be orders of magnitude simpler than most Rails 2 -> 3 transitions, say.
21:30 < alindeman> And we get to standardize on a (IMHO) nicer overall syntax
21:31 <@outoftime> but this thing we've been designing, I want to be what people use by default going forward, unless they have existing Sunspot integrations they want to keep, or they just prefer the Sunspot 1.x DSL
21:31 < nz> alindeman: I think a lot of people stay on Rails 2 precisely because of the magnitude of that change :)
21:31 < nz> maybe I can simplify my main concern here...
21:31 <@outoftime> so if it's just a semantic distinction, continuing to tie the Sunspot brand to the current API, I'm totally open to that
21:31 < nz> if Sunspot 2.0 isn't a 100% backwards compatible upgrade, a lot of people won't upgrade
21:31 <@outoftime> understood
21:32 <@outoftime> but the day after we release whatever this thing is
21:32 < nz> and we'll be answering questions like "I'm still on Sunspot 1.2, and..." :)
21:32 <@outoftime> I want people to be installing our new thing and using our new API
21:32 < nz> cool
21:32 < nz> so I think we're converging on the idea of...
21:32 <@outoftime> not because they want to go "under the hood" -- because it's better
21:32 < nz> aggressively push forward
21:32 < nz> in an opinionated way
21:32 < nz> and make sure to support the old syntax
21:33 <@outoftime> yeah, I think from my standpoint that all needs to be invariant
21:33 < nz> cool
21:33 < nz> well, I can dig that for sure
21:33 < nz> oh, well, I guess before I move on...
21:34 < nz> maybe I should talk more about my SolrMapper concepts?
21:34 <@outoftime> by "opinionated" you mean opinionated about the schema, right?
21:34 < nz> I still would like to have a "ORM" of sorts that uses Solr as a persistence layer
21:34 <@outoftime> and i guess about how, for instance, fulltext searches should be performed (via dismax)
21:34 < nz> outoftime: yeah, opinionated in terms of schema, and DSL in the models
21:35 < nz> "you *should* be searching with Foo.search.where(:category => 'Bar').facet(:color)"
21:35 <@outoftime> I think I get it, but what's the alternative?
21:35 <@outoftime> exposing the Solr API directly with no translation?
21:35 < nz> no, also, opinionated in the sense of "this new API is better than the old one"
21:36 <@outoftime> gotcha.
21:36 <@outoftime> yes.
21:36 < nz> I was flirting with the idea of Sunspot purely maintaining its current DSL
21:36 < nz> and us forking the other fun stuff into some other lib
21:36 <@outoftime> although I wouldn't mind constructing the internals such that the DSL is easily swappable
21:36 < nz> well, we'll see what kind of burden that presents ;)
21:36 <@outoftime> (Sunspot 1 actually largely achieves that design goal, I think)
21:37 <@outoftime> (but I think a lot of what I had in mind for Sunspot 2 would, if anything, make that more difficult)
21:37 < alindeman> Remember as well that the more abstractions and historical knowledge required, the harder for new contributors to the code as well
21:37 < nz> with a general trend toward better introspection, I don't think that will be too bad...
21:37 <@outoftime> alindeman: good point
21:37 < nz> yeah, I think that's one of the few upsides of a rewrite
21:37 < nz> your history gets a lot shorter ;)
21:38 <@outoftime> sunspot needs it
21:38 <@outoftime> the internals were structured for a simpler library
21:38 <@outoftime> also I kind of wrote it not giving a shit about it being easy to use
21:38 <@outoftime> needs the rewrite
21:39 < alindeman> One of the other projects I contributed to: v1.x was terribly written, v2.x was an amazing piece of architecture, but few people could understand it fully, v3.x was well written but didn't take every abstraction to the max .. and the project is flourishing lots better
21:39 <@outoftime> I'd like to be shooting for v3.x then
21:39 <@outoftime> I'd like to think we skipped v1.x : )
21:40 <@outoftime> OK how about this
21:40 < nz> acts_as_solr was 1.x :)
21:40 <@outoftime> haha, fair enough
21:40 < alindeman> lol
21:41 <@outoftime> how about solrskates (because come on, that's an awesome name) becomes what we have envisioned as our next-generation Solr client. with some opinions, but also keeping its public API robust such that advanced users can make use of it without conforming to its opinions
21:41 <@outoftime> (ditching the idea of a standalone low-level library like Soolr)
21:42 <@outoftime> then Sunspot 2 becomes a fairly thin wrapper on top of Solrskates that just exposes Sunspot's API
21:42 <@outoftime> testing Sunspot 2 should be easy/DRY because we can just dogfood the Solrskates introspection capabilities
21:43 < nz> so... Option A! ;)
21:43 < alindeman> HAHA, yep
21:43 < nz> Solrskates is pretty sweet though
21:44 < nz> I'm seeing a chalk-on-pavement kind of aesthetic
21:44 <@outoftime> yeah nick I'm basically proposing what you proposed : )
21:44 < nz> with rollerstake wheels throwing off gnarly sparks!
21:44 <@outoftime> except ditching the standalone lower-level library
21:44 <@outoftime> because let's be honest, nobody wants to write that thing.
21:44 <@outoftime> or at least, the idea doesn't especially excite me.
21:44 < nz> yeah you seem pretty over soolr ;)
21:44 <@outoftime> oh I am way over soolr
21:44 <@outoftime> god, hours of my life wasted
21:45 < nz> not wasted!
21:45 < nz> exploring new concepts
21:45 < squeegy> why was it such a waste though?
21:45 <@outoftime> but even like my initial proposal for Solrskates, I'm kind of over
21:45 <@outoftime> squeegy: basically it ended up that I was documenting the Solr API by way of writing Ruby classes. it felt like writing Java.
21:46 < nz> I'm a big fan of hard-coding as little API knowledge as possible
21:46 <@outoftime> nz: yes I have very much come around to that idea
21:46 < nz> push that down to the docs. good docs are where it's at anyway
21:46 <@outoftime> nz: although I do think it's appropriate for a high-level abstraction like Sunspot itself.
21:47 < nz> yeah
21:47 <@outoftime> nz: at a certain point, it's better for a handful of us to learn the Solr API and all its quirks than for thousands of application programmers to have to do so.
21:47 < nz> yeah, true enough
21:47 <@outoftime> nz: would that be the case if we were writing against ElasticSearch? perhaps not.
21:47 < squeegy> outoftime: i was just going to say, i use sunspot and i know very little about solr
21:47 < nz> Sunspot = backwards compatible, no knowledge of Solr required? Solrskates = quick, light, and more flexible if you know Solr?
21:48 <@outoftime> nz: no : (
21:48 < squeegy> as our application has grown, i've had to learn more, but sunspot does a good job of keeping me dumb :)
21:48 < nz> outoftime: swing and a miss, then ;)
21:48 < nz> my what has become of me? I think of marketing before anything else these days ;)
21:48 <@outoftime> nz: what I've been trying to get across is that Solrskates should do the job that Sunspot does now.
21:49 <@outoftime> that's kind of a red line for me
21:49 <@outoftime> that we release something that does what sunspot does, but does it better. including with a sweet new API designed by a cool guy I know ; )
21:50 < nz> well if we're going to do searches like Post.search.fq(...).qf(...), that implies a certain level of knowledge about Solr right there
21:50 < nz> I think the requisite level of Solr knowledge we're assuming is a pretty legit conceptual point to make
21:50 < squeegy> at a certain point knowledge about boost (and other details) is hard to avoid
21:51 <@outoftime> nz: ok yes two different things are being confused
21:51 < nz> a benefit to less conceptual mapping, is that usage docs sort of double to explain Solr concepts
21:51 < nz> yeah, we're probably agreeing ;)
21:51 <@outoftime> first proposal for Solrskates: Solrskates.q('great pizza').fq('category_id:1')
21:52 <@outoftime> lightweight, transparently exposes the Solr API
21:52 <@outoftime> just makes life a bit easier
21:52 <@outoftime> second proposal for Solrskates: just a new name for Sunspot 2.0
21:52 <@outoftime> with Sunspot 2.0 being the name of the compatibility layer
21:53 <@outoftime> am I making sense? I realize it's incredibly confusing to use the same name for two entirely different ideas
21:54 < nz> maybe we let it simmer?
21:54 <@outoftime> damn, really?
21:54 <@outoftime> ok
21:54 < nz> I lean toward Solrskates lightly exposing/mapping the Solr API
21:54 <@outoftime> and how does the rest of the Sunspot universe look in that case?
21:54 < nz> which Sunspot 2.0 uses to map to the Sunspot DSL
21:55 < nz> with the same, lovable "searchable" and "search" blocks it's always had
21:55 <@outoftime> and what about the new API?
21:55 < nz> largely in Solrskates?
21:56 < nz> oh right
21:56 < nz> now I see where we've circled back
21:56 <@outoftime> yes : )
21:56 <@outoftime> basically we have three ideas but two names
21:56 < nz> I feel like we're trying to communicate in the wrong language!
21:56 < nz> haha
21:56 <@outoftime> hontou nee.
21:57 <@outoftime> so we've got
21:58 <@outoftime> 1) Lightweight wrapper with transparent API access
21:58 <@outoftime> 2) Library that does what Sunspot does better -- new non-transparent API, but better introspection, easier setup, etc.
21:58 <@outoftime> 3) The next version of Sunspot, which retains compatibility with the Sunspot 1.x API
21:59 < nz> yeah that sounds about right
22:00 <@outoftime> so from my standpoint, I wouldn't mind merging #1 and #2 into a single library, and just exposing lower-level functionality as part of the public API for those who want it.
22:00 <@outoftime> or at the very least, build #2 first, and then if it seems like one could cleanly extract #1, do it then
22:01 <@outoftime> but I could also be swayed to stay the course in building #1 first, and then building #2 on top of it from the outset
22:02 < nz> I personally favor starting with #1
22:03 <@outoftime> okay. I'm cool with that.
22:03 < nz> partly because I have other crazy ideas
22:03 <@outoftime> yes.
22:03 < nz> which represent a bit of a conceptual break
22:03 <@outoftime> yes.
22:03 <@outoftime> yes I would like to have #1 available for exactly that sort of thing
22:03 <@outoftime> okay, so I cut you off as we began to discuss SolrMapper, although I've glad we've got the above cleared up
22:04 <@outoftime> so can you dive into that again?
22:05 < nz> why certainly!
22:05 < nz> expect a gist momentarily
22:05 < nz> two of the things that I like here...
22:05 < nz> 1. Solr as a persistence layer
22:05 < nz> I have a lot of solr cores laying around, and I want to just throw data into them and get it back out again
22:06 < nz> from plain old Ruby scripts
22:06 < nz> also
22:06 < nz> I like opinionated schemas, but I am no stranger to tweaking them
22:06 < nz> I want my schema to define my DSL a little bit
22:07 < nz> ie. <dynamicField name="*_text" .../> gives me the "text" class method to define a text field
22:08 < nz> gist incoming
22:08 <@outoftime> okay
22:11 <@outoftime> gotta run in about 20 mins fyi : )
22:12 < nz> https://gist.github.com/1055137
22:13 <@outoftime> so it seems like SolrMapper would converge with #2 in the indexing structure
22:13 <@outoftime> but how much?
22:14 <@outoftime> most of the work in constructing the Solr document would still be up to #2, right?
22:14 <@outoftime> once you've got that, passing it to RSolr isn't really a big deal...
22:15 < nz> I just put in some updates, feel free to refresh there
22:16 < nz> gist-1 is about SolrMapper as a reasonably convenient and complete Solr client
22:16 <@outoftime> right
22:16 < nz> gist-2 points out that SolrMapper could be low-level enough to not care about anything. Just give it a reasonable hash that corresponds to your schema.xml
22:16 <@outoftime> as a persistence layer
22:16 <@outoftime> sure
22:16 < nz> that's the bit that could be reused, say, in a Sunspot 2.0 DSL mapper
22:17 <@outoftime> right, but again, once you've got that hash, how much value is SolrMapper adding?
22:17 < nz> instead of RSolr.add, use SolrMapper::Document.create
22:17 < nz> relative to the Solr API? or relative to our other ideas? ;)
22:17 < nz> I think there is a lot of overlap here with what we've been discussing
22:17 <@outoftime> what's the value add of SolrMapper::Document.create instead of RSolr.add
22:18 < nz> none
22:18 <@outoftime> sooooooo
22:18 < nz> I might not have named things suuuper well in the example
22:18 < nz> but my example Post.create...
22:18 <@outoftime> I mean, don't get me wrong, I think this is a great idea. I'm just not seeing the compelling argument for integrating it into Sunspot
22:19 <@outoftime> (or whatever #2 ends up being called)
22:19 < nz> introspects the class's field definitions, assembles that hash, sends it to Solr
22:19 <@outoftime> but a class that's searchable via Sunspot wouldn't inherit from SolrMapper::Document
22:19 <@outoftime> and thus SolrMapper wouldn't have field definitions for it
22:20 <@outoftime> unless...
22:21 < nz> unless maybe the searchable block was turned into a class that itself extends/includes SolrMapper::Document
22:21  * alindeman had to get back to work, will read backlog though
22:21 < nz> that's where the interesting potential for reuse happens
22:21 <@outoftime> if SolrMapper had something equivalent to the existing Sunspot::Setup
22:22 <@outoftime> so basically you could create a SolrMapper setup without actually instantiating a subclass of SolrMapper::Document
22:23 <@outoftime> then #2 would create a SolrMapper::Setup for each class being indexed
22:23 < nz> something like that
22:23 <@outoftime> with a bit of special behavior -- in particular, it would default to non-stored fields, whereas I assume SolrMapper itself would default to stored fields
22:24 < nz> right
22:24 < nz> tbh,
22:24 < nz> the part that I most care about
22:24 <@outoftime> then Sunspot proxies searches to SolrMapper, gets back a collection of SolrMapper::Documents just containing the IDs (and whatever other stored fields)
22:24 < nz> is being able to fire up a script, require a file, and then create classes like the ones in that gist-1 example :)
22:25 <@outoftime> and Sunspot (or #2) takes care of loading the model classes out of the main persistence layer
22:25 < nz> yes
22:25 < nz> exactl
22:25 < nz> y
22:25 <@outoftime> a compelling idea.
22:25 <@outoftime> I need to let that sink in -- I'm wondering if the overlap might not in fact be so great that it should be one library
22:26 < nz> so Sunspot is one way or another creating, then searching, a bunch of lightweight standalone Solr Documents
22:26 <@outoftime> (especially thinking from a documentation standpoint)
22:26 <@outoftime> right
22:26 < nz> which it then uses to do ActiveRecordy things
22:26 <@outoftime> right
22:26 < nz> hence my repeated return to the preference of forking the "fun" stuff into a new library
22:26 < nz> ;)
22:26 <@outoftime> nah
22:26 <@outoftime> let's put it all in #2
22:27 < nz> btw, this is already a WIP for search.rubygems.org
22:27 < nz> a light indexing DSL to describe a configuration
22:27 < nz> which indexing then uses to create a hash for RSolr
22:27 < nz> works well that way, since our only interface with rubygems.org proper will be webhooks
22:27 < nz> no access to its database
22:28 <@outoftime> really the diff between SolrMapper and what we have in mind for #2 is just exposing a public API for standalone documents
22:28 <@outoftime> adding standalone documents to #2 would not be a huge addition
22:28 < nz> yeah
22:28 < nz> I can see that
22:29 <@outoftime> right? then you get your wish, and we get to keep the meat of the code in one library, which keeps things simple from a documentation/contribution standpoint
22:30 < nz> yep!
22:30 <@outoftime> in fact, to be honest, I think you could probably do this with adapters.
22:31 <@outoftime> class Sunspot::StandaloneDocumentAdapter ; def load(doc) ; doc ; end ; end
22:31 <@outoftime> not saying that's necessarily the right approach, but just as a demonstration of how neatly the idea could fit into what Sunspot / #2 is already doing
22:31 < nz> sure
22:33 <@outoftime> so maybe give that some thought? I'm definitely coming around to the potential for overlap here, but I'd hate to have #2 be a separate library built on top of SolrMapper with all the meat in SolrMapper, again mostly from a documentation/contribution standpoint
22:35 < nz> I think I like where we've arrived on this point
22:35 <@outoftime> awesome. let's let it simmer to a rich golden brown
22:35 <@outoftime> and catch up soon with further inspiration
22:36 < nz> so you need to head out
22:36 < nz> but first
22:36 <@outoftime> yes
22:36 < nz> what's the status with running the integration tests? ;)
22:36 < nz> because that's not working for me lately
22:36 <@outoftime> oh um
22:36 <@outoftime> i mean i haven't tried to do that in months
22:36 < nz> I can run the sunspot specs, but not the integration specs
22:36 <@outoftime> soooooooooooooo your guess is probably considerably better than mine : )
22:36 < nz> hmm
22:36 < nz> bummer
22:37 <@outoftime> if an answer doesn't make itself available, shoot me an email tomorrow and I'll see what I can figure out
22:37 < nz> have you seen travisci?
22:37 < nz> https://github.com/travis-ci/travis-ci
22:37 <@outoftime> nope
22:37 < nz> http://travis-ci.org/
22:37 < nz> community CI tool
22:37 < nz> works great
22:37 < nz> we should use it
22:37 <@outoftime> that looks awesome
22:37 <@outoftime> i back it
22:38 < nz> we'd need to get our tests to a point of 'bundle install; bundle exec rake'
22:38 <@outoftime> we can probably make that happen
22:38 < nz> yeah
22:38 <@outoftime> assuming their servers have java
22:38 <@outoftime> i'm not sure java is packaged as a gem : )
22:38 < nz> or I can provide an index for the integration testing stuff
22:39 <@outoftime> ohhh
22:39 <@outoftime> not bad
22:39 <@outoftime> not bad at all.
22:39 <@outoftime> (although I'd still prefer to have it be self-contained if possible)
22:39 < nz> sure, less hassle to change versions and whatnot
22:40 <@outoftime> yep
22:40 <@outoftime> ok rly gotta go now!!
22:40 <@outoftime> talk to you soon
22:40 < nz> late!