7a9ceada3a8da9e8a6aa51ab76ebdb7e357a1d8a John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/50044d069d0c75047a5aab726629295097f0650d 50044d069d0c75047a5aab726629295097f0650d 2009-12-13T16:27:37-08:00 2009-12-13T16:27:15-08:00 Tests for multi-arch and multi-Ruby version XapianDocValueAccessor should read stored values from databases created by various versions of Ruby on various architectures d3974bd725a643c3ee1868e249f1ef91bb372f8a John Leach john@johnleach.co.uk 799afac2383b48d3741608438ac843600d10c9dd John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/7a9ceada3a8da9e8a6aa51ab76ebdb7e357a1d8a 7a9ceada3a8da9e8a6aa51ab76ebdb7e357a1d8a 2009-12-11T17:56:29-08:00 2009-12-11T17:56:29-08:00 Fix value storage architecture independence * Change to_xapian_fu_storage_value instance method to class method, makes more sense * Improve tests * Raise XapianFu::ValueOutOfBounds when Bignum is too big to be stored (+/-9,007,199,254,740,991) 95dc61a537ada67123d20482bcba526bc1ee74b8 John Leach john@johnleach.co.uk f8ea4147baf9e6f20d13582aa147faed8ea015e1 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/799afac2383b48d3741608438ac843600d10c9dd 799afac2383b48d3741608438ac843600d10c9dd 2009-12-08T16:55:42-08:00 2009-12-08T16:55:42-08:00 Ruby 1.9.1 support - changes on-disk format! Adding Ruby 1.9.1 support mostly just involved tweaking the specs - there are still some issues with stemming and utf8 though. Unfortunately, the way values numbers are calculated from key names for storage had to change. String#hash is not consistent between VM executions under 1.9.1, so it's now useless as a way of converting key names to value numbers (as they change every execution!). I changed this to use Zlib.crc32. Thanks to McClain Looney of LoonSoft LLC who found this and wrote the initial code to fix it. This update changes the on-disk format - all stored document values are essentially lost (as their value numbers change to likely non-existant values). If you're using XapianFu to store original data, then you need to do some work to convert your databases from xapian_fu 1.0.1 or earlier. Otherwise, just rebuild your index from scratch and you'll be fine. fe40dbd3106e4d5da228ab37b0b202274cc4b146 John Leach john@johnleach.co.uk 330db08909f3d76bf575d2478eaad4ecc919fd98 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/f8ea4147baf9e6f20d13582aa147faed8ea015e1 f8ea4147baf9e6f20d13582aa147faed8ea015e1 2009-12-08T14:11:36-08:00 2009-12-08T14:11:36-08:00 Fix value storing on 64bit(!) On 64bit Ruby, String#hash seems to returns an integer bigger than Xapian can use for value numbers. For now I'm masking the extra bits - simple and backwards compatible. 3e7420cee7362cb4746446162ced79b4e1a1c83d John Leach john@johnleach.co.uk 1a4aad811f3a018e51ffc72b81f3fd3da81f2fca John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/330db08909f3d76bf575d2478eaad4ecc919fd98 330db08909f3d76bf575d2478eaad4ecc919fd98 2009-12-08T14:01:00-08:00 2009-12-08T14:01:00-08:00 Add require and include lines to examples in README a66e0e2fdacdf3cf9b986fb0381dc2602a7278e7 John Leach john@johnleach.co.uk 105603d83c452c1e95340b09800edd1a9688c54b John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/1a4aad811f3a018e51ffc72b81f3fd3da81f2fca 1a4aad811f3a018e51ffc72b81f3fd3da81f2fca 2009-12-08T13:55:07-08:00 2009-12-08T13:55:07-08:00 Support require 'xapian-fu' as well ass 'xapian_fu' e1e2a986f63157d8e7389c417f88983ea6a71731 John Leach john@johnleach.co.uk a2ed266050003c817f6f668364e16d085202d0b6 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/105603d83c452c1e95340b09800edd1a9688c54b 105603d83c452c1e95340b09800edd1a9688c54b 2009-09-09T15:10:02-07:00 2009-09-09T15:10:02-07:00 XapianDb should lazily create the on-disk database 86d4f0dfbba1b6f5e009f169d5f0eef235af9ea6 John Leach john@johnleach.co.uk 9b2d5ba7ceb6f8cfa46056ba6d33842a1b57a47a John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/a2ed266050003c817f6f668364e16d085202d0b6 a2ed266050003c817f6f668364e16d085202d0b6 2009-09-09T14:42:14-07:00 2009-09-09T14:42:14-07:00 xapian-fu 1.0 9f526ca3f31daa19134db9bed5b3fb36a922e7b5 John Leach john@johnleach.co.uk 427eb6a5ae3bcb670975157d2895c8a65dc1230b John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/9b2d5ba7ceb6f8cfa46056ba6d33842a1b57a47a 9b2d5ba7ceb6f8cfa46056ba6d33842a1b57a47a 2009-09-07T16:03:28-07:00 2009-09-07T16:03:28-07:00 More documentation, ready for version 1.0 release 968fa567617e4180154a62e975b9b9cc9aec2773 John Leach john@johnleach.co.uk 8389581312b4ca193bb8898b9284484ea74b120a John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/427eb6a5ae3bcb670975157d2895c8a65dc1230b 427eb6a5ae3bcb670975157d2895c8a65dc1230b 2009-09-06T14:45:23-07:00 2009-09-06T14:45:23-07:00 Proper indexing of Time, Date and DateTime objects fdea9bb50e53e2874c2149c7fca04fc2e9a45aeb John Leach john@johnleach.co.uk c115d15ae4019a53d1449dd534940220bf2d7e20 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/8389581312b4ca193bb8898b9284484ea74b120a 8389581312b4ca193bb8898b9284484ea74b120a 2009-08-31T11:04:20-07:00 2009-08-31T11:04:20-07:00 Value storage system overhaul - Changes on-disk format, not backwards compatible - Reversible storage for values. Put in a Time and get out a Time. - More efficient storage for Integers, Time ane Date values. No more leading zeros. - Use class/instance methods for value conversions. Replaces ungainly case statement and is easily extendible. - More documentation b34b59a9601f6d172669c19c342aa01621fb8648 John Leach john@johnleach.co.uk be0b105a93286e22f4595f46ee5d89fae38f71a2 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/c115d15ae4019a53d1449dd534940220bf2d7e20 c115d15ae4019a53d1449dd534940220bf2d7e20 2009-08-23T10:19:26-07:00 2009-08-23T10:19:26-07:00 New :fields option format - Allow specifying of field type (useful later) - Allow defining fields as not to be indexed a7e5d6d32e4aacca5d9f2f1d82b401d44215c81b John Leach john@johnleach.co.uk e22a90940ea2e8d17142949726ed533ec97d3f6c John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/be0b105a93286e22f4595f46ee5d89fae38f71a2 be0b105a93286e22f4595f46ee5d89fae38f71a2 2009-08-16T06:07:55-07:00 2009-08-16T06:07:55-07:00 Update the spider.rb and query.rb examples - Updated for new API. - store the filesize - allow queries on fields - --order-by-filesize option on query.rb - improve benchmark measurements - replace Dir.glob with index queue for faster start 7c06824fefb757537316286cb0fad959744dce1e John Leach john@johnleach.co.uk 378f908792db1a92e4f4b0affe7e0713a610f67a John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/e22a90940ea2e8d17142949726ed533ec97d3f6c e22a90940ea2e8d17142949726ed533ec97d3f6c 2009-08-16T04:31:41-07:00 2009-08-16T03:41:39-07:00 Field search support, and more documentation. XapianDb documentation spec: QueryParser parse_query: - should use the :fields option to set field names - should use the database's field names as prefixes spec: XapianDb search should do searches with and without field names c283fd3db67562054acd7e7cf08db39764984071 John Leach john@johnleach.co.uk 045ee81f345be5423f127c9426cf81c245191623 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/378f908792db1a92e4f4b0affe7e0713a610f67a 378f908792db1a92e4f4b0affe7e0713a610f67a 2009-08-16T03:39:26-07:00 2009-08-16T03:39:26-07:00 Removed query_parser_factory.rb. Wakey wakey sleepy-head. b5d45f43ae758f53552d0a1498349db145e8a55d John Leach john@johnleach.co.uk a3c9574f375d907d88156d344e260001cbb2bb46 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/045ee81f345be5423f127c9426cf81c245191623 045ee81f345be5423f127c9426cf81c245191623 2009-08-16T03:34:00-07:00 2009-08-16T03:34:00-07:00 Imported missing query_parser_factory.rb (duh) 155f8bf8a34af4c7247e4299fc3b1af7d92af730 John Leach john@johnleach.co.uk 0a237154a706270a7ae3ea8553cb09633df5d166 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/a3c9574f375d907d88156d344e260001cbb2bb46 a3c9574f375d907d88156d344e260001cbb2bb46 2009-08-09T06:35:08-07:00 2009-08-09T06:35:08-07:00 Documentation for more QueryParser options b5d45f43ae758f53552d0a1498349db145e8a55d John Leach john@johnleach.co.uk 55b13b91c25cc8a99cb6bd4fb7b40aa012c22766 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/0a237154a706270a7ae3ea8553cb09633df5d166 0a237154a706270a7ae3ea8553cb09633df5d166 2009-08-09T06:25:25-07:00 2009-08-09T06:07:23-07:00 Refactored query parsing. Moved code to new files. New XapianFu::QueryParser class to handle setting up the Xapian::QueryParser. Moved XapianDocumentsAccessor code out of xapian_db.rb Moved ResultSet code out of xapian_db.rb b54685ecd3f282833306477d4767a7b7f1659b47 John Leach john@johnleach.co.uk 813eebfd03c55925dc5d7bf9e77ce217c73cdb63 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/55b13b91c25cc8a99cb6bd4fb7b40aa012c22766 55b13b91c25cc8a99cb6bd4fb7b40aa012c22766 2009-08-08T11:27:31-07:00 2009-08-08T11:27:26-07:00 Separate indexing of fields spec: XapianDoc to_xapian_document should tokenize the fields of a hash separately 3c5ca9013b0de7a674b28d6fc00b22aa7f24e1bc John Leach john@johnleach.co.uk 703e49819fc5f3a7af991790e4a9880b3a1957ac John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/813eebfd03c55925dc5d7bf9e77ce217c73cdb63 813eebfd03c55925dc5d7bf9e77ce217c73cdb63 2009-08-02T15:45:32-07:00 2009-08-02T15:45:32-07:00 Tweak term generation to use less ram b1543fccb37e457d45d3fc73d51009a11ac57f31 John Leach john@johnleach.co.uk 07997ad3547f6aa5df10c4e2ba68adffcbb8def3 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/703e49819fc5f3a7af991790e4a9880b3a1957ac 703e49819fc5f3a7af991790e4a9880b3a1957ac 2009-08-02T15:33:43-07:00 2009-08-02T15:33:43-07:00 Fix :store option and update README spec: XapianDb add_doc should store fields declared as an array to be stored as values c0963aa856c615f6d7af2fced4331ad8b65ff148 John Leach john@johnleach.co.uk 03a1d837043451aba526a7a8e251a781daa8fe61 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/07997ad3547f6aa5df10c4e2ba68adffcbb8def3 07997ad3547f6aa5df10c4e2ba68adffcbb8def3 2009-08-02T13:42:02-07:00 2009-08-02T13:42:02-07:00 Refactor field storage and retrieval Redesigned field storage. Now all fields that need storing are stored as Xapian values, rather than being Marshalled and stored in the Xapian data blob. They're also now lazily loaded, which should increase performance when getting a lot of results from a query. d44d33d15bfd2d8cd67eaefc7b7c9054f0510c70 John Leach john@johnleach.co.uk 227c60cb36d4d82c41098c490344d0ea2526697f John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/03a1d837043451aba526a7a8e251a781daa8fe61 03a1d837043451aba526a7a8e251a781daa8fe61 2009-08-02T03:44:37-07:00 2009-08-02T03:44:37-07:00 spec: XapianDoc stopper: - should return the same stopper as the database by default - should return a stopper for the document language, overriding the db - should return a stopper set by the :stopper option, overriding the :language option and the db stopper spec: XapianDoc stemmer: - should return the same stemmer as the database by default - should return a stemmer for the document language, overriding the db - should return a stemmer set by the :stemmer option, overriding the :language option and the db stemmer Refactored XapianDc stemmer and stopper to be lazily set up b64f908095900be17e53cb4e9abb8c7430d33cf5 John Leach john@johnleach.co.uk 018aa0b28c03c0d73b39a36a5efe51d61c7b0d00 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/227c60cb36d4d82c41098c490344d0ea2526697f 227c60cb36d4d82c41098c490344d0ea2526697f 2009-08-01T08:23:37-07:00 2009-08-01T08:23:37-07:00 spec: XapianDb stemmer should return an english stemmer by default spec: XapianDb stemmer should return a stemmer for the database language spec: XapianDb stopper should return an english stopper by default spec: XapianDb stopper should return a stopper for the database language bbc153e72477caa3b13a922deb55bb0a960d2364 John Leach john@johnleach.co.uk 4bdb4d0698a6e013af732e579b345df3cead7056 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/018aa0b28c03c0d73b39a36a5efe51d61c7b0d00 018aa0b28c03c0d73b39a36a5efe51d61c7b0d00 2009-07-25T05:36:21-07:00 2009-07-25T05:36:21-07:00 Added Danish, Dutch, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish and Swedish stop word lists spec: StopperFactory#stop_words_for should return characters in utf8 spec: XapianDoc should allow setting the stopper on initialisation spec: XapianDoc should support stop words encoded in utf8 2905a7434c2a5f6604924653eb86c3a53299c590 John Leach john@johnleach.co.uk bb222129a00b8f94ae66356301dfbcf5d35c2fa9 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/4bdb4d0698a6e013af732e579b345df3cead7056 4bdb4d0698a6e013af732e579b345df3cead7056 2009-07-19T16:22:46-07:00 2009-07-19T16:22:34-07:00 spec: XapianDoc to_xapian_document should not stem english stop words by default Implemented StopperFactory to created Xapian::SimpleStopper objects from Snowball library stopword lists (currently included English stopword list) 459f252d7ab6a377a6d5c8694b5b013bc1cc9439 John Leach john@johnleach.co.uk 6b11e7a721ce8304a522618a851437854e227602 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/bb222129a00b8f94ae66356301dfbcf5d35c2fa9 bb222129a00b8f94ae66356301dfbcf5d35c2fa9 2009-07-12T10:07:33-07:00 2009-07-12T10:07:33-07:00 spec: XapianDoc should inherit the databases stemmer by default spec: XapianDoc should stem German, French, Swedish, Danish, Russian and Italian words when the :stemmer option is set accordingly 06629601b8be407e6f7a9d1f385be983caf721c0 John Leach john@johnleach.co.uk 7f08919f637e18fa09f5101155d1a7a4d003c4a2 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/6b11e7a721ce8304a522618a851437854e227602 6b11e7a721ce8304a522618a851437854e227602 2009-07-12T05:41:21-07:00 2009-07-12T05:41:21-07:00 XapianDoc to_xapian_document should stem english words by default XapianDoc to_xapian_document should not stem words when stemmer is set to false 6f7bd03482b4c744e673181b6fb25a30b3b4a72f John Leach john@johnleach.co.uk 9074f93a0157f93742038f9efd83c4a01d407a8c John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/7f08919f637e18fa09f5101155d1a7a4d003c4a2 7f08919f637e18fa09f5101155d1a7a4d003c4a2 2009-07-12T05:12:29-07:00 2009-07-12T05:12:29-07:00 spec: XapianDb#documents#new should return a new XapianDoc with the db set on new Moved tokenization specs into XapianDoc specs and use new documents#new method b1e003e7dc376171d1a4bf83e88653e6b75e631e John Leach john@johnleach.co.uk 31940b59cc5a8539d5c4ea3e6c02f8f0462a51e3 John Leach john@johnleach.co.uk http://github.com/johnl/xapian-fu/commit/9074f93a0157f93742038f9efd83c4a01d407a8c 9074f93a0157f93742038f9efd83c4a01d407a8c 2009-07-12T04:52:08-07:00 2009-07-11T07:41:08-07:00 Refactor add_doc * moved to DocumentAccessor * moved term generation into XapianDoc * moved actual saving to XapianDoc (save, update, create methods) f8fe97928346e4a6cb73f1511b226948bd8f8da8 John Leach john@johnleach.co.uk