Stopwords

[assem-ch]

http://trac.xapian.org/ticket/269#no1

• Gathering the list of Arabic Stop-words
• Include the stop-word lists from Snowball Project

[ojwb]

You don't need to actually include <fstream> if you only reference things from it in a documentation comment.

[assem-ch]

I removed it

[ojwb]

I guess you're trying to use $'\t' here? That's a bash-ism, and will only work as you expect if SHELL gets set to bash, which certainly won't happen on systems without bash installed. I think you've not managed to get it quite right for bash either - if I un-double the $ and echo the result to see how it expands in bash, I get:

$ echo sed 's/[$'\t' ]*|.*//;/^[$'\t' ]*$/d'
sed s/[$t ]*|.*//;/^[$t ]*$/d

But what you want is simply a literal tab character in this Makefile.mk. How to insert one depends on what editor you're using. Maybe just hitting the tab key will do it, but often that's mapped to some sort of smart indent function. For example, in vim it's Ctrl+V tab to insert a literal tab.

[assem-ch]

I fixed it

[ojwb]

There's a space after the backslash on this line, which confuses the preautoreconf script which collates a list of the sources to feed to doxygen, and results in make choking on docs/Makefile:

make[2]: Entering directory '/home/olly/git/xapian-svn/xapian-core/docs'
Makefile:390: *** recipe commences before first target.  Stop.

This may be something GNU make 4.0 is fussier about than 3.x, but generally spaces or tabs after a \ intended to continue the line are asking for trouble, as they may be interpreted as escaping the whitespace instead.

[ojwb]

OK, merged almost all of this via SVN, and committed the whitespace fixup to languages/Makefile.mk as well.

I haven't merged the bindings changes yet, but it's past time I went to bed.

[ojwb]

I just noticed there are a number of duplicates in your arabic stopword list (number is how many times it occurs):

      2 إذ
      2 إذا
      2 أما
      2 أنى
      3 أي
      2 إياك
      2 أيان
      3 أين
      2 بكم
      2 ذا
      2 سوى
      2 كل
      2 كلا
      2 كم
      3 لست
      3 لكن
      2 لما
      3 متى
      2 هيا
      2 وا

List was generated by:

sort arabic/stop.list|uniq -c|grep -v '^ *1\>'

[ojwb]

OK, I've tweaked the stop word prep makefile rule to sort and uniq the words, so any duplicates in the sources will get eliminated, though unless there's a good reason to have them in the sources it's probably better to remove them from there too.

A few of the snowball lists also had duplicates which I've removed and I will open a PR with snowball.

The Russian list seemed to be KOI8-R, so I've converted that to UTF-8.

And I've merged the stemmer change, reworked slightly to wrap it as a constructor so it's more similar to the C++ API. Also added a quick testcase.

So everything's now merged - sorry it's taken a while.