#1921 listview filtering/search config

[project707]

created simple "data-filter-skipspecial" option that allows for skipping all special chars in input and filtered text. Defaults to false.

[toddparker]

Thanks for the pull. Instead of creating a special case for this one situation, we'd be better off having a pluggable solution that lets' people use whatever parsing logic they want. I can see the current approach ballooning as people want to add scoping and more complex searching logic.

[project707]

I agree that would be better than this simple generic method.

Do you think having the options set on mobileinit would be the right way to do that (I haven't used those methods yet)? Maybe something like:

$(document).bind("mobileinit", function(){
$.mobile.searchFilterReplace = "/[^a-zA-Z 0-9]+/g,''"; //user's choice of .replace regex
$.mobile.searchFilterTrimWhitespace = true; //should be default true if above is set
});

[toddparker]

I asked some other team members for an opinion on this. If don't hear back in a few days, please ping us here.

[jeff-ph]

I have two thoughts on this:

1. Using a simple keyword is my first choice. Matching the start of the string to me would be the most typical scenario for a list filter. So from my post List data-filter type #1982 I would retain the idea of having a data-filter-type attribute with possible values being start, contains, end, and regex -- start would be the default if the data-filter-type attribute is not explicitly defined in the markup. The regex type would then require another attribute of data-filter-regex that would contain the actual regex to use.

2. Inherent in item 1 is the ability to set the filter type/regex for each list, rather than forcing it as one filtering method across the application which I believe is what would happen if set on mobileinit.

[toddparker]

We discussed @jeffholmes idea and that seems like a decent solution. Do either of you want to try and code this up as a pull?

To summarize "a data-filter-type attribute with possible values being start, contains, end, and regex -- start would be the default if the data-filter-type attribute is not explicitly defined in the markup. "

[project707]

I'm not really sure I follow. It seems to me that matching on the beginning of any term whether that be at the start or end is default functionality, and has no problems with case or anything of that nature. regex just seems like it would do the rest. What am I missing?

Everything but matching of non alphanumeric characters in the native keyboard range seems to work just fine for me unless I want to work with mixed terms. If mixing terms at different positions was the issue, I guess I would suggest separating terms into an array that has no set order, or even taking it a step further into the type of edge n-gram filtering available with the complex indexing available in apps like solr.

[project707]

OK I think this now functions as everyone expects. I had a hard time understanding the need to only match on the first or last term, or that that was what was meant, but that's in there now and you can specify your own regex for what to exclude.

So, here's how it works:

-default is current default functionality and this is the same as data-filter-type="contains"

-data-filter-type="start" ensures that it is only matching on the first term in the string

-data-filter-type="end" ensures that it is only matching on the last term in the string

-data-filter-type="regex" by default excludes all but english alpha and numeric characters [^a-zA-Z 0-9]

-adding data-filter-regex="[^some regex here]" allows you to supply data-filter-type="regex" with your own regex

Seems correct and simple enough right?

[jeff-ph]

I haven't had a chance to test, but just observing -- 707 I think you're thinking about it more as a search utility as opposed to a filter. For me, the list will usually already have some logic to it, then filtering is just to narrow down the results. There are a lot of different possibilities so the more flexible the better.

I just want to confirm that the "start" functionality matches the beginning characters (it shouldn't matter how many terms it actually filters on). For example:

LIST
Abardo
Barbados
Bar Bon
Bar Cloud
Carbarlo
Dunbarton

All terms contain "bar", so FILTER TEXT of "bar" should only bring up "Barbados", "Bar Bon" and "Bar Cloud".

FILTER TEXT of "bar b" should only bring up "Bar Bon".

Is that how you assembled it?

[project707]

ah no. "Start" is not exactly like that - I have it matching on the first segment of the string. However, I think it would be super easy to get it working like you mention.

based on that how were you imagining "end" would work? where would you start matching?

As far as looking at it as a search utility is concerned - that's really what it is, it just starts with all results displayed. You have to figure there will be cases where users know exactly the string they're searching for, but probably more cases where they only know a word or two that should likely get them the result they need from a very long list. Also if you're trying to imitate the functionality of native search that displays a listview (often done this way in ios and android) this is still the best way to go if your result sets are somewhat limited, because then you can return the initial set and not make additional calls unless absolutely needed.

[jeff-ph]

My thinking of start and end was that it would work the same as simple pattern matching in SQL with the LIKE keyword and % wildcard, such as:

listitem LIKE '[filtertext]%' (start)
listitem LIKE '%[filtertext]%' (contains)
listitem LIKE '%[filtertext]' (end)

The reason I felt start would be the default is mostly because with long lists, entering a few characters to filter on would probably not narrow down the list enough to make it usable. I had this problem in an app I was developing, which is what sparked the whole data-filter-type idea for me since I wasn't able to filter on just the beginning of the string.

I agree that the search functionality is needed, too, and I'm really indifferent about what functionality is the default, I just know I need to be able to match on the start of the string somehow. :)

[project707]

OK. I get it.
The problem is, with "end" functioning that way you would lose all results until you had completed typing in the last character since the filter responds on keyup, and you can't enter it in reverse, so until then it wouldn't match since it's a filter from the end character back. The difference in running that as an sql query is that you're sending the fragment/term all at once. "End" functioning this way seems like a very special case, but I'm sure I can think of a way to do it if that's what you need.

My take on making filtering down a list usable is in promoting the user to use non generic terms. If they know the specifics of what they're looking for, there shouldn't be much of a problem but that's not a programming based solution of course...

I guess why I don't really "get" the usage of the start or end features is because more often than not you might have an item start with a brand name or even a common word like "the" or "a". if you have a variety of cases like this choosing start or end doesn't help you, but I could see if you have some kind of set of filtered items that always somehow only starts with the most unique part of the string then it could be totally useful. But it could only be useful if they know to type the characters that are part of that term anyhow.

I'm going to look into a way to make "end" work like you are thinking.

[project707]

Ok @jeffholmes - this now functions exactly as you described. I would say that "end" will have more overhead than the others, but I think it's as simple and straightforward as it could be.

[gseguin]

I find data-filter-type "regex" a bit confusing. The first time I saw it I thought it would offer the user the possibility of entering the regexp to filter the results when in fact it's a "contains" search where items have been pre-processed by a trimming regexp.

How about data-filter-type="contains" and data-filter-trimming-regexp="[^a-zA-Z 0-9]" ?

[project707]

You may have missed that the additional parameter specifies the regex (of what should be ignored) itself - as had been suggested. All I did was have it default to a common alphanumeric filter if you didn't supply your own in "data-filter-regex=..." but did state that data-filter-type="regex"

I do get what you are saying, but the default without the parameter "data-filter-type" specified does not actually apply any regex.

[gseguin]

I haven't missed the second parameter, just the naming is confusing I think because really the data-filter-type regex is actually a contains with a pre-processing of the list.

So what I'm proposing is that data-filter-trimming-regexp defaults to "" (empty string) and that you always pre-process.

[project707]

I see, so put simply you think that data-filter-trimming-regexp should not be a sub-param of data-filter-type, but rather always run. That's how I started this actually, but I had been trying to not go too far with this and not do any processing on anything that wasn't already happening unless initiated by the user (though I know there likely wouldn't be any real difference)

The one thing I had done is to not require the user to state "contains" since that is the default already since it's using a simple indexOf.

So let's go through this, as I only have time for so many revisions of this:

• list (and input) is always pre-processed
• if you specify regex in "data-filter-trimming-regexp" param, then the above pre-processing happens using this to exclude those characters from being necessary to enter
• "data-filter-type" can be set = "start" or "end" (or left blank for "contains") and the above would still apply to it

Correct?

Just FYI this isn't the end of this as I have another pull for "terms" based filtering rather than full string, and I'm sure something will come up :)
Sure would be nice to just combine the two.

[gseguin]

That is correct. We can then add filter types like "word", "wordstart" and "wordend".

[project707]

If you're talking about in conjunction with the term-based filtering that naming convention would make some sense. The funny thing is, I had already essentially created "wordstart" and "wordend" filtering to limit it to those terms, before realizing that the expectation was for it to function like SQL LIKE start% and %end which it is now. Myself I can't really see a case for the %end functionality as it requires a complete match to the last letter, meaning everything gets filtered until complete, but the recommendation had been to go with his methodology.

Although I want to make everybody happy, I really need to get this all locked down soon. Who all agrees that if I get it functioning as stated in my last comment that we're all good on this part?

Also, what is the usual convention for something like the terms based filtering option which is tightly interwoven with this functionality? Would it make sense to move it into this same pull since it is technically part of the original concept of "Customize the filtering algorithm"?

[gseguin]

I think keeping the 2 pull requests separated is good. The first one being the ground work and the other one will be adding a feature. This way we can get stuff in quicker, what do you think?

[project707]

Fine with me. At this point I just need to make sure everybody involved is on the same page with the recent changes you have suggested so I'm not spending time on additional iterations. I'm not totally sure how to go about getting consensus from everyone with an interest in this feature...

[johnbender]

@project707

I'm going to take a deeper look at this some time today hopefully and provide feedback.

[johnbender]

Sorry to respond so slowly guys, but upon review here I'm leaning towards just providing a call back with a default. In this case you can customize your search behavior as you like. I'm going to consult with the others during tomorrow's meeting with my recommendation looking like the following.

johnbender/jquery-mobile@jquery:master...johnbender:listview-filter-callback

The main reason is that we aren't a text search library and there are an enormous number of use cases when it comes to searching content, be it in the listview or otherwise.

Whether we merge the request here or not, thanks for participating.

[johnbender]

I should point out that my implementation was the simplest thing that would get the tests passing and will probably change in the final implementation. If you have ideas about how a callback should operate, we'll post here with the solution for commentary.

[project707]

I think I see what the custom callback could accomplish, though I would beg to differ in that if you offer a search method (which is precisely what a filter is) it seems that it would be an important factor to natively make it more usable over time.

That said, I would actually argue that allowing for filtering based on partial terms is actually a bit more important in this respect, and it actually looks to me like this might allow one to intercept "val" and split that into sub-terms like I have in:
#2093

May I ask which tests you managed to get working using this method? I will try a few of my own later this evening.

[gseguin]

@project707 we just landed #2216 which you can leverage to suit your needs. Thank you for raising the issue and the time you spent on thinking it through.

[johnbender]

@project707, @jeffholmes

Just wanted to let you know that we really appreciate your hard work on this. The team decided that it was better to delegate this work out to you guys on your individual projects for two primary reasons:

1. As the length of the discussion here indicates, search semantics are complicated to explain and implement. The documentation will almost always fall short in complex cases for people trying to understand how it will work.
2. As we approach Beta 3 and our release candidate work, we need to start curtailing our supportable feature addition so that we can get the issue/bug list into a manageable place. In short we're prioritizing bugs at this point but we still wanted to give you the flexibility to do what you needed to do in your projects.

I hope you haven't been put off by our decision, and continue to submit pull requests as the project continues.

[project707]

Thanks guys. It looks like I would easily be able to get regex implemented on input and list item text using this - that's much appreciated. This particular problem is solved I think.

It's too bad it wouldn't also provide a hook into the portion of the code where an optional loop could be implemented per term per my other pull request, but that would be a difficult thing to implement through a simple callback. We are unfortunately at the budget event horizon on this particular project, so I think we will have to move on with the current hacked version for now.