Allow regex as search arguments

[alexandernst]

MongoDB is able to search regex directly, but currently there is no way (or I couldn't find one) to make Yii2 to actually tell MongoDB to use a regex instead of a String.

Example: MyModel::findAll(["path" => "/^1/2/3//"]);

I'd expect, somehow, to make Yii2 to tell MongoDB that I'm actually searching the /^1/2/3// regex, and not the "/^1/2/3//" string.

[klimov-paul]

yiisoft/yii2#2337

[alexandernst]

@klimov-paul I think this is not a proper fix as you can't always use $regex see docs.

On the other hand, I just tried with

$current_path = "/70000/";
$query = new Query;
$query->select(['short_id', 'path'])->from('category')->where(["REGEX", "path", "/^$current_path/"]); // Serch all paths that starts with "/70000/"
$subcategories = $query->all();

And the query that was ran in MongoDB is:

find({
    "ns":"tests.category",
    "limit":0,"batchSize":0,"skip":0,"flags":0,
    "query":{
        "path":{
            "regex":"^/70000/","flags":""
        }
    },
    "fields":{"short_id":true,"path":true},
    "started_iterating":false}
)

Which doesn't seem to be valid. I can't find anywhere in MongoDB's docs the regex operator. I find only the $regex operator.

I tried running that same query from mongo-client (debian 8) and I got no results:

> db.category.find({ "path": { "regex": "^/70000/", "flags": "" } })
> db.category.find({ "path": { $regex: "^/70000/" } })
{ "_id" : ObjectId("554c80d127e11b99338b456e"), "short_id" : NumberLong(80000), "path" : "/70000/", "name" : { "en-US" : "Category 8 (80000)" }, "slug" : { "en-US" : "category-8" } }
{ "_id" : ObjectId("554d3b2a27e11b2e678b4568"), "short_id" : "c72674", "path" : "/70000/", "name" : { "en-US" : "Sf SWrf s W SWff" } }
{ "_id" : ObjectId("554d1c3827e11b2e678b4567"), "short_id" : "333fe6", "path" : "/70000/80000/", "name" : { "en-US" : "New category" } }

Am I doing something wrong?

[alexandernst]

Can I get some attention, please? I think this is a genuine issue that should be addressed somehow (and get this issue reopened). (or if it's not, I'd like to hear where am I failing) @klimov-paul

[klimov-paul]

There is a Official Yii forum for question asking.

[alexandernst]

@klimov-paul I'm not asking for coding help. I'm trying to report a bug and you're constantly misreading all my issues and closing them. I already explained to you that the fix you pointed me to is not a proper fix as there are situations in which MongoDB requires an actual regex expression instead of a $regex operator. Please re-read my last comment and most importantly, please read what official MongoDB docs say about the $regex operator vs regex strings.

I'll sum that up:

You cannot use $regex operator expressions inside an $in.

[klimov-paul]

Regular expression usage is covered by unit tests. Its correct function confirmed at yiisoft/yii2#2337. What bug are you reporting here? Or you consider yourself the first person ever using this feature?

[bagrat]

@klimov-paul I will agree with @alexandernst. What if I want to query a collection using simple key that has the name regex?

[alexandernst]

@klimov-paul Ok, I'm not really sure what MongoDB version are you testing against, but I just ran the unit tests against MongoDB 3.0.x and it's failing.

There was 1 error:

1) yiiunit\extensions\mongodb\CollectionTest::testFullTextSearch
yii\mongodb\Exception: no such command: text

/tmp/test/yii2-mongodb/Collection.php:726
/tmp/test/yii2-mongodb/Collection.php:720
/tmp/test/yii2-mongodb/tests/CollectionTest.php:418
/usr/bin/phpunit:42

Caused by
yii\mongodb\Exception: no such command: text

/tmp/test/yii2-mongodb/Collection.php:751
/tmp/test/yii2-mongodb/Collection.php:720
/tmp/test/yii2-mongodb/tests/CollectionTest.php:418
/usr/bin/phpunit:42

--

There were 3 failures:

1) yiiunit\extensions\mongodb\ActiveRecordTest::testFind
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 1
-    1 => 2
-    2 => 3
-    3 => 4
-    4 => 5
-    5 => 6
-    6 => 7
-    7 => 8
-    8 => 9
-    9 => 10
 )

/tmp/test/yii2-mongodb/tests/ActiveRecordTest.php:97
/usr/bin/phpunit:42

2) yiiunit\extensions\mongodb\CollectionTest::testDistinct
Failed asserting that true is false.

/tmp/test/yii2-mongodb/tests/CollectionTest.php:491
/usr/bin/phpunit:42

3) yiiunit\extensions\mongodb\file\ActiveRecordTest::testFind
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 1
-    1 => 2
-    2 => 3
-    3 => 4
-    4 => 5
-    5 => 6
-    6 => 7
-    7 => 8
-    8 => 9
-    9 => 10
 )

/tmp/test/yii2-mongodb/tests/file/ActiveRecordTest.php:113
/usr/bin/phpunit:42

--

There was 1 skipped test:

1) yiiunit\extensions\mongodb\ActiveRecordTest::testQueryByIntegerField
This test depends on "yiiunit\extensions\mongodb\ActiveRecordTest::testFind" to pass.

FAILURES!                                                       
Tests: 114, Assertions: 356, Failures: 3, Errors: 1, Skipped: 1.

So, back to the issue. I'll try to explain it again. Please read carefully what I'll write because I already wrote the same thing twice.

There are 2 issues with the regex search.

The first one is that the code is using (or generating) regex keyword/operator to make the query, but that keyword/argument doesn't exist. My test is very simple. A simple controller with an action in it. The action looks like this:

//Look for paths starting with /30000
$query = new Query;
$query->select(['short_id', 'path'])->from('category')->where(["REGEX", "path", "/^/30000/"]);
$subcategories = $query->all();

Now, this query generates this find command:

find({
    "ns":"tests.category",
    "limit":0,"batchSize":0,"skip":0,"flags":0,
    "query":{
    "path":{
        "regex":"^/30000/","flags":"" // <--- here is the issue, "regex" op doesn't exist
        }
    },
    "fields":{"short_id":true,"path":true},
    "started_iterating":false}
)

You see? That regex keyword/operator just doesn't exist (not in MongoDB 3.x at least) and the query isn't returning anything. Note that $regex however does exists.

The second issue is that you're failing to provide a method to use an actual regex pattern instead of the $regex operator. This is required because the $regex operator can't be used inside $in, which I already said in my previous comments. I also already linked you to the exact fragment of the documentation of MongoDB that clearly says that there are certain cases where $regex can't be used and a real regex pattern is required.

[klimov-paul]

I'm not really sure what MongoDB version are you testing against, but I just ran the unit tests against MongoDB 3.0.x and it's failing.

Unit tests are passed via Travis successfully:
https://travis-ci.org/yiisoft/yii2-mongodb
They fails for you because you have no 'full text search' support enabled at your MongoDb server.

Now, this query generates this find command: ...

If you ever bother to actually read the docs, you may find this topic: https://github.com/yiisoft/yii2-mongodb/blob/master/docs/guide/basic-usage.md
which contains following text:

This extension supports logging and profiling, however log messages does not contain actual text of the performed queries, they contains only a “close approximation” of it composed on the values which can be extracted from PHP Mongo extension classes. If you need to see actual query text, you should use specific tools for that.

What if I want to query a collection using simple key that has the name regex?

You are absolutely free to do so.

[alexandernst]

@klimov-paul Ok, I installed mongotail (which monitors all queries) and I must say that, indeed, you're right about the first issue. Yii is sending a $regex operator, and not regex as I though. Sorry for the confusion.

Anyways, I insist that the second issue is real. There is no way to use $regex inside an $in.

Consider this: If I wanted to query all documents that start with 300 or 500, in a MongoDB shell I'd do it like this:

db.tests.find({ "path": $in: [ /^300/, /^500/ ] });

But there is no way to do this with Yii, as $regex can't be used inside $in.

[klimov-paul]

What does prevent you from write following condition:

[
    'path' => ['$in' => [new \MongoRegex('/^300/'), new \MongoRegex('/^500/')]]
]

?

[alexandernst]

@klimov-paul Oh! I wasn't aware that Yii would understand (and play well) with plain PHP-MongoDB classes. That is why I didn't use MongoRegex in the first place. I'm sorry for wasting your time.