FC-0057: implement search test cases

[Ali-Salman29]

The PR includes the test cases for the search API along with some fixes in additional files.

• Implemented test cases similar to those in the cs_comments_service/spec/api/search_spec.rb.
• Added MongoDB indexes.
• Fixed issues in the search API view.
• By default, the test cases use fixtures to get dummy data. To test with the actual Elasticsearch, follow the guidelines mentioned in test_search.py.
• Added relevant settings i.e. FORUM_DEFAULT_PAGE and FORUM_DEFAULT_PER_PAGE.

close #53

[Ali-Salman29]

@regisb To reproduce this error, first start the Elasticsearch container. Here, you can see that the port is different from the one we are using in the edx.

docker run --rm --name elasticsearch_container -p 5200:9200 -p 5300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" elasticsearch:7.17.13

To enable the use of elastic search instead of fixtures, update the value of FORUM_ENABLE_ELASTIC_SEARCH to True in the tests/conftest.py:33.

We are focused on fixing the issue mentioned in the forum/search/search_manager.py at line 80, and the relevant test case is check_correction in the file test_search.

To reproduce the error, comment out the temporary fix in the search_manager.py.

After doing so, the first test case will fail because Elasticsearch won't be able to find "pineapples" in the comments index.
Use the following test case to verify the error: check_correction("pinapples", "pineapples").

Use this command to run test cases for search only: pytest tests/test_views/test_search.py

[regisb]

I believe that the issue stems from the construction of the indices in v2. The only difference between v1 and v2 I managed to find was the content of the indices.

To identify the difference, I queried the content of the comments and comment_thread indices.

In v1:

$ curl http://elasticsearch:9200/comment_threads,comments/_search?pretty
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "comment_threads_20240904150622420",
        "_type" : "_doc",
        "_id" : "66d8776f8c6533121e36067b",
        "_score" : 1.0,
        "_source" : {
          "abuse_flaggers" : [ ],
          "anonymous" : false,
          "anonymous_to_peers" : false,
          "at_position_list" : [ ],
          "author_id" : "cordell.mueller_5",
          "author_username" : "cordell.mueller_5",
          "body" : "a thread about green artichokes",
          "close_reason_code" : null,
          "closed" : false,
          "closed_by_id" : null,
          "comment_count" : 0,
          "commentable_id" : "test_commentable",
          "context" : "course",
          "course_id" : "test/course/id",
          "created_at" : "2024-09-04T15:06:23Z",
          "group_id" : null,
          "historical_abuse_flaggers" : [ ],
          "last_activity_at" : "2024-09-04T15:06:23Z",
          "pinned" : null,
          "retired_username" : null,
          "thread_type" : "discussion",
          "title" : "a thread about green artichokes",
          "updated_at" : "2024-09-04T15:06:23Z",
          "visible" : true,
          "votes" : {
            "up" : [ ],
            "down" : [ ],
            "up_count" : 0,
            "down_count" : 0,
            "count" : 0,
            "point" : 0
          }
        }
      },
      {
        "_index" : "comments_20240904150622420",
        "_type" : "_doc",
        "_id" : "66d8776f8c6533121e36067d",
        "_score" : 1.0,
        "_source" : {
          "abuse_flaggers" : [ ],
          "anonymous" : false,
          "anonymous_to_peers" : false,
          "at_position_list" : [ ],
          "author_id" : "cordell.mueller_5",
          "author_username" : "cordell.mueller_5",
          "body" : "a comment about greed pineapples",
          "child_count" : null,
          "comment_thread_id" : "66d8776f8c6533121e36067b",
          "commentable_id" : "test_commentable",
          "course_id" : "test/course/id",
          "created_at" : "2024-09-04T15:06:23Z",
          "depth" : 0,
          "endorsed" : false,
          "endorsement" : null,
          "historical_abuse_flaggers" : [ ],
          "parent_id" : null,
          "parent_ids" : [ ],
          "retired_username" : null,
          "sk" : "66d8776f8c6533121e36067d",
          "updated_at" : "2024-09-04T15:06:23Z",
          "visible" : true,
          "votes" : {
            "up" : [ ],
            "down" : [ ],
            "up_count" : 0,
            "down_count" : 0,
            "count" : 0,
            "point" : 0
          }
        }
      }
    ]
  }
}

In v2:

$ curl http://localhost:5200/comment_threads,comments/_search?pretty
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "comment_threads_20240904095228",
        "_type" : "_doc",
        "_id" : "66d8742decb2a86851511014",
        "_score" : 1.0,
        "_source" : {
          "id" : "66d8742decb2a86851511014",
          "title" : "a thread about green artichokes",
          "body" : "text",
          "created_at" : "2024-09-04T09:52:29.049000",
          "updated_at" : "2024-09-04T09:52:29.049000",
          "last_activity_at" : "2024-09-04T09:52:29.049000",
          "comment_count" : 0,
          "votes_point" : 0,
          "context" : "course",
          "course_id" : "course_id",
          "commentable_id" : "test_commentable",
          "author_id" : "1",
          "group_id" : null,
          "thread_id" : "66d8742decb2a86851511014"
        }
      },
      {
        "_index" : "comments_20240904095228",
        "_type" : "_doc",
        "_id" : "66d8742decb2a86851511015",
        "_score" : 1.0,
        "_source" : {
          "body" : "a comment about greed pineapples",
          "course_id" : "course_id",
          "comment_thread_id" : "66d8742decb2a86851511014",
          "commentable_id" : null,
          "group_id" : null,
          "context" : "course",
          "created_at" : "2024-09-04T09:52:29.067000",
          "updated_at" : "2024-09-04T09:52:29.068000",
          "title" : null
        }
      }
    ]
  }
}

The index contents are very different. In particular, I think that the field mappings should be identical between comment_threads and comments. This is something that you could investigate (I think).

[Ali-Salman29]

I implement it this way. There is a bug in the cs_comment service, which I have fixed in the V2 version. They store the exact MongoDB data to the index, but the mapping they created does not match the object. Also, in the search function, they are not utilizing any of the additional fields. We can still try this, but I don't think it will have a significant impact. However, I will give it a try.

https://github.com/openedx/cs_comments_service/blob/master/models/comment_thread.rb#L47

[Faraz32123]

LGTM!

[regisb]

nit: The max(...) part is no longer necessary here. It used to be, because the // division could result in a zero result, but this is no longer the case here with ceil.

[regisb]

Good job figuring out this one :) Some parts of this PR were hardcore 🤘.