Test for search functionality

.travis.yml

@@ -3,21 +3,23 @@ python:
   - 2.7
   - 3.6
 sudo: false
+env:
+  - ES_VERSION=1.3.9 ES_DOWNLOAD_URL=https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
 matrix:
   include:
     - python: 2.7
       env: TOXENV=docs
     - python: 2.7
       env: TOXENV=lint
+      script: tox
     - python: 2.7
       env: TOXENV=eslint
-script:
-  - tox
 cache:
   directories:
     - ~/.cache/pip
     - ~/.nvm/nvm.sh
 install:
+  - ./scripts/travis/install_elasticsearch.sh
   - pip install tox-travis
   - curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash
   - source ~/.nvm/nvm.sh
@@ -27,7 +29,7 @@ install:
   - npm install
   - bower install
 script:
-  - tox
+  - ./scripts/travis/run_tests.sh
 notifications:
   slack:
     rooms:

conftest.py

@@ -0,0 +1,9 @@
+def pytest_addoption(parser):
+    parser.addoption('--including-search', action='store_true', dest="searchtests",
+                     default=False, help="enable search tests")
+
+
+def pytest_configure(config):
+    if not config.option.searchtests:
+        # Include `not search` to parameters so that search test do not perform
+        setattr(config.option, 'markexpr', 'not search')

readthedocs/search/indexes.py

@@ -107,6 +107,11 @@ def create_index(self, index=None):
         }
         self.es.indices.create(index=index, body=body)
 
+    def refresh_index(self, index=None):
+        index = index or self._index
+
+        self.es.indices.refresh(index=index)
+
     def put_mapping(self, index=None):
         index = index or self._index
         self.es.indices.put_mapping(self._type, self.get_mapping(), index)
@@ -155,6 +160,10 @@ def index_document(self, data, index=None, parent=None, routing=None):
             kwargs['routing'] = routing
         self.es.index(**kwargs)
 
+    def delete_index(self, index_name):
+
+        self.es.indices.delete(index=index_name)
+
     def delete_document(self, body, index=None, parent=None, routing=None):
         kwargs = {
             'index': index or self._index,

readthedocs/search/tests/conftest.py

@@ -0,0 +1,59 @@
+import pytest
+from django_dynamic_fixture import G
+from faker import Faker
+
+from readthedocs.projects.models import Project
+from readthedocs.search.indexes import Index, ProjectIndex, PageIndex, SectionIndex
+from .dummy_data import DUMMY_PAGE_JSON, ALL_PROJECTS
+
+fake = Faker()
+
+
+@pytest.fixture(autouse=True)
+def mock_elastic_index(mocker):
+    mocker.patch.object(Index, '_index', fake.word().lower())
+
+
+@pytest.fixture(autouse=True)
+def search():
+    # Create the index.
+    index = Index()
+    index_name = index.timestamped_index()
+    index.create_index(index_name)
+    index.update_aliases(index_name)
+    # Update mapping
+    proj = ProjectIndex()
+    proj.put_mapping()
+    page = PageIndex()
+    page.put_mapping()
+    sec = SectionIndex()
+    sec.put_mapping()
+
+    yield index
+    index.delete_index(index_name=index_name)
+
+
+@pytest.fixture
+def all_projects():
+    return [G(Project, slug=project_name, name=project_name) for project_name in ALL_PROJECTS]
+
+
+@pytest.fixture
+def project(all_projects):
+    # Return a single project
+    return all_projects[0]
+
+
+def get_dummy_page_json(version, *args, **kwargs):
+    dummy_page_json = DUMMY_PAGE_JSON
+    project_name = version.project.name
+    return dummy_page_json.get(project_name)
+
+
+@pytest.fixture
+def mock_parse_json(mocker):
+
+    # patch the function from `projects.tasks` because it has been point to there
+    # http://www.voidspace.org.uk/python/mock/patch.html#where-to-patch
+    mocked_function = mocker.patch('readthedocs.projects.tasks.process_all_json_files')
+    mocked_function.side_effect = get_dummy_page_json

readthedocs/search/tests/data/kuma/docker.json

@@ -0,0 +1,25 @@
+{
+  "content": "Docker\nDocker is used for development and (soon) for deployment.\nDocker Images\nDocker images are used in development, usually with the local working files mounted in the images to set behaviour.\nImages are built by Jenkins, after tests pass, and are published to quay.io. We try to store the configuration in the environment, so that the published images can be used in deployments by setting environment variables to deployment-specific values.\nHere are some of the images used in the Kuma project:\nkuma\nThe kuma Docker image builds on the kuma_base image, installing a kuma branch and building the assets needed for running as a webservice. The environment can be customized for different deployments.\nThe image can be recreated locally with make build-kuma.\nThe image tagged latest is used by default for development. It can be created locally with make build-kuma VERSION=latest. The official latest image is created from the master branch in Jenkins and published to quay.io.\nkuma_base\nThe kuma_base Docker image contains the OS and libraries (C, Python, and Node.js) that support the kuma project. The kuma image extends this by installing the kuma source and building assets needed for production.\nThe image can be recreated locally with make build-base.\nThe image tagged latest is used by default for development. It can be created localled with make build-base VERSION=latest. The official latest image is created from the master branch in Jenkins and published to quay.io\nkumascript\nThe kumascript Docker image contains the kumascript rendering engine and support files. The environment can be customized for different deployments.\nThe image can be recreated locally with make build-kumascript.\nThe image tagged latest is used by default for development. It can be created locally with make build-kumascript KS_VERSION=latest. The official latest image is created from the master branch in Jenkins and published to quay.io.\nintegration-tests\nThe integration-tests Docker image contains browser-based integration tests that check the functionality of a running Kuma deployment.\nThe image can be recreated locally with docker build -f docker/images/integration-tests/ ., but this is only necessary for image development. Most developer will follow the Client-side testing documentation to develop and run these integration tests.\nThe image is built and used in Jenkins in the stage-integration-tests and prod-integration-tests pipelines, configured by scripts in the Jenkinsfiles folder. It is not published to quay.io.",
+  "headers": [
+    "Docker",
+    "Docker Images",
+    "kuma",
+    "kuma_base",
+    "kumascript",
+    "integration-tests"
+  ],
+  "title": "Docker",
+  "sections": [
+    {
+      "content": "\n<a class=\"reference external\" href=\"https://www.docker.com\">Docker</a> is used for development and (soon) for deployment.\n",
+      "id": "docker",
+      "title": "Docker"
+    },
+    {
+      "content": "\n<h2>Docker Images<a class=\"headerlink\" href=\"#docker-images\" title=\"Permalink to this headline\">\u00b6</a></h2>\n<p>Docker images are used in development, usually with the local\nworking files mounted in the images to set behaviour.</p>\n<p>Images are built by <a class=\"reference external\" href=\"https://ci.us-west.moz.works\">Jenkins</a>, after tests pass, and are\npublished to <a class=\"reference external\" href=\"https://quay.io/repository/mozmar/kuma\">quay.io</a>.  We try to\n<a class=\"reference external\" href=\"https://12factor.net/config\">store the configuration in the environment</a>, so that the\npublished images can be used in deployments by setting\nenvironment variables to deployment-specific values.</p>\n<p>Here are some of the images used in the Kuma project:</p>\n<div class=\"section\" id=\"kuma\">\n<h3>kuma<a class=\"headerlink\" href=\"#kuma\" title=\"Permalink to this headline\">\u00b6</a></h3>\n<p>The kuma Docker image builds on the kuma_base image, installing a kuma branch\nand building the assets needed for running as a webservice.  The environment\ncan be customized for different deployments.</p>\n<p>The image can be recreated locally with <code class=\"docutils literal notranslate\"><span class=\"pre\">make</span> <span class=\"pre\">build-kuma</span></code>.</p>\n<p>The image tagged <code class=\"docutils literal notranslate\"><span class=\"pre\">latest</span></code> is used by default for development. It can be\ncreated locally with <code class=\"docutils literal notranslate\"><span class=\"pre\">make</span> <span class=\"pre\">build-kuma</span> <span class=\"pre\">VERSION=latest</span></code>. The official latest\nimage is created from the master branch in <a class=\"reference external\" href=\"https://ci.us-west.moz.works/blue/organizations/jenkins/mdn_multibranch_pipeline/branches/\">Jenkins</a> and published to\n<a class=\"reference external\" href=\"https://quay.io/repository/mozmar/kuma_base\">quay.io</a>.</p>\n</div>\n<div class=\"section\" id=\"kuma-base\">\n<h3>kuma_base<a class=\"headerlink\" href=\"#kuma-base\" title=\"Permalink to this headline\">\u00b6</a></h3>\n<p>The kuma_base Docker image contains the OS and libraries (C, Python, and\nNode.js) that support the kuma project.  The kuma image extends this by\ninstalling the kuma source and building assets needed for production.</p>\n<p>The image can be recreated locally with <code class=\"docutils literal notranslate\"><span class=\"pre\">make</span> <span class=\"pre\">build-base</span></code>.</p>\n<p>The image tagged <code class=\"docutils literal notranslate\"><span class=\"pre\">latest</span></code> is used by default for development. It can be\ncreated localled with <code class=\"docutils literal notranslate\"><span class=\"pre\">make</span> <span class=\"pre\">build-base</span> <span class=\"pre\">VERSION=latest</span></code>. The official\nlatest image is created from the master branch in <a class=\"reference external\" href=\"https://ci.us-west.moz.works/blue/organizations/jenkins/mdn_multibranch_pipeline/branches/\">Jenkins</a> and published to\n<a class=\"reference external\" href=\"https://quay.io/repository/mozmar/kuma_base\">quay.io</a></p>\n</div>\n<div class=\"section\" id=\"kumascript\">\n<h3>kumascript<a class=\"headerlink\" href=\"#kumascript\" title=\"Permalink to this headline\">\u00b6</a></h3>\n<p>The kumascript Docker image contains the kumascript rendering engine and\nsupport files.  The environment can be customized for different deployments.</p>\n<p>The image can be recreated locally with <code class=\"docutils literal notranslate\"><span class=\"pre\">make</span> <span class=\"pre\">build-kumascript</span></code>.</p>\n<p>The image tagged <code class=\"docutils literal notranslate\"><span class=\"pre\">latest</span></code> is used by default for development. It can be\ncreated locally with <code class=\"docutils literal notranslate\"><span class=\"pre\">make</span> <span class=\"pre\">build-kumascript</span> <span class=\"pre\">KS_VERSION=latest</span></code>. The official\nlatest image is created from the master branch in <a class=\"reference external\" href=\"https://ci.us-west.moz.works/blue/organizations/jenkins/mdn_multibranch_pipeline/branches/\">Jenkins</a> and published to\n<a class=\"reference external\" href=\"https://quay.io/repository/mozmar/kumascript\">quay.io</a>.</p>\n</div>\n<div class=\"section\" id=\"integration-tests\">\n<h3>integration-tests<a class=\"headerlink\" href=\"#integration-tests\" title=\"Permalink to this headline\">\u00b6</a></h3>\n<p>The integration-tests Docker image contains browser-based integration tests\nthat check the functionality of a running Kuma deployment.</p>\n<p>The image can be recreated locally with\n<code class=\"docutils literal notranslate\"><span class=\"pre\">docker</span> <span class=\"pre\">build</span> <span class=\"pre\">-f</span> <span class=\"pre\">docker/images/integration-tests/</span> <span class=\"pre\">.</span></code>, but this is only\nnecessary for image development. Most developer will follow the\n<a class=\"reference external\" href=\"https://kuma.readthedocs.io/en/latest/tests-ui.html\">Client-side testing documentation</a> to develop and run these integration tests.</p>\n<p>The image is built and used in <a class=\"reference external\" href=\"https://ci.us-west.moz.works/blue/organizations/jenkins/mdn_multibranch_pipeline/activity\">Jenkins</a> in the <code class=\"docutils literal notranslate\"><span class=\"pre\">stage-integration-tests</span></code> and\n<code class=\"docutils literal notranslate\"><span class=\"pre\">prod-integration-tests</span></code> pipelines, configured by scripts in the\n<code class=\"docutils literal notranslate\"><span class=\"pre\">Jenkinsfiles</span></code> folder.  It is not published to quay.io.</p>\n</div>\n",
+      "id": "docker-images",
+      "title": "Docker Images"
+    }
+  ],
+  "path": "docker"
+}

readthedocs/search/tests/data/kuma/documentation.json

@@ -0,0 +1,22 @@
+{
+  "content": "Documentation\nThis documentation is generated and published at Read the Docs whenever the master branch is updated.\nGitHub can render our .rst documents as ReStructuredText, which is close enough to Sphinx for most code reviews, without features like links between documents.\nIt is occasionally necessary to generate the documentation locally. It is easiest to do this with a virtualenv on the host system, using Docker only to regenerate the MDN Sphinx template. If you are not comfortable with that style of development, it can be done entirely in Docker using docker-compose.\nGenerating documentation\nSphinx uses a Makefile in the docs subfolder to build documentation in several formats. MDN only uses the HTML format, and the generated document index is at docs/_build/html/index.html.\nTo generate the documentation in a virtualenv on the host machine, first install the requirements:\npip install -r requirements/docs.txt\nThen switch to the docs folder to use the Makefile:\ncd docs make html python -m webbrowser file://${PWD}/_build/html/index.html\nTo generate the documentation with Docker:\ndocker-compose run --rm --user $(id -u) web sh -c \"\\ virtualenv /tmp/.venvs/docs && \\ . /tmp/.venvs/docs/bin/activate && \\ pip install -r /app/requirements/docs.txt && \\ cd /app/docs && \\ make html\" python -m webbrowser file://${PWD}/docs/_build/html/index.html\nA virtualenv is required, to avoid a pip bug when changing the version of a system-installed package.",
+  "headers": [
+    "Documentation",
+    "Generating documentation",
+    "Installation"
+  ],
+  "title": "Documentation",
+  "sections": [
+    {
+      "content": "\nThis documentation is generated and published at\n<a class=\"reference external\" href=\"https://kuma.readthedocs.io/en/latest/\">Read the Docs</a> whenever the master branch is updated.\n\nGitHub can render our <code class=\"docutils literal notranslate\"><span class=\"pre\">.rst</span></code> documents as <a class=\"reference external\" href=\"https://en.wikipedia.org/wiki/ReStructuredText\">ReStructuredText</a>, which is\nclose enough to <a class=\"reference external\" href=\"https://en.wikipedia.org/wiki/Sphinx_(documentation_generator)\">Sphinx</a> for most code reviews, without features like links\nbetween documents.\n\nIt is occasionally necessary to generate the documentation locally. It is\neasiest to do this with a <a class=\"reference external\" href=\"https://virtualenv.pypa.io/en/stable/\">virtualenv</a> on the host system, using Docker only to\nregenerate the MDN Sphinx template.  If you are not comfortable with that style\nof development, it can be done entirely in Docker using <code class=\"docutils literal notranslate\"><span class=\"pre\">docker-compose</span></code>.\n",
+      "id": "documentation",
+      "title": "Documentation"
+    },
+    {
+      "content": "\n<h2>Generating documentation<a class=\"headerlink\" href=\"#generating-documentation\" title=\"Permalink to this headline\">\u00b6</a></h2>\n<p>Sphinx uses a <code class=\"docutils literal notranslate\"><span class=\"pre\">Makefile</span></code> in the <code class=\"docutils literal notranslate\"><span class=\"pre\">docs</span></code> subfolder to build documentation in\nseveral formats.  MDN only uses the HTML format, and the generated document\nindex is at <code class=\"docutils literal notranslate\"><span class=\"pre\">docs/_build/html/index.html</span></code>.</p>\n<p>To generate the documentation in a virtualenv on the host machine, first\ninstall the requirements:</p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span/><span class=\"n\">pip</span> <span class=\"n\">install</span> <span class=\"o\">-</span><span class=\"n\">r</span> <span class=\"n\">requirements</span><span class=\"o\">/</span><span class=\"n\">docs</span><span class=\"o\">.</span><span class=\"n\">txt</span>\n</pre></div>\n</div>\n<p>Then switch to the <code class=\"docutils literal notranslate\"><span class=\"pre\">docs</span></code> folder to use the <code class=\"docutils literal notranslate\"><span class=\"pre\">Makefile</span></code>:</p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span/>cd docs\nmake html\npython -m webbrowser file://${PWD}/_build/html/index.html\n</pre></div>\n</div>\n<p>To generate the documentation with Docker:</p>\n<div class=\"highlight-default notranslate\"><div class=\"highlight\"><pre><span/>docker-compose run --rm --user $(id -u) web sh -c \"\\\n  virtualenv /tmp/.venvs/docs &amp;&amp; \\\n  . /tmp/.venvs/docs/bin/activate &amp;&amp; \\\n  pip install -r /app/requirements/docs.txt &amp;&amp; \\\n  cd /app/docs &amp;&amp; \\\n  make html\"\npython -m webbrowser file://${PWD}/docs/_build/html/index.html\n</pre></div>\n</div>\n<p>A <code class=\"docutils literal notranslate\"><span class=\"pre\">virtualenv</span></code> is required, to avoid a <code class=\"docutils literal notranslate\"><span class=\"pre\">pip</span></code> bug when changing the version\nof a system-installed package.</p>\n",
+      "id": "generating-documentation",
+      "title": "Generating documentation"
+    }
+  ],
+  "path": "documentation"
+}