Comprehensive datajoint-docs updates: content, structure, and configuration

.github/workflows/development.yml

@@ -14,7 +14,7 @@ jobs:
       - uses: actions/checkout@v2
       - name: Compile docs static artifacts
         run: |
-          GITHUB_TOKEN=$DJBOT_GH_TOKEN MODE=BUILD HOST_UID=$(id -u) docker compose up --exit-code-from docs --build
+          BOT_PAT=$DJBOT_GH_TOKEN MODE=BUILD HOST_UID=$(id -u) docker compose up --exit-code-from docs --build
       - name: Commit documentation changes
         run: |
           git clone https://github.com/${GITHUB_REPOSITORY}.git \

.markdownlint.yaml

@@ -1,18 +1,24 @@
 # https://github.com/DavidAnson/markdownlint
 # https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md
+MD007: false # Unordered list indentation
 MD009: false # permit trailing spaces
-MD013: 
-    line_length: "88" # Line length limits
+MD013:
+    # previously we defined line_length to 88 which is better for python
+    # but not for markdown
+    line_length:
+        - strict: false
     tables: false # disable for tables
     headings: false # disable for headings
+MD029: false # Ordered list item prefix
 MD030: false # Number of spaces after a list
+MD032: false # Lists should be surrounded by blank lines
 MD033: # HTML elements allowed
-    allowed_elements: 
+    allowed_elements:
         - "div"
         - "span"
         - "a"
         - "br"
-        - "sup"  
+        - "sup"
         - "figure"
 MD034: false # Bare URLs OK
 MD031: false # Spacing w/code blocks. Conflicts with `??? Note` and code tab styling

.vscode/extensions.json

@@ -0,0 +1,7 @@
+{
+    "recommendations": [
+        "streetsidesoftware.code-spell-checker",
+        "DavidAnson.vscode-markdownlint",
+        "stkb.rewrap"
+    ]
+}

.vscode/settings.json

@@ -1,5 +1,5 @@
 {
-    "[markdown]" : {
+    "[markdown]": {
         "editor.rulers": [88],
         "editor.formatOnPaste": true,
         "editor.formatOnSave": true,
@@ -11,7 +11,7 @@
     },
     // https://github.com/DavidAnson/markdownlint
     "editor.codeActionsOnSave": {
-        "source.fixAll.markdownlint":true
+        "source.fixAll.markdownlint": "always"
     },
-    "markdownlint.focusMode": 5, // ignore issues around the cursor
-}
+    "markdownlint.focusMode": 5 // ignore issues around the cursor
+}

Dockerfile

@@ -0,0 +1,10 @@
+FROM python:3-alpine
+
+WORKDIR /main
+COPY mkdocs.yaml mkdocs.yaml
+COPY src/ src/
+COPY pip_requirements.txt pip_requirements.txt
+
+RUN \
+    apk add --no-cache git && \
+    pip install --no-cache-dir -r /main/pip_requirements.txt

README.md

@@ -2,32 +2,91 @@
 
 This is the home for DataJoint software documentation as hosted at https://datajoint.com/docs
 
-## Test Locally
-
-To run locally, ensure you have `Docker` and `Docker Compose` installed.
-
-Then run the following:
-
-`MODE="LIVE" HOST_UID=$(id -u) docker compose up --build`
-
-Navigate to `http://localhost/` to preview the changes.
-
-This setup supports live-reloading so all that is needed is to save the markdown files
-and/or `mkdocs.yaml` file to trigger a reload.
-
-## Linters and Settings
+## VSCode Linter Extensions and Settings
 
 The following extensions were used in developing these docs, with the corresponding
 settings files:
 
-- [MarkdownLinter](https://github.com/DavidAnson/markdownlint):
+- Recommended extensions are already specified in `.vscode/extensions.json`, it will ask you to install them when you open the project if you haven't installed them.
+- settings in `.vscode/settings.json`
+- [MarkdownLinter](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint):
   - `.markdownlint.yaml` establishes settings for various
   [linter rules](https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md)
   - `.vscode/settings.json` formatting on save to fix linting
 
-- [CSpell](https://github.com/streetsidesoftware/vscode-spell-checker): `cspell.json`
+- [CSpell](https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker): `cspell.json`
 has various ignored words.
 
-- [ReWrap](https://github.com/stkb/Rewrap/): `.vscode/settings.json` allows toggling
+- [ReWrap](https://marketplace.visualstudio.com/items?itemName=stkb.rewrap): `.vscode/settings.json` allows toggling
 automated hard wrapping for files at 88 characters. This can also be keymapped to be
 performed on individual paragraphs, see documentation.
+
+## With Virtual Environment
+
+conda
+```bash
+conda create -n djdocs -y
+conda activate djdocs
+```
+venv
+```bash
+python -m venv .venv
+source .venv/bin/activate
+```
+
+Then install the required packages:
+```bash
+pip install -r pip_requirements.txt
+```
+
+Run mkdocs at: http://127.0.0.1:8000/docs/
+```bash
+# It will automatically reload the docs when changes are made
+mkdocs serve --config-file ./mkdocs.yaml
+```
+
+## With Docker
+
+> We mostly use Docker to simplify docs deployment
+
+Ensure you have `Docker` and `Docker Compose` installed.
+
+Then run the following:
+```bash
+# It will automatically reload the docs when changes are made
+MODE="LIVE" docker compose up --build
+```
+
+Navigate to http://127.0.0.1:8000/docs/ to preview the changes.
+
+This setup supports live-reloading so all that is needed is to save the markdown files
+and/or `mkdocs.yaml` file to trigger a reload.
+
+## Mkdocs Warning Explanation
+
+> TL;DR: We need to do it this way for hosting, please keep it as is.
+
+```log
+WARNING -  A reference to 'core/datajoint-python/' is included in the 'nav' configuration, which is not found
+           in the documentation files.
+INFO    -  Doc file 'index.md' contains an unrecognized relative link './core/datajoint-python/', it was left
+           as is.
+```
+
+- We use reverse proxy to proxy our docs sites, here is the proxy flow:
+  - You hit `datajoint.com/*` on your browser
+  - It'll bring you to the reverse proxy server first, that you wouldn't notice
+  - when your URL ends with:
+    - `/` is the company page
+    - `/docs/` is the landing/navigation page hosted by datajoint/datajoint-docs's github pages
+    - `/docs/core/datajoint-python/` is the actual docs site hosted by datajoint/datajoint-python's github pages
+    - `/docs/elements/element-*/` is the actual docs site hosted by each element's github pages
+
+
+```log
+WARNING -  Doc file 'partnerships/openephysgui.md' contains a link
+           '../../images/community-partnerships-openephysgui-logo.png', but the target
+           '../images/community-partnerships-openephysgui-logo.png' is not found among documentation files.
+           Did you mean '../images/community-partnerships-openephysgui-logo.png'?
+```
+- We use Github Pages to host our docs, the image references needs to follow the mkdocs's build directory structure, under `site/` directory once you run mkdocs.

docker-compose.yaml

@@ -1,29 +1,22 @@
-# MODE="LIVE|BUILD" HOST_UID=$(id -u) docker compose up --build
-# 
-# Navigate to http://localhost/
-version: "2.4"
+# MODE="LIVE|BUILD" docker compose up --build
+#
+# Navigate to http://localhost/docs/
 services:
   docs:
-    build:
-      dockerfile: .docker/Dockerfile
-      context: .
-      args:
-        - GITHUB_TOKEN
     image: datajoint/datajoint-docs
     environment:
       - MODE
     volumes:
       - .:/main
-    user: ${HOST_UID}:anaconda
     ports:
-      - 80:80
+      - 8000:8000
     command:
       - sh
       - -c
       - |
         set -e
         if echo "$${MODE}" | grep -i live &>/dev/null; then
-            mkdocs serve --config-file ./mkdocs.yaml -a 0.0.0.0:80
+            mkdocs serve --config-file ./mkdocs.yaml -a 0.0.0.0:8000
         elif echo "$${MODE}" | grep -i build &>/dev/null; then
             mkdocs build --config-file ./mkdocs.yaml
         else

mkdocs.yaml

@@ -6,6 +6,8 @@ repo_name: datajoint/datajoint-docs
 repo_url: https://github.com/datajoint/datajoint-docs
 nav:
   - Welcome: index.md
+  # relative site url, not pointing to any docs in the repo
+  # it's for reverse proxy to proxy datajoint-python docs
   - DataJoint Python: core/datajoint-python/
   - DataJoint Elements:
       - elements/index.md
@@ -35,6 +37,7 @@ nav:
       - Open Ephys GUI: partnerships/openephysgui.md
       - Suite2p: partnerships/suite2p.md
   - About:
+      - About: about/about.md
       - History: about/history.md
       - Team: about/datajoint-team.md
       - Citation Guidelines: about/citation.md
@@ -69,9 +72,10 @@ theme:
 plugins:
   - search
   - section-index
-  - redirects:
-      redirect_maps:
-        "index.md": "welcome.md"
+  # There is no welcome.md anymore
+  # - redirects:
+  #     redirect_maps:
+  #       "index.md": "welcome.md"
   - exclude:
       glob:
         - archive/*
@@ -85,8 +89,8 @@ markdown_extensions:
       options:
         custom_icons:
           - .overrides/.icons
-      emoji_index: !!python/name:materialx.emoji.twemoji
-      emoji_generator: !!python/name:materialx.emoji.to_svg
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
   - mdx_truly_sane_lists
   - pymdownx.superfences:
       custom_fences:
@@ -103,12 +107,16 @@ markdown_extensions:
       custom_checkbox: true
 extra:
   generator: false # Disable watermark
-  version:
-    provider: mike
+  # There is no version for this doc
+  # version:
+  #   provider: mike
   social:
     - icon: main/company-logo
       link: https://www.datajoint.com
       name: DataJoint
+    - icon: main/company-logo
+      link: https://datajoint.com/docs/
+      name: DataJoint Documentation      
     - icon: fontawesome/brands/slack
       link: https://datajoint.slack.com
       name: Slack
@@ -131,7 +139,7 @@ extra:
       link: https://stackoverflow.com/questions/tagged/datajoint
       name: StackOverflow
     - icon: fontawesome/brands/youtube
-      link: https://www.youtube.com/channel/UCdeCuFOTCXlVMRzh6Wk-lGg
+      link: https://www.youtube.com/@datajoint
       name: YouTube
 extra_css:
   - assets/stylesheets/extra.css

src/about/about.md

@@ -0,0 +1,11 @@
+DataJoint Elements, DataJoint documentation, and DataJoint Web GUIs and APIs are supported by the NIH grant [**NIH U24 NS116470**](https://reporter.nih.gov/project-details/10547509) for disseminating open-source software for neuroscience research.
+
+The goal is to systematize and disseminate data pipeline designs from leading neuroscience projects using the DataJoint framework.
+
+### Aim 1: DataJoint Pipelines for Neurophysiology
+
+Extract and systematize essential design motifs from published DataJoint-based projects as a collection of simple modules.
+
+### Aim 2: Access and Training Resources
+
+Support a dedicated resource for accessing and using DataJoint Pipelines for Neurophysiology.