Merge pull request #2833 from centerofci/dev_docs_organize

Clean up dev docs

CONTRIBUTING.md

@@ -1 +1,73 @@
-Please see our [Contributing to Mathesar](https://wiki.mathesar.org/en/community/contributing) wiki page.
+# Contributor Guide
+
+Mathesar's development happens on [GitHub](https://github.com/centerofci/mathesar). We welcome contributions of all kinds!
+
+## Joining the Community
+
+We highly recommend joining our [Matrix community](https://wiki.mathesar.org/en/community/matrix) and our [developer mailing list](https://wiki.mathesar.org/en/community/mailing-lists) before making contributions. This is where most of the core team's conversations about building Mathesar happen.
+
+## Contributing code
+
+1. **Get Mathesar [running locally](./DEVELOPER_GUIDE.md#local-development-setup).**
+
+    Make sure to **do this before moving on**. If you need help, ask in [Matrix](https://wiki.mathesar.org/en/community/matrix), taking care to form *specific* questions that people can answer asynchronously.
+
+1. **Find an [issue](https://github.com/centerofci/mathesar/issues) to work on.**
+
+    - ✅ Our easiest issues are labeled [good first issue](https://github.com/centerofci/mathesar/issues?q=is%3Aopen+is%3Aissue+no%3Aassignee+label%3A%22good+first+issue%22) and are a great place to start. However keep in mind that we're not always entirely sure of the necessary steps to solve a problem when we open an issue. 
+    - ✅ Slightly more challenging issues are still labeled [help wanted](https://github.com/centerofci/mathesar/issues?q=is%3Aopen+is%3Aissue+no%3Aassignee+label%3A%22help+wanted%22). These can be a good place to start if you have some experience coding but are not yet familiar with our codebase.
+    - ❌ Issues are not appropriate if they meet any of the following criteria:
+        - already assigned to someone
+        - labeled with a `restricted: ...` label
+        - labeled with any `status: ...` label other than `status: ready`
+    - ⚠️ Some issues fall into a middle ground, not being labeled "help wanted" or "restricted". These tickets are more challenging and are only appropriate for community contributors who are familiar with our codebase.
+
+    If you want to work on something for which there is no GitHub issue open yet, create an issue and propose your change there. A Mathesar [team member](https://wiki.mathesar.org/en/team) will evaluate your issue and decide whether we'll accept a pull request for the issue.
+
+1. ***(Optionally)* Claim the issue.**
+
+    If the Mathesar team has already merged one of your PRs, then you may wish to ear-mark the issue for yourself so that other do not work on it concurrently.
+
+    However, **if you are brand new to Mathesar, we recommend skipping this step** and making your changes *without* claiming the issue. (A core team member will assign the issue to you *after* you open a PR.) This recommendation is designed to reduce the overhead the core team has experienced from a high volume of contributors failing to follow through with their intent to submit PRs.
+
+    If you decide to claim the issue:
+
+    1. Comment on the ticket, saying *"I'd like to work on this"* or similar. If you're relatively new to the community, mentioning your PR(s) that we have already merged would be helpful.
+    1. A core team member will assign you to the ticket.
+    1. At this point **you have one week to follow up.** If we don't hear from you by then, we will unassign you from the ticket so that others may claim it. If you need more time, you can ask for an extension, explaining the progress you've made and the challenges you've encountered. If you have not begun work at all, then we will need to unassign you.
+
+    Please do not claim more than 2 issues concurrently before submitting PRs.
+
+1. **Begin making your changes.**
+
+    - Refer to our **[Developer Guide](./DEVELOPER_GUIDE.md)** for questions about the code.
+    - Make sure to follow our [front end code standards](./mathesar_ui/STANDARDS.md) and [API standards](./mathesar/api/STANDARDS.md) where applicable.
+    - If you are not familiar with GitHub or pull requests, please follow [GitHub's "Hello World" guide](https://guides.github.com/activities/hello-world/) first. Make sure to commit your changes on a new git branch named after the ticket you claimed. Base that new branch on our `develop` branch.
+    - Commit early, commit often. Write good commit messages. Try to keep pull requests small if possible, since it makes review easier.
+    - If you expect your work to last longer than 1 week, open a draft pull request for your in-progress work.
+
+1. **Open a PR.**
+
+    When you are ready for a core team member to review your changes, open a pull request (or mark your draft PR as ready for review). If you have already been corresponding with a core team member about the issue, then you may request a review from that person. Otherwise, you may leave your PR without any review requests and a core team member will assign someone to review it.
+
+1. **Address critique from PR review.**
+
+    - When making changes to address review critique, feel free to reply to threads within the PR (especially to point to specific commits which you think should address the critique), but do not click the "Resolve conversation" button on threads which other people have started.
+    - If you are ready for a subsequent round of review, comment on the PR requesting another review and tagging the original reviewer.
+
+## Contributing PR reviews
+
+We encourage and appreciate code review by contributors. Feel free to review any open pull requests. Follow our [code review guidelines](https://wiki.mathesar.org/en/engineering/code-review).
+
+## Contributing documentation
+
+- Help improve our user and administrator documentation (published to https://docs.mathesar.org/) by editing the markdown files in the `/docs` directory of this repo. See our [Documentation guide](./docs/README.md) for more info.
+
+- Developer-focused documentation lives in other markdown files throughout this repo, and we welcome PRs to improve this content. All PRs should target the `develop` branch.
+
+## Contributing to UX and graphic design
+
+Due to limited capacity, we are currently unable to accept design volunteers. Please return to this page for updates.
+
+Please read through our [Design](https://wiki.mathesar.org/en/design) section to learn more about our design process.
+

DEVELOPER_GUIDE.md

@@ -0,0 +1,187 @@
+# Developer Guide
+
+## Stack
+
+Mathesar is built using:
+
+- [PostgreSQL](https://www.postgresql.org/) for the data storage
+- [Python](https://www.python.org/) for the backend
+- [Django](https://www.djangoproject.com/) for the web application
+- [SQLAlchemy](https://www.sqlalchemy.org/) to talk to the database
+- [Django REST Framework](https://www.django-rest-framework.org/) for the API 
+- [Svelte](https://svelte.dev/) and [TypeScript](https://www.typescriptlang.org/) for the frontend
+
+## Local development setup
+
+> **Note:** If you are developing on Windows, first install [WSL](https://learn.microsoft.com/en-us/windows/wsl/install). Then do all of your development work from within the WSL shell, including running commands like `git clone` and `docker compose`.
+
+1. Ensure ensure that you have [Docker](https://docs.docker.com/get-docker/) installed.
+
+1. Clone the repository and `cd` into it.
+
+1. Copy the `.env.example` file to `.env` like so:
+
+    ```
+    cp .env.example .env
+    ```
+
+1. From the repository's root directory, run:
+
+    ```
+    docker compose -f docker-compose.yml -f docker-compose.dev.yml up dev-service
+    ```
+
+    > **Note:** You may need to run docker `sudo` depending on how you have it installed.
+
+    You should now have a web server and database server running.
+
+1. Login at http://localhost:8000/ with the following credentials:
+
+    - username: `admin`
+    - password: `password`
+
+1. Keep Docker running while making your code changes. The app will update automatically with your new code.
+
+## Contribution guidelines
+
+Before getting started with your code changes, read our [Contributor guide](./CONTRIBUTING.md) to understand our processes for handling issues and and PRs.
+
+## Loading sample data
+
+For sample table data, you can create a new table in the UI using the `patents.csv` file found in `/mathesar/tests/data`.
+
+<!-- TODO add more content about sample data -->
+
+## API
+
+See our [API guide](./mathesar/api/README.md) for more information on API usage and development.
+
+## Back end development
+
+- The `db` directory contains low-level code for interacting with the database.
+- The `mathesar` directory contains the Django application and API, which we sometimes refer to as the "service layer".
+
+### Running backend tests
+
+We use [pytest](https://docs.pytest.org) for our backend tests.
+
+- Run all backend tests:
+
+    ```
+    docker exec mathesar_service_dev pytest mathesar/ db/
+    ```
+
+- Run a specific test, by name:
+
+    ```
+    docker exec mathesar_service_dev pytest -k "test_name"
+    ```
+
+- See the [pytest documentation](https://docs.pytest.org/en/latest/how-to/usage.html), or run pytest with the `--help` flag to lear about more options for running tests.
+
+## Front end development
+
+- All the front end code is in the `mathesar_ui` directory.
+- If you are modifying front end code, read more the [Front end development](./mathesar_ui/README.md) guide.
+
+## Full-stack linting
+
+To lint the front end and back end code at the same time, run the `lint.sh` script from the root of the repository. The script requires that the Python virtual environment with `flake8` be activated and that Node modules be installed in `mathesar_ui/`. Alternatively, ESLint and Flake8 should be installed globally on the system.
+
+```
+./lint.sh
+```
+
+- By default, the script lints both Python and Node.js (if changes are staged), but this can be overridden with the `-p` and `-n` flags respectively.
+
+    ```
+    ./lint.sh -p false
+    ```
+
+- You may wish to symlink the script as a pre-commit hook to lint your changes before committing.
+
+    ```
+    ln -s ../../lint.sh .git/hooks/pre-commit
+    ```
+
+## Usage with preexisting databases
+
+If you want to use Mathesar with a preexisting Postgres DB, modify the `DATABASES.mathesar_tables` entry of the `config/settings.py` file with appropriate connection details before installing the Mathesar types and functions by running `install.py` as described in the previous step.
+
+## Rebuilding the Docker images
+
+Sometimes you may need to rebuild your Docker images after pulling new code changes or switching branches. Do so via:
+
+```
+docker compose -f docker-compose.yml -f docker-compose.dev.yml up dev-service --force-recreate --build dev-service
+```
+
+
+## Demo mode
+
+Mathesar can be run in "demo mode" to meet the specific needs of our [live demo site](https://demo.mathesar.org).
+
+See our [Live demo mode](./demo/README.md) guide for more information on enabling live demo mode locally
+
+
+## Opening a shell in the container
+
+- If you need to do some work within the container you can open a bash shell via:
+
+    ```
+    docker exec -it mathesar_service_dev bash
+    ```
+
+- To open a PostgreSQL [psql](https://www.postgresql.org/docs/current/app-psql.html) terminal for the data in Mathesar:
+
+    ```
+    docker exec -it mathesar_db psql -U mathesar
+    ```
+
+## Troubleshooting
+
+### Permissions within Windows
+
+- Running Script in powershell is disabled by default in windows , you have to change permission to run scripts  [Official Docs ](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.security/set-executionpolicy?view=powershell-7.2) 
+
+### Fixing line endings on Windows
+
+If you happen to clone the git repository outside of WSL, then you fix it by running these commands from within WSL.
+
+```
+git config --global core.autocrlf input
+sudo apt-get install -y dos2unix
+sudo find . -type f -exec dos2unix {} \;
+```
+
+These commands install the `dos2unix` utility, which converts text files from the DOS/Microsoft Windows format (with CRLF line endings) to the Unix/Linux format (with LF line endings). Next, the find utility is used to locate all files (-type f) in the current directory (.) and its subdirectories, and the dos2unix command is then executed on each of them (-exec dos2unix ;).
+
+
+### Live reloading front end code on Windows
+
+Hot module replacement for front end code does not work when the project is present on a windows filesystem and WSL is used to run docker. [Read more](./mathesar_ui/README.md#live-reloading-on-windows).
+
+### PostgreSQL server is running on your host machine
+
+If you you see the following error after attempting to start Docker, then the port used by Postgres is already in use on your host machine.
+
+> ERROR: for db Cannot start server db: driver failed programming external connectivity on endpoint mathesar_db (70c521f468cf2bd54014f089f0051ba28e2514667): Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use.
+
+1. First stop Postgres on your host machine.
+
+    ```
+    sudo service postgresql stop
+    ```
+
+1. Then try starting Mathesar via Docker again.
+
+Note that you'll need to manually start your Postgres server on your host machine again if you want to continue working on other projects which rely on that. And the next time you restart your machine, you'll probably need to stop Postgres again before you can begin working on Mathesar.
+
+### Invalid function definition SQL errors
+
+Upon starting Mathesar, you may notice errors similar to:
+```
+mathesar_service_dev  | psycopg.errors.InvalidFunctionDefinition: cannot change name of input parameter "tab_id"
+mathesar_service_dev  | HINT:  Use DROP FUNCTION msar.drop_table(oid,boolean,boolean) first.
+```
+In this case, it's probable that a function parameter name was changed in the `develop` branch at some point. To fix this, you must drop the `msar` and `__msar` schemas from the PostgreSQL database you're using for development using either `psql` or a different client. After doing this, simply stop and start Mathesar using the appropriate `docker compose` commands.

README.md

@@ -30,7 +30,6 @@ You can use Mathesar to build **data models**, **enter data**, and even **build
 - [Live Demo](#live-demo)
 - [Features](#features)
 - [Self-hosting](#self-hosting)
-- [Local development setup](#local-development-setup)
 - [Our motivation](#our-motivation)
 - [Contributing](#contributing)
 - [License](#license)
@@ -78,9 +77,6 @@ Check out a [live demo of Mathesar here](https://demo.mathesar.org)!
 ## Self-hosting
 Please see [our documentation](https://docs.mathesar.org/) for instructions on installing Mathesar on your own server.
 
-## Local development setup
-Please see [our guide in our documentation](http://docs.mathesar.org/contributing/local-dev/).
-
 ## Our motivation
 Mathesar is a non-profit project. Our goal is to make understanding and working with data easy for everyone.
 
@@ -89,7 +85,7 @@ Databases have been around for a long time and solve common data problems really
 We want to make existing database functionality more accessible, for users of all technical skill levels.
 
 ## Contributing
-We actively encourage contribution! Read through [our contributing guidelines](https://wiki.mathesar.org/community/contributing) to get started.
+We actively encourage contribution! Get started by reading our [Contributor Guide](./CONTRIBUTING.md).
 
 ## License
 Mathesar is open source under the GPLv3 license - see [LICENSE](LICENSE). It also contains derivatives of third-party open source modules licensed under the MIT license. See the list and respective licenses in [THIRDPARTY](THIRDPARTY).

demo/README.md

@@ -1,8 +1,14 @@
-# Live Demo mode
+# Demo mode
 
-This folder includes code related to enabling "live demo mode" for Mathesar.
+This folder includes code related to enabling "demo mode" for Mathesar.
 
-To run Mathesar in "live demo mode" locally, add the following to your local `.env` file:
+When Mathesar is in demo mode it has the following special behavior:
+
+- A new database is created for every user session
+- A banner displays atop the UI
+- Analytics are enabled, sending data to the Mathesar team
+
+To run Mathesar in demo mode locally, add the following to your local `.env` file:
 
 ```
 DJANGO_SETTINGS_MODULE=demo.settings

docs/README.md

@@ -1,6 +1,8 @@
-# Contributing to documentation
+# Mathesar's Documentation
 
-## Edit documentation locally
+This directory contains the source code for Mathesar's user and administrator documentation published to https://docs.mathesar.org/
+
+## Preview your documentation edits locally
 
 1. Install requirements
 
@@ -14,6 +16,22 @@
     mkdocs serve -a localhost:9000
     ```
 
+1. Preview the docs at http://localhost:9000
+
+1. Keep mkdocs running while you edit and your local preview will refresh automatically.
+
+## Contribution process
+
+- For small documentation improvements, we welcome PRs without any prior issues. For larger edits, please [open an issue](https://github.com/centerofci/mathesar/issues/new/choose) first to discuss your changes and get approval from the team before proceeding.
+
+- Take care when choosing the git branch on which to base your edits
+    - **Target `master`** if you have an important fix which needs to be published for the _currently-released_ version of Mathesar.
+    - **Target `develop`** if you are adding/updating documentation along with yet-to-be-released changes to the product.
+
+    The docs site is published from the `master` branch. This is important because we want to ensure that it reflects the latest *released* version of Mathesar so that docs readers who are installing or using Mathesar don't see content before it's actually applicable.
+
+See our [Contribution guidelines](../CONTRIBUTING.md) for more information about our pull-request workflow.
+
 ## Reference
 
 - Our docs run on a distribution of [`mkdocs`](https://www.mkdocs.org/) called [`mkdocs-material`](https://squidfunk.github.io/mkdocs-material/). For basics of doc writing, see the [Writing your docs](https://www.mkdocs.org/user-guide/writing-your-docs/) section of `mkdocs` user guide.