Merge branch 'develop' into pre/2.7

.github/workflows/release.yml

@@ -12,15 +12,17 @@ jobs:
   github-release:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v4
+      with:
+        ref: ${{ github.ref }}
     - name: Exit if any RC release
       if: contains(${{ github.ref }}, 'rc') == false
       uses: everlytic/branch-merge@1.1.2
       with:
         github_token: ${{ secrets.GITHUB_TOKEN }}
         source_ref: ${{ github.ref }}
         target_branch: "latest"
-        commit_message_template: 'RELEASE: :shipit: :boom: :tada: Merged {source_ref} into target {target_branch}'
+        commit_message_template: ':tada: RELEASE: Merged {source_ref} into target {target_branch}'
     - name: Release
       uses: softprops/action-gh-release@v1
       with:
@@ -30,7 +32,9 @@ jobs:
   pypi-release:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
+      with:
+        ref: ${{ github.ref }}
     - uses: actions/setup-python@v2
     - name: Install dependencies
       run: |
@@ -43,3 +47,17 @@ jobs:
       run: |
         python -m build
         python -m twine upload --repository pypi dist/*
+  sync_to_develop:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        ref: "latest"
+    - name: Exit if any RC release
+      if: contains(${{ github.ref }}, 'rc') == false
+      uses: everlytic/branch-merge@1.1.2
+      with:
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        source_ref: "latest"
+        target_branch: "develop"
+        commit_message_template: ':tada: RELEASE: Synced latest into develop'

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ repos:
       name: black <<< NOTE:, if you get a message stating that "All done! n file(s) reformatted", you must git add and commit the changes again to complete the commit. >>>
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.1.5"
+    rev: "v0.3.2"
     hooks:
      - id: ruff
        args: [ --fix ]

CHANGELOG.md

@@ -8,29 +8,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 - 2D heat simulations are now fully supported. 
 - `tidy3d.plugins.adjoint.web.run_local` used in place of `run` will skip validators that restrict the size or number of `input_structures`.
-
-### Fixed
-- Better error message when trying to transform a geometry with infinite bounds.
+- Introduced the `microwave` plugin which includes `ImpedanceCalculator` for computing the characteristic impedance of transmission lines.
+- `Simulation` now accepts `LumpedElementType`, which currently only supports the `LumpedResistor` type. `LumpedPort` together with `LumpedResistor` make up the new `TerminalComponentModeler` in the `smatrix` plugin.
+- Uniaxial medium Lithium Niobate to material library.
+- Added support for conformal mesh methods near PEC structures that can be specified through the field `pec_conformal_mesh_spec` in the `Simulation` class.
 
 ### Changed
+- `run_time` of the adjoint simulation is set more robustly based on the adjoint sources and the forward simulation `run_time` as `sim_fwd.run_time + c / fwdith_adj` where `c=10`.
+- `FieldTimeMonitor` restriction to record at a maximum of 5000 time steps if the monitor is not zero-dimensional, to avoid creating unnecessarily large amounts of data.
 - Bumped `trimesh` version to `>=4,<4.2`.
 
-## [2.6.1] - 2024-03-07
+### Fixed
+- Bug in PolySlab intersection if slab bounds are `inf` on one side.
+- Better error message when trying to transform a geometry with infinite bounds.
+
+## [2.6.3] - 2024-04-02
 
 ### Added
-- Introduced the `microwave` plugin which includes `ImpedanceCalculator` for computing the characteristic impedance of transmission lines.
-- `Simulation` now accepts `LumpedElementType`, which currently only supports the `LumpedResistor` type. `LumpedPort` together with `LumpedResistor` make up the new `TerminalComponentModeler` in the `smatrix` plugin.
-- Uniaxial medium Lithium niobate to material library.
-- Added support for conformal mesh methods near PEC structures that can be specified through the field `pec_conformal_mesh_spec` in the `Simulation` class.
+- Added new validators in `HeatSimulation`: no structures with dimensions of zero size, no all-Neumann boundary conditions, non-empty simulation domain.
 
 ### Changed
-- `run_time` of the adjoint simulation is set more robustly based on the adjoint sources and the forward simulation `run_time` as `sim_fwd.run_time + c / fwdith_adj` where `c=10`.
-- `FieldTimeMonitor` restriction to record at a maximum of 5000 time steps if the monitor is not zero-dimensional, to avoid creating unnecessarily large amounts of data.
+- Revert forbidden `"` in component names.
+
+## [2.6.2] - 2024-03-21
+
+### Changed
+- Characters `"` and `/` not allowed in component names.
+- Change error when `JaxPolySlab.sidewall_angle != 0.0` to a warning, enabling optimization with slanted sidewalls if a lower accuracy gradient is acceptable.
 
 ### Fixed
+- Compute time stepping speed shown `tidy3d.log` using only the number of time steps that was run in the case of early shutoff. Previously, it was using the total number of time steps.
+- Bug in PolySlab intersection if slab bounds are `inf` on one side.
+- Divergence in the simultaneous presence of PML, absorber, and symmetry.
+- Fixed validator for `ModeSpec.bend_radius == 0`, which was not raising an error.
+
+## [2.6.1] - 2024-03-07
+
+### Added
 - `tidy3d.plugins.design.Results` store the `BatchData` for batch runs in the `.batch_data` field.
 - Prompting user to check solver log when loading solver data if warnings were found in the log, or if the simulation diverged or errored.
-- Bug in PolySlab intersection if slab bounds are `inf` on one side.
 
 ### Changed
 - Slightly reorganized `web.run` logging when `verbose=True` to make it clearer.
@@ -1146,6 +1162,9 @@ which fields are to be projected is now determined automatically based on the me
 
 [Unreleased]: https://github.com/flexcompute/tidy3d/compare/v2.6.0...pre/2.7
 [Unreleased]: https://github.com/flexcompute/tidy3d/compare/v2.6.1...develop
+[Unreleased]: https://github.com/flexcompute/tidy3d/compare/v2.6.3...develop
+[2.6.3]: https://github.com/flexcompute/tidy3d/compare/v2.6.2...v2.6.3
+[2.6.2]: https://github.com/flexcompute/tidy3d/compare/v2.6.1...v2.6.2
 [2.6.1]: https://github.com/flexcompute/tidy3d/compare/v2.6.0...v2.6.1
 [2.6.0]: https://github.com/flexcompute/tidy3d/compare/v2.5.2...v2.6.0
 [2.5.2]: https://github.com/flexcompute/tidy3d/compare/v2.5.1...v2.5.2

docs/conf.py

@@ -74,7 +74,7 @@
 copybutton_prompt_is_regexp = True
 custom_sitemap_excludes = [r"/notebooks/"]
 # divparams_enable_postprocessing = True # TODO FIX
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "**.ipynb_checkpoints"]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "**.ipynb_checkpoints", "faq/_faqs/*"]
 extensions = [
     "IPython.sphinxext.ipython_directive",
     "IPython.sphinxext.ipython_console_highlighting",

docs/development/installation.rst

@@ -1,17 +1,22 @@
 Installation
 ==============
 
+Beginners Guide
+^^^^^^^^^^^^^^^
+
+
+
 The Fast Lane
 ^^^^^^^^^^^^^
 
 Maybe you already have ``tidy3d`` installed in some form. After installing version ``tidy3d>=2.6``, you can use a few terminal commands to set you up on the correct environment and perform common development tasks. Just run in your terminal, :code:`tidy3d develop` to get the latest list of commands.
 
-It does not matter how you have installed ``tidy3d`` before as long as you have any form of `tidy3d>=2.6`` in your environment. This can help you transition from a standard user installation to a development environment installation.
+It does not matter how you have installed ``tidy3d`` before as long as you have any form of ``tidy3d>=2.6`` in your environment. This can help you transition from a standard user installation to a development environment installation.
 
 Quick Start
 ''''''''''''
 
-Instructions for anyone who wants to test the new development flow before it gets included as part of the pre-release:
+Instructions for anyone who wants to migrate to the development flow from a version before 2.6:
 
 For ubuntu:
 
@@ -30,8 +35,8 @@ For ubuntu:
 Now you can run the following ``tidy3d`` cli commands to test them.
 
 
-Automatic Environment Installation (Beta)
-'''''''''''''''''''''''''''''''''''''''''''''
+Automatic Environment Installation *Beta*
+''''''''''''''''''''''''''''''''''''''''''
 
 If you are transitioning from the old development flow, to this new one, there are a list of commands you can run to make your life easier and set you up well:
 
@@ -40,7 +45,7 @@ If you are transitioning from the old development flow, to this new one, there a
     # Automatically check and install requirements like pipx, poetry, pandoc
     tidy3d develop install-dev-environment
 
-Note that this is just a automatic script implementation of the `The Detailed Lane`_ instructions. It is intended to help you and raise warnings with suggestions of how to fix an environment setup issue. You do not have to use this helper function and can just follow the instructions in  `The Detailed Lane`_. All commands are echo-ed in the terminal so you will be able to observe and reproduce what is failing if you desire.
+Note that this is just a automatic script implementation of the :ref:`The Detailed Lane` instructions. It is intended to help you and raise warnings with suggestions of how to fix an environment setup issue. You do not have to use this helper function and can just follow the instructions in  :ref:`The Detailed Lane`. All commands are echo-ed in the terminal so you will be able to observe and reproduce what is failing if you desire.
 
 The way this command works is dependent on the operating system you are running. There are some prerequisites for each platform, but the command line tool will help you identify and install the tools it requires. You should rerun the command after you have installed any prerequisite as it will just progress with the rest of the tools installation. If you already have the tool installed, it will verify that it conforms to the supported versions.
 
@@ -49,14 +54,14 @@ This command will first check if you already have installed the development requ
 Environment Verification
 ''''''''''''''''''''''''
 
-If you rather install ``poetry``, ``pipx`` and ``pandoc`` yourself, you can run the following command to verify that your environment conforms to the reproducible development environment which would be equivalent to the one installed automatically above and described in `The Detailed Lane`_.
+If you rather install ``poetry``, ``pipx`` and ``pandoc`` yourself, you can run the following command to verify that your environment conforms to the reproducible development environment which would be equivalent to the one installed automatically above and described in :ref:`The Detailed Lane`.
 
 .. code::
 
     tidy3d develop verify-dev-environment
 
 
-.. _detailed_lane:
+.. _The Detailed Lane::
 
 The Detailed Lane
 ^^^^^^^^^^^^^^^^^

docs/development/project_structure.rst

@@ -11,6 +11,9 @@ As of ``tidy3d>=2.6``, the frontend has been restructured to improve the develop
         notebooks/
             # Git submodule repository
             # Checks out github.com/flexcompute/tidy3d-notebooks
+        faq/
+            # Git submodule repository
+            # Checks out github.com/flexcompute/tidy3d-faq
     tests/
         # pytest source and docs
         # pytest notebooks
@@ -27,6 +30,36 @@ It is important to note the new tools we are using to manage our development env
 - ``poetry``
 - ``pipx``
 
+Important Branches
+------------------
+
+We currently have *three* main branches that have to be kept track of when creating a release, each with different functionality.
+
+.. list-table:: Project Branches
+    :header-rows: 1
+    :widths: 10 45 45
+
+    * - Name
+      - Description
+      - Caveats
+    * - ``latest``
+      - Contains the latest version of the docs. Version release tags are created from this branch.
+      - Feature PRs should not be made to this branch as will cause divergence. Only in important documentation patches.
+    * - ``develop``
+      - Contains the "staging" version of the project. Patch versions and development occurs from these branches.
+      - Docs PRs that are non-crucial for the current version should be made to this branch.
+    * - ``pre/^*``
+      - Contains the next version of the project.
+      - Documentation and source code that will only go live in the next version should be updated here.
+
+Sometimes, hopefully infrequently, the `latest` and `develop` branches might diverge.
+It is important to bring them back together. However, what happens if we rebase develop into latest?
+
+It could be argued that all the commits in the `latest` branch should have constructed within the `develop` branch.
+Then, there is the question if we want to maintain the commit history accordingly. If we just want to maintain the content,
+then rebasing and fixing up all the branches works fine. The problem with a merge commit is that it inserts the commits at the historical period in which they were made, rather than the commit period in which we desire to add them.
+Hence, it makes sense to merge the `develop` and `latest` branches in order to maintain the same history, assuming the commits should in theory have been in both branches.
+
 
 
 

docs/development/recommendations.rst

@@ -1,8 +1,56 @@
 Recommendations
 =================
 
+Standardised Commit Messages
+----------------------------
+
+Now, realistically, this is a `matter of preference <https://medium.com/walmartglobaltech/semantic-commit-messages-with-emojis-dba2541cea9a>`_.
+However, it could be argued there is something nice in having standard commit messages that can be easily searched through
+to understand the role of each change, and also render nicely in the git history. Also, having a commit standard maybe makes people
+looking through our code feel that we take pride in our work and also like to make it nice. It is debatable whether this is a way to do this, however, we can update these recommendations depending on how we consider best.
+
+However, if we do decide to commit with emojis, I believe it would be worth having a standard, so that it does not get polluted with different emojis (as I have been guilty of before) and also as can be seen in other open-source projects.
+
+.. list-table:: Commit Standard
+    :header-rows: 1
+    :widths: 25 15 15 45
+
+    * - Purpose
+      - Emoji
+      - Types
+      - Example
+    * - |:sparkles:| New Feature
+      - ``:sparkles:``
+      - ``FEAT:``
+      - ``:sparkles: FEAT: <my commit message>``
+    * - |:wrench:| Fix Broken Code
+      - ``:wrench:``
+      - ``FIX:``
+      - ``:wrench: FIX: <my commit message>``
+    * - |:package:| Packaging-related
+      - ``:package:``
+      - ``BUILD:``
+      - ``:package: BUILD: <my commit message>``
+    * - |:book:| Documentation-related
+      - ``:book:``
+      - ``DOCS:``
+      - ``:book: DOCS: <my commit message>``
+    * - |:rocket:| Refactor code
+      - ``:rocket:``
+      - ``REFC:``
+      - ``:rocket: REFC: <my commit message>``
+    * - |:test_tube:| Testing related
+      - ``:test_tube:``
+      - ``TEST:``
+      - ``:test_tube: TEST: <my commit message>``
+    * - |:tada:| Release commit
+      - ``:tada:``
+      - ``RELEASE:``
+      - ``:tada: RELEASE: <my commit message>``
+
+
 Package Speedup Best Practices
-''''''''''''''''''''''''''''''
+----------------------------
 
 ``tidy3d`` is a pretty big project already, and will get bigger. We want to optimise the performance of the codebase throughout the multiple operations that we perform.
 
@@ -29,15 +77,16 @@ This is because the latter will import the entire package, which is not necessar
 
 
 Managing Optional Dependencies On-The-Fly
-''''''''''''''''''''''''''''''''''''''''''
+----------------------------
+
 
 If you look within ``pyproject.toml``, it is possible to see that we have different packages relating to different functionalities that are optional.
 
 Some examples from these are ``[vtk, jax, trimesh, gdstk, gdspy]`` etc. What we want to do is improve the import speed of the core-package in order to minimise small core operations. As we scale into a bigger package, decoupling these type of imports from the total pacakge import is essential.
 
 
 Benchmarking Package Import
-''''''''''''''''''''''''''''
+----------------------------
 
 We want to make the tidy3d package be as light as possible for a given set of operations. As such, it is important to understand exactly where a given set of operations is expending computational power.
 

docs/development/release/documentation.rst

@@ -0,0 +1,26 @@
+Documentation Release
+---------------------
+
+The `tidy3d-docs` repository automatically mirrors the `tidy3d` repository. Specifically, these branches are automatically synced.
+
+- main
+- latest
+- develop
+- 'pre/*'
+- 'v*'
+
+These branches are synced to the tidy3d-docs repo through the sync-readthedocs-repo Github action.
+You can read the latest versions synced in the action file.
+However, you need to configure how they appear in the documentation build in the readthedocs admin page.
+Only latest is the public version, others are private.
+
+The `latest` branch holds the state of the docs that we want to host in `latest` version on the website. These are the latest docs (including new notebooks, typo fixes, etc.) related to the last official release (not pre-release).
+
+The `stable` version of the docs on our website is built based on the last version tag which is not a pre-release tag (no `rc`  ending).
+
+Hot Fix & Submodule Updates
+'''''''''''''''''''''''''''
+
+To make a “hot fix” (eg fix a typo, add a notebook, update the release FAQ), just update the ``latest`` branch in ``tidy3d`` repo. This should automatically sync to `tidy3d-docs`, and trigger a docs rebuild. **However, we should avoid this as this will cause the ``develop`` and ``latest branches`` to diverge.** Ideally, these hot fixes could wait until the next pre/post-release to be propagated through.
+
+NOTE: To avoid conflicts, ideally we should only update ``latest`` by merging ``develop`` in it, or at the very least we should make sure changes are propagated to both branches.

docs/development/release/flow.rst

@@ -1,12 +1,16 @@
 Feature Development Workflow
 ------------------------------
 
+Currently most of our release development flow is made under the latest ``pre/*`` branch under the main frontend
+tidy3d repository. You want to fork from this latest branch to develop your feature in order for it to be included
+under that release.
+
 We are using a variation of the `gitflow
 workflow <https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow>`__
 - so this is the first thing to familiarize yourselves with. The
 splitting of branches into ``main``, ``develop`` and separate feature
 branches is as explained there. Most importantly, **all contributions
-should happen through a PR from a feature branch into the ``develop``
+should happen through a PR from a feature branch into the develop
 branch.**
 
 The extra step that we have in our workflow is to
@@ -16,11 +20,8 @@ history clean, but it does require a little more work. As an extra
 advantage, once you get the hang of rebasing it also becomes a very
 useful tool to prune your commits and write more meaningful commit
 messages when you’re done with the work. The main purpose of this page
-is to give an example of the workflow.
-
-   for more information on the difference between rebasing vs merging,
-   see this
-   `article <https://www.atlassian.com/git/tutorials/merging-vs-rebasing>`__.
+is to give an example of the workflow. For more information on the difference between rebasing vs merging,
+see this `article <https://www.atlassian.com/git/tutorials/merging-vs-rebasing>`__.
 
 The first thing to do when starting a new batch of work is to start from
 a clean branch on your machine.
@@ -39,7 +40,7 @@ Create your feature rebase
 Before rebasing, you should make sure you have the latest version
 of ``develop``, in case other work has been merged meanwhile.
 
-::
+.. code-block:: bash
 
    git checkout develop
    git pull origin develop