Merge branch 'pre/2.7' into develop

.github/ISSUE_TEMPLATE/autograd_bug.md

@@ -0,0 +1,20 @@
+---
+name: Autograd Bug Report
+about: Create a report to help us improve our autograd feature
+title: 'Autograd Bug'
+labels: 'autograd'
+assignees: 'tylerflex'
+
+---
+
+**Description**
+Please paste the error message displayed in your warning when running `tidy3d.web.run()` or `tidy3d.web.run_async()`
+
+**Simulation Setup**
+
+Please describe your simulation setup as specifically as you can, including code snippets or uploading the `simulation.to_file('sim.json')` file. 
+
+Also please include what options were passed to the `web` function.
+
+**Other information**
+Anything else you'd like to add to help us look into it?

.github/workflows/release.yml

@@ -16,7 +16,7 @@ jobs:
       with:
         ref: ${{ github.ref }}
     - name: Exit if any RC release
-      if: contains(${{ github.ref }}, 'rc') == false
+      if: contains(github.ref, 'rc') == false
       uses: everlytic/branch-merge@1.1.2
       with:
         github_token: ${{ secrets.GITHUB_TOKEN }}
@@ -54,7 +54,7 @@ jobs:
       with:
         ref: "latest"
     - name: Exit if any RC release
-      if: contains(${{ github.ref }}, 'rc') == false
+      if: contains(github.ref, 'rc') == false
       uses: everlytic/branch-merge@1.1.2
       with:
         github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/run_tests.yml

@@ -11,20 +11,22 @@ on:
       - 'pre/*'
 
 jobs:
+
   pre-commit:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           submodules: 'recursive'
       - name: Test pre-commit hooks
         run: |
           python -m pip install --upgrade pip
           pip install pre-commit
           pre-commit run # this should be really more agressive
+
   test-latest-submodules:
     runs-on: ubuntu-latest
     steps:
@@ -39,12 +41,19 @@ jobs:
         id: get_branch
         run: |
           if [[ "${{ github.event_name }}" == "pull_request" ]]; then
-            echo "BRANCH_NAME=${{ github.event.pull_request.base.ref }}" >> $GITHUB_ENV
+            # Extract the base branch of the PR
+            BRANCH_NAME="${{ github.event.pull_request.base.ref }}"
+            echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
           else
-            echo "BRANCH_NAME=$(echo $GITHUB_REF | sed 's|refs/heads/||')" >> $GITHUB_ENV
+            # Assume it's a push event, extract the branch name from $GITHUB_REF
+            BRANCH_NAME=$(echo $GITHUB_REF | sed 's|refs/heads/||')
+            echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
           fi
-          echo $BRANCH_NAME
+          # Now echoing the BRANCH_NAME to verify
+          echo "BRANCH_NAME: $BRANCH_NAME"
         shell: bash
+        env:
+          GITHUB_REF: ${{ github.ref }}
 
       - name: Initialize and update submodule
         run: |
@@ -59,17 +68,18 @@ jobs:
           # Checking out Notebooks submodule with the same branch as the main project
           echo "Checking $NOTEBOOKS_PATH for updates..."
           cd $NOTEBOOKS_PATH
+          NOTEBOOKS_CURRENT_COMMIT=$(git rev-parse HEAD)
           echo $(git fetch --all --verbose)
           echo $(git remote get-url origin)
-          git checkout origin/$BRANCH_NAME
           if git show-ref --verify refs/remotes/origin/$BRANCH_NAME; then
            echo "Branch $BRANCH_NAME exists."
           else
             echo "::error::Branch $BRANCH_NAME does not exist on remote."
             exit 1
           fi
           NOTEBOOKS_LATEST_COMMIT=$(git rev-parse refs/remotes/origin/${{ env.BRANCH_NAME }})
-          NOTEBOOKS_CURRENT_COMMIT=$(git rev-parse HEAD)
+          echo "NOTEBOOKS_LATEST_COMMIT: $NOTEBOOKS_LATEST_COMMIT"
+          echo "NOTEBOOKS_CURRENT_COMMIT: $NOTEBOOKS_CURRENT_COMMIT"
           
           
           cd ../..
@@ -83,22 +93,27 @@ jobs:
           # Checking FAQs only on the develop branch.
           echo "Checking $FAQ_PATH for updates..."
           cd $FAQ_PATH
-          FAQ_LATEST_COMMIT=$(git rev-parse origin/develop)
           FAQ_CURRENT_COMMIT=$(git rev-parse HEAD)
+          echo $(git fetch --all --verbose)
+          echo $(git remote get-url origin)
+          FAQ_LATEST_COMMIT=$(git rev-parse refs/remotes/origin/develop)
+          echo "FAQ_LATEST_COMMIT: $FAQ_LATEST_COMMIT"
+          echo "FAQ_CURRENT_COMMIT: $FAQ_CURRENT_COMMIT"
           cd ../..
           if [ "$FAQ_LATEST_COMMIT" != "$FAQ_CURRENT_COMMIT" ]; then
             echo "::error ::Submodule $FAQ_PATH is not up to date. Please update it."
             exit 1
           else
             echo "Submodule $FAQ_PATH is up to date."
           fi
+
   build:
     name: test ${{ matrix.python-version }} - ${{ matrix.platform }}
     runs-on: ${{ matrix.platform }}
     strategy:
       matrix:
-        python-version: ['3.9', '3.10', '3.11']
-        platform: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ['3.9', '3.10', '3.11', '3.12']
+        platform: [ubuntu-latest, windows-latest, macos-latest]
     defaults:
       run:
         shell: bash
@@ -124,13 +139,16 @@ jobs:
     - name: Install library
       run: |
         poetry --version
+        poetry run pip install --upgrade pip wheel setuptools
+        poetry run pip install gdstk --only-binary gdstk
         poetry install -E dev
+
     #----------------------------------------------
     #    add matrix specifics and run test suite
     #----------------------------------------------
     - name: Run tests
       run: |
-        poetry run black . --check --diff
+        poetry run ruff format . --check --diff
         poetry run ruff check tidy3d --fix --exit-non-zero-on-fix
         poetry run pytest -rA tests
         poetry run pytest -rA tests/_test_data/_test_datasets_no_vtk.py

.github/workflows/sync-to-readthedocs-repo.yaml

@@ -1,6 +1,5 @@
 name: "sync-to-readthedocs-repo"
-permissions:
-  contents: write
+
 on:
   workflow_dispatch:
   push:
@@ -9,43 +8,72 @@ on:
       - latest
       - develop
       - 'pre/*'
-      - 'v*'
       - 'production/test/*'
+    tags:
+      - 'v*'
+      - 'demo/*'
+
 jobs:
-  extract_branch:
+  extract_branch_or_tag:
     outputs:
-      branch: ${{ steps.extractbranch.outputs.branch }}
+      ref_name: ${{ steps.extract.outputs.ref_name }}
     runs-on: ubuntu-latest
     steps:
-      - id: extractbranch
-        name: Extract branch name
+      - id: extract
+        name: Extract branch or tag name
         shell: bash
         run: |
-          echo "::set-output name=branch::${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}"
-          echo "${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}"
-          echo "${{ steps.extractbranch.outputs.branch }}"
+          REF_NAME="${GITHUB_REF#refs/*/}"
+          echo "::set-output name=ref_name::$REF_NAME"
+          echo "Extracted ref: $REF_NAME"
+
   build-and-deploy:
-    needs: extract_branch
+    permissions:
+      contents: write
+    needs: extract_branch_or_tag
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout
+      # Conditional Checkout for Branch
+      - name: Checkout Branch
+        if: contains(github.ref, 'refs/heads/')
+        uses: actions/checkout@v3
+        with:
+          submodules: true
+          token: ${{ secrets.GH_PAT }}
+          ref: ${{ needs.extract_branch_or_tag.outputs.ref_name }}
+
+      # Conditional Checkout for Tag
+      - name: Checkout Tag
+        if: contains(github.ref, 'refs/tags/')
         uses: actions/checkout@v3
         with:
           submodules: true
-      - name: Create new branch if doesn't exist
+          token: ${{ secrets.GH_PAT }}
+          fetch-depth: 0
+          ref: ${{ needs.extract_branch_or_tag.outputs.ref_name }}
+          fetch-tags: true
+
+      - name: Create new branch or tag
         uses: GuillaumeFalourd/create-other-repo-branch-action@v1.5
+        if: contains(github.ref, 'refs/heads/')
         with:
           repository_owner: flexcompute-readthedocs
           repository_name: tidy3d-docs
-          new_branch_name: ${{ needs.extract_branch.outputs.branch }}
+          new_branch_name: ${{ needs.extract_branch_or_tag.outputs.ref_name }}
           access_token: ${{ secrets.GH_PAT }}
           new_branch_ref: default_clean_sync_branch
           ignore_branch_exists: true
+
+      - name: Tag and push
+        if: contains(github.ref, 'refs/tags/')
+        run: |
+          git push --force --tags https://${{ secrets.GH_PAT }}@github.com/flexcompute-readthedocs/tidy3d-docs.git
+
       - uses: JamesIves/github-pages-deploy-action@v4
         with:
           folder: .
           token: ${{ secrets.GH_PAT }}
           repository-name: flexcompute-readthedocs/tidy3d-docs
           target-folder: .
-          branch: ${{ needs.extract_branch.outputs.branch }}
+          branch: ${{ needs.extract_branch_or_tag.outputs.ref_name }}
           force: true

.github/workflows/update_coverage.yml

@@ -82,7 +82,7 @@ jobs:
     - name: Run tests
       run: |
         source $VENV
-        black . --check --diff
+        ruff format . --check --diff
         ruff check tidy3d --fix --exit-non-zero-on-fix
         coverage run -m pytest -rA tests
         coverage report -m

.gitignore

@@ -80,6 +80,9 @@ pip-log.txt
 .dmypy.json
 .mypy_cache/
 
+# pickle files
+*.pkl
+
 # PyInstaller
 *.manifest
 *.spec
@@ -109,6 +112,7 @@ _autosummary
 
 # Test and temporary files
 tests/tmp/*
+tests/data/invdes_history.*
 
 # Translations
 *.mo
@@ -129,3 +133,7 @@ htmlcov/
 
 # Specific file and folder exclusions
 .idea
+.vscode
+
+# cProfile output
+*.prof

.pre-commit-config.yaml

@@ -1,12 +1,7 @@
 repos:
-  - repo: https://github.com/psf/black
-    rev: 23.12.1
-    hooks:
-    - id: black
-      name: black <<< NOTE:, if you get a message stating that "All done! n file(s) reformatted", you must git add and commit the changes again to complete the commit. >>>
-
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.3.2"
+    rev: "v0.4.8"
     hooks:
      - id: ruff
-       args: [ --fix ]
+       args: [ --fix ]
+     - id: ruff-format

CHANGELOG.md

@@ -5,6 +5,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- EME solver through `EMESimulation` class.
+- 2D heat simulations are now fully supported. 
+- `tidy3d.plugins.adjoint.web.run_local` used in place of `run` will skip validators that restrict the size or number of `input_structures`.
+- Introduces the `microwave` plugin which includes `ImpedanceCalculator` for computing the characteristic impedance of transmission lines.
+- `Simulation` now accepts `LumpedElementType`, which currently only supports the `LumpedResistor` type. `LumpedPort` together with `LumpedResistor` make up the new `TerminalComponentModeler` in the `smatrix` plugin.
+- Uniaxial medium Lithium Niobate to material library.
+- Properties `num_time_steps_adjoint` and `tmesh_adjoint` to `JaxSimulation` to estimate adjoint run time.
+- Ability to add `path` to `updated_copy()` method to recursively update sub-components of a tidy3d model. For example `sim2 = sim.updated_copy(size=new_size, path="structures/0/geometry")` creates a recursively updated copy of `sim` where `sim.structures[0].geometry` is updated with `size=new_size`.
+- Python 3.12 support. Python 3.8 deprecation. Updated dependencies.
+- Tidy3D objects may store arbitrary metadata in an `.attrs` dictionary.
+- `JaxSimulation` now supports the following GDS export methods: `to_gds()`, `to_gds_file()`, `to_gdspy()`, and `to_gdstk()`.
+- `RunTimeSpec` accepted by `Simulation.run_time` to adaptively set the run time based on Q-factor, propagation length, and other factors.
+- `JaxDataArray` now supports selection by nearest value via `JaxDataArray.sel(..., method="nearest")`.
+- Convenience method `constant_loss_tangent_model` in `FastDispersionFitter` to fit constant loss tangent material model.
+- Classmethods in `DispersionFitter` to load complex-valued permittivity or loss tangent data.
+- Pre-upload validator to check that mode sources overlap with more than 2 grid cells.
+- Support `2DMedium` for `Transformed`/`GeometryGroup`/`ClipOperation` geometries.
+- `num_proc` argument to `tidy3d.plugins.adjoint.web.run_local` to control the number of processes used on the local machine for gradient processing.
+- Support for complex and self-intersecting polyslabs in adjoint module via `JaxComplexPolySlab`.
+- Support for `.gz` files in `Simulation` version updater.
+- Warning if a nonuniform custom medium is intersecting `PlaneWave`, `GaussianBeam`, `AstigmaticGaussianBeam`, `FieldProjectionCartesianMonitor`, `FieldProjectionAngleMonitor`, `FieldProjectionKSpaceMonitor`, and `DiffractionMonitor`.
+- Added a `CoaxialLumpedPort` and `CoaxialLumpedResistor` for coaxial type transmission lines and excitations.
+- Automatic differentiation supported natively in Tidy3D components, and through the `web.run()` and `web.run_async()` functions through `autograd`.
+- A batch of `ModeSolver` objects can be run concurrently using `tidy3d.plugins.mode.web.run_batch()`
+- `RectangularWaveguide.plot_field` optionally draws geometry edges over fields. 
+- `RectangularWaveguide` supports layered cladding above and below core.
+- `SubpixelSpec` accepted by `Simulation.subpixel` to select subpixel averaging methods separately for dielectric, metal, and PEC materials. Specifically, added support for conformal mesh methods near PEC structures that can be specified through the field `pec` in the `SubpixelSpec` class. Note: previously, `subpixel=False` was implementing staircasing for every material except PEC. Now, `subpixel=False` implements direct staircasing for all materials. For PEC, the behavior of `subpixel=False` in Tidy3D < 2.7 is now achieved through `subpixel=SubpixelSpec(pec=HeuristicPECStaircasing())`, while `subpixel=True` in Tidy3D < 2.7 is now achieved through `subpixel=SubpixelSpec(pec=Staircasing())`. The default is `subpixel=SubpixelSpec(pec=PECConformal())` for more accurate PEC modelling.
+- Lossless `Green2008` variant for crystalline silicon added to material library.
+
+### Changed
+- IMPORTANT NOTE: differentiable fields in the `adjoint` plugin (`JaxBox.size`, `JaxBox.center`, `JaxPolySlab.vertices`) no longer store the derivative information after the object is initialized. For example, if using JaxPolySlab.vertices directly in an objective function, the vertices will have no effect on the gradient. Instead, this information is now stored in a field of the same name with `_jax` suffix, eg. `JaxPolySlab.vertices_jax`. For some applications, such as evaluating penalty values, please change to `radius_penalty.evaluate(polyslab.vertices_jax)` or use the vertices as generated by your parameterization functions (`make_vertices(params)`).
+- `run_time` of the adjoint simulation is set more robustly based on the adjoint sources and the forward simulation `run_time` as `sim_fwd.run_time + c / fwdith_adj` where `c=10`.
+- `FieldTimeMonitor` restriction to record at a maximum of 5000 time steps if the monitor is not zero-dimensional, to avoid creating unnecessarily large amounts of data.
+- Bumped `trimesh` version to `>=4,<4.2`.
+- Make directories in path passed to `.to_gds_file()` methods, if they don't exist.
+- Changed sign of mode solver PML conductivity to match the `exp(-1j * omega * t)` convention used everywhere else. Previously, loss due to PML was coming out as negative `k_eff` of the mode, while now it comes out as positive `k_eff`, in line with material loss.
+- Augmented the PML mode solver profile to the form `s(x) = kappa(x) + 1j * sigma(x) / (omega * EPSILON_0)`. Previously, `kappa` was not used. The parameters are currently set to `kappa_min = 1`, `kappa_max = 3`, `sigma_max = 2`. These are not yet exposed to the user.
+- Also scaling `sigma_max` by the average relative impedance in the mode solver PML region.
+- `tidy3d convert` from `.lsf` files to tidy3d scripts has moved to another repository at `https://github.com/hirako22/Lumerical-to-Tidy3D-Converter`.
+- Relax validation of smallest triangle area in `TriangleMesh`.
+- Default variant for silicon dioxide in material library switched from `Horiba` to `Palik_Lossless`.
+- Sources and monitors which are exactly at the simulation domain boundaries will now error. They can still be placed very close to the boundaries, but need to be on the inside of the region.
+
+### Fixed
+- Bug in PolySlab intersection if slab bounds are `inf` on one side.
+- Better error message when trying to transform a geometry with infinite bounds.
+- `JaxSimulation.epsilon` properly handles `input_structures`.
+- `FieldData.flux` in adjoint plugin properly returns `JaxDataArray` containing frequency coordinate `f` instead of summing over values.
+- Proper error message when trying to compute Poynting vector for adjoint field data.
+- Bug in plotting and computing tilted plane intersections of transformed 0 thickness geometries.
+- `Simulation.to_gdspy()` and `Simulation.to_gdstk()` now place polygons in GDS layer `(0, 0)` when no `gds_layer_dtype_map` is provided, instead of erroring.
+- `task_id` now properly stored in `JaxSimulationData`.
+- Bug in `FastDispersionFitter` when poles move close to input frequencies.
+- Bug in plotting polarization vector of angled sources.
+- Bug in `SpatialDataArray.reflect()` that was causing errors for older versions of `xarray`.
+- `ModeSolver.plot_field` correctly returning the plot axes.
+- Avoid error if non-positive refractive index used for integration resolution in adjoint.
+- Make `Batch.monitor` robust if the run status is not found.
+
 ## [2.6.4] - 2024-04-23
 
 ### Fixed
@@ -16,6 +76,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - Added new validators in `HeatSimulation`: no structures with dimensions of zero size, no all-Neumann boundary conditions, non-empty simulation domain.
+- `web.Batch` uses multi-threading for upload and download operations. The total time for many tasks is reduced by an order of magnitude.
 
 ### Changed
 - Revert forbidden `"` in component names.
@@ -25,6 +86,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 - Characters `"` and `/` not allowed in component names.
 - Change error when `JaxPolySlab.sidewall_angle != 0.0` to a warning, enabling optimization with slanted sidewalls if a lower accuracy gradient is acceptable.
+- Simplified output and logging in `web.Batch` with `verbose=True`.
 
 ### Fixed
 - Compute time stepping speed shown `tidy3d.log` using only the number of time steps that was run in the case of early shutoff. Previously, it was using the total number of time steps.
@@ -1150,7 +1212,9 @@ which fields are to be projected is now determined automatically based on the me
 - Job and Batch classes for better simulation handling (eventually to fully replace webapi functions).
 - A large number of small improvements and bug fixes.
 
-[Unreleased]: https://github.com/flexcompute/tidy3d/compare/v2.6.4...develop
+[Unreleased]: https://github.com/flexcompute/tidy3d/compare/v2.7.0rc2...pre/2.7
+[2.7.0rc2]: https://github.com/flexcompute/tidy3d/compare/v2.6.4...v2.7.0rc2
+[2.7.0rc1]: https://github.com/flexcompute/tidy3d/compare/v2.6.3...v2.7.0rc1
 [2.6.4]: https://github.com/flexcompute/tidy3d/compare/v2.6.3...v2.6.4
 [2.6.3]: https://github.com/flexcompute/tidy3d/compare/v2.6.2...v2.6.3
 [2.6.2]: https://github.com/flexcompute/tidy3d/compare/v2.6.1...v2.6.2