Updating to PartitionedArrays v0.5#157
Open
JordiManyer wants to merge 53 commits into
Open
Conversation
[Diff since v0.4.12](v0.4.12...v0.4.13) **Merged pull requests:** - Add Hcurl-projection test + fix BUG (#199) (@amartinhuertas)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #157 +/- ##
==========================================
- Coverage 84.72% 79.38% -5.34%
==========================================
Files 16 19 +3
Lines 4249 4255 +6
==========================================
- Hits 3600 3378 -222
- Misses 649 877 +228
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…One MPI process is spawned by testset
Closed
Member
Author
|
@amartinhuertas I think this starts being ready. There are still a couple things I want to do, but they are optional. All tests we have in master pass, I believe. So (no rush) I think it is time to start reviewing this. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates GridapDistributed’s distributed linear algebra and assembly pipeline to align with PartitionedArrays v0.5, including a major refactor of parallel assembly strategies and the test/documentation structure.
Changes:
- Bump
PartitionedArraysdependency to0.5and adapt distributed indices/ownership handling accordingly. - Replace/rename parallel assembly strategies with
Assembled,SubAssembled, andLocallyAssembled, and rework matrix/vector assembly internals. - Restructure MPI tests into per-test mpiexec “driver” wrappers and rewrite documentation into topic-based pages.
Reviewed changes
Copilot reviewed 105 out of 107 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/sequential/runtests.jl | Add testcase logging; formatting tweaks |
| test/PullbackTests.jl | Update assembly strategy name |
| test/PLaplacianTests.jl | Update assembly strategy names |
| test/PeriodicBCsTests.jl | Adds top-level main call (problematic) |
| test/parrays_update.jl | Adds standalone PA v0.5 experiment script |
| test/mpi/ZeroMeanFESpacesTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/VisualizationTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/TransientMultiFieldDistributedCellFieldTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/TransientDistributedCellFieldTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/SurfaceCouplingTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/StokesOpenBoundaryTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/StokesHdivDGTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/runtests.jl | New MPI testcase dispatcher layout |
| test/mpi/runtests_np4.jl | Removed old monolithic MPI runner |
| test/mpi/runtests_np4_body.jl | Removed old monolithic MPI runner body |
| test/mpi/run_mpi_driver.jl | New shared mpiexec runner helper |
| test/mpi/PolytopalCoarseningTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/PoissonTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/PLaplacianTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/PeriodicBCsTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/MultiFieldTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/MacroDiscreteModelsTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/HeatEquationTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/HcurlProjectionTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/GeometryTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/FESpacesTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/ConstantFESpacesTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/CellDataTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/BlockSparseMatrixAssemblersTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/BlockPartitionedArraysTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/AutodiffTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/AdaptivityUnstructuredTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/AdaptivityTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/AdaptivityMultiFieldTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/AdaptivityCartesianTests.jl | New mpiexec wrapper test entrypoint |
| test/mpi/drivers/ZeroMeanFESpacesTests.jl | New MPI driver module |
| test/mpi/drivers/VisualizationTests.jl | New MPI driver module |
| test/mpi/drivers/TransientMultiFieldDistributedCellFieldTests.jl | New MPI driver module |
| test/mpi/drivers/TransientDistributedCellFieldTests.jl | New MPI driver module |
| test/mpi/drivers/SurfaceCouplingTests.jl | New MPI driver module |
| test/mpi/drivers/StokesOpenBoundaryTests.jl | New MPI driver module |
| test/mpi/drivers/StokesHdivDGTests.jl | New MPI driver module |
| test/mpi/drivers/PolytopalCoarseningTests.jl | New MPI driver module |
| test/mpi/drivers/PoissonTests.jl | New MPI driver module |
| test/mpi/drivers/PLaplacianTests.jl | New MPI driver module |
| test/mpi/drivers/PeriodicBCsTests.jl | New MPI driver module |
| test/mpi/drivers/MultiFieldTests.jl | New MPI driver module |
| test/mpi/drivers/MacroDiscreteModelsTests.jl | New MPI driver module |
| test/mpi/drivers/HeatEquationTests.jl | New MPI driver module |
| test/mpi/drivers/HcurlProjectionTests.jl | New MPI driver module |
| test/mpi/drivers/GeometryTests.jl | New MPI driver module |
| test/mpi/drivers/FESpacesTests.jl | New MPI driver module |
| test/mpi/drivers/ConstantFESpacesTests.jl | New MPI driver module |
| test/mpi/drivers/CellDataTests.jl | New MPI driver module |
| test/mpi/drivers/BlockSparseMatrixAssemblersTests.jl | New MPI driver module |
| test/mpi/drivers/BlockPartitionedArraysTests.jl | New MPI driver module |
| test/mpi/drivers/AutodiffTests.jl | New MPI driver module |
| test/mpi/drivers/AdaptivityUnstructuredTests.jl | New MPI driver module |
| test/mpi/drivers/AdaptivityTests.jl | New MPI driver module |
| test/mpi/drivers/AdaptivityMultiFieldTests.jl | New MPI driver module |
| test/mpi/drivers/AdaptivityCartesianTests.jl | New MPI driver module |
| test/HeatEquationTests.jl | Update assembly strategy name |
| test/FESpacesTests.jl | Add GID generation tests; strategy rename |
| test/BlockSparseMatrixAssemblersTests.jl | Update assembly strategy name |
| test/BlockPartitionedArraysTests.jl | Minor module footer change |
| test/AutodiffTests.jl | Improve matrix equality checking |
| test/AdaptivityTests.jl | Rename local-to-local map helper |
| test/AdaptivityMultiFieldTests.jl | Rename local-to-local map helper |
| test/AdaptivityCartesianTests.jl | Rename local-to-local map helper |
| src/Visualization.jl | Add docstring for DistributedPvd |
| src/Redistribution.jl | Update vector assembly cache construction |
| src/Pullbacks.jl | Use foreach where return value unused |
| src/Partitions.jl | New partition/GID generation utilities |
| src/PArraysExtras.jl | Add PA v0.5 helpers (incl. unpermute) |
| src/MultiField.jl | Switch multifield gid concat to vcat_gids |
| src/MacroDiscreteModels.jl | Update cell gids + docs formatting |
| src/GridapExtras.jl | New tracked allocation + COO axis helpers |
| src/GridapDistributed.jl | Export new strategies; include new modules |
| src/Geometry.jl | New docstrings; strategy-based ghost filtering |
| src/FESpaces.jl | Refactor assembly + GID partitioning utilities |
| src/CellData.jl | Add docstrings + minor formatting fixes |
| src/BlockPartitionedArrays.jl | Update PSparse allocation; add ops + centralize |
| src/Assembly.jl | New assembly strategy + builder implementation |
| src/Algebra.jl | Remove old assembly; simplify LocalView mapping |
| src/Adaptivity.jl | Update mapping helper + GID partition updates |
| Project.toml | Bump PartitionedArrays compat to 0.5 |
| docs/src/Visualization.md | Remove old autodocs page |
| docs/src/visualization.md | New narrative visualization docs |
| docs/src/MultiField.md | Remove old autodocs page |
| docs/src/index.md | Rewrite landing page and doc structure |
| docs/src/GridapDistributed.md | Remove old autodocs page |
| docs/src/getting-started.md | Remove old getting started page |
| docs/src/Geometry.md | Remove old autodocs geometry page |
| docs/src/geometry.md | New narrative geometry docs |
| docs/src/FESpaces.md | Remove old autodocs fespaces page |
| docs/src/fespaces.md | New narrative FE spaces docs |
| docs/src/changelog.md | Add (currently empty) changelog page |
| docs/src/CellData.md | Remove old autodocs celldata page |
| docs/src/backends.md | New backends + philosophy page |
| docs/src/assembly.md | New assembly strategies docs |
| docs/src/Algebra.md | Remove old autodocs algebra page |
| docs/src/algebra.md | New narrative algebra docs |
| docs/src/Adaptivity.md | Remove old autodocs adaptivity page |
| docs/src/adaptivity.md | New narrative adaptivity docs |
| docs/Project.toml | Add Gridap as docs dependency |
| docs/make.jl | Update pages list; enable warnonly |
| .gitignore | Broaden ignore patterns; add agent files |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+54
to
+60
| function unpermute(indices::AbstractLocalIndices) | ||
| @notimplemented | ||
| end | ||
|
|
||
| unpermute(indices::PermutedLocalIndices) = unpermute(indices.indices) | ||
| unpermute(indices::PartitionedArrays.LocalIndicesInBlockPartition) = indices | ||
| unpermute(indices::OwnAndGhostIndices) = indices |
Comment on lines
+38
to
+39
| main(DebugArray,(2,2)) | ||
|
|
|
|
||
| ## Global DOF numbering and ownership | ||
|
|
||
| Global DOF IDs are stored as a `PRange` (see [Algebra](@ref algebra)), which can be accessed via `get_free_dof_gids(V)`. Some general considerations: |
Comment on lines
+4
to
+10
| repodir = normpath(joinpath(@__DIR__, "..", "..")) | ||
| cmd = mpiexec() | ||
| if MPI.MPI_LIBRARY == "OpenMPI" || (isdefined(MPI, :OpenMPI) && MPI.MPI_LIBRARY == MPI.OpenMPI) | ||
| run(`$cmd -n $procs --oversubscribe $(Base.julia_cmd()) --project=$repodir $(file)`) | ||
| else | ||
| run(`$cmd -n $procs $(Base.julia_cmd()) --project=$repodir $(file)`) | ||
| end |
Comment on lines
+1
to
+10
|
|
||
| using Gridap | ||
| using PartitionedArrays | ||
| using GridapDistributed | ||
|
|
||
| using Gridap.FESpaces, Gridap.Algebra | ||
|
|
||
| np = (1,2) | ||
| ranks = with_debug() do distribute | ||
| distribute(LinearIndices((prod(np),))) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We have been detached from the development of PartitionedArrays for a while, currently stuck at v0.3. This is also related to #137 .
This PR is an update to PartitionedArrays v0.5, with a substantial rework of the re-assembly process... This comes with several upsides, such as
Change Log
FESpace free dof ids are now given by a permuted instance of
LocalIndicesWithVariableBlockSize. We were already doing this (so our numbering strategy does not change), but it's now explicit. This has two advantages:LocalIndicesWithVariableBlockSize.scanoperation). This is quite handy for assembly, since expanding the ghost ids in the index partition can be done using PArrays'sfind_ownerandunion_ghostmethods.With PA 0.5, new assembly strategies are available. Also, we've been wanting to fix the confusing names we currently have. We now have the following assembly strategies:
Assembled: We integrate on owned cells and collect contributions for owned and ghost row ids, then exchange ghost contributions and assemble. CurrentSubAssembledRows.SubAssembled: We integrate on owned cells and collect contributions for owned and ghost row ids, but no exchange. The resulting matrix is sub-assembled, i.e each local matrix contains the contribution for it's owned and ghost rows/cols resulting from integrating over the owned part of the domain.LocallyAssembled: This assumes all contributions can be found by integrating on the local portion of the domain. We integrate on local cells (owned and ghost), and keep contributions for owned row ids. No exchange is needed. CurrentFullyAssembledRows. Comes at your own risk.The indices in the matrices and vectors are now
LocalIndicesWithVariableBlockSize, so that the owner of a dof can be locally deduced by it's gid. This will save us some communication.To-Do
Assembled, but for which extra ghost rows are not allocated. This strategy will re-assemble using COO matrices (instead of directly inserting in the matrix). This can save binary searches, which can be quite costly. The re-assembly caches will be stored by the user (this will be an advanced api).SubAssembled()assembly strategy.Assembly cache reuse strategy
The most involved part of this PR is the assembly cache re-use. The issue is that so far we had been using the output
PSparseMatrixas the cache, by allocating extra ghost rows that were only used in the sub-assembly process. This has quite a lot of downsides (see #137), so we are pivoting towards an external assembly cache model.However, having external caches does not go well with the current Gridap API. Here is a possible solution:
assemble_matrix!will not be supported, except if anAssembleris provided (see next point).Assemblerwill hold the caches. To be able to reuse an assembler for multiple matrices, we will use aDictand the matrix object-id to hold multiple caches (tied to a single matrix each). Cache re-use will be activated by boolean variablereusethat is held by theAssembler.FEOperatorswill keep an instance of their assembler, allowing re-use by default for nonlinear and transientFEOperators.