Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable extrapolation in PETSc #2082

Merged
merged 5 commits into from Mar 5, 2018

Conversation

Projects
None yet
2 participants
@endJunction
Copy link
Member

endJunction commented Feb 20, 2018

Although the extrapolation is not correct for partitioned domains, it is correct for non-partitioned projects and some people are using PETSc for its linear solvers in single threaded setup.

@endJunction endJunction force-pushed the endJunction:EnableExtrapolationInPetsc branch 2 times, most recently from 54324e6 to eadb81e Feb 20, 2018

@wenqing

This comment has been minimized.

Copy link
Member

wenqing commented Feb 21, 2018

As a temporary solution, you can change #ifndef USE_PETSC to #ifdef USE_PETSC with a constraint as

#ifdef USE_PETSC
int petsc_np;
MPI_comm_size(PETSC_COMM_WORLD,  petsc_np);
if (petsc_np == 1)
{
#endif // end of #ifdef USE_PETSC`

...
piece of the source code for sequential computing
...

#ifdef USE_PETSC
}
#endif
@endJunction

This comment has been minimized.

Copy link
Member Author

endJunction commented Feb 21, 2018

Good point, Wenqing. I'll come back to this if the more correct implementation turns out to be more complex...

@endJunction endJunction force-pushed the endJunction:EnableExtrapolationInPetsc branch 3 times, most recently from 1fd594c to 71b70e3 Feb 21, 2018

@endJunction

This comment has been minimized.

Copy link
Member Author

endJunction commented Feb 21, 2018

OK. Seems to be working, at least what is tested.
There is a problem with residuals which shows up in the TES-mpi ctest. The problem is about the access of values w/o finalize, or copy before finalize calls. I'd not trust the residual output for petsc now. I'm working on this, but it does not look very promising right now.
Update: The error was found and fixed. Dereference a pointer to matrix in finalizeAssembly call.

@endJunction endJunction removed the WIP 🏗 label Feb 21, 2018

@endJunction

This comment has been minimized.

Copy link
Member Author

endJunction commented Feb 21, 2018

Some of the changes are just cleanup. see #2085

@wenqing
Copy link
Member

wenqing left a comment

👍

@@ -47,12 +47,39 @@ void LocalLinearLeastSquaresExtrapolator::extrapolate(
{
auto const num_nodal_dof_result =
_dof_table_single_component.dofSizeWithoutGhosts() * num_components;

std::vector<GlobalIndexType> ghost_indices;

This comment has been minimized.

@wenqing

wenqing Feb 22, 2018

Member

This is only for DDC. How about

#ifndef USE_PETSC
    if (!_nodal_values ||  _nodal_values->size() != num_nodal_dof_result)
    {
         _nodal_values = MathLib::MatrixVectorTraits<GlobalVector>::newInstance(
            MathLib::MatrixSpecifications{
                num_nodal_dof_result, num_nodal_dof_result, nullptr, nullptr});         
    }
#else // For USE_PETSC
    std::vector<GlobalIndexType> ghost_indices;
    {  // Create num_components times version of ghost_indices arranged by
       // location. For example for 3 components and ghost_indices {5,6,10} we
       // compute {15, 16, 17,  18, 19, 20,  30, 31, 32}.
        auto const& single_component_ghost_indices =
            _dof_table_single_component.getGhostIndices();
        auto const single_component_ghost_indices_size =
            single_component_ghost_indices.size();
        ghost_indices.reserve(single_component_ghost_indices_size *
                              num_components);
        for (unsigned i = 0; i < single_component_ghost_indices_size; ++i)
        {
            for (unsigned c = 0; c < num_components; ++c)
            {
                ghost_indices.push_back(
                    single_component_ghost_indices[i] * num_components + c);
            }
        }
    }

     if (!_nodal_values ||  static_cast<std::size_t>(_nodal_values->getLocalSize() +
                                 _nodal_values->getGhostSize())
            != num_nodal_dof_result)
     {
         _nodal_values = MathLib::MatrixVectorTraits<GlobalVector>::newInstance(
           {num_nodal_dof_result, num_nodal_dof_result, &ghost_indices,
            nullptr});
     }
#endif

@endJunction endJunction force-pushed the endJunction:EnableExtrapolationInPetsc branch 2 times, most recently from aa42a05 to fb85bcf Feb 22, 2018

endJunction added some commits Feb 19, 2018

[NL] LeastSqExt: Fix global vector creation.
Especially the creation of ghost nodes.
[MaL] LinAlg; Remove default impl of finalizeAsm.
The default impl., which is not doing anything, is error prone,
because passing an object of not matrix type, e.g. std::unique_ptr<PETScVector>,
the default impl. will be taken.

@endJunction endJunction force-pushed the endJunction:EnableExtrapolationInPetsc branch from fb85bcf to 99232bb Mar 5, 2018

@endJunction endJunction merged commit dfc04ef into ufz:master Mar 5, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details

@endJunction endJunction deleted the endJunction:EnableExtrapolationInPetsc branch Mar 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.