ci: add mgard dependency to spack builds

[vicentebolea]

Closes #3752

• It adds MGARD as a dependency.
• Since it requires a recent (but not too recent) MGARD it uses a newer E4S release.
• It fixes a static build issue with duplicated symbols from our pugixml dependency.
• It adds a way to add non-adios2 dependencies to our spack builds.
• It disable Serial tests in MPI builds that have an equivalent dedicated Serial build.
• It removed some OMPI tweaks that breaks newer OMPI.

[vicentebolea]

@scottwittenburg

[scottwittenburg]

This all looks good to me. Now that you are the first one to change the E4S base image for the ubuntu builds, can you comment on:

• Did you build with --progress=plain or similar to see how the buildcache hit rate was affected?
• Is the time to build and push all the images affected much?
• Did the size of the resulting images change much?

You'll have to push again to get rid of the tmp in the image tags everywhere. I forget if, in this project, that will dismiss my review. Just ping me if you need me to re-approve after pushing.

[vicentebolea]

This all looks good to me. Now that you are the first one to change the E4S base image for the ubuntu builds, can you comment on:

* Did you build with `--progress=plain` or similar to see how the buildcache hit rate was affected?

* Is the time to build and push all the images affected much?

* Did the size of the resulting images change much?

You'll have to push again to get rid of the tmp in the image tags everywhere. I forget if, in this project, that will dismiss my review. Just ping me if you need me to re-approve after pushing.

Oh, I had the auto-merge enabled for this and it merged before renaming the tags.

I will follow up in another MR and report the information asked by you

[vicentebolea]

followup #3768