Campaign manager #3932
Merged
Campaign manager #3932
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…at this point. Added CampaignReader engine as a skeleton reader.
…ors and different ADIOS objects.
…stuck on empty m_AvailableStepBlockIndexOffsets
…o that we can use the actual engine and IO functions. bpls works properly showing dimensions now. Min/max is missing. Reading is not supported. BeginStep/EndStep not supported.
…based on hostname match)
* master: (126 commits) ReadMe.md: Mention 2.9.2 release Cleanup server output a bit (ornladios#3914) ci: set openmpi and openmp params Example using Kokkos buffers with SST Changes to MallocV to take into consideration the memory space of a variable Change install directory of Gray scott files again ci,crusher: increase supported num branches ci: add shellcheck coverage to source and testing Change install directory of Gray scott files Only rank 0 should print the initialization message in perfstub Defining and computing derived variables (ornladios#3816) Add Remote "-status" command to see if a server is running and where (ornladios#3911) examples,hip: use find_package(hip) once in proj Add Steps Tutorial Add Operators Tutorial Add Attributes Tutorial Add Variables Tutorial Add Hello World Tutorial Add Tutorials' Download and Build section Add Tutorials' Overview section Improve bpStepsWriteRead* examples Rename bpSZ to bpOperatorSZWriter Convert bpAttributeWriter to bpAttributeWriteRead Improve bpWriter/bpReader examples Close file after reading for hello-world.py Fix names of functions in engine Fix formatting warnings Add dataspaces.rst in the list of engines Add query.rst cmake: find threads package first docs: update new_release.md Bump version to v2.9.2 ci: update number of task for mpich build clang-format: Correct format to old style Merge pull request ornladios#3878 from anagainaru/test-null-blocks Merge pull request ornladios#3588 from vicentebolea/fix-mpi-dp Adding tests for writing null blocks with and without compression bp5: make RecMap an static anon namespaced var Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. (ornladios#3877) Fix data length calculation for hash (ornladios#3875) Merge pull request ornladios#3823 from eisenhauer/SstMemSel Merge pull request ornladios#3805 from pnorbert/fix-bpls-string-scalar Merge pull request ornladios#3804 from pnorbert/fix-aws-version Merge pull request ornladios#3759 from pnorbert/bp5dbg-metadata new attempt to commit query support of local array. (ornladios#3868) MPI::MPI_Fortran should be INTERFACE not PUBLIC Fix hip example compilation error (ornladios#3865) Server Improvements (ornladios#3862) ascent,ci: remove unshallow flag Remove Slack as a contact mechanism (ornladios#3866) bug fix: syntax error in json output (ornladios#3857) Update the bpWriterReadHip example's cmake to run on crusher Examples: Use BPFile instead of BP3/4/5 for future-proof inlineMWE example: Close files at the end Examples: Add BeginStep/EndStep wherever it was missing BP5Serializer: handle local variables that use operators (ornladios#3859) gha,ci: update checkout to v4 Blosc2 USE ON: Fix Module Fallback cmake: correct prefer_shared_blosc behavior cmake: correct info.h installation path ci: disable MGARD static build operators: fix module library ci: add downloads readthedocs cmake: Add Blosc2 2.10.1 compatibility. Blosc2 USE ON: Fix Module Fallback (ornladios#3774) Fix destdir install test (ornladios#3850) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) MPI_DP: do not call MPI_Init (ornladios#3847) install: export adios2 device variables (ornladios#3819) Merge pull request ornladios#3799 from vicentebolea/support-new-yaml-cpp Merge pull request ornladios#3737 from vicentebolea/fix-evpath-plugins-path SST,MPI,DP: soft handle peer error SST,MPI,DP: improve uniq identifier Fix destdir install test (ornladios#3850) cmake: include ctest before detectoptions ci: enable tau check Add/Improve the ReadMe.md files in examples directory Disable BUILD_TESTING and ADIOS2_BUILD_EXAMPLES by default Remove testing based on ADIOS2-examples Fix formatting issue in DetectOptions.cmake Add examples from ADIOS2-Examples Improve existing examples MPI_DP: do not call MPI_Init (ornladios#3847) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) Tweak Remote class and test multi-threaded file remote access (ornladios#3834) Add prototype testing of remote functionality (ornladios#3830) Try always using the MPI version Try always using the MPI version Import tests from bp to staging common, implement memory selection in SST ci: fix codeql ignore path (ornladios#3772) install: export adios2 device variables (ornladios#3819) added support to query BP5 files (ornladios#3809) Partial FFS Upstream, only changes to type_id ffs 2023-09-19 (67e411c0) Fix abs/rel step in BP5 DoCount fix dummy Win build Pass Array Order of reader to remote server for proper Get() operation ...
* master: Example using DataMan with Kokkos buffers Propagating the GPU logic inside the DataMan engine ci: Use mpich built with ch3:sock:tp for faster tests
…ader engine. - convert info to json at Close() and write out. - add example to write many different files from different communicators and different ADIOS objects. - add explicit map for which engine supports ReadRandomAccess mode - save campaign data (in json) in adios-campaign/ - use SQLite3 database for campaign archive, compresses metadata - CampaignReader uses SQLite3 - check ctime of individual files and save from DB only if it is older - enable using multiple directories in same host - new parameter RemoteDataPath for BP5 engine, and Campaign Engine controls this parameter when opening a remote file (with metadata files locally present in another (local) path)
* master: CI Contract: Build examples with external ADIOS
* master: Have HDF5 write raise error if operator(s) requested (ornladios#3951) fix for ASAN issue related to JoinedDimArray handling in BP5 deserializer (ornladios#3963) New operator MDR, for refactoring floating point arrays using MGARD's new MDR extension. (ornladios#3826) restricted http transport from windows builds. XMLConfigTest: Add RemoveIO test adios2::core::ADIOS: Initialize new IO objects with config file removed unsused variable Update readme for heat transfer example with new location and build instructions Ignore tests with defects for now Adapt libfabric dataplane of SST to Cray CXI provider (ornladios#3672) ci: fix path to lsan suppressions, fix broken gh status post Use adios2_mode_readRandomAccess in matlab open to make it work for BP5 (ornladios#3956) Add Global Array Capabilities and Limitations Add Section for Anatomy of an ADIOS Program Enable Shell-Check for gh-actions scripts Enable Shell-Check for circle CI scripts Enable Shell-Check for tau contract scripts Enable Shell-Check for scorpio contract scripts Enable Shell-Check for lammps contract scripts Delete VTK code in examples Fix MATLAB bindings for MacOS (ornladios#3950) Set the compiler for the Kokkos DataMan example to what is used to build Kokkos Fix the HIP architecture CMAKE variable (ornladios#3931) perfstubs 2023-11-27 (845d0702) (ornladios#3944) Revert "Only rank 0 should print the initialization message in perfstub" Formatting Formatting Revision Added buffered data receive in the client side. A socket version of HTTP connector. Proxy server host is hardwired to "localhost" and port to 9999 Remote bpls: bpls -E bp4 -T "Library=HTTP" /remote_path/myVector_cpp.bp -d bpInts
…se empty functions for campaign recorder.
…can be added to a single campaign. It also supports restarting the application and adding more steps, so that a campaign archive can be updated later.
dmitry-ganyushin
approved these changes
Dec 15, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Keeping campaign manager up to date with master.