Remote Server Improvements #3862
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
eisenhauer
commented
Oct 20, 2023
eisenhauer
commented
- Add idle timeout to server. Unless the -no_timeout flag is specified, the server will exit after 10 minutes of idle time
- Specify the server port with a single constant in remote_common.h, rather than several explicit constants.
eisenhauer
force-pushed
the
serverimprovements
branch
from
ff9ebac
to
b03a61e
Compare
eisenhauer
force-pushed
the
serverimprovements
branch
from
b03a61e
to
95c1e5b
Compare
anagainaru
approved these changes
Oct 21, 2023
There was a problem hiding this comment.
@spyridon97 fixed the reason crusher is failing, it's not related to this PR so you can ignore it.
One small comment, isn't CManager_close(cm) called at the destruction of the Singleton? Can we call this multiple times?
CManager does have a reference count, so we can do it multiple times. It might not be quite right in this case, but this is really just a backstop to prevent an unfortunate situation. Sometimes when you use ssh port forwarding to run the server and kill ssh with ^C, the server process remains running forever on whatever random login node had been selected by the ssh. The timeout is just to make sure that those things go away in relatively short order. A perfectly clean shutdown isn't necessary isn't the first priority. |
dmitry-ganyushin
added a commit
to dmitry-ganyushin/ADIOS2
that referenced
this pull request
Nov 1, 2023
* master: (183 commits) Adding tests for writing null blocks with and without compression Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. Merge pull request ornladios#3805 from pnorbert/fix-bpls-string-scalar Merge pull request ornladios#3804 from pnorbert/fix-aws-version Merge pull request ornladios#3759 from pnorbert/bp5dbg-metadata new attempt to commit query support of local array. (ornladios#3868) MPI::MPI_Fortran should be INTERFACE not PUBLIC Fix hip example compilation error (ornladios#3865) Server Improvements (ornladios#3862) ascent,ci: remove unshallow flag Remove Slack as a contact mechanism (ornladios#3866) bug fix: syntax error in json output (ornladios#3857) Update the bpWriterReadHip example's cmake to run on crusher Examples: Use BPFile instead of BP3/4/5 for future-proof inlineMWE example: Close files at the end Examples: Add BeginStep/EndStep wherever it was missing BP5Serializer: handle local variables that use operators (ornladios#3859) Blosc2 USE ON: Fix Module Fallback (ornladios#3774) SST,MPI,DP: soft handle peer error SST,MPI,DP: improve uniq identifier Fix destdir install test (ornladios#3850) cmake: include ctest before detectoptions ci: enable tau check Add/Improve the ReadMe.md files in examples directory Disable BUILD_TESTING and ADIOS2_BUILD_EXAMPLES by default Remove testing based on ADIOS2-examples Fix formatting issue in DetectOptions.cmake Add examples from ADIOS2-Examples Improve existing examples MPI_DP: do not call MPI_Init (ornladios#3847) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) Tweak Remote class and test multi-threaded file remote access (ornladios#3834) Add prototype testing of remote functionality (ornladios#3830) Try always using the MPI version Try always using the MPI version Import tests from bp to staging common, implement memory selection in SST ci: fix codeql ignore path (ornladios#3772) install: export adios2 device variables (ornladios#3819) added support to query BP5 files (ornladios#3809) ffs 2023-09-19 (67e411c0) Fix abs/rel step in BP5 DoCount fix dummy Win build Pass Array Order of reader to remote server for proper Get() operation Fix the ADIOS_USE_{_} variable names to use ADIOS2 Remove BP5 BetweenStepPairs variable that hides Engine.h counterpart Delete experimental examples yaml-cpp: support 0.8.0 version Mod not to overload to make some compilers happy Mod to keep MinBlocksInfo private to BP5Writer, not reuse Engine.h API. gha,ci: update checkout to v4 Limit testing changes to BP5 tests: fix windows 64bit type issues Add location of hdf5.dll to PATH for windows tests Fix more compiler warnings from ompi build clang format Fix size_t -> int conversion warnings Allow building with Visual Studio and HDF5 shared libs Fixup local var reading by block with test, master branch use documented initial cache variable to find hdf5 on windows ci: Add HDF5 to a windows build Remove unused SelectionType values Update ADIOS2 HDF5 VOL with basic set of capability flags defines reorder Remove file before writing, catch exceptions format Reader-side Profiling removed a comment ci: disable MGARD static build cmake: fix ffs dependency removed comments and cleaned up more code bp5: remove ADIOS2_USE_BP5 option operators: fix module library ci: Create static minimal build cmake: correct prefer_shared_blosc behavior removing warning from auto clang-format match type of timestep for h5 engine to size_t (same as adios VariableBase class) when storing to hdf5, check the limit of size_t and match to the right type (uint, ulong, default ulonglong) BP5Deserialize: modify changes to keep abi compt Rename test. fix windows compile errors. add codeql workflow Fix ChunkV maintaining CurOffset when downsizing current chunk in Allocate() and creating the next chunk. ci: fix docker images names removed kwsys gitattributes KWSys 2023-05-25 (c9f0da47) fixup! ci: add mgard dependency to spack builds ci: remove unused script part cmake: remove redundant pugixml dependency ci: add mgard dependency to spack builds cmake: Remove enet config warning cmake: resolve cmake python deprecation enet 2023-08-15 (7587eb15) EVPath 2023-08-15 (657c7fa4) dill 2023-08-15 (235dadf0) atl 2023-08-15 (7286dd11) remove data.py since BP5 data file has no format format bp5dbg parse records and check sizes during it for mmd.0 and md.0. No decoding FFS records. cmake: correct info.h installation path ...
pnorbert
added a commit
to pnorbert/ADIOS2
that referenced
this pull request
Nov 20, 2023
* master: (126 commits) ReadMe.md: Mention 2.9.2 release Cleanup server output a bit (ornladios#3914) ci: set openmpi and openmp params Example using Kokkos buffers with SST Changes to MallocV to take into consideration the memory space of a variable Change install directory of Gray scott files again ci,crusher: increase supported num branches ci: add shellcheck coverage to source and testing Change install directory of Gray scott files Only rank 0 should print the initialization message in perfstub Defining and computing derived variables (ornladios#3816) Add Remote "-status" command to see if a server is running and where (ornladios#3911) examples,hip: use find_package(hip) once in proj Add Steps Tutorial Add Operators Tutorial Add Attributes Tutorial Add Variables Tutorial Add Hello World Tutorial Add Tutorials' Download and Build section Add Tutorials' Overview section Improve bpStepsWriteRead* examples Rename bpSZ to bpOperatorSZWriter Convert bpAttributeWriter to bpAttributeWriteRead Improve bpWriter/bpReader examples Close file after reading for hello-world.py Fix names of functions in engine Fix formatting warnings Add dataspaces.rst in the list of engines Add query.rst cmake: find threads package first docs: update new_release.md Bump version to v2.9.2 ci: update number of task for mpich build clang-format: Correct format to old style Merge pull request ornladios#3878 from anagainaru/test-null-blocks Merge pull request ornladios#3588 from vicentebolea/fix-mpi-dp Adding tests for writing null blocks with and without compression bp5: make RecMap an static anon namespaced var Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. Replace LookupWriterRec's linear search on RecList with an unordered_map. For 250k variables, time goes from 21sec to ~1sec in WSL. The order of entries in RecList was not necessary for the serializer to work correctly. (ornladios#3877) Fix data length calculation for hash (ornladios#3875) Merge pull request ornladios#3823 from eisenhauer/SstMemSel Merge pull request ornladios#3805 from pnorbert/fix-bpls-string-scalar Merge pull request ornladios#3804 from pnorbert/fix-aws-version Merge pull request ornladios#3759 from pnorbert/bp5dbg-metadata new attempt to commit query support of local array. (ornladios#3868) MPI::MPI_Fortran should be INTERFACE not PUBLIC Fix hip example compilation error (ornladios#3865) Server Improvements (ornladios#3862) ascent,ci: remove unshallow flag Remove Slack as a contact mechanism (ornladios#3866) bug fix: syntax error in json output (ornladios#3857) Update the bpWriterReadHip example's cmake to run on crusher Examples: Use BPFile instead of BP3/4/5 for future-proof inlineMWE example: Close files at the end Examples: Add BeginStep/EndStep wherever it was missing BP5Serializer: handle local variables that use operators (ornladios#3859) gha,ci: update checkout to v4 Blosc2 USE ON: Fix Module Fallback cmake: correct prefer_shared_blosc behavior cmake: correct info.h installation path ci: disable MGARD static build operators: fix module library ci: add downloads readthedocs cmake: Add Blosc2 2.10.1 compatibility. Blosc2 USE ON: Fix Module Fallback (ornladios#3774) Fix destdir install test (ornladios#3850) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) MPI_DP: do not call MPI_Init (ornladios#3847) install: export adios2 device variables (ornladios#3819) Merge pull request ornladios#3799 from vicentebolea/support-new-yaml-cpp Merge pull request ornladios#3737 from vicentebolea/fix-evpath-plugins-path SST,MPI,DP: soft handle peer error SST,MPI,DP: improve uniq identifier Fix destdir install test (ornladios#3850) cmake: include ctest before detectoptions ci: enable tau check Add/Improve the ReadMe.md files in examples directory Disable BUILD_TESTING and ADIOS2_BUILD_EXAMPLES by default Remove testing based on ADIOS2-examples Fix formatting issue in DetectOptions.cmake Add examples from ADIOS2-Examples Improve existing examples MPI_DP: do not call MPI_Init (ornladios#3847) cmake: update minimum cmake to 3.12 (ornladios#3849) MPI: add timeout for conf test for MPI_DP (ornladios#3848) Tweak Remote class and test multi-threaded file remote access (ornladios#3834) Add prototype testing of remote functionality (ornladios#3830) Try always using the MPI version Try always using the MPI version Import tests from bp to staging common, implement memory selection in SST ci: fix codeql ignore path (ornladios#3772) install: export adios2 device variables (ornladios#3819) added support to query BP5 files (ornladios#3809) Partial FFS Upstream, only changes to type_id ffs 2023-09-19 (67e411c0) Fix abs/rel step in BP5 DoCount fix dummy Win build Pass Array Order of reader to remote server for proper Get() operation ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.