-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multispecies cosmology #104
Multispecies cosmology #104
Conversation
Mesh - added Box class for computing loop indices - replaced loop_limits in FieldFace with Box Cello - fix: changed std::numeric_limits<T>::digits to digits10 Charm - removed obsolete sum_long_double_[4-8]_type reductions; switched to n_type Data - added rank parameter to FieldFace constructor for Box class, and since cello:rank() not available in test_FieldFace.cpp - removed unused FluxData methods Enzo - implementd EnzoProlong to call ENZO interpolate() but with local data (extends coarse region by duplicating edge values) - changed EnzoSolverDiagonal to use temporary instead of "diagonal_D"
Cello - bug fix: use digits10 not digits in cello::digits_max() Charm - add missing array indexing [in] to static EnzoBlock::NumberOfBaryonFields Refresh - removed "new" from refresh-related identifiers - removed obsolete debugging code
Build - added linux_clang config Refresh - Adding support for EnzoProlong's interpolation region overlapping multiple extra blocks Problem - Added virtual Prolong::padding() - Removed Refresh::id_solver_ Mesh - Cleaning Box API - Added support for third "extra" Block to Box - Converted Box attributes to arrays
Refresh - Implemented most of refresh_extra_field_faces_ Mesh - separated Box::compute_block_start from Box::compute_region() - Added support for fourth "array" Block type to Box - set gr3_[] default to g3_ - added Box::restrict_limits() to confine loop limits to Block
Charm - Added optional MsgRefresh::oname_[block|type]_ for debugging Refresh - Added TRACE_COMM debugging - Implemented and debugged refresh_load_extra_face_ (still needs field value copies) - Implemented refresh_extra_send_() Mesh - added Block::padded_face_array_[] to store padded arrays - added Block::padded_face_array() - added Block::padded_face_array_[de]allocate() - added Block::name(Index) to get name of neighboring blocks Data - added DataMsg::padded_face_ - added DataMsg::padded_face_field_list_ - added DataMsg::set_padded_face() - added padded_face_field_list_ Enzo - added missing array copy for 3D problems Tools - changed obsolete xrange to range in plot_mesh.py
Testing - removed "dump" from Output in PPML tests which was failing in random queueing Debugging - added temporary cello::sum() and cello::copy() used in enzo_prolong debugging - added cello::hex_string() to create pseudo-random hex string for MsgRefresh::tag_ and DataMsg::tag_ for debugging messages - updated debugging code in MsgRefresh - updated debugging code in control_refresh - updated debugging code in DataMsg EnzoProlong - added Block::refresh_extra_apply_ to call EnzoProlong only after all expected face data have been received - added refresh_type parameter to refresh_load_extra_face_ Mesh - Updated Box to not overwrite receive block with extra since needed for EnzoProlong array receive - Added Box_type to Box class for - Added Box::get_region_size() - Added rank,level,face,child to Box constructor instead of separate set_face(), set_child(), etc. Method - fixed timestepping in EnzoMethodGravity to match ENZO (had grouping error) - removed const from virtual double Method::timestep() required by change to ENzoMethodGravity::timestep() Data - added ENABLE_PADDING to FieldFace - Moved padded_face_array_ from Data to FieldData::padded_array_ - added padded_array_dimensions_[] - added padded_array_fields_[] - removed FieldFace::box_ since doesn't need to be stored - cleaned padded array code in FieldFace::[array|face]_to_[array|face]() - added code to FieldFace to copy send-recv face zones to padded array in FieldFace::[array|face]_to_[array|face]() - fixed typo c3[3] = {3} in FieldFace::face_to_face() Messages - updated padded_array attributes in DataMsg ENZO - added int() around sign() calls in some fortran code (calcdiss)
Build - removed obsolete ncsa-bw check in build.sh that points to user home directory Testing - added test_Box (empty) - updated test_Field for coarse fields Debugging - removed control_refresh.cpp TRACE_COMM debugging - updated debugging in DataMsg - cleaned FieldFace debugging - added temporary ENABLE_EXTRA to control_refresh.cpp - added MsgRefresh::set_block_[name|type] Mesh - cleaning Box API Prolong - converting face padded_arrays to single coarse field per block (incomplete) Data - added FieldData::coarse_values() - added FieldData::[de]allocate_coarse() - added FieldData::coarse_dimensions()
Documentation - formatting Debugging - updated MsgRefresh.cpp, control_refresh.cpp - cleaned DataMsg EnzoProlong - renamed refresh_extra_*() as refresh_coarse_*() - updated call to refresh_load_coarse_face_() - updated refresh_load_coarse_face_() parameters - added main S->r send for padded prolong Mesh - refactored Box API: split get_limits() into get_start_stop() and get_start_size() - refactored Box to clarify role of get_start_*() parameters (first defines block to intersect region, second to define index coordinate system) - added Box::coarse_size_[] and coarse_ghost_[] initialized by set_coarse() - bugfix in FieldData::coarse_dimensions() for odd ghost depths Charm - renamed and updated DataMsg::set_padded_face() as set_coarse_array() - updated DataMsg coarse_array parameters Prolong - renamed Prolong::padding() as Prolong::coarse_padding() - removed padding code from FieldFace since handled by FieldData::coarse_array_ - removed unused Prolong::monotonic_ and ::positive_ - changed Prolong::apply() to void return - added virtual Prolong::array_sizes_valid (true default) Fortran - removed obsolete [DEBUG_ERROR_WARNING]_MESSAGE defines
Cello - cleaning macros for sizing, saving, loading message data - updated random number generator in cello::hex_string() to not be the same on different processors Refresh - removed redundant Box method calls - fixed interpolation factor: add not average - finished implementing Block::refresh_coarse_apply_() Data - renamed DataMsg::field_[array|data}_ as field_[array|data]_u_ to indicate union - replaced manual size/load/save with cello.hpp macros - fixed data size check ASSERT call position - updated print() to include tag_ - added Box BlockType::none for not intersecting region EnzoProlong - finished implementing Tools - updated valgrind to include additional issues in generated valgrind.org file
Problem - added MethodRefresh to manually add refresh operations between other Methods Refresh - added debug to control_refresh.cpp - fixed logic for calling refresh_load_[field|coarse]_face_ and corresponding counters (note count expected receives not number sent) - fixed BlockType argument to box_sr.get_start_stop for (iam3,iap3) indices - fixed BlockType arguments to box_[er|se].get_start_stop() for (ifms3,ifps3) indices - fixed rr scaling to be num_children() only when levels actually differ - fixed loop limits in DataMsg::update() - fixed (re)setting pad = 0 in else of if (refresh_fine) in FieldFace face_to_array() array_to_face() - simplified calls to prolong->apply in FieldFace::[array|field]_to_field() - adding Refresh::set_min_face_rank() and ::set_ghost_depth(), ::sync() (not called yet) - added return_bypass to return_enum for Blocks that have nothing to do in a Solver Mesh - split BlockType::coarse into BlockType::coarse_receive and BlockType::coarse_extra Config - added MethodRefresh parameters to Config - method_refresh_field_list; - method_refresh_particle_list; - method_refresh_ghost_depth; - method_refresh_min_face_rank; - method_refresh_all_fields; - method_refresh_all_particles; - made Config::index_schedule public for consistency Enzo - adding debugging to EnzoSolverBiCgStab - incorporating ENZO Fortran code error handling into EnzoMethodPpm to exit immediately on errors - adding ENZO errors - added missing ierror parameter to flux_[hll|twoshock] argument lists (was in call) - added returns after errors in [xyz]euler_sweeps to exit with error earlier and not try to "compute" with nan's
Testing - fixed B_COPY -> B_copy in test_cosmo.incl - implemented test_Box.cpp (no checks, currently for comparing with expected values by hand) - updated test_FieldFace for change to FieldFace::set_ghost() arguments (int instead of bool) - add src/Enzo/test_interpolate.F for testing setup for calling interpolate.F (old but hadn't added to git) Mesh - changed Block::create_face() arguments from logical lg3 to values g3 - added lpad argument to Box::get_start_[stop|size]() whether to include padding (need false for fine blocks true for coarse) - removed temporary set_padding(0) in refresh_coarse_apply_ (not needed with lpad argument) - moved box set_padding() inside set_box_() - fixed coarse loop indices in array_to_face() - update FieldFace::ghost_ from bool to int for size - fixed logic for computing pad (0 if not refresh_fine) - changed Box to add padding when get_start_[stop|size] called not when computing region (since not always used) - added Box::apply_padding_() Method - renamed i_f as index_field in MethodDebug - updated EnzoSolverDd::pack_field_() for including ghosts only when refresh_type == refresh_fine - added EXIT_ON_ERROR to SolveHydroEquations (EnzoMethodPpm) - removed obsolete setting ierror to constants in [yz]euler_sweep.F Refresh - BUG FIX: fixed use of i_f index instead of index_field in refresh_coarse_apply_ - fixed FieldData::coarse dimensions() for non-centered fields (cx was used instead of cy)
Input - removed prolong parameters for testing - added input/schedule_time_0.01.incl - removed old/unused fields from test_cosmo.incl Testing - removed mesh-balanced test since redundant with adapt_L5 - run adapt_L5 regression test to 0.05 instead of 0.1 to speed up regression - changed gravity solver tolerance from 0.01 to 0.1 to speed up regression - updated test/index.php for above changes - cleaning: removed commented-out tests Prolong - readded TRACE_PROLONG output to Prolong objects for verifying type Parameters - removed obsolete interpolation_method parameter
Debug - consolidating debug macros in cello.hpp - removing debug code from control_refresh.cpp Refresh - incorporate refresh->field_list_dst() as well as field_list_src() for padded prolong refresh - duplicated Box object in refresh_coarse_apply_() that was used for two separate index computations - moved FieldFace::accumulate_() into Refresh::accumulate() - Added Refresh::box_accumulate_adjust() Output - added use_min_max parameter to OutputImage() - fixed OutputImage pup()--added missing attributes - changed log() in OutputImage to log(fabs()) Prolong - consolidated duplicated code in ProlongLinear::apply_() for accumulate = 0 and 1 - added support for accumulate in EnzoProlong Method - removed non-accumulated fields from ir_post refresh in EnzoMethodGravity (accelerations and density) - added DEBUG_COPY_DENSITIES to EnzoMethodGravity - removed errant debug code from EnzoMethodPmDeposit Testing - removed obsolete collapse tests test_method_gravity_cg-[18].unit
- Cleaning Sync, preparing to add error checking Initial - Moved initialize_[prolong|restrict] before initialize_[method|solver] since initialize_[prolong|restrict] create "default" prolong/restrict which must be first Prolong - moved coarse_padding() from Prolong to Refresh, so can set to 0 if refresh accumulate_ is true - Made Prolong::coarse_padding() protected but friend to Refresh::coarse_padding() Problem - changed Problem::[prolong|restrict]_ to lists [prolong|restrict]_list_ - changed all NULL's to nullptr's in problem_Problem.?pp - changed initialize_[prolong|restrict]_ to create default prolong/restrict check that it's creating the first prolong/restrict objects in the lists - changed Problem::[prolong|restrict]() methods to accept optional argument i, with default 0 to return default first in list - added "use_linear" argument to EnzoProlong constructor Refresh - adding prolong and restrict to Refresh Testing - cleaning test_Sync Parameters - added Prolong : enzo : use_linear parameter for debugging (EnzoProlong::apply() calls ProlongLinear::apply()) Solvers - Changed restrict/prolong arguments to EnzoSolver[Dd|Mg0] to index_prolong/index_restrict
- adding debugging to Sync to test for over-counting (INCOMPLETE) - moved [mul|div]_by_density() from FieldFace class to cello:: namespace Debug - removed tag_ attribute and other debugging code from MsgRefresh - and DataMsg addressed some minor compiler warning messages (-Wall) - add checks when copying fields for debugging in EnzoMethodGravity - and EnzoMethodPpm Rferesh - Added index_[prolong|refresh] to Refresh - Removed Prolong / Restrict from FieldFace (use from Refresh instead) Parameters - Changed default Field:prolong from "linear" to "enzo" Solver - Added prolong/restrict arguments to Solvers - changed EnzoSolverBiCgStab::res_tol_ from long double to double (suspect this was causing compiler issues with optimization)
Control - bug fix: Moved deallocating fluxes from MethodFluxCorrect.cpp to control_compute.cpp to avoid memory leak if MethodFluxCorrect isn't called Debug - Updating MethodDebug for added parameters (incomplete) - Added parameters to "debug" method: "print", "coarse", "ghost" (incomplete) - added debug code for copying fields in EnzoSolverBiCgStab Parameters - Added method_refresh_prolong Method - EnzoMethodGravity updates - Added index_prolong parameter to EnzoMethodGravity - Re-added accelerations to ir_post refresh in EnzoMethodGravity - fixed typo: naming ir_exit_ refresh Solver - EnzoSolverBiCgStab can produce poor solutions at mesh level jumps if restart != 1: resetting to 1 with WARNING message to bypass until this is addressed Cleaning - Removed obsolete EnzoMethodCheckGravity - Removed obsolete files in tools directory - Added tools/index.org file to document remaining available tools
Mesh - Added BlockTrace class to aid traversing distributed array-of-octrees partition (range of elements in root-block array) - Added [pr]_method_output_<foo>() entry methods for MethodOutput Parameters - Added initial_hdf5_* parameters for InitialHdf5 - Renamed method_refresh_<foo> to method_<foo> - Added method_output_blocking parameter to define scope of domain for each output file Testing - Added test_BlockTrace unit tests - Removed unused code from test/index.php Method / Output - Added MethodOutput "method" for writing HDF5 output Cello - fixed LOAD_ARRAY_TYPE() Charm - Added MsgOutput Charm++ message for use by MethodOutput - cleaned MsgRefresh: removed debug code Initial - Added InitialHdf5 stub Cleaning - addressed some -Wall compiler warnings
Build - removed obsolete "new_output" from SConstruct Output - Removed unused index parameter from Io class hierarchy - removed unused Io::data_count_ - Copied expand_name() and directory() from Output to Cello so can use in MethodOutput - removed NEW_OUTPUT from control_output.cpp Mesh - Added index_home to BlockTrace (not necessarily the same as index_root) - Added data_size, [save|load]_data to Index Cello - Added string, object, and object_ptr to size/load/save type macros Charm - Updating MsgOutput for block name, file pointer, and tag (for debugging) - Converting manual size/load/save in MsgOutput::[un]pack() to use cello.hpp macros Parameters - Added method_output_[file|path]_name parameters - Fixed method_field_list to be list of string not integer Method - Implementing MethodOutput (sequencing works, file created and closed, need to add HDF5 write calls) - Added MethodOutput ScalarData for file counter (needed per-block since may have multiple Blocks writing files on same pe - Updated MethodRefresh field and particle lists to be string not int Simulation - Removed Simulation sync objects for obsolete "new_output"
Input - Removed flux_correct from test_double_mach to avoid crashing Charm - Added IoBlock, Block lower/upper to MsgOutput Problem - Added Factory argument to Problem::initialize_method() (required for MethodOutput) - Made argument orderings in Problem::create_[*]() more consistent Output - Added data_size() and [load|save]_data() to Io class hierarchy for including in MsgOutput - Made Io class hierarchy pup-able - Updated Io classes to store metadata and not access via Block / Hierarchy classes (since IoBlock may be accessed on different Block than where created in MethodOutput) - Finished Block and field data output in MethodOutput (particle data still not implemented) (untested at scale) Mesh - Added Index::operator[] - Added virtual Block::factory() for use by MethodOutput() Tools - Made plot_mesh.py more forgiving, supporting any text file containing "words" that look like Block names B0:0_1:00
Method - removed debugging code from MethodOutput - added support to output particle data
Output - bug fix: check for any fields or particles in MethodOutput lists before setting in DataMsg - bug fix: added missing HDF5 file close to MethodOutput - added "writing data file" monitor output to MethodOutput - Cleaning and optimizing OutputImage (incomplete) - Added trace_memory debugging to OutputImage Testing - added new-output to test-adapt-L5-P1 regression test - fixed typo in test/index.php Parameters - removed image_block_size parameter - replaced image_block_size with image_size in input files - changed image_size default from [0,0] to [512,512] Documentation - marked image_block_size parameter as depreciated Data - added FieldFace::ghost() (for debugging MethodImage) Tools - Tweaked valgrind-org to check src not source for source files Cleaning - Removed unused file control_Control.hpp - Spell-checked comment in control_stopping.cpp
Checkpoint - Adding MethodCheckpoint (incomplete) - Added Simulation_r_write_checkpoint_[output|method] - Added Main::p_[output|method]_checkpoint Data - bug fix: removede delete particle_data_ in update (deleted in destructor if needed) Parameters - renamed parameters method_output_[file|path]_name as method_[file|path]_name Initial - moved InitialHdf5 to EnzoInitialHdf5 - removed obsolete EnzoMethodHydro - fixed array initializers in EnzoInitialMusic Config - Split EnzoConfig::read() into subfunctions
@WillHicks96 there's something funny with this PR - there are 289 changed files, which doesn't align with the nominal content of the PR. Looking at the commits, this seems to include a lot of @jobordner 's WIP PR (#97 ), and probably some other stuff. Would you mind issuing a cleaner PR? |
@WillHicks96 @bwoshea I'm pretty sure all of those file changes from PR #97 will persist until that PR is merged into |
Charm - Added MsgInitial message type for EnzoInitialHdf5 - Added Block::r_initial_new_[next|continue]() - Added EnzoBlock::p_initial_hdf5_recv() barrier to start of EnzoInitialHdf5 to prevent sending data to non-existing Blocks Input - Reverted to "music" instead of "hdf5" for initial_music.incl - Updated test_cosmo.incl for "hdf5" input Cleaning - Moved TAG_LEN message tag length declaration to cello.hpp - Removed unused functions in cello namespace - Moved common code into apply_initial_() - Removed unused Simulation::r_initialize_hierarchy() Mesh - Added Block::initial() - Updated Block::apply_initial_ to support new initialization - bug fix: delay incrementing Block::index_initial_ until after Initial::enforce_block() is called Initial - Finished implementing EnzoInitialHdf5 for unigrid IC's - Added initial_new_* methods for updated initialization supporting communication (currently unigrid only) - Added calls to Block::initial_done() required for new initialization Parameters - Added Initial:new parameter to enable "new" initialization supporting communication within an initialization step (currently unigrid only). Default is false - Added Initial:hdf5:max_level maximum level for HDF5 initialization (untested over 0 (default)) - Added Initial:hdf5:format for "enzo", "music", or "inits" (only "music" supported) - Added Initial:hdf5:blocking analagous to MethodOutput for EnzoInitialHdf5 - Added Initial:hdf5:max_level - Removed initial_hdf5 parameters from Config (initial_hdf5 parameters are in EnzoConfig) Disk - Added File::file_read_scalar for non-array meta-data
@WillHicks96 Given that there are so many files modified (with the inclusion of files from PR #97 ), could you comment on the files that you modified for this specific PR? |
@bwoshea @mabruzzo Oh yes, I can definitely see how this is confusing--I added these changes on top of @jobordner's branch that includes the commits from PR #97, so the hundreds of file changes associated with that PR should go away once it goes through. The changes I made are limited only to the following 4 files: |
Just a heads up: Because PR #97 branched off from the main branch so long ago, this doesn't seem to include some improvements that I previously introduced to It might take a little work to resolve these differences. I'm happy to help, if you run into any problems. |
Build - Removed checking for VERSION file in Charm++ for compatability with Charm++ v7.0.0 Charm - Updated FieldMsg to include child_index() Solver - Added buffering and synchronization for "self" for restrict in DD (already in prolong)
Solver - Added buffering and synchronization for "self" for restrict in Mg0 - Global change NULL -> nullptr in EnzoSolverMg0
Charm - Changed MsgInitial::data_delete_ default from false to true to address memory leak Parameters - Added missing Grackle chemistry_data attributes to operator|(PUP) in enzo_EnzoConfig.hpp (values are already in enzo-project/main) Grackle - Removed trailing _ from parameter and local variables: reserved by convention for private member attributes - Changed "*c = _set_default_chemistry_parameters()" call to "set_default_chemistry_parameters(c)" to avoid unnecessary copy
Input - Added velocities to "conserved" field group (needed for momentum) - Updated Testing : time_final for checkpoint_ppm tests for updates to "conserved" field group Charm - Added MsgOutput::operator=() to complete "the big three" (all-or-none of destructor, assignment operator, and copy constructor); moved common code into MsgOutput::copy_() Stopping - Added printing of counts of remaining messages for MsgInitial and MsgOutput - Set DataMsg::face_fluxes_delete_[] to true in load_data() instead of copying since locally created - Delete face_fluxes_list_ elements explicitly instead of clearing list Data - Replace fluxes_ std::vector with delete_fluxes_ and raw fluxes_ array instead for pointing into single-array allocation by FluxData - updated FaceFluxes and FluxData (partial) data methods (data_size(), save_data(), load_data()) with cello macros - added FaceFluxes::print() for debugging - add missing initializers to FaceFluxes constructor - completed FaceFluxes "big three" - added FluxData single_array parameter to constructor - FluxData allocates a single array for all FaceFluxes objects if single_array parameter is true - Updated FaceFluxes::get_size() to return total size and allow optional arguments - Added FaceFluxes::copy_() to avoid common code in big three Output - Bug fix: image_ghost=true parameter to OutputImage modified root_size parameter: added const and use copy for input Parameters root_size and root_blocks Enzo Compute - added single_flux_array parameter to EnzoMethodPpm, SolveHydroEquations - updated SolveHydroEquations for single_flux_array Parameters - added method_flux_correct_single_array (Method : flux_correct : single_array) parameter
Parameters - Removed check on memory usage in test_cosmo.incl: was failing with false positive in SMP mode Data - Added mutex node lock for FieldFace for field scaling by density (fixes sporadic race condition errors in SMP mode) Memory - Replaced free() and malloc() with std::free() and std::malloc() in new/delete override Initial - Added mutex node lock for InitialValue (fixes threading errors in SMP mode related to evaluating parameter expressions ) Cello - Added bounds checking to MaskPng class (failed in SMP mode) Method - Removed debug code from pgas2d_dual.F
Input - adding group_list = ["has_mass"] for merging with enzo-project/main Charm - Converting public attributes to private in MsgOutput Adapt - Updated adapt to not delay delete (bug workaround for old Charm++) which fixes SMP bug on Frontera (adapt-L5 crashes at first coarsen in cycle 291) - removed Block : delete_ attribute Checkpoint - Added optional argument to CkStartCheckpoint for number of simultaneous writers (1) Method - Only add acceleration Field in EnzoMethodGravity if rank sufficiently large (bug fix for enzo-prolong + new-output merge) Solver - reordered i_msg_* attributes to address -Wall out-of-order initialization warnings Testing - Fixed test/index.php prolong_linear bug
… multispecies_cosmology
dadecfe
to
90e9f6b
Compare
1987588
to
4a56c5d
Compare
Build - added "new_adapt" variable to define NEW_ADAPT - added default for CELLO_VAR in config files Adapt - Added quiescence detection (QD) after doneInserting() call - Added p_adapt_update() entry methods (Main and Block) for additional QD - Changed Block::adapt_ to local variable Checkpoint - Added single-thread to CkStartCheckpoint (should be generalized with parameter) Cleaning - Removed unused r_adapt_*() entry methods Monitor - Added NEW_ADAPT to Monitor::header() Testing - Fixed Balance tests to include parallel_arg
… multispecies_cosmology
Hi Will, now that PR #97 has been merged, what is the status of this PR? |
This contains changes necessary for using Grackle with cosmology problems (working out of @jobordner's
enzo-prolong
branch). The changes are as follows:enzo_SolveHydroEquations.cpp
, wherecolindex
was initialized as NULL and never updated, which caused segfaults when doing color field advection.enzo_EnzoInitialCosmology.cpp
initialize_grackle_chemistry_data()
andsetup_grackle_units()
happen beforeenzo_units
are set to comoving