SWIFT stuck on first step

[titanbin]

Hello, I am trying to run SWIFT on an isolated galaxy setup on the TGCC cluster, but after running the steps before starting the configuration, it seems to be stuck forever after this message in the log :

[00009.8] main: Running on 2500000 gas particles, 0 sink particles, 13000000 stars particles 0 black hole particles, 0 neutrino particles, and 4998152 DM particles (20498152 gravity particles)
[00009.8] main: from t=0.000e+00 until t=1.000e+01 with 1 ranks, 16 threads / rank and 128 task queues / rank (dt_min=1.000e-09, dt_max=1.000e-02)...

My config setup

--with-fftw=/ccc/products/fftw3-3.3.9/intel--20.0.0__openmpi--4.0.1/default \
--enable-parallel-hdf5 \
--enable-optimization=yes \
--enable-mpi-mesh-gravity \
--with-hydro=sphenix \
--with-cooling=EAGLE \
--with-chemistry=EAGLE \
--with-feedback=EAGLE \
--with-entropy-floor=EAGLE \
--with-stars=EAGLE \
--with-star-formation=EAGLE \
--with-tracers=EAGLE \
--with-tbbmalloc \
--with-parmetis=/ccc/products/parmetis-4.0.3/intel--20.0.0__openmpi--4.0.1/default \
--with-metis=/ccc/products/metis-5.1.0/intel--20.0.0/default \
--with-numa=/ccc/products/numaprof-1.0.0/system/default \
--with-gsl=/ccc/products/gsl-2.6/intel--20.0.0/default

My run setup

# Use -N node
#MSUB -N 1
# Use -n MPI ranks in total
#MSUB -n 1
# -c cores per MPI rank
#MSUB -c 16
# Exclusive use of the node
#MSUB -x

set -x

ccc_mprun ../swift_eagle_2 --threads=16 --feedback --self-gravity --stars --star-formation --cooling --hydro --limiter --sync Happy2.yml 2>&1 | tee output.log

My simulation has 2.5M hydro parts, 13M star parts and 5M DM parts. I waited several hours and not a single step has completed. I'm guessing it has to do with the initial domain decomposition ?

Any idea how I can fix this ?

Thanks !

[MatthieuSchaller]

Hi,
Great that you are playing with the code.

Could you share the parameter file too?

That setup does not use MPI, so it's not going to be a domain decomposition or communication problem.

Could you run with -v 1 to get more information in the log file?

Also, any reason you set to run with 128 queues explicitly? We normally never touch that parameter and let the code decide automatically what to do (normally 1 queue / thread).

[titanbin]

Hi, Great that you are playing with the code.

Could you share the parameter file too?

That setup does not use MPI, so it's not going to be a domain decomposition or communication problem.

Could you run with -v 1 to get more information in the log file?

Also, any reason you set to run with 128 queues explicitly? We normally never touch that parameter and let the code decide automatically what to do (normally 1 queue / thread).

Thanks for the quick answer ! Here's my parameter file below, and my output.log with -v 1.

I was trying to change some parameters of the scheduler a bit blindly to try and fix the issue, so I tried changing Scheduler:nr_queues, but not touching this parameter also gives me the issue.

MetaData:
  run_name:   test

# Define the system of units to use internally.
InternalUnitSystem:
  UnitMass_in_cgs:     1.98848e43    # 10^10 M_sun in grams
  UnitLength_in_cgs:   3.08567758e21 # 1 kpc in cm
  UnitVelocity_in_cgs: 1e5           # 1 km/s in cm/s
  UnitCurrent_in_cgs:  1             # Amperes
  UnitTemp_in_cgs:     1             # Kelvin

# Parameters for the self-gravity scheme
Gravity:
  eta:          0.025                 # Constant dimensionless multiplier for time integration.
  MAC:          geometric
  theta_cr:     0.7                   # Opening angle (Multipole acceptance criterion).
  use_tree_below_softening:  1
  max_physical_DM_softening:     0.01 # Physical softening length (in internal units).
  max_physical_baryon_softening: 0.01 # Physical softening length (in internal units).

# Parameters governing the time integration (Set dt_min and dt_max to the same value for a fixed time-step run.)
TimeIntegration:
  time_begin:        0.    # The starting time of the simulation (in internal units).
  time_end:          10.0  # The end time of the simulation (in internal units).
  dt_min:            1e-9  # The minimal time-step size of the simulation (in internal units).
  dt_max:            1e-2  # The maximal time-step size of the simulation (in internal units).

# Parameters governing the snapshots
Snapshots:
  subdir:    outputs_test
  basename: output
  time_first:  0.          # Time of the first output if non-cosmological time-integration (in internal units)
  delta_time:  0.01023     # Time difference between consecutive outputs (in internal units)  0.001023 TU = 1 Myr
  compression: 4           # Compress the snapshots
  recording_triggers_part: [-1, -1]   # Not recording as we have many snapshots
  recording_triggers_bpart: [-1, -1]  # Not recording as we have many snapshots 
  
Restarts:
  enable:             1
  save:               1          # Keep copies
  onexit:             1
  subdir:             restarts_test
  basename:           restart
  delta_hours:        0.45

# Parameters governing the conserved quantities statistics
Statistics:
  delta_time:           1e-2     # Time between statistics output
  time_first:              0     # (Optional) Time of the first stats output if non-cosmological time-integration (in internal units)

# Parameters related to the initial conditions
InitialConditions:
  file_name:               SWIFT.test.hdf5
  periodic:                0            # Are we running with periodic ICs?
  stars_smoothing_length:  0.5
  shift:    [675, 675, 675]
  
# Parameters for the hydrodynamics scheme
SPH:
  resolution_eta:        1.2348   # Target smoothing length in units of the mean inter-particle separation (1.2348 == 48Ngbs with the cubic spline kernel).
  CFL_condition:         0.1      # Courant-Friedrich-Levy condition for time integration.
  h_min_ratio:           0.1      # Minimal smoothing in units of softening.
  h_max:                 10.
  minimal_temperature:   100.

# Parameters for the stars neighbour search
Stars:
  resolution_eta:        1.1642  # Target smoothing length in units of the mean inter-particle separation
  h_tolerance:           7e-3
  overwrite_birth_time:    1     # Make sure the stars in the ICs do not do any feedback
  birth_time:             -1.    # by setting all of their birth times to -1  
  timestep_age_threshold_Myr: 10.   # Age at which stars switch from young to old (in Mega-years).
  max_timestep_young_Myr:     0.1   # Maximal time-step length of young stars (in Mega-years).
  max_timestep_old_Myr:       1.0   # Maximal time-step length of old stars (in Mega-years).
  luminosity_filename:   photometry

EAGLECooling:
  dir_name:                coolingtables     # Location of the Wiersma+09 cooling tables
  H_reion_z:               7.5               # Redshift of Hydrogen re-ionization
  H_reion_eV_p_H:          2.0               # Energy inject by Hydrogen re-ionization in electron-volt per Hydrogen atom
  He_reion_z_centre:       3.5               # Redshift of the centre of the Helium re-ionization Gaussian
  He_reion_z_sigma:        0.5               # Spread in redshift of the  Helium re-ionization Gaussian
  He_reion_eV_p_H:         2.0               # Energy inject by Helium re-ionization in electron-volt per Hydrogen atom

# PS2020 cooling parameters
PS2020Cooling:
  dir_name:                UV_dust1_CR1_G1_shield1.hdf5 # Location of the Ploeckinger+20 cooling tables
  H_reion_z:               7.5               # Redshift of Hydrogen re-ionization (Planck 2018)
  H_reion_eV_p_H:          2.0
  He_reion_z_centre:       3.5               # Redshift of the centre of the Helium re-ionization Gaussian
  He_reion_z_sigma:        0.5               # Spread in redshift of the  Helium re-ionization Gaussian
  He_reion_eV_p_H:         2.0               # Energy inject by Helium re-ionization in electron-volt per Hydrogen atom
  delta_logTEOS_subgrid_properties: 0.3      # delta log T above the EOS below which the subgrid properties use Teq assumption
  rapid_cooling_threshold:          0.333333 # Switch to rapid cooling regime for dt / t_cool above this threshold.

#EAGLEChemistry:
EAGLEChemistry:
  init_abundance_metal:     0.
  init_abundance_Hydrogen:  0.752
  init_abundance_Helium:    0.248
  init_abundance_Carbon:    0.0
  init_abundance_Nitrogen:  0.0
  init_abundance_Oxygen:    0.0
  init_abundance_Neon:      0.0
  init_abundance_Magnesium: 0.0
  init_abundance_Silicon:   0.0
  init_abundance_Iron:      0.0

# EAGLE star formation parameters
EAGLEStarFormation:
  SF_threshold:                      Zdep         # Zdep (Schaye 2004) or Subgrid
  SF_model:                          PressureLaw  # PressureLaw (Schaye et al. 2008) or SchmidtLaw
  KS_normalisation:                  1.515e-4     # The normalization of the Kennicutt-Schmidt law in Msun / kpc^2 / yr.
  KS_exponent:                       1.4          # The exponent of the Kennicutt-Schmidt law.
  min_over_density:                  100.0        # The over-density above which star-formation is allowed.
  KS_high_density_threshold_H_p_cm3: 1e3          # Hydrogen number density above which the Kennicut-Schmidt law changes slope in Hydrogen atoms per cm^3.
  KS_high_density_exponent:          2.0          # Slope of the Kennicut-Schmidt law above the high-density threshold.
  EOS_entropy_margin_dex:            0.3          # When using Z-based SF threshold, logarithm base 10 of the maximal entropy above the EOS at which stars can form.
  threshold_norm_H_p_cm3:            0.1          # When using Z-based SF threshold, normalisation of the metal-dependant density threshold for star formation in Hydrogen atoms per cm^3.
  threshold_Z0:                      0.002        # When using Z-based SF threshold, reference metallicity (metal mass fraction) for the metal-dependant threshold for star formation.
  threshold_slope:                   -0.64        # When using Z-based SF threshold, slope of the metal-dependant star formation threshold
  threshold_max_density_H_p_cm3:     10.0         # When using Z-based SF threshold, maximal density of the metal-dependant density threshold for star formation in Hydrogen atoms per cm^3.
  threshold_temperature1_K:          1000         # When using subgrid-based SF threshold, subgrid temperature below which gas is star-forming.
  threshold_temperature2_K:          31622        # When using subgrid-based SF threshold, subgrid temperature below which gas is star-forming if also above the density limit.
  threshold_number_density_H_p_cm3:  10           # When using subgrid-based SF threshold, subgrid number density above which gas is star-forming if also below the second temperature limit.
  
# Parameters for the EAGLE "equation of state"
EAGLEEntropyFloor:
  Jeans_density_threshold_H_p_cm3: 1e-4      # Physical density above which the EAGLE Jeans limiter entropy floor kicks in expressed in Hydrogen atoms per cm^3.
  Jeans_over_density_threshold:    10.       # Overdensity above which the EAGLE Jeans limiter entropy floor can kick in.
  Jeans_temperature_norm_K:        800       # Temperature of the EAGLE Jeans limiter entropy floor at the density threshold expressed in Kelvin.
  Jeans_gamma_effective:           1.3333333 # Slope the of the EAGLE Jeans limiter entropy floor
  Cool_density_threshold_H_p_cm3: 1e-5       # Physical density above which the EAGLE Cool limiter entropy floor kicks in expressed in Hydrogen atoms per cm^3.
  Cool_over_density_threshold:    10.        # Overdensity above which the EAGLE Cool limiter entropy floor can kick in.
  Cool_temperature_norm_K:        10.        # Temperature of the EAGLE Cool limiter entropy floor at the density threshold expressed in Kelvin. (NOTE: This is below the min T and hence this floor does nothing)
  Cool_gamma_effective:           1.         # Slope the of the EAGLE Cool limiter entropy floor

# EAGLE feedback model with constant feedback energy fraction
EAGLEFeedback:
  use_SNII_feedback:                    1               # Global switch for SNII thermal (stochastic) feedback.
  use_SNIa_feedback:                    1               # Global switch for SNIa thermal (continuous) feedback.
  use_AGB_enrichment:                   1               # Global switch for enrichement from AGB stars.
  use_SNII_enrichment:                  1               # Global switch for enrichement from SNII stars.
  use_SNIa_enrichment:                  1               # Global switch for enrichement from SNIa stars.
  filename:                             yieldtables/  # Path to the directory containing the EAGLE yield tables.
  IMF_min_mass_Msun:                    0.1             # Minimal stellar mass considered for the Chabrier IMF in solar masses.
  IMF_max_mass_Msun:                  100.0             # Maximal stellar mass considered for the Chabrier IMF in solar masses.
  SNII_min_mass_Msun:                   8.0             # Minimal mass considered for SNII stars in solar masses.
  SNII_max_mass_Msun:                 100.0             # Maximal mass considered for SNII stars in solar masses.
  SNII_feedback_model:                  MinimumDistance # Feedback modes: Random, Isotropic, MinimumDistance, MinimumDensity
  SNII_sampled_delay:                   1               # Sample the SNII lifetimes to do feedback.
  SNII_delta_T_K:                       3.16228e7       # Change in temperature to apply to the gas particle in a SNII thermal feedback event in Kelvin.
  SNII_delta_v_km_p_s:                  200             # Velocity kick applied by the stars when doing SNII feedback (in km/s).
  SNII_energy_erg:                      1.0e51          # Energy of one SNII explosion in ergs.
  SNII_energy_fraction_function:        EAGLE           # Type of functional form to use for scaling the energy fraction with density and metallicity ('EAGLE', 'Separable', or 'Independent').
  SNII_energy_fraction_min:             1.0             # Minimal fraction of energy applied in a SNII feedback event.
  SNII_energy_fraction_max:             1.0             # Maximal fraction of energy applied in a SNII feedback event.
  SNII_energy_fraction_Z_0:             0.0012663729    # Pivot point for the metallicity dependance of the SNII energy fraction (metal mass fraction).
  SNII_energy_fraction_n_0_H_p_cm3:     1.4588          # Pivot point for the birth density dependance of the SNII energy fraction in cm^-3.
  SNII_energy_fraction_n_Z:             0.8686          # Power-law for the metallicity dependance of the SNII energy fraction.
  SNII_energy_fraction_n_n:             0.8686          # Power-law for the birth density dependance of the SNII energy fraction.
  SNII_energy_fraction_use_birth_density: 0             # Are we using the density at birth to compute f_E or at feedback time?
  SNII_energy_fraction_use_birth_metallicity: 0         # Are we using the metallicity at birth to compuote f_E or at feedback time?
  SNIa_DTD:                              Exponential
  SNIa_DTD_delay_Gyr:                   0.04            # Stellar age after which SNIa start in Gyr (40 Myr corresponds to stars ~ 8 Msun).
  SNIa_DTD_exp_timescale_Gyr:           2.0             # Time-scale of the exponential decay of the SNIa rates in Gyr.
  SNIa_DTD_exp_norm_p_Msun:             0.002           # Normalisation of the SNIa rates in inverse solar masses.
  SNIa_energy_erg:                     1.0e51           # Energy of one SNIa explosion in ergs.
  AGB_ejecta_velocity_km_p_s:          10.0             # Velocity of the AGB ejectas in km/s.
  stellar_evolution_age_cut_Gyr:        0.1             # Age in Gyr above which the enrichment is downsampled.
  stellar_evolution_sampling_rate:       10             # Number of time-steps in-between two enrichment events for a star above the age threshold.
  SNII_yield_factor_Hydrogen:           1.0             # (Optional) Correction factor to apply to the Hydrogen yield from the SNII channel.
  SNII_yield_factor_Helium:             1.0             # (Optional) Correction factor to apply to the Helium yield from the SNII channel.
  SNII_yield_factor_Carbon:             0.5             # (Optional) Correction factor to apply to the Carbon yield from the SNII channel.
  SNII_yield_factor_Nitrogen:           1.0             # (Optional) Correction factor to apply to the Nitrogen yield from the SNII channel.
  SNII_yield_factor_Oxygen:             1.0             # (Optional) Correction factor to apply to the Oxygen yield from the SNII channel.
  SNII_yield_factor_Neon:               1.0             # (Optional) Correction factor to apply to the Neon yield from the SNII channel.
  SNII_yield_factor_Magnesium:          2.0             # (Optional) Correction factor to apply to the Magnesium yield from the SNII channel.
  SNII_yield_factor_Silicon:            1.0             # (Optional) Correction factor to apply to the Silicon yield from the SNII channel.
  SNII_yield_factor_Iron:               0.5             # (Optional) Correction factor to apply to the Iron yield from the SNII channel.

output.log

 Welcome to the cosmological hydrodynamical code
    ______       _________________
   / ___/ |     / /  _/ ___/_  __/
   \__ \| | /| / // // /_   / /   
  ___/ /| |/ |/ // // __/  / /    
 /____/ |__/|__/___/_/    /_/     
 SPH With Inter-dependent Fine-grained Tasking

 Version : 2026.04
 Revision: unknown, Branch: unknown, Date: unknown
 Webpage : www.swiftsim.com

 Config. options: '--disable-compiler-warnings --disable-doxygen-doc --with-fftw=/ccc/products/fftw3-3.3.9/intel--20.0.0__openmpi--4.0.1/default --enable-parallel-hdf5 --enable-optimization=yes --enable-mpi-mesh-gravity --with-hydro=sphenix --with-cooling=EAGLE --with-chemistry=EAGLE --with-feedback=EAGLE --with-entropy-floor=EAGLE --with-stars=EAGLE --with-star-formation=EAGLE --with-tracers=EAGLE --with-tbbmalloc --with-parmetis=/ccc/products/parmetis-4.0.3/intel--20.0.0__openmpi--4.0.1/default --with-metis=/ccc/products/metis-5.1.0/intel--20.0.0/default --with-numa=/ccc/products/numaprof-1.0.0/system/default --with-gsl=/ccc/products/gsl-2.6/intel--20.0.0/default'

 Compiler: ICC, Version: 19.10.20191121
 CFLAGS  : '-flto -O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -funroll-loops -march=znver2 -mavx2 -pthread'

 HDF5 library version     : 1.12.0
 FFTW library version     : 3.x (details not available)
 GSL library version      : 2.6

engine_numa_policies: NUMA nodes in use: [0 . . . . . . . ]
[00000.0] main: CPU frequency used for tick conversion: 2599988844 Hz
[00000.0] main: Running on: irene4220
[00000.0] main: sizeof(part)          is  256 bytes.
[00000.0] main: sizeof(xpart)         is  192 bytes.
[00000.0] main: sizeof(sink)          is   96 bytes.
[00000.0] main: sizeof(spart)         is  480 bytes.
[00000.0] main: sizeof(bpart)         is  224 bytes.
[00000.0] main: sizeof(gpart)         is   96 bytes.
[00000.0] main: sizeof(gpart_foreign) is   40 bytes.
[00000.0] main: sizeof(multipole)     is  192 bytes.
[00000.0] main: sizeof(grav_tensor)   is  144 bytes.
[00000.0] main: sizeof(task)          is   96 bytes.
[00000.0] main: sizeof(cell)          is 1120 bytes.
[00000.0] main: Reading runtime parameters from file 'Happy2.yml'
[00000.0] main: Internal unit system: U_M = 1.988480e+43 g.
[00000.0] main: Internal unit system: U_L = 3.085678e+21 cm.
[00000.0] main: Internal unit system: U_t = 3.085678e+16 s.
[00000.0] main: Internal unit system: U_I = 1.000000e+00 A.
[00000.0] main: Internal unit system: U_T = 1.000000e+00 K.
[00000.0] phys_const_print:    Gravitational constant = 4.301069e+04
[00000.0] phys_const_print:            Speed of light = 2.997925e+05
[00000.0] phys_const_print:           Planck constant = 1.079902e-96
[00000.0] phys_const_print:        Boltzmann constant = 6.943238e-70
[00000.0] phys_const_print:     Thomson cross-section = 6.986845e-68
[00000.0] phys_const_print:             Electron-Volt = 8.057293e-66
[00000.0] phys_const_print:       Vacuum permeability = 1.950021e-33
[00000.0] phys_const_print:               Proton mass = 8.411560e-68
[00000.0] phys_const_print:                      Year = 1.022690e-09
[00000.0] phys_const_print:                    Parsec = 1.000000e-03
[00000.0] phys_const_print:         Astronomical Unit = 4.848137e-09
[00000.0] phys_const_print:              Earth radius = 2.067001e-13
[00000.0] phys_const_print:                Solar mass = 9.999648e-11
[00000.0] phys_const_print:          Solar luminosity = 5.940202e-04
[00000.0] phys_const_print:    H_0 / h = 100 km/s/Mpc = 1.000000e-01
[00000.0] phys_const_print:                    T_CMB0 = 2.725500e+00
[00000.5] feedback_props_init: Feedback model is EAGLE (EAGLE)
[00000.5] feedback_props_init: Feedback energy fraction min=1.000000, max=1.000000
[00000.5] feedback_props_init: Feedback energy fraction powers: n_n=0.868600, n_Z=0.868600
[00000.5] feedback_props_init: Feedback energy fraction widths: s_n=0.499994, s_Z=0.499994
[00000.5] feedback_props_init: Feedback energy fraction pivots: Z_0=0.001266, n_0_cgs=1.458800
[00000.5] get_cooling_redshifts: Found cooling tables at 49 redhsifts
[00000.6] cooling_print_backend: Cooling function is 'EAGLE'.
[00000.6] starformation_print_backend: Star formation model is EAGLE
[00000.6] starformation_print_backend: Density threshold follows Schaye (2004)
[00000.6] starformation_print_backend: the normalization of the density threshold is given by 1.000000e-01 #/cm^3, with metallicity slope of -6.400000e-01, and metallicity normalization of 2.000000e-03, the maximum density threshold is given by 1.000000e+01 #/cm^3
[00000.6] starformation_print_backend: Temperature threshold is given by Dalla Vecchia and Schaye (2012)
[00000.6] starformation_print_backend: The temperature threshold offset from the EOS is given by: 3.000000e-01 dex
[00000.6] starformation_print_backend: Star formation law is a pressure law (Schaye & Dalla Vecchia 2008): 
[00000.6] starformation_print_backend: With properties: normalization = 1.515000e-04 Msun/kpc^2/yr, slope of theKennicutt-Schmidt law = 1.400000e+00 and gas fraction = 1.000000e+00 
[00000.6] starformation_print_backend: At densities of 1.000000e+03 H/cm^3 the slope changes to 2.000000e+00.
[00000.6] starformation_print_backend: Running with a direct conversion density of: 3.402823e+38 #/cm^3
[00000.6] starformation_print_backend: Will form 1 stars per gas particle.
[00000.6] chemistry_print_backend: Chemistry model is 'EAGLE' tracking 9 elements.
[00000.6] main: Reading ICs from file '/ccc/store/cont003/gen10154/bointris/V3/SWIFT.V3.hdf5'
[00000.7] io_read_unit_system: Reading IC units from ICs.
[00000.7] read_ic_single: Conversion needed from:
[00000.7] read_ic_single: (ICs) Unit system: U_M =      1.988480e+43 g.
[00000.7] read_ic_single: (ICs) Unit system: U_L =      3.085678e+21 cm.
[00000.7] read_ic_single: (ICs) Unit system: U_t =      3.085678e+16 s.
[00000.7] read_ic_single: (ICs) Unit system: U_I =      1.000000e+00 A.
[00000.7] read_ic_single: (ICs) Unit system: U_T =      1.000000e+00 K.
[00000.7] read_ic_single: to:
[00000.7] read_ic_single: (internal) Unit system: U_M = 1.988480e+43 g.
[00000.7] read_ic_single: (internal) Unit system: U_L = 3.085678e+21 cm.
[00000.7] read_ic_single: (internal) Unit system: U_t = 3.085678e+16 s.
[00000.7] read_ic_single: (internal) Unit system: U_I = 1.000000e+00 A.
[00000.7] read_ic_single: (internal) Unit system: U_T = 1.000000e+00 K.
[00000.7] ic_info_read_hdf5: Metadata group ICs_parameters not found in ICs file
[00008.0] main: Reading initial conditions took 7372.816 ms.
[00008.0] main: Read 2500000 gas particles, 0 sink particles, 13000000 star particles, 0 black hole particles, 4998152 DM particles, 0 DM background particles, and 0 neutrino DM particles from the ICs.
[00008.0] space_init: max_size set to 8000000 split_size set to 400
[00008.0] space_init: subdepth_grav set to 4
[00008.0] space_init: sub_size_pair_hydro set to 256000000, sub_size_self_hydro set to 32000
[00008.0] space_init: sub_size_pair_grav set to 256000000, sub_size_self_grav set to 32000
[00008.0] space_init: Imposing a star smoothing length of 5.000000e-01
[00008.0] space_init: Shifting particles by [6.750000e+02 6.750000e+02 6.750000e+02]
[00008.9] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[00008.9] space_regrid: set cell dimensions to [ 12 12 12 ].
[00008.9] space_regrid: took 80.586 ms.
[00009.1] main: space_init took 1087.520 ms.
[00009.1] potential_print_backend: External potential is 'No external potential'.
[00009.1] forcing_terms_print: Forcing terms is 'No forcing terms'.
[00009.1] main: space dimensions are [ 1200.000 1200.000 1200.000 ].
[00009.1] main: space isn't periodic.
[00009.1] main: highest-level cell dimensions are [ 12 12 12 ].
[00009.1] main: 2500000 parts in 1728 cells.
[00009.1] main: 20498152 gparts in 1728 cells.
[00009.1] main: 0 sinks in 1728 cells.
[00009.1] main: 13000000 sparts in 1728 cells.
[00009.1] main: 0 bparts in 1728 cells.
[00009.1] main: maximum depth is 0.
[00009.1] main: map_cellcheck picked up 0 parts.
[00009.1] main: nr of cells at depth 0 is 1728.
[00009.1] engine_init: took 0.065 ms.
[00009.1] engine_config: Running simulation 'Happy2'.
[00009.1] engine_config: Number of task queues set to 128
[00009.1] engine_config: no processor affinity used
[00009.1] engine_policy: engine policies are [  'steal'  'keep'  'numa affinity'  'hydro'  'self gravity'  'cooling'  'stars'  'star formation'  'feedback'  'time-step limiter'  'time-step sync'  ]
[00009.1] eos_print: Equation of state: Ideal gas.
[00009.1] eos_print: Adiabatic index gamma: 1.666667.
[00009.1] hydro_props_print: Hydrodynamic scheme: SPHENIX (Borrow+ 2020) in 3D.
[00009.1] hydro_props_print: Hydrodynamic kernel: Cubic spline (M4) with eta=1.234800 (48.00 neighbours).
[00009.1] hydro_props_print: Hydrodynamic relative tolerance in h: 0.00010 (+/- 0.0144 neighbours).
[00009.1] hydro_props_print: Hydrodynamic integration: CFL parameter: 0.1000.
[00009.1] hydro_props_print: Hydrodynamic integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[00009.1] hydro_props_print: Neighbour number definition: Unweighted.
[00009.1] hydro_props_print: Maximal smoothing length allowed: 10.0000
[00009.1] hydro_props_print: Maximal time-bin difference between neighbours: 2
[00009.1] hydro_props_print: Minimal gas temperature set to 100.000000
[00009.1] hydro_props_print: No particle splitting
[00009.1] viscosity_print: Artificial viscosity parameters set to alpha: 0.100, max: 2.000, min: 0.000, length: 0.050.
[00009.1] diffusion_print: Artificial diffusion parameters set to alpha: 0.000, max: 1.000, min: 0.000, beta: 1.000.
[00009.1] hydro_props_print: MHD scheme: No MHD scheme.
[00009.1] pressure_floor_print: Pressure floor is 'none'
[00009.1] entropy_floor_print: Entropy floor is 'EAGLE' with:
[00009.1] entropy_floor_print: Jeans limiter with slope n=1.333 at rho=3.286324e-07 (1.000000e-04 H/cm^3) and T=800.0 K
[00009.1] entropy_floor_print:  Cool limiter with slope n=1.000 at rho=3.286324e-08 (1.000000e-05 H/cm^3) and T=10.0 K
[00009.1] gravity_props_print: Self-gravity scheme: With per-particle softening
[00009.1] gravity_props_print: Self-gravity scheme: FMM-MM with m-poles of order 4
[00009.1] gravity_props_print: Self-gravity time integration: eta=0.0250
[00009.1] gravity_props_print: Self-gravity opening angle scheme:  fixed
[00009.1] gravity_props_print: Self-gravity opening angle:  theta_cr=0.7000
[00009.1] gravity_props_print: Self-gravity softening functional form: Wendland-C2
[00009.1] gravity_props_print: Self-gravity DM comoving softening: epsilon=0.030000 (Plummer equivalent: 0.010000)
[00009.1] gravity_props_print: Self-gravity DM maximal physical softening:    epsilon=0.030000 (Plummer equivalent: 0.010000)
[00009.1] gravity_props_print: Self-gravity baryon comoving softening: epsilon=0.030000 (Plummer equivalent: 0.010000)
[00009.1] gravity_props_print: Self-gravity baryon maximal physical softening:    epsilon=0.030000 (Plummer equivalent: 0.010000)
[00009.1] gravity_props_print: Self-gravity neutrino DM comoving softening: epsilon=0.000000 (Plummer equivalent: 0.000000)
[00009.1] gravity_props_print: Self-gravity neutrino DM maximal physical softening:    epsilon=0.000000 (Plummer equivalent: 0.000000)
[00009.1] gravity_props_print: Self-gravity mesh side-length: N=0
[00009.1] gravity_props_print: Self-gravity mesh smoothing-scale: a_smooth=0.000000
[00009.1] gravity_props_print: Self-gravity distributed mesh enabled: 0
[00009.1] gravity_props_print: Self-gravity tree cut-off ratio: r_cut_max=0.000000
[00009.1] gravity_props_print: Self-gravity truncation cut-off ratio: r_cut_min=0.000000
[00009.1] gravity_props_print: Self-gravity mesh truncation function: Gadget-like (using erfc())
[00009.1] gravity_props_print: Self-gravity tree update frequency: f=0.010000
[00009.1] stars_props_print: Stars kernel: Cubic spline (M4) with eta=1.164200 (40.22 neighbours).
[00009.1] stars_props_print: Stars relative tolerance in h: 0.00700 (+/- 0.8506 neighbours).
[00009.1] stars_props_print: Stars integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[00009.1] stars_props_print: Maximal iterations in ghost task set to 30
[00009.1] stars_props_print: Stars' birth time read from the ICs will be overwritten to -1.000000
[00009.1] stars_props_print: Stars' age threshold for unlimited dt: 0.000000e+00 [U_t]
[00009.1] stars_props_print: Stars' young/old age threshold: 1.022712e-02 [U_t]
[00009.1] stars_props_print: Max time-step size of young stars: 1.022712e-04 [U_t]
[00009.1] stars_props_print: Max time-step size of old stars: 1.022712e-03 [U_t]
[00009.1] engine_config: Absolute minimal timestep size: 6.938894e-17
[00009.1] engine_config: Minimal timestep size (on time-line): 5.820766e-10
[00009.1] engine_config: Maximal timestep size (on time-line): 9.765625e-03
[00009.2] engine_compute_next_snapshot_time: Next snapshot time set to t=1.023000e-02.
[00009.2] engine_compute_next_statistics_time: Next output time for stats set to t=1.000000e-02.
[00009.2] engine_config: Restarts will be dumped every 0.450000 hours
[00009.2] engine_config: Restarts will be dumped after the final step
[00009.2] engine_config: Using 16 threads in the thread-pool
[00009.2] engine_estimate_nr_tasks: tasks per cell estimated as: 236.00, maximum tasks: 407808
[00009.2] engine_config: runner 0 using qid=0 no cpuid.
[00009.2] engine_config: runner 1 using qid=8 no cpuid.
[00009.2] engine_config: runner 2 using qid=16 no cpuid.
[00009.2] engine_config: runner 3 using qid=24 no cpuid.
[00009.2] engine_config: runner 4 using qid=32 no cpuid.
[00009.2] engine_config: runner 5 using qid=40 no cpuid.
[00009.2] engine_config: runner 6 using qid=48 no cpuid.
[00009.2] engine_config: runner 7 using qid=56 no cpuid.
[00009.2] engine_config: runner 8 using qid=64 no cpuid.
[00009.2] engine_config: runner 9 using qid=72 no cpuid.
[00009.2] engine_config: runner 10 using qid=80 no cpuid.
[00009.2] engine_config: runner 11 using qid=88 no cpuid.
[00009.2] engine_config: runner 12 using qid=96 no cpuid.
[00009.2] engine_config: runner 13 using qid=104 no cpuid.
[00009.2] engine_config: runner 14 using qid=112 no cpuid.
[00009.2] engine_config: runner 15 using qid=120 no cpuid.
[00009.2] engine_config: took 108.353 ms.
[00009.2] main: Running on 2500000 gas particles, 0 sink particles, 13000000 stars particles 0 black hole particles, 0 neutrino particles, and 4998152 DM particles (20498152 gravity particles)
[00009.2] main: from t=0.000e+00 until t=1.000e+01 with 1 ranks, 16 threads / rank and 128 task queues / rank (dt_min=1.000e-09, dt_max=1.000e-02)...
[00009.2] engine_init_particles: Setting particles to a valid state...
[00009.3] space_first_init_parts: took 79.837 ms.
[00009.5] space_first_init_gparts: took 179.309 ms.
[00009.8] space_first_init_sparts: took 389.524 ms.
[00009.8] space_first_init_bparts: took 0.001 ms.
[00009.8] space_first_init_sinks: took 0.000 ms.
[00009.8] engine_first_init_particles: took 648.734 ms.
[00009.8] engine_init_particles: Computing initial gas densities and approximate gravity.
[00009.8] scheduler_report_task_times: took 0.001 ms.
[00009.8] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[00009.8] space_free_cells: took 0.184 ms.
[00009.8] space_regrid: took 0.193 ms.
[00009.8] space_allocate_extras: Currently have 2500000/20498152/0/13000000/0 real particles.
[00009.8] space_allocate_extras: Currently have 0/0/0/0/0 spaces for extra particles.
[00009.8] space_allocate_extras: Requesting space for future 0/345600/0/345600/0 part/gpart/sinks/sparts/bparts.
[00009.8] space_allocate_extras: Re-allocating gparts array from 20498152 to 25012502
[00011.2] space_allocate_extras: Re-allocating sparts array from 13000000 to 16014720
[00014.9] space_parts_get_cell_index: took 20.922 ms.
[00015.0] space_gparts_get_cell_index: took 92.573 ms.
[00015.1] space_sparts_get_cell_index: took 107.491 ms.
[00015.1] space_rebuild: Moving non-local particles took 0.020 ms.
[00015.4] space_rebuild: Have 93 local top-level cells with particles (total=1728)
[00015.4] space_rebuild: Have 1728 local top-level cells (total=1728)
[00015.4] space_rebuild: hooking up cells took 0.097 ms.
[00026.4] space_split: Max tree depth after split: 10
[00026.4] space_split: took 10875.275 ms.
[00026.4] space_rebuild: took 16560.305 ms.
[00026.4] engine_rebuild: Nr. of top-level cells: 1728 Nr. of local cells: 203955 memory use: 219 MB.
[00026.4] engine_rebuild: Nr. of top-level mpoles: 1728 Nr. of local mpoles: 203955 memory use: 81 MB.
[00026.4] engine_rebuild: Space has memory for 2500000/25012502/16014720/0/0 part/gpart/spart/sink/bpart (610/2289/7330/0/0 MB)
[00026.4] engine_rebuild: Space holds 2500000/20843752/13345600/0/0 part/gpart/spart/sink/bpart (fracs: 1.000000/0.833333/0.833333/0.000000/0.000000)
[00026.4] engine_rebuild: updating particle counts took 0.000 ms.
[00026.4] engine_estimate_nr_tasks: tasks per cell estimated as: 15.00, maximum tasks: 3059325
[00026.4] engine_maketasks: Making hydro tasks took 0.296 ms.
[00026.4] engine_maketasks: Making gravity tasks took 0.750 ms.
[00026.4] scheduler_splittasks: space_subsize_self_hydro= 32000
[00026.4] scheduler_splittasks: space_subsize_pair_hydro= 256000000
[00026.4] scheduler_splittasks: space_subsize_self_stars= 32000
[00026.4] scheduler_splittasks: space_subsize_pair_stars= 256000000
[00026.4] scheduler_splittasks: space_subsize_self_grav= 32000
[00026.4] scheduler_splittasks: space_subsize_pair_grav= 256000000
[00026.4] engine_maketasks: Splitting tasks took 7.020 ms.
[00026.4] engine_maketasks: Counting and linking tasks took 4.101 ms.
[00026.4] engine_maketasks: Setting super-pointers took 13.673 ms.
[00026.5] engine_maketasks: Making extra hydroloop tasks took 15.929 ms.
[00026.5] engine_maketasks: Linking gravity tasks took 10.315 ms.
[00026.5] engine_maketasks: Nr. of tasks: 156603 allocated tasks: 3059325 ratio: 0.051189 memory use: 280 MB.
[00026.5] engine_maketasks: Nr. of links: 196990 allocated links: 957200 ratio: 0.205798 memory use: 14 MB.
[00026.5] engine_maketasks: Actual usage: tasks/cell: 0.767831 links/task: 1.257894
[00026.5] engine_maketasks: Setting unlocks took 12.316 ms.
[00026.5] engine_maketasks: Ranking the tasks took 5.393 ms.
[00026.5] scheduler_reweight: took 10.080 ms.
[00026.5] engine_maketasks: took 90.931 ms (including reweight).
[00026.5] space_list_useful_top_level_cells: Have 93 local top-level cells with tasks (total=1728)
[00026.5] space_list_useful_top_level_cells: Have 93 top-level cells with particles (total=1728)
[00026.5] space_list_useful_top_level_cells: took 0.079 ms.
[00026.5] engine_unskip: took 53.770 ms.
[00026.5] engine_print_task_counts: Total = 156603 (per cell = 0.77)
[00026.5] engine_print_task_counts: task counts are [ none=0 sort=15 self=5014 pair=78988 init_grav=93 init_grav_out=394 ghost_in=13 ghost=5241 ghost_out=13 extra_ghost=13 drift_part=15 drift_spart=15 drift_sink=0 drift_bpart=0 drift_gpart=93 drift_gpart_out=394 hydro_end_force=13 kick1=93 kick2=93 timestep=93 timestep_limiter=15 timestep_sync=15 collect=93 send=0 recv=0 pack=0 unpack=0 grav_long_range=93 grav_mm=1216 grav_down_in=394 grav_down=93 grav_end_force=93 cooling=592 cooling_in=13 cooling_out=13 star_formation=13 star_formation_in=0 star_formation_out=0 star_formation_sink=0 csds=0 stars_in=15 stars_out=15 stars_ghost_in=9 stars_density_ghost=45815 stars_ghost_out=9 stars_prep_ghost1=0 hydro_prep_ghost1=0 stars_prep_ghost2=0 stars_sort=9 stars_resort=13 bh_in=0 bh_out=0 bh_density_ghost=0 bh_swallow_ghost1=0 bh_swallow_ghost2=0 bh_swallow_ghost3=0 fof_self=0 fof_pair=0 fof_attach_self=0 fof_attach_pair=0 neutrino_weight=0 sink_in=0 sink_density_ghost=0 sink_ghost1=0 sink_ghost2=0 sink_out=0 rt_in=0 rt_out=0 sink_formation=0 rt_ghost1=0 rt_ghost2=0 rt_transport_out=0 rt_tchem=0 rt_advance_cell_time=0 rt_sorts=0 rt_collect_times=0 skipped=17482 ]
[00026.5] engine_print_task_counts: nr_parts = 2500000.
[00026.5] engine_print_task_counts: nr_gparts = 20843752.
[00026.5] engine_print_task_counts: nr_sink = 0.
[00026.5] engine_print_task_counts: nr_sparts = 13345600.
[00026.5] engine_print_task_counts: nr_bparts = 0.
[00026.5] engine_print_task_counts: took 0.426 ms.
[00026.5] engine_rebuild: took 16705.671 ms.
[00026.6] engine_print_task_counts: Total = 156603 (per cell = 0.77)
[00026.6] engine_print_task_counts: task counts are [ none=0 sort=15 self=2481 pair=42861 init_grav=93 init_grav_out=394 ghost_in=13 ghost=5241 ghost_out=13 extra_ghost=0 drift_part=0 drift_spart=0 drift_sink=0 drift_bpart=0 drift_gpart=0 drift_gpart_out=394 hydro_end_force=0 kick1=0 kick2=0 timestep=0 timestep_limiter=0 timestep_sync=0 collect=0 send=0 recv=0 pack=0 unpack=0 grav_long_range=93 grav_mm=1216 grav_down_in=394 grav_down=93 grav_end_force=93 cooling=0 cooling_in=13 cooling_out=13 star_formation=0 star_formation_in=0 star_formation_out=0 star_formation_sink=0 csds=0 stars_in=0 stars_out=0 stars_ghost_in=0 stars_density_ghost=0 stars_ghost_out=0 stars_prep_ghost1=0 hydro_prep_ghost1=0 stars_prep_ghost2=0 stars_sort=9 stars_resort=0 bh_in=0 bh_out=0 bh_density_ghost=0 bh_swallow_ghost1=0 bh_swallow_ghost2=0 bh_swallow_ghost3=0 fof_self=0 fof_pair=0 fof_attach_self=0 fof_attach_pair=0 neutrino_weight=0 sink_in=0 sink_density_ghost=0 sink_ghost1=0 sink_ghost2=0 sink_out=0 rt_in=0 rt_out=0 sink_formation=0 rt_ghost1=0 rt_ghost2=0 rt_transport_out=0 rt_tchem=0 rt_advance_cell_time=0 rt_sorts=0 rt_collect_times=0 skipped=103174 ]

[MatthieuSchaller]

It seems to be stuck running the tasks. Have you managed to run one of the examples that come in the code?

[titanbin]

It seems to be stuck running the tasks. Have you managed to run one of the examples that come in the code?

Yes I can ! In fact, with this parameter file + my ICs but with 10 times less particles, SWIFT is able to run fine and fast.

[titanbin]

Because SWIFT doesn't get stuck if I use less particles, I thought I'd try to run it in MPI mode on more cores, but then I get this error instead :
[0000] [00003.1] engine_maketasks.c:engine_maketasks():4064: We have particles but no hydro or gravity tasks were created.

[MatthieuSchaller]

Could you try switching on --enable-debugging-checks ?

[titanbin]

After switching on --enable-debugging-checks, this is my full output.log:

[0000] main: MPI is up and running with 4 node(s).

 Welcome to the cosmological hydrodynamical code
    ______       _________________
   / ___/ |     / /  _/ ___/_  __/
   \__ \| | /| / // // /_   / /   
  ___/ /| |/ |/ // // __/  / /    
 /____/ |__/|__/___/_/    /_/     
 SPH With Inter-dependent Fine-grained Tasking

 Version : 2026.04
 Revision: unknown, Branch: unknown, Date: unknown
 Webpage : www.swiftsim.com

 Config. options: '--disable-compiler-warnings --disable-doxygen-doc --with-fftw=/ccc/products/fftw3-3.3.9/intel--20.0.0__openmpi--4.0.1/default --enable-parallel-hdf5 --enable-optimization=yes --enable-mpi-mesh-gravity --enable-debugging-checks --with-hydro=sphenix --with-cooling=EAGLE --with-chemistry=EAGLE --with-feedback=EAGLE --with-entropy-floor=EAGLE --with-stars=EAGLE --with-star-formation=EAGLE --with-tracers=EAGLE --with-tbbmalloc --with-parmetis=/ccc/products/parmetis-4.0.3/intel--20.0.0__openmpi--4.0.1/default --with-metis=/ccc/products/metis-5.1.0/intel--20.0.0/default --with-numa=/ccc/products/numaprof-1.0.0/system/default --with-gsl=/ccc/products/gsl-2.6/intel--20.0.0/default'

 Compiler: ICC, Version: 19.10.20191121
 CFLAGS  : '-flto -O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -funroll-loops -march=znver2 -mavx2 -pthread'

 HDF5 library version     : 1.12.0
 FFTW library version     : 3.x (details not available)
 GSL library version      : 2.6
 MPI library version      : Open MPI v4.1.4 (MPI std v3.1)
 METIS library version    : 5.1.0
 ParMETIS library version : 4.0.3

[0000] engine_numa_policies: NUMA nodes in use: [0 1 . . . . . . ]
[0001] engine_numa_policies: NUMA nodes in use: [. . 2 3 . . . . ]
[0002] engine_numa_policies: NUMA nodes in use: [. . . . 4 5 . . ]
[0003] engine_numa_policies: NUMA nodes in use: [. . . . . . 6 7 ]
[0000] [00000.0] main: CPU frequency used for tick conversion: 2599995269 Hz
[0000] [00000.0] main: Rank 0 running on: irene4016
[0000] [00000.0] main: WARNING: Debugging checks activated. Code will be slower !
[0000] [00000.0] main: sizeof(part)          is  288 bytes.
[0000] [00000.0] main: sizeof(xpart)         is  192 bytes.
[0000] [00000.0] main: sizeof(sink)          is  128 bytes.
[0000] [00000.0] main: sizeof(spart)         is  480 bytes.
[0000] [00000.0] main: sizeof(bpart)         is  224 bytes.
[0000] [00000.0] main: sizeof(gpart)         is  136 bytes.
[0000] [00000.0] main: sizeof(gpart_foreign) is   48 bytes.
[0000] [00000.0] main: sizeof(multipole)     is  200 bytes.
[0000] [00000.0] main: sizeof(grav_tensor)   is  168 bytes.
[0000] [00000.0] main: sizeof(task)          is  128 bytes.
[0000] [00000.0] main: sizeof(cell)          is 1344 bytes.
[0000] [00000.0] main: Reading runtime parameters from file 'Happy2.yml'
[0000] [00000.0] main: Using ParMETIS partitioning:
[0000] [00000.0] main:   initial partitioning: memory and edge balanced cells using particle weights
[0000] [00000.0] main:   repartitioning: edge and vertex task cost weights
[0000] [00000.0] main: Internal unit system: U_M = 1.988480e+43 g.
[0000] [00000.0] main: Internal unit system: U_L = 3.085678e+21 cm.
[0000] [00000.0] main: Internal unit system: U_t = 3.085678e+16 s.
[0000] [00000.0] main: Internal unit system: U_I = 1.000000e+00 A.
[0000] [00000.0] main: Internal unit system: U_T = 1.000000e+00 K.
[0000] [00000.0] phys_const_print:    Gravitational constant = 4.301069e+04
[0000] [00000.0] phys_const_print:            Speed of light = 2.997925e+05
[0000] [00000.0] phys_const_print:           Planck constant = 1.079902e-96
[0000] [00000.0] phys_const_print:        Boltzmann constant = 6.943238e-70
[0000] [00000.0] phys_const_print:     Thomson cross-section = 6.986845e-68
[0000] [00000.0] phys_const_print:             Electron-Volt = 8.057293e-66
[0000] [00000.0] phys_const_print:       Vacuum permeability = 1.950021e-33
[0000] [00000.0] phys_const_print:               Proton mass = 8.411560e-68
[0000] [00000.0] phys_const_print:                      Year = 1.022690e-09
[0000] [00000.0] phys_const_print:                    Parsec = 1.000000e-03
[0000] [00000.0] phys_const_print:         Astronomical Unit = 4.848137e-09
[0000] [00000.0] phys_const_print:              Earth radius = 2.067001e-13
[0000] [00000.0] phys_const_print:                Solar mass = 9.999648e-11
[0000] [00000.0] phys_const_print:          Solar luminosity = 5.940202e-04
[0000] [00000.0] phys_const_print:    H_0 / h = 100 km/s/Mpc = 1.000000e-01
[0000] [00000.0] phys_const_print:                    T_CMB0 = 2.725500e+00
[0000] [00000.9] feedback_props_init: Feedback model is EAGLE (EAGLE)
[0000] [00000.9] feedback_props_init: Feedback energy fraction min=1.000000, max=1.000000
[0000] [00000.9] feedback_props_init: Feedback energy fraction powers: n_n=0.868600, n_Z=0.868600
[0000] [00000.9] feedback_props_init: Feedback energy fraction widths: s_n=0.499994, s_Z=0.499994
[0000] [00000.9] feedback_props_init: Feedback energy fraction pivots: Z_0=0.001266, n_0_cgs=1.458800
[0000] [00000.9] get_cooling_redshifts: Found cooling tables at 49 redhsifts
[0000] [00001.0] cooling_print_backend: Cooling function is 'EAGLE'.
[0000] [00001.0] starformation_print_backend: Star formation model is EAGLE
[0000] [00001.0] starformation_print_backend: Density threshold follows Schaye (2004)
[0000] [00001.0] starformation_print_backend: the normalization of the density threshold is given by 1.000000e-01 #/cm^3, with metallicity slope of -6.400000e-01, and metallicity normalization of 2.000000e-03, the maximum density threshold is given by 1.000000e+01 #/cm^3
[0000] [00001.0] starformation_print_backend: Temperature threshold is given by Dalla Vecchia and Schaye (2012)
[0000] [00001.0] starformation_print_backend: The temperature threshold offset from the EOS is given by: 3.000000e-01 dex
[0001] [00000.9] get_cooling_redshifts: Found cooling tables at 49 redhsifts
[0002] [00000.9] get_cooling_redshifts: Found cooling tables at 49 redhsifts
[0003] [00000.9] get_cooling_redshifts: Found cooling tables at 49 redhsifts
[0000] [00001.0] starformation_print_backend: Star formation law is a pressure law (Schaye & Dalla Vecchia 2008): 
[0000] [00001.0] starformation_print_backend: With properties: normalization = 1.515000e-04 Msun/kpc^2/yr, slope of theKennicutt-Schmidt law = 1.400000e+00 and gas fraction = 1.000000e+00 
[0000] [00001.0] starformation_print_backend: At densities of 1.000000e+03 H/cm^3 the slope changes to 2.000000e+00.
[0000] [00001.0] starformation_print_backend: Running with a direct conversion density of: 3.402823e+38 #/cm^3
[0000] [00001.0] starformation_print_backend: Will form 1 stars per gas particle.
[0000] [00001.0] chemistry_print_backend: Chemistry model is 'EAGLE' tracking 9 elements.
[0000] [00001.0] main: Reading ICs from file '/ccc/store/cont003/gen10154/bointris/summer_26/SWIFT.V4.hdf5'
[0000] [00001.0] io_read_unit_system: Reading IC units from ICs.
[0000] [00001.0] read_ic_parallel: Conversion needed from:
[0000] [00001.0] read_ic_parallel: (ICs) Unit system: U_M =      1.988480e+43 g.
[0000] [00001.0] read_ic_parallel: (ICs) Unit system: U_L =      3.085678e+21 cm.
[0000] [00001.0] read_ic_parallel: (ICs) Unit system: U_t =      3.085678e+16 s.
[0000] [00001.0] read_ic_parallel: (ICs) Unit system: U_I =      1.000000e+00 A.
[0000] [00001.0] read_ic_parallel: (ICs) Unit system: U_T =      1.000000e+00 K.
[0000] [00001.0] read_ic_parallel: to:
[0000] [00001.0] read_ic_parallel: (internal) Unit system: U_M = 1.988480e+43 g.
[0000] [00001.0] read_ic_parallel: (internal) Unit system: U_L = 3.085678e+21 cm.
[0000] [00001.0] read_ic_parallel: (internal) Unit system: U_t = 3.085678e+16 s.
[0000] [00001.0] read_ic_parallel: (internal) Unit system: U_I = 1.000000e+00 A.
[0000] [00001.0] read_ic_parallel: (internal) Unit system: U_T = 1.000000e+00 K.
[0000] [00001.0] ic_info_read_hdf5: Metadata group ICs_parameters not found in ICs file
[0000] [00001.5] main: Reading initial conditions took 562.758 ms.
[0000] [00001.6] part_verify_links: All links OK
[0000] [00001.6] part_verify_links: took 18.433 ms.
[0000] [00001.6] main: Read 250000 gas particles, 0 sink particles, 1300000 star particles, 0 black hole particles, 499828 DM particles, 0 DM background particles, and 0 neutrino DM particles from the ICs.
[0000] [00001.6] space_init: max_size set to 8000000 split_size set to 400
[0000] [00001.6] space_init: subdepth_grav set to 4
[0000] [00001.6] space_init: sub_size_pair_hydro set to 256000000, sub_size_self_hydro set to 32000
[0000] [00001.6] space_init: sub_size_pair_grav set to 256000000, sub_size_self_grav set to 32000
[0000] [00001.6] space_init: Imposing a star smoothing length of 5.000000e-01
[0000] [00001.6] space_init: Shifting particles by [6.750000e+02 6.750000e+02 6.750000e+02]
[0000] [00001.6] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[0000] [00001.6] space_regrid: (re)griding space cdim=(12 12 12)
[0001] [00001.6] part_verify_links: All links OK
[0001] [00001.6] part_verify_links: took 18.149 ms.
[0001] [00001.6] space_init: Imposing a star smoothing length of 5.000000e-01
[0001] [00001.6] space_init: Shifting particles by [6.750000e+02 6.750000e+02 6.750000e+02]
[0001] [00001.6] space_regrid: (re)griding space cdim=(12 12 12)
[0002] [00001.6] part_verify_links: All links OK
[0002] [00001.6] part_verify_links: took 18.120 ms.
[0002] [00001.6] space_init: Imposing a star smoothing length of 5.000000e-01
[0002] [00001.6] space_init: Shifting particles by [6.750000e+02 6.750000e+02 6.750000e+02]
[0002] [00001.6] space_regrid: (re)griding space cdim=(12 12 12)
[0003] [00001.6] part_verify_links: All links OK
[0003] [00001.6] part_verify_links: took 18.078 ms.
[0003] [00001.6] space_init: Imposing a star smoothing length of 5.000000e-01
[0003] [00001.6] space_init: Shifting particles by [6.750000e+02 6.750000e+02 6.750000e+02]
[0003] [00001.6] space_regrid: (re)griding space cdim=(12 12 12)
[0000] [00001.6] space_regrid: set cell dimensions to [ 12 12 12 ].
[0000] [00001.6] space_regrid: took 2.852 ms.
[0000] [00001.6] main: space_init took 33.473 ms.
[0000] [00001.6] potential_print_backend: External potential is 'No external potential'.
[0000] [00001.6] forcing_terms_print: Forcing terms is 'No forcing terms'.
[0000] [00001.6] main: space dimensions are [ 1200.000 1200.000 1200.000 ].
[0000] [00001.6] main: space isn't periodic.
[0000] [00001.6] main: highest-level cell dimensions are [ 12 12 12 ].
[0000] [00001.6] main: 62500 parts in 1728 cells.
[0000] [00001.6] main: 512457 gparts in 1728 cells.
[0000] [00001.6] main: 0 sinks in 1728 cells.
[0000] [00001.6] main: 325000 sparts in 1728 cells.
[0000] [00001.6] main: 0 bparts in 1728 cells.
[0000] [00001.6] main: maximum depth is 0.
[0000] [00001.6] main: map_cellcheck picked up 0 parts.
[0000] [00001.6] main: nr of cells at depth 0 is 1728.
[0000] [00001.6] engine_init: took 0.059 ms.
[0000] [00001.6] engine_config: Running simulation 'Happy2'.
[0000] [00001.6] engine_config: no processor affinity used
[0000] [00001.6] engine_policy: engine policies are [  'steal'  'keep'  'mpi'  'numa affinity'  'hydro'  'self gravity'  'cooling'  'stars'  'star formation'  'feedback'  'time-step limiter'  'time-step sync'  ]
[0000] [00001.6] eos_print: Equation of state: Ideal gas.
[0000] [00001.6] eos_print: Adiabatic index gamma: 1.666667.
[0000] [00001.6] hydro_props_print: Hydrodynamic scheme: SPHENIX (Borrow+ 2020) in 3D.
[0000] [00001.6] hydro_props_print: Hydrodynamic kernel: Cubic spline (M4) with eta=1.234800 (48.00 neighbours).
[0000] [00001.6] hydro_props_print: Hydrodynamic relative tolerance in h: 0.00010 (+/- 0.0144 neighbours).
[0000] [00001.6] hydro_props_print: Hydrodynamic integration: CFL parameter: 0.1000.
[0000] [00001.6] hydro_props_print: Hydrodynamic integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[0000] [00001.6] hydro_props_print: Neighbour number definition: Unweighted.
[0000] [00001.6] hydro_props_print: Maximal smoothing length allowed: 10.0000
[0000] [00001.6] hydro_props_print: Maximal time-bin difference between neighbours: 2
[0000] [00001.6] hydro_props_print: Minimal gas temperature set to 100.000000
[0000] [00001.6] hydro_props_print: No particle splitting
[0000] [00001.6] viscosity_print: Artificial viscosity parameters set to alpha: 0.100, max: 2.000, min: 0.000, length: 0.050.
[0000] [00001.6] diffusion_print: Artificial diffusion parameters set to alpha: 0.000, max: 1.000, min: 0.000, beta: 1.000.
[0000] [00001.6] hydro_props_print: MHD scheme: No MHD scheme.
[0000] [00001.6] pressure_floor_print: Pressure floor is 'none'
[0000] [00001.6] entropy_floor_print: Entropy floor is 'EAGLE' with:
[0000] [00001.6] entropy_floor_print: Jeans limiter with slope n=1.333 at rho=3.286324e-07 (1.000000e-04 H/cm^3) and T=800.0 K
[0000] [00001.6] entropy_floor_print:  Cool limiter with slope n=1.000 at rho=3.286324e-08 (1.000000e-05 H/cm^3) and T=10.0 K
[0000] [00001.6] gravity_props_print: Self-gravity scheme: With per-particle softening
[0000] [00001.6] gravity_props_print: Self-gravity scheme: FMM-MM with m-poles of order 4
[0000] [00001.6] gravity_props_print: Self-gravity time integration: eta=0.0250
[0000] [00001.6] gravity_props_print: Self-gravity opening angle scheme:  fixed
[0000] [00001.6] gravity_props_print: Self-gravity opening angle:  theta_cr=0.7000
[0000] [00001.6] gravity_props_print: Self-gravity softening functional form: Wendland-C2
[0000] [00001.6] gravity_props_print: Self-gravity DM comoving softening: epsilon=0.030000 (Plummer equivalent: 0.010000)
[0000] [00001.6] gravity_props_print: Self-gravity DM maximal physical softening:    epsilon=0.030000 (Plummer equivalent: 0.010000)
[0000] [00001.6] gravity_props_print: Self-gravity baryon comoving softening: epsilon=0.030000 (Plummer equivalent: 0.010000)
[0000] [00001.6] gravity_props_print: Self-gravity baryon maximal physical softening:    epsilon=0.030000 (Plummer equivalent: 0.010000)
[0000] [00001.6] gravity_props_print: Self-gravity neutrino DM comoving softening: epsilon=0.000000 (Plummer equivalent: 0.000000)
[0000] [00001.6] gravity_props_print: Self-gravity neutrino DM maximal physical softening:    epsilon=0.000000 (Plummer equivalent: 0.000000)
[0000] [00001.6] gravity_props_print: Self-gravity mesh side-length: N=0
[0000] [00001.6] gravity_props_print: Self-gravity mesh smoothing-scale: a_smooth=0.000000
[0000] [00001.6] gravity_props_print: Self-gravity distributed mesh enabled: 0
[0000] [00001.6] gravity_props_print: Self-gravity tree cut-off ratio: r_cut_max=0.000000
[0000] [00001.6] gravity_props_print: Self-gravity truncation cut-off ratio: r_cut_min=0.000000
[0000] [00001.6] gravity_props_print: Self-gravity mesh truncation function: Gadget-like (using erfc())
[0000] [00001.6] gravity_props_print: Self-gravity tree update frequency: f=0.010000
[0000] [00001.6] stars_props_print: Stars kernel: Cubic spline (M4) with eta=1.164200 (40.22 neighbours).
[0000] [00001.6] stars_props_print: Stars relative tolerance in h: 0.00700 (+/- 0.8506 neighbours).
[0000] [00001.6] stars_props_print: Stars integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[0000] [00001.6] stars_props_print: Maximal iterations in ghost task set to 30
[0000] [00001.6] stars_props_print: Stars' birth time read from the ICs will be overwritten to -1.000000
[0000] [00001.6] stars_props_print: Stars' age threshold for unlimited dt: 0.000000e+00 [U_t]
[0000] [00001.6] stars_props_print: Stars' young/old age threshold: 1.022712e-02 [U_t]
[0000] [00001.6] stars_props_print: Max time-step size of young stars: 1.022712e-04 [U_t]
[0000] [00001.6] stars_props_print: Max time-step size of old stars: 1.022712e-03 [U_t]
[0000] [00001.6] engine_config: Absolute minimal timestep size: 6.938894e-17
[0000] [00001.6] engine_config: Minimal timestep size (on time-line): 5.820766e-10
[0000] [00001.6] engine_config: Maximal timestep size (on time-line): 9.765625e-03
[0000] [00001.6] engine_compute_next_snapshot_time: Next snapshot time set to t=1.023000e-02.
[0000] [00001.6] engine_compute_next_statistics_time: Next output time for stats set to t=1.000000e-02.
[0000] [00001.6] engine_config: Restarts will be dumped every 0.450000 hours
[0000] [00001.6] engine_config: Restarts will be dumped after the final step
[0000] [00001.6] engine_config: Using 16 threads in the thread-pool
[0000] [00001.6] engine_estimate_nr_tasks: tasks per cell estimated as: 250.00, maximum tasks: 432000
[0000] [00001.6] engine_config: runner 0 using qid=0 no cpuid.
[0000] [00001.6] engine_config: runner 1 using qid=1 no cpuid.
[0000] [00001.6] engine_config: runner 2 using qid=2 no cpuid.
[0000] [00001.6] engine_config: runner 3 using qid=3 no cpuid.
[0000] [00001.6] engine_config: runner 4 using qid=4 no cpuid.
[0000] [00001.6] engine_config: runner 5 using qid=5 no cpuid.
[0000] [00001.6] engine_config: runner 6 using qid=6 no cpuid.
[0000] [00001.6] engine_config: runner 7 using qid=7 no cpuid.
[0000] [00001.6] engine_config: runner 8 using qid=8 no cpuid.
[0000] [00001.6] engine_config: runner 9 using qid=9 no cpuid.
[0000] [00001.6] engine_config: runner 10 using qid=10 no cpuid.
[0000] [00001.6] engine_config: runner 11 using qid=11 no cpuid.
[0000] [00001.6] engine_config: runner 12 using qid=12 no cpuid.
[0000] [00001.6] engine_config: runner 13 using qid=13 no cpuid.
[0000] [00001.6] engine_config: runner 14 using qid=14 no cpuid.
[0000] [00001.6] engine_config: runner 15 using qid=15 no cpuid.
[0000] [00001.6] engine_config: took 17.968 ms.
[0000] [00001.6] main: Running on 250000 gas particles, 0 sink particles, 1300000 stars particles 0 black hole particles, 0 neutrino particles, and 499828 DM particles (2049828 gravity particles)
[0000] [00001.6] main: from t=0.000e+00 until t=1.000e+01 with 4 ranks, 16 threads / rank and 16 task queues / rank (dt_min=1.000e-09, dt_max=1.000e-02)...
[0000] [00001.6] partition_count_edges: Found 37576 total edges in the adjacency graph
[0000] [00001.6] partition_count_edges: Maximum number of edges on a cell was 26
[0000] [00001.6] partition_count_edges: took 0.042 ms.
[0000] [00001.6] partition_accumulate_sizes: clipped gravity part counts of 2 cells
[0000] [00001.6] check_complete: Region 0 is not present in partition
[0000] [00001.6] check_complete: Region 2 is not present in partition
[0000] [00001.6] check_complete: Region 3 is not present in partition
[0000] [00001.6] partition_initial_partition: METIS initial partition failed, using a vectorised partition
[0000] [00001.6] partition_initial_partition: took 0.023 ms.
[0000] [00001.6] partition_initial_partition: took 45.645 ms.
[0000] [00001.6] engine_makeproxies: Looking for proxies up to 5 top-level cells away (delta_m=5 delta_p=5)
[0000] [00001.7] engine_makeproxies: took 30.111 ms.
[0000] [00001.8] engine_redistribute: 187499 of 250000 (75.00%) of particles moved
[0000] [00001.8] engine_redistribute: 1537308 of 2049828 (75.00%) of g-particles moved
[0000] [00001.8] engine_redistribute: 975000 of 1300000 (75.00%) of s-particles moved
[0000] [00002.4] part_verify_links: All links OK
[0000] [00002.4] part_verify_links: took 0.006 ms.
[0000] [00002.4] engine_redistribute: node 0 now has 0 parts, 0 sparts, 0 bparts, 0 sinks and 0 gparts in 288 cells.
[0000] [00002.4] engine_redistribute: took 679.177 ms.
[0000] [00002.4] engine_split: took 754.973 ms.
[0000] [00002.4] engine_init_particles: Setting particles to a valid state...
[0000] [00002.4] space_first_init_parts: took 0.000 ms.
[0000] [00002.4] space_first_init_gparts: took 0.000 ms.
[0000] [00002.4] space_first_init_sparts: took 0.001 ms.
[0000] [00002.4] space_first_init_bparts: took 0.000 ms.
[0000] [00002.4] space_first_init_sinks: took 0.000 ms.
[0000] [00002.4] engine_first_init_particles: took 0.013 ms.
[0000] [00002.4] engine_init_particles: Computing initial gas densities and approximate gravity.
[0000] [00002.4] scheduler_report_task_times: took 0.000 ms.
[0000] [00002.4] space_rebuild: (re)building space
[0000] [00002.5] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[0000] [00002.5] space_free_cells: took 0.489 ms.
[0000] [00002.5] space_regrid: took 155.798 ms.
[0000] [00002.5] space_allocate_extras: Currently have 0/0/0/0/0 real particles.
[0000] [00002.5] space_allocate_extras: Currently have 0/0/0/0/0 spaces for extra particles.
[0000] [00002.5] space_allocate_extras: Requesting space for future 0/57600/0/57600/0 part/gpart/sinks/sparts/bparts.
[0000] [00002.5] space_allocate_extras: Re-allocating gparts array from 0 to 69120
[0000] [00002.5] space_allocate_extras: Re-allocating sparts array from 0 to 69120
[0000] [00002.5] space_gparts_get_cell_index: took 0.554 ms.
[0000] [00002.5] space_sparts_get_cell_index: took 0.341 ms.
[0000] [00002.5] space_rebuild: Moving non-local particles took 0.166 ms.
[0000] [00002.7] engine_exchange_strays: sent out 0/0/0/0/0 parts/gparts/sparts/bparts/sinks, got 0/0/0/0/0 back.
[0000] [00002.7] engine_exchange_strays: took 125.705 ms.
[0000] [00002.7] part_verify_links: All links OK
[0000] [00002.7] part_verify_links: took 0.668 ms.
[0000] [00002.7] space_rebuild: Have 0 local top-level cells with particles (total=1728)
[0000] [00002.7] space_rebuild: Have 288 local top-level cells (total=1728)
[0000] [00002.7] space_rebuild: hooking up cells took 0.104 ms.
[0000] [00002.7] space_split: Max tree depth after split: 0
[0000] [00002.7] space_split: took 0.067 ms.
[0000] [00002.7] space_rebuild: took 339.949 ms.
[0000] [00002.7] engine_rebuild: Nr. of top-level cells: 1728 Nr. of local cells: 1728 memory use: 4 MB.
[0000] [00002.7] engine_rebuild: Nr. of top-level mpoles: 1728 Nr. of local mpoles: 1728 memory use: 1 MB.
[0000] [00002.7] engine_rebuild: Space has memory for 0/69120/69120/0/0 part/gpart/spart/sink/bpart (0/8/31/0/0 MB)
[0000] [00002.7] engine_rebuild: Space holds 0/57600/57600/0/0 part/gpart/spart/sink/bpart (fracs: 0.000000/0.833333/0.833333/0.000000/0.000000)
[0000] [00004.3] engine_rebuild: updating particle counts took 1601.330 ms.
[0000] [00004.3] part_verify_links: All links OK
[0000] [00004.3] part_verify_links: took 0.608 ms.
[0000] [00004.4] engine_exchange_top_multipoles: took 106.270 ms.
[0000] [00004.4] proxy_cells_exchange: Counting cells to send took 0.416 ms.
[0000] [00004.4] proxy_cells_exchange: Packing cells took 0.098 ms.
[0000] [00004.4] proxy_cells_exchange: Un-packing cells took 31.549 ms.
[0000] [00004.4] engine_exchange_cells: took 36.295 ms.
[0000] [00004.4] engine_estimate_nr_tasks: tasks per cell estimated as: 16.00, maximum tasks: 622848
[0000] [00004.4] engine_maketasks: Making hydro tasks took 0.072 ms.
[0000] [00004.4] engine_maketasks: Making gravity tasks took 0.520 ms.
[0000] [00004.4] engine_maketasks.c:engine_maketasks():4064: We have particles but no hydro or gravity tasks were created.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[2026-07-28T16:30:55.807] error: *** STEP 5298401.0 ON irene4016 CANCELLED AT 2026-07-28T16:30:55 DUE to SIGNAL Killed ***
srun: Job step aborted: Waiting up to 302 seconds for job step to finish.
srun: error: irene4016: task 0: Exited with exit code 255
srun: error: irene4016: tasks 1-3: Killed

The compile config script:

make clean

./configure --disable-compiler-warnings --disable-doxygen-doc \
--with-fftw=/ccc/products/fftw3-3.3.9/intel--20.0.0__openmpi--4.0.1/default \
--enable-parallel-hdf5 \
--enable-optimization=yes \
--enable-mpi-mesh-gravity \
--enable-debugging-checks \
--with-hydro=sphenix \
--with-cooling=EAGLE \
--with-chemistry=EAGLE \
--with-feedback=EAGLE \
--with-entropy-floor=EAGLE \
--with-stars=EAGLE \
--with-star-formation=EAGLE \
--with-tracers=EAGLE \
--with-tbbmalloc \
--with-parmetis=/ccc/products/parmetis-4.0.3/intel--20.0.0__openmpi--4.0.1/default \
--with-metis=/ccc/products/metis-5.1.0/intel--20.0.0/default \
--with-numa=/ccc/products/numaprof-1.0.0/system/default \
--with-gsl=/ccc/products/gsl-2.6/intel--20.0.0/default

read -n 1 -p "Input Selection:"

make -j4

The run command:

ccc_mprun swift_mpi -v 1 --threads=16 --feedback --self-gravity --stars --star-formation --cooling --hydro --limiter --sync Happy2.yml 2>&1 | tee output.log

[MatthieuSchaller]

Sorry, I meant without MPI first

[titanbin]

No problem, here's the log without MPI, it actually crashed after 10 minutes:

output.log

 Welcome to the cosmological hydrodynamical code
    ______       _________________
   / ___/ |     / /  _/ ___/_  __/
   \__ \| | /| / // // /_   / /   
  ___/ /| |/ |/ // // __/  / /    
 /____/ |__/|__/___/_/    /_/     
 SPH With Inter-dependent Fine-grained Tasking

 Version : 2026.04
 Revision: unknown, Branch: unknown, Date: unknown
 Webpage : www.swiftsim.com

 Config. options: '--disable-compiler-warnings --disable-doxygen-doc --with-fftw=/ccc/products/fftw3-3.3.9/intel--20.0.0__openmpi--4.0.1/default --enable-parallel-hdf5 --enable-optimization=yes --enable-mpi-mesh-gravity --enable-debugging-checks --with-hydro=sphenix --with-cooling=EAGLE --with-chemistry=EAGLE --with-feedback=EAGLE --with-entropy-floor=EAGLE --with-stars=EAGLE --with-star-formation=EAGLE --with-tracers=EAGLE --with-tbbmalloc --with-parmetis=/ccc/products/parmetis-4.0.3/intel--20.0.0__openmpi--4.0.1/default --with-metis=/ccc/products/metis-5.1.0/intel--20.0.0/default --with-numa=/ccc/products/numaprof-1.0.0/system/default --with-gsl=/ccc/products/gsl-2.6/intel--20.0.0/default'

 Compiler: ICC, Version: 19.10.20191121
 CFLAGS  : '-flto -O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -funroll-loops -march=znver2 -mavx2 -pthread'

 HDF5 library version     : 1.12.0
 FFTW library version     : 3.x (details not available)
 GSL library version      : 2.6

engine_numa_policies: NUMA nodes in use: [0 . . . . . . . ]
[00000.0] main: CPU frequency used for tick conversion: 2599997463 Hz
[00000.0] main: Running on: irene5487
[00000.0] main: WARNING: Debugging checks activated. Code will be slower !
[00000.0] main: sizeof(part)          is  288 bytes.
[00000.0] main: sizeof(xpart)         is  192 bytes.
[00000.0] main: sizeof(sink)          is  128 bytes.
[00000.0] main: sizeof(spart)         is  480 bytes.
[00000.0] main: sizeof(bpart)         is  224 bytes.
[00000.0] main: sizeof(gpart)         is  136 bytes.
[00000.0] main: sizeof(gpart_foreign) is   48 bytes.
[00000.0] main: sizeof(multipole)     is  200 bytes.
[00000.0] main: sizeof(grav_tensor)   is  168 bytes.
[00000.0] main: sizeof(task)          is   96 bytes.
[00000.0] main: sizeof(cell)          is 1280 bytes.
[00000.0] main: Reading runtime parameters from file 'Happy2.yml'
[00000.0] main: Internal unit system: U_M = 1.988480e+43 g.
[00000.0] main: Internal unit system: U_L = 3.085678e+21 cm.
[00000.0] main: Internal unit system: U_t = 3.085678e+16 s.
[00000.0] main: Internal unit system: U_I = 1.000000e+00 A.
[00000.0] main: Internal unit system: U_T = 1.000000e+00 K.
[00000.0] phys_const_print:    Gravitational constant = 4.301069e+04
[00000.0] phys_const_print:            Speed of light = 2.997925e+05
[00000.0] phys_const_print:           Planck constant = 1.079902e-96
[00000.0] phys_const_print:        Boltzmann constant = 6.943238e-70
[00000.0] phys_const_print:     Thomson cross-section = 6.986845e-68
[00000.0] phys_const_print:             Electron-Volt = 8.057293e-66
[00000.0] phys_const_print:       Vacuum permeability = 1.950021e-33
[00000.0] phys_const_print:               Proton mass = 8.411560e-68
[00000.0] phys_const_print:                      Year = 1.022690e-09
[00000.0] phys_const_print:                    Parsec = 1.000000e-03
[00000.0] phys_const_print:         Astronomical Unit = 4.848137e-09
[00000.0] phys_const_print:              Earth radius = 2.067001e-13
[00000.0] phys_const_print:                Solar mass = 9.999648e-11
[00000.0] phys_const_print:          Solar luminosity = 5.940202e-04
[00000.0] phys_const_print:    H_0 / h = 100 km/s/Mpc = 1.000000e-01
[00000.0] phys_const_print:                    T_CMB0 = 2.725500e+00
[00000.7] feedback_props_init: Feedback model is EAGLE (EAGLE)
[00000.7] feedback_props_init: Feedback energy fraction min=1.000000, max=1.000000
[00000.7] feedback_props_init: Feedback energy fraction powers: n_n=0.868600, n_Z=0.868600
[00000.7] feedback_props_init: Feedback energy fraction widths: s_n=0.499994, s_Z=0.499994
[00000.7] feedback_props_init: Feedback energy fraction pivots: Z_0=0.001266, n_0_cgs=1.458800
[00000.8] get_cooling_redshifts: Found cooling tables at 49 redhsifts
[00000.9] cooling_print_backend: Cooling function is 'EAGLE'.
[00000.9] starformation_print_backend: Star formation model is EAGLE
[00000.9] starformation_print_backend: Density threshold follows Schaye (2004)
[00000.9] starformation_print_backend: the normalization of the density threshold is given by 1.000000e-01 #/cm^3, with metallicity slope of -6.400000e-01, and metallicity normalization of 2.000000e-03, the maximum density threshold is given by 1.000000e+01 #/cm^3
[00000.9] starformation_print_backend: Temperature threshold is given by Dalla Vecchia and Schaye (2012)
[00000.9] starformation_print_backend: The temperature threshold offset from the EOS is given by: 3.000000e-01 dex
[00000.9] starformation_print_backend: Star formation law is a pressure law (Schaye & Dalla Vecchia 2008): 
[00000.9] starformation_print_backend: With properties: normalization = 1.515000e-04 Msun/kpc^2/yr, slope of theKennicutt-Schmidt law = 1.400000e+00 and gas fraction = 1.000000e+00 
[00000.9] starformation_print_backend: At densities of 1.000000e+03 H/cm^3 the slope changes to 2.000000e+00.
[00000.9] starformation_print_backend: Running with a direct conversion density of: 3.402823e+38 #/cm^3
[00000.9] starformation_print_backend: Will form 1 stars per gas particle.
[00000.9] chemistry_print_backend: Chemistry model is 'EAGLE' tracking 9 elements.
[00000.9] main: Reading ICs from file '/ccc/store/cont003/gen10154/bointris/V3/SWIFT.V3.hdf5'
[00000.9] io_read_unit_system: Reading IC units from ICs.
[00000.9] read_ic_single: Conversion needed from:
[00000.9] read_ic_single: (ICs) Unit system: U_M =      1.988480e+43 g.
[00000.9] read_ic_single: (ICs) Unit system: U_L =      3.085678e+21 cm.
[00000.9] read_ic_single: (ICs) Unit system: U_t =      3.085678e+16 s.
[00000.9] read_ic_single: (ICs) Unit system: U_I =      1.000000e+00 A.
[00000.9] read_ic_single: (ICs) Unit system: U_T =      1.000000e+00 K.
[00000.9] read_ic_single: to:
[00000.9] read_ic_single: (internal) Unit system: U_M = 1.988480e+43 g.
[00000.9] read_ic_single: (internal) Unit system: U_L = 3.085678e+21 cm.
[00000.9] read_ic_single: (internal) Unit system: U_t = 3.085678e+16 s.
[00000.9] read_ic_single: (internal) Unit system: U_I = 1.000000e+00 A.
[00000.9] read_ic_single: (internal) Unit system: U_T = 1.000000e+00 K.
[00000.9] ic_info_read_hdf5: Metadata group ICs_parameters not found in ICs file
[00009.2] main: Reading initial conditions took 8318.238 ms.
[00010.2] part_verify_links: All links OK
[00010.2] part_verify_links: took 695.652 ms.
[00010.2] main: Read 2500000 gas particles, 0 sink particles, 13000000 star particles, 0 black hole particles, 4998152 DM particles, 0 DM background particles, and 0 neutrino DM particles from the ICs.
[00010.2] space_init: max_size set to 8000000 split_size set to 400
[00010.2] space_init: subdepth_grav set to 4
[00010.2] space_init: sub_size_pair_hydro set to 256000000, sub_size_self_hydro set to 32000
[00010.2] space_init: sub_size_pair_grav set to 256000000, sub_size_self_grav set to 32000
[00010.2] space_init: Imposing a star smoothing length of 5.000000e-01
[00010.2] space_init: Shifting particles by [6.750000e+02 6.750000e+02 6.750000e+02]
[00011.2] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[00011.2] space_regrid: (re)griding space cdim=(12 12 12)
[00011.2] space_regrid: set cell dimensions to [ 12 12 12 ].
[00011.2] space_regrid: took 80.431 ms.
[00011.4] main: space_init took 1218.826 ms.
[00011.4] potential_print_backend: External potential is 'No external potential'.
[00011.4] forcing_terms_print: Forcing terms is 'No forcing terms'.
[00011.4] main: space dimensions are [ 1200.000 1200.000 1200.000 ].
[00011.4] main: space isn't periodic.
[00011.4] main: highest-level cell dimensions are [ 12 12 12 ].
[00011.4] main: 2500000 parts in 1728 cells.
[00011.4] main: 20498152 gparts in 1728 cells.
[00011.4] main: 0 sinks in 1728 cells.
[00011.4] main: 13000000 sparts in 1728 cells.
[00011.4] main: 0 bparts in 1728 cells.
[00011.4] main: maximum depth is 0.
[00011.4] main: map_cellcheck picked up 0 parts.
[00011.4] main: nr of cells at depth 0 is 1728.
[00011.4] engine_init: took 0.058 ms.
[00011.4] engine_config: Running simulation 'Happy2'.
[00011.4] engine_config: no processor affinity used
[00011.4] engine_policy: engine policies are [  'steal'  'keep'  'numa affinity'  'hydro'  'self gravity'  'cooling'  'stars'  'star formation'  'feedback'  'time-step limiter'  'time-step sync'  ]
[00011.4] eos_print: Equation of state: Ideal gas.
[00011.4] eos_print: Adiabatic index gamma: 1.666667.
[00011.4] hydro_props_print: Hydrodynamic scheme: SPHENIX (Borrow+ 2020) in 3D.
[00011.4] hydro_props_print: Hydrodynamic kernel: Cubic spline (M4) with eta=1.234800 (48.00 neighbours).
[00011.4] hydro_props_print: Hydrodynamic relative tolerance in h: 0.00010 (+/- 0.0144 neighbours).
[00011.4] hydro_props_print: Hydrodynamic integration: CFL parameter: 0.1000.
[00011.4] hydro_props_print: Hydrodynamic integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[00011.4] hydro_props_print: Neighbour number definition: Unweighted.
[00011.4] hydro_props_print: Maximal smoothing length allowed: 10.0000
[00011.4] hydro_props_print: Maximal time-bin difference between neighbours: 2
[00011.4] hydro_props_print: Minimal gas temperature set to 100.000000
[00011.4] hydro_props_print: No particle splitting
[00011.4] viscosity_print: Artificial viscosity parameters set to alpha: 0.100, max: 2.000, min: 0.000, length: 0.050.
[00011.4] diffusion_print: Artificial diffusion parameters set to alpha: 0.000, max: 1.000, min: 0.000, beta: 1.000.
[00011.4] hydro_props_print: MHD scheme: No MHD scheme.
[00011.4] pressure_floor_print: Pressure floor is 'none'
[00011.4] entropy_floor_print: Entropy floor is 'EAGLE' with:
[00011.4] entropy_floor_print: Jeans limiter with slope n=1.333 at rho=3.286324e-07 (1.000000e-04 H/cm^3) and T=800.0 K
[00011.4] entropy_floor_print:  Cool limiter with slope n=1.000 at rho=3.286324e-08 (1.000000e-05 H/cm^3) and T=10.0 K
[00011.4] gravity_props_print: Self-gravity scheme: With per-particle softening
[00011.4] gravity_props_print: Self-gravity scheme: FMM-MM with m-poles of order 4
[00011.4] gravity_props_print: Self-gravity time integration: eta=0.0250
[00011.4] gravity_props_print: Self-gravity opening angle scheme:  fixed
[00011.4] gravity_props_print: Self-gravity opening angle:  theta_cr=0.7000
[00011.4] gravity_props_print: Self-gravity softening functional form: Wendland-C2
[00011.4] gravity_props_print: Self-gravity DM comoving softening: epsilon=0.030000 (Plummer equivalent: 0.010000)
[00011.4] gravity_props_print: Self-gravity DM maximal physical softening:    epsilon=0.030000 (Plummer equivalent: 0.010000)
[00011.4] gravity_props_print: Self-gravity baryon comoving softening: epsilon=0.030000 (Plummer equivalent: 0.010000)
[00011.4] gravity_props_print: Self-gravity baryon maximal physical softening:    epsilon=0.030000 (Plummer equivalent: 0.010000)
[00011.4] gravity_props_print: Self-gravity neutrino DM comoving softening: epsilon=0.000000 (Plummer equivalent: 0.000000)
[00011.4] gravity_props_print: Self-gravity neutrino DM maximal physical softening:    epsilon=0.000000 (Plummer equivalent: 0.000000)
[00011.4] gravity_props_print: Self-gravity mesh side-length: N=0
[00011.4] gravity_props_print: Self-gravity mesh smoothing-scale: a_smooth=0.000000
[00011.4] gravity_props_print: Self-gravity distributed mesh enabled: 0
[00011.4] gravity_props_print: Self-gravity tree cut-off ratio: r_cut_max=0.000000
[00011.4] gravity_props_print: Self-gravity truncation cut-off ratio: r_cut_min=0.000000
[00011.4] gravity_props_print: Self-gravity mesh truncation function: Gadget-like (using erfc())
[00011.4] gravity_props_print: Self-gravity tree update frequency: f=0.010000
[00011.4] stars_props_print: Stars kernel: Cubic spline (M4) with eta=1.164200 (40.22 neighbours).
[00011.4] stars_props_print: Stars relative tolerance in h: 0.00700 (+/- 0.8506 neighbours).
[00011.4] stars_props_print: Stars integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[00011.4] stars_props_print: Maximal iterations in ghost task set to 30
[00011.4] stars_props_print: Stars' birth time read from the ICs will be overwritten to -1.000000
[00011.4] stars_props_print: Stars' age threshold for unlimited dt: 0.000000e+00 [U_t]
[00011.4] stars_props_print: Stars' young/old age threshold: 1.022712e-02 [U_t]
[00011.4] stars_props_print: Max time-step size of young stars: 1.022712e-04 [U_t]
[00011.4] stars_props_print: Max time-step size of old stars: 1.022712e-03 [U_t]
[00011.4] engine_config: Absolute minimal timestep size: 6.938894e-17
[00011.4] engine_config: Minimal timestep size (on time-line): 5.820766e-10
[00011.4] engine_config: Maximal timestep size (on time-line): 9.765625e-03
[00011.5] engine_compute_next_snapshot_time: Next snapshot time set to t=1.023000e-02.
[00011.5] engine_compute_next_statistics_time: Next output time for stats set to t=1.000000e-02.
[00011.5] engine_config: Restarts will be dumped every 0.450000 hours
[00011.5] engine_config: Restarts will be dumped after the final step
[00011.5] engine_config: Using 16 threads in the thread-pool
[00011.5] engine_estimate_nr_tasks: tasks per cell estimated as: 236.00, maximum tasks: 407808
[00011.5] engine_config: runner 0 using qid=0 no cpuid.
[00011.5] engine_config: runner 1 using qid=1 no cpuid.
[00011.5] engine_config: runner 2 using qid=2 no cpuid.
[00011.5] engine_config: runner 3 using qid=3 no cpuid.
[00011.5] engine_config: runner 4 using qid=4 no cpuid.
[00011.5] engine_config: runner 5 using qid=5 no cpuid.
[00011.5] engine_config: runner 6 using qid=6 no cpuid.
[00011.5] engine_config: runner 7 using qid=7 no cpuid.
[00011.5] engine_config: runner 8 using qid=8 no cpuid.
[00011.5] engine_config: runner 9 using qid=9 no cpuid.
[00011.5] engine_config: runner 10 using qid=10 no cpuid.
[00011.5] engine_config: runner 11 using qid=11 no cpuid.
[00011.5] engine_config: runner 12 using qid=12 no cpuid.
[00011.5] engine_config: runner 13 using qid=13 no cpuid.
[00011.5] engine_config: runner 14 using qid=14 no cpuid.
[00011.5] engine_config: runner 15 using qid=15 no cpuid.
[00011.5] engine_config: took 110.489 ms.
[00011.5] main: Running on 2500000 gas particles, 0 sink particles, 13000000 stars particles 0 black hole particles, 0 neutrino particles, and 4998152 DM particles (20498152 gravity particles)
[00011.5] main: from t=0.000e+00 until t=1.000e+01 with 1 ranks, 16 threads / rank and 16 task queues / rank (dt_min=1.000e-09, dt_max=1.000e-02)...
[00011.5] engine_init_particles: Setting particles to a valid state...
[00011.6] space_first_init_parts: took 90.112 ms.
[00011.9] space_first_init_gparts: took 247.918 ms.
[00012.3] space_first_init_sparts: took 435.092 ms.
[00012.3] space_first_init_bparts: took 0.001 ms.
[00012.3] space_first_init_sinks: took 0.000 ms.
[00012.3] engine_first_init_particles: took 773.197 ms.
[00012.3] engine_init_particles: Computing initial gas densities and approximate gravity.
[00012.3] scheduler_report_task_times: took 0.001 ms.
[00012.3] space_rebuild: (re)building space
[00012.3] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[00012.3] space_free_cells: took 0.184 ms.
[00012.3] space_regrid: took 0.195 ms.
[00012.3] space_allocate_extras: Currently have 2500000/20498152/0/13000000/0 real particles.
[00012.3] space_allocate_extras: Currently have 0/0/0/0/0 spaces for extra particles.
[00012.3] space_allocate_extras: Requesting space for future 0/345600/0/345600/0 part/gpart/sinks/sparts/bparts.
[00012.3] space_allocate_extras: Re-allocating gparts array from 20498152 to 25012502
[00014.1] space_allocate_extras: Re-allocating sparts array from 13000000 to 16014720
[00018.6] part_verify_links: All links OK
[00018.6] part_verify_links: took 686.963 ms.
[00018.6] space_parts_get_cell_index: took 21.629 ms.
[00018.7] space_gparts_get_cell_index: took 123.255 ms.
[00018.8] space_sparts_get_cell_index: took 107.397 ms.
[00018.8] space_rebuild: Moving non-local particles took 13.167 ms.
[00020.4] part_verify_links: All links OK
[00020.4] part_verify_links: took 801.085 ms.
[00020.4] space_rebuild: Have 93 local top-level cells with particles (total=1728)
[00020.4] space_rebuild: Have 1728 local top-level cells (total=1728)
[00020.4] space_rebuild: hooking up cells took 0.118 ms.
[00035.9] space_split: Max tree depth after split: 10
[00035.9] space_split: took 15176.266 ms.
[00039.2] space_rebuild: took 26930.512 ms.
[00039.2] engine_rebuild: Nr. of top-level cells: 1728 Nr. of local cells: 203955 memory use: 251 MB.
[00039.2] engine_rebuild: Nr. of top-level mpoles: 1728 Nr. of local mpoles: 203955 memory use: 87 MB.
[00039.2] engine_rebuild: Space has memory for 2500000/25012502/16014720/0/0 part/gpart/spart/sink/bpart (686/3244/7330/0/0 MB)
[00039.2] engine_rebuild: Space holds 2500000/20843752/13345600/0/0 part/gpart/spart/sink/bpart (fracs: 1.000000/0.833333/0.833333/0.000000/0.000000)
[00039.2] engine_rebuild: updating particle counts took 0.001 ms.
[00040.2] part_verify_links: All links OK
[00040.2] part_verify_links: took 1011.794 ms.
[00040.2] engine_estimate_nr_tasks: tasks per cell estimated as: 15.00, maximum tasks: 3059325
[00040.2] engine_maketasks: Making hydro tasks took 0.220 ms.
[00040.2] engine_maketasks: Making gravity tasks took 1.106 ms.
[00040.2] scheduler_splittasks: space_subsize_self_hydro= 32000
[00040.2] scheduler_splittasks: space_subsize_pair_hydro= 256000000
[00040.2] scheduler_splittasks: space_subsize_self_stars= 32000
[00040.2] scheduler_splittasks: space_subsize_pair_stars= 256000000
[00040.2] scheduler_splittasks: space_subsize_self_grav= 32000
[00040.2] scheduler_splittasks: space_subsize_pair_grav= 256000000
[00040.2] engine_maketasks: Splitting tasks took 5.906 ms.
[00040.2] engine_maketasks: Counting and linking tasks took 3.835 ms.
[00040.3] engine_maketasks: Setting super-pointers took 13.786 ms.
[00040.3] engine_maketasks: Making extra hydroloop tasks took 15.292 ms.
[00040.3] engine_maketasks: Linking gravity tasks took 9.455 ms.
[00040.3] engine_maketasks: Nr. of tasks: 156603 allocated tasks: 3059325 ratio: 0.051189 memory use: 280 MB.
[00040.3] engine_maketasks: Nr. of links: 196990 allocated links: 957200 ratio: 0.205798 memory use: 14 MB.
[00040.3] engine_maketasks: Actual usage: tasks/cell: 0.767831 links/task: 1.257894
[00041.1] engine_maketasks: Setting unlocks took 856.125 ms.
[00041.2] engine_maketasks: Ranking the tasks took 6.322 ms.
[00041.2] scheduler_reweight: took 10.892 ms.
[00041.2] engine_maketasks: took 934.083 ms (including reweight).
[00041.2] space_list_useful_top_level_cells: Have 93 local top-level cells with tasks (total=1728)
[00041.2] space_list_useful_top_level_cells: Have 93 top-level cells with particles (total=1728)
[00041.2] space_list_useful_top_level_cells: took 0.103 ms.
[00043.4] engine_unskip: took 55.615 ms.
[00043.4] engine_print_task_counts: Total = 156603 (per cell = 0.77)
[00043.4] engine_print_task_counts: task counts are [ none=0 sort=15 self=5014 pair=78988 init_grav=93 init_grav_out=394 ghost_in=13 ghost=5241 ghost_out=13 extra_ghost=13 drift_part=15 drift_spart=15 drift_sink=0 drift_bpart=0 drift_gpart=93 drift_gpart_out=394 hydro_end_force=13 kick1=93 kick2=93 timestep=93 timestep_limiter=15 timestep_sync=15 collect=93 send=0 recv=0 pack=0 unpack=0 grav_long_range=93 grav_mm=1216 grav_down_in=394 grav_down=93 grav_end_force=93 cooling=592 cooling_in=13 cooling_out=13 star_formation=13 star_formation_in=0 star_formation_out=0 star_formation_sink=0 csds=0 stars_in=15 stars_out=15 stars_ghost_in=9 stars_density_ghost=45815 stars_ghost_out=9 stars_prep_ghost1=0 hydro_prep_ghost1=0 stars_prep_ghost2=0 stars_sort=9 stars_resort=13 bh_in=0 bh_out=0 bh_density_ghost=0 bh_swallow_ghost1=0 bh_swallow_ghost2=0 bh_swallow_ghost3=0 fof_self=0 fof_pair=0 fof_attach_self=0 fof_attach_pair=0 neutrino_weight=0 sink_in=0 sink_density_ghost=0 sink_ghost1=0 sink_ghost2=0 sink_out=0 rt_in=0 rt_out=0 sink_formation=0 rt_ghost1=0 rt_ghost2=0 rt_transport_out=0 rt_tchem=0 rt_advance_cell_time=0 rt_sorts=0 rt_collect_times=0 skipped=17482 ]
[00043.4] engine_print_task_counts: nr_parts = 2500000.
[00043.4] engine_print_task_counts: nr_gparts = 20843752.
[00043.4] engine_print_task_counts: nr_sink = 0.
[00043.4] engine_print_task_counts: nr_sparts = 13345600.
[00043.4] engine_print_task_counts: nr_bparts = 0.
[00043.4] engine_print_task_counts: took 0.759 ms.
[00043.4] engine_rebuild: took 31153.678 ms.
[00043.5] engine_print_task_counts: Total = 156603 (per cell = 0.77)
[00043.5] engine_print_task_counts: task counts are [ none=0 sort=15 self=2481 pair=42861 init_grav=93 init_grav_out=394 ghost_in=13 ghost=5241 ghost_out=13 extra_ghost=0 drift_part=0 drift_spart=0 drift_sink=0 drift_bpart=0 drift_gpart=0 drift_gpart_out=394 hydro_end_force=0 kick1=0 kick2=0 timestep=0 timestep_limiter=0 timestep_sync=0 collect=0 send=0 recv=0 pack=0 unpack=0 grav_long_range=93 grav_mm=1216 grav_down_in=394 grav_down=93 grav_end_force=93 cooling=0 cooling_in=13 cooling_out=13 star_formation=0 star_formation_in=0 star_formation_out=0 star_formation_sink=0 csds=0 stars_in=0 stars_out=0 stars_ghost_in=0 stars_density_ghost=0 stars_ghost_out=0 stars_prep_ghost1=0 hydro_prep_ghost1=0 stars_prep_ghost2=0 stars_sort=9 stars_resort=0 bh_in=0 bh_out=0 bh_density_ghost=0 bh_swallow_ghost1=0 bh_swallow_ghost2=0 bh_swallow_ghost3=0 fof_self=0 fof_pair=0 fof_attach_self=0 fof_attach_pair=0 neutrino_weight=0 sink_in=0 sink_density_ghost=0 sink_ghost1=0 sink_ghost2=0 sink_out=0 rt_in=0 rt_out=0 sink_formation=0 rt_ghost1=0 rt_ghost2=0 rt_transport_out=0 rt_tchem=0 rt_advance_cell_time=0 rt_sorts=0 rt_collect_times=0 skipped=103174 ]
[00043.5] engine_print_task_counts: nr_parts = 2500000.
[00043.5] engine_print_task_counts: nr_gparts = 20843752.
[00043.5] engine_print_task_counts: nr_sink = 0.
[00043.5] engine_print_task_counts: nr_sparts = 13345600.
[00043.5] engine_print_task_counts: nr_bparts = 0.
[00043.5] engine_print_task_counts: took 0.603 ms.
[00043.5] space_init_parts: took 31.792 ms.
[00043.6] space_init_gparts: took 156.094 ms.
[00043.8] space_init_sparts: took 165.930 ms.
[00043.8] space_init_bparts: took 0.001 ms.
[00043.8] space_init_sinks: took 0.000 ms.
[00043.8] get_cooling_table: Reading cooling table 'z_0.000.hdf5'
[00043.8] get_cooling_table: Reading cooling table 'z_0.049.hdf5'
[00044.3] get_cooling_table: Done reading in general cooling table
[00205.6] engine_launch: (tasks) took 161331.123 ms.
[00205.6] engine_init_particles: Converting internal energy variable.
[00205.6] space_convert_quantities: took 25.277 ms.
[00205.6] space_post_init_parts: took 0.386 ms.
[00205.7] engine_init_particles: Running initial fake time-step.
[00205.7] scheduler_report_task_times: *** CPU time spent in different task categories:
[00205.7] scheduler_report_task_times: ***                drift:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                sorts: 20352.30 ms (0.79 %)
[00205.7] scheduler_report_task_times: ***               resort:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                hydro: 27935.46 ms (1.08 %)
[00205.7] scheduler_report_task_times: ***              gravity: 334699.25 ms (12.97 %)
[00205.7] scheduler_report_task_times: ***             feedback: 1566975.88 ms (60.70 %)
[00205.7] scheduler_report_task_times: ***          black holes:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***              cooling:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***       star formation:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***              limiter:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                 sync:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***     time integration:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                  mpi:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                 pack:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                  fof:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***               others:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***             neutrino:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                 sink:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                   RT:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***                 CSDS:     0.00 ms (0.00 %)
[00205.7] scheduler_report_task_times: ***            dead time: 631334.88 ms (24.46 %)
[00205.7] scheduler_report_task_times: ***                total: 2581297.75 ms (100.00 %)
[00205.7] scheduler_report_task_times: took 0.461 ms.
[00205.7] space_rebuild: (re)building space
[00205.7] space_regrid: h_max is 4.945e+01 (cell_min=9.931e+01).
[00205.7] space_free_cells: took 33.539 ms.
[00205.7] space_regrid: took 33.547 ms.
[00205.7] space_allocate_extras: Currently have 2500000/20498152/0/13000000/0 real particles.
[00205.7] space_allocate_extras: Currently have 0/345600/0/345600/0 spaces for extra particles.
[00205.7] space_allocate_extras: Requesting space for future 0/345600/0/345600/0 part/gpart/sinks/sparts/bparts.
[00206.7] part_verify_links: All links OK
[00206.7] part_verify_links: took 1014.112 ms.
[00206.7] space_parts_get_cell_index: took 20.561 ms.
[00206.9] space_gparts_get_cell_index: took 122.914 ms.
[00207.0] space_sparts_get_cell_index: took 107.559 ms.
[00207.0] space_rebuild: Moving non-local particles took 13.136 ms.
[00208.5] part_verify_links: All links OK
[00208.5] part_verify_links: took 1013.602 ms.
[00208.5] space_rebuild: Have 93 local top-level cells with particles (total=1728)
[00208.5] space_rebuild: Have 1728 local top-level cells (total=1728)
[00208.5] space_rebuild: hooking up cells took 0.118 ms.
[00215.5] space_split: Max tree depth after split: 10
[00215.5] space_split: took 6677.256 ms.
[00218.9] space_rebuild: took 13213.655 ms.
[00218.9] engine_rebuild: Nr. of top-level cells: 1728 Nr. of local cells: 203955 memory use: 251 MB.
[00218.9] engine_rebuild: Nr. of top-level mpoles: 1728 Nr. of local mpoles: 203955 memory use: 87 MB.
[00218.9] engine_rebuild: Space has memory for 2500000/25012502/16014720/0/0 part/gpart/spart/sink/bpart (686/3244/7330/0/0 MB)
[00218.9] engine_rebuild: Space holds 2500000/20843752/13345600/0/0 part/gpart/spart/sink/bpart (fracs: 1.000000/0.833333/0.833333/0.000000/0.000000)
[00218.9] engine_rebuild: updating particle counts took 0.000 ms.
[00219.9] part_verify_links: All links OK
[00219.9] part_verify_links: took 1015.703 ms.
[00219.9] engine_estimate_nr_tasks: tasks per cell estimated as: 15.00, maximum tasks: 3059325
[00219.9] engine_maketasks: Making hydro tasks took 0.416 ms.
[00219.9] engine_maketasks: Making gravity tasks took 0.729 ms.
[00219.9] scheduler_splittasks: space_subsize_self_hydro= 32000
[00219.9] scheduler_splittasks: space_subsize_pair_hydro= 256000000
[00219.9] scheduler_splittasks: space_subsize_self_stars= 32000
[00219.9] scheduler_splittasks: space_subsize_pair_stars= 256000000
[00219.9] scheduler_splittasks: space_subsize_self_grav= 32000
[00219.9] scheduler_splittasks: space_subsize_pair_grav= 256000000
[00219.9] engine_maketasks: Splitting tasks took 4.704 ms.
[00219.9] engine_maketasks: Counting and linking tasks took 3.142 ms.
[00219.9] engine_maketasks: Setting super-pointers took 14.030 ms.
[00219.9] engine_maketasks: Making extra hydroloop tasks took 2.142 ms.
[00220.0] engine_maketasks: Linking gravity tasks took 8.355 ms.
[00220.0] engine_maketasks: Nr. of tasks: 90599 allocated tasks: 3059325 ratio: 0.029614 memory use: 280 MB.
[00220.0] engine_maketasks: Nr. of links: 70570 allocated links: 679100 ratio: 0.103917 memory use: 10 MB.
[00220.0] engine_maketasks: Actual usage: tasks/cell: 0.444211 links/task: 0.778927
[00220.3] engine_maketasks: Setting unlocks took 343.132 ms.
[00220.3] engine_maketasks: Ranking the tasks took 1.723 ms.
[00220.3] scheduler_reweight: took 5.665 ms.
[00220.3] engine_maketasks: took 393.722 ms (including reweight).
[00220.3] space_list_useful_top_level_cells: Have 93 local top-level cells with tasks (total=1728)
[00220.3] space_list_useful_top_level_cells: Have 93 top-level cells with particles (total=1728)
[00220.3] space_list_useful_top_level_cells: took 0.076 ms.
[00222.6] engine_unskip: took 28.085 ms.
[00222.6] engine_print_task_counts: Total = 90599 (per cell = 0.44)
[00222.6] engine_print_task_counts: task counts are [ none=0 sort=15 self=1738 pair=31432 init_grav=93 init_grav_out=394 ghost_in=13 ghost=5241 ghost_out=13 extra_ghost=13 drift_part=15 drift_spart=15 drift_sink=0 drift_bpart=0 drift_gpart=93 drift_gpart_out=394 hydro_end_force=13 kick1=93 kick2=93 timestep=93 timestep_limiter=15 timestep_sync=15 collect=93 send=0 recv=0 pack=0 unpack=0 grav_long_range=93 grav_mm=1216 grav_down_in=394 grav_down=93 grav_end_force=93 cooling=592 cooling_in=13 cooling_out=13 star_formation=13 star_formation_in=0 star_formation_out=0 star_formation_sink=0 csds=0 stars_in=15 stars_out=15 stars_ghost_in=9 stars_density_ghost=45815 stars_ghost_out=9 stars_prep_ghost1=0 hydro_prep_ghost1=0 stars_prep_ghost2=0 stars_sort=9 stars_resort=13 bh_in=0 bh_out=0 bh_density_ghost=0 bh_swallow_ghost1=0 bh_swallow_ghost2=0 bh_swallow_ghost3=0 fof_self=0 fof_pair=0 fof_attach_self=0 fof_attach_pair=0 neutrino_weight=0 sink_in=0 sink_density_ghost=0 sink_ghost1=0 sink_ghost2=0 sink_out=0 rt_in=0 rt_out=0 sink_formation=0 rt_ghost1=0 rt_ghost2=0 rt_transport_out=0 rt_tchem=0 rt_advance_cell_time=0 rt_sorts=0 rt_collect_times=0 skipped=2310 ]
[00222.6] engine_print_task_counts: nr_parts = 2500000.
[00222.6] engine_print_task_counts: nr_gparts = 20843752.
[00222.6] engine_print_task_counts: nr_sink = 0.
[00222.6] engine_print_task_counts: nr_sparts = 13345600.
[00222.6] engine_print_task_counts: nr_bparts = 0.
[00222.6] engine_print_task_counts: took 0.631 ms.
[00222.6] engine_rebuild: took 16878.873 ms.
[00222.6] space_init_parts: took 31.689 ms.
[00222.8] space_init_gparts: took 155.939 ms.
[00222.9] space_init_sparts: took 165.776 ms.
[00222.9] space_init_bparts: took 0.001 ms.
[00222.9] space_init_sinks: took 0.000 ms.
[00222.9] engine_print_task_counts: Total = 90599 (per cell = 0.44)
[00222.9] engine_print_task_counts: task counts are [ none=0 sort=15 self=1738 pair=31432 init_grav=93 init_grav_out=394 ghost_in=13 ghost=5241 ghost_out=13 extra_ghost=13 drift_part=0 drift_spart=0 drift_sink=0 drift_bpart=0 drift_gpart=0 drift_gpart_out=394 hydro_end_force=13 kick1=93 kick2=93 timestep=93 timestep_limiter=15 timestep_sync=15 collect=93 send=0 recv=0 pack=0 unpack=0 grav_long_range=93 grav_mm=1216 grav_down_in=394 grav_down=93 grav_end_force=93 cooling=592 cooling_in=13 cooling_out=13 star_formation=13 star_formation_in=0 star_formation_out=0 star_formation_sink=0 csds=0 stars_in=15 stars_out=15 stars_ghost_in=9 stars_density_ghost=45815 stars_ghost_out=9 stars_prep_ghost1=0 hydro_prep_ghost1=0 stars_prep_ghost2=0 stars_sort=9 stars_resort=13 bh_in=0 bh_out=0 bh_density_ghost=0 bh_swallow_ghost1=0 bh_swallow_ghost2=0 bh_swallow_ghost3=0 fof_self=0 fof_pair=0 fof_attach_self=0 fof_attach_pair=0 neutrino_weight=0 sink_in=0 sink_density_ghost=0 sink_ghost1=0 sink_ghost2=0 sink_out=0 rt_in=0 rt_out=0 sink_formation=0 rt_ghost1=0 rt_ghost2=0 rt_transport_out=0 rt_tchem=0 rt_advance_cell_time=0 rt_sorts=0 rt_collect_times=0 skipped=2433 ]
[00222.9] engine_print_task_counts: nr_parts = 2500000.
[00222.9] engine_print_task_counts: nr_gparts = 20843752.
[00222.9] engine_print_task_counts: nr_sink = 0.
[00222.9] engine_print_task_counts: nr_sparts = 13345600.
[00222.9] engine_print_task_counts: nr_bparts = 0.
[00222.9] engine_print_task_counts: took 0.445 ms.
[00222.9] scheduler_write_dependencies: Printing task graph took 14.488 ms.
[00575.3] ./feedback/EAGLE_thermal/feedback_iact.h:runner_iact_nonsym_feedback_apply():227: Invalid fraction of material to distribute for star ID=7664460 Omega_frac=1.364168e+00 count since last enrich=-1
srun: error: irene5487: task 0: Exited with exit code 1

[MatthieuSchaller]

Ok, so it is actually trying to run.

How did you set the smoothing lengths of your stars in the ICs? I worry that they are much too large and the code takes ages to correct them in the first step.

[titanbin]

Ok, so it is actually trying to run.

How did you set the smoothing lengths of your stars in the ICs? I worry that they are much too large and the code takes ages to correct them in the first step.

I'm using smoothing lengths of 0.05 in internal units, so 50pc.

[titanbin]

I tried setting the parameter InitialConditions:cleanup_smoothing_lengths to 1, which results in a crash with the error:

[00041.7] cell.c:cell_check_spart_drift_point():899: depth_h set incorrectly! c->depth=6 p->depth_h=6 h=3.565902e-01 h_min=4.279083e-01 h_max=8.558165e-01

[titanbin]

If this can help, I have tried disabling the --feedback flag, which results in SWIFT being able to run, so I am guessing this is somehow related to feedback. These are my feedback parameters, which I think are pretty generic:

EAGLEFeedback:
  use_SNII_feedback:                    1               # Global switch for SNII thermal (stochastic) feedback.
  use_SNIa_feedback:                    1               # Global switch for SNIa thermal (continuous) feedback.
  use_AGB_enrichment:                   1               # Global switch for enrichement from AGB stars.
  use_SNII_enrichment:                  1               # Global switch for enrichement from SNII stars.
  use_SNIa_enrichment:                  1               # Global switch for enrichement from SNIa stars.
  filename:                             yieldtables/    # Path to the directory containing the EAGLE yield tables.
  IMF_min_mass_Msun:                    0.1             # Minimal stellar mass considered for the Chabrier IMF in solar masses.
  IMF_max_mass_Msun:                  100.0             # Maximal stellar mass considered for the Chabrier IMF in solar masses.
  SNII_min_mass_Msun:                   8.0             # Minimal mass considered for SNII stars in solar masses.
  SNII_max_mass_Msun:                 100.0             # Maximal mass considered for SNII stars in solar masses.
  SNII_feedback_model:           	MinimumDistance     # Feedback modes: Random, Isotropic, MinimumDistance, MinimumDensity
  SNII_sampled_delay:                   1               # Sample the SNII lifetimes to do feedback.
  SNII_delta_T_K:                       3.16228e7	    # Change in temperature to apply to the gas particle in a SNII thermal feedback event in Kelvin.
  SNII_delta_v_km_p_s:                  200             # Velocity kick applied by the stars when doing SNII feedback (in km/s).
  SNII_energy_erg:                      1.0e51          # Energy of one SNII explosion in ergs.
  SNII_energy_fraction_function:        EAGLE           # Type of functional form to use for scaling the energy fraction with density and metallicity ('EAGLE', 'Separable', or 'Independent').
  SNII_energy_fraction_min:             0.3             # Minimal fraction of energy applied in a SNII feedback event.
  SNII_energy_fraction_max:             3.0             # Maximal fraction of energy applied in a SNII feedback event.
  SNII_energy_fraction_Z_0:             0.0012663729    # Pivot point for the metallicity dependance of the SNII energy fraction (metal mass fraction).
  SNII_energy_fraction_n_0_H_p_cm3:     1.4588          # Pivot point for the birth density dependance of the SNII energy fraction in cm^-3.
  SNII_energy_fraction_n_Z:             0.8686          # Power-law for the metallicity dependance of the SNII energy fraction.
  SNII_energy_fraction_n_n:             0.8686          # Power-law for the birth density dependance of the SNII energy fraction.
  SNII_energy_fraction_use_birth_density: 0             # Are we using the density at birth to compute f_E or at feedback time?
  SNII_energy_fraction_use_birth_metallicity: 0         # Are we using the metallicity at birth to compuote f_E or at feedback time?
  SNIa_DTD:                              Exponential
  SNIa_DTD_delay_Gyr:                   0.04            # Stellar age after which SNIa start in Gyr (40 Myr corresponds to stars ~ 8 Msun).
  SNIa_DTD_exp_timescale_Gyr:           2.0             # Time-scale of the exponential decay of the SNIa rates in Gyr.
  SNIa_DTD_exp_norm_p_Msun:             0.002           # Normalisation of the SNIa rates in inverse solar masses.
  SNIa_energy_erg:                      1.0e51          # Energy of one SNIa explosion in ergs.
  AGB_ejecta_velocity_km_p_s:           10.0            # Velocity of the AGB ejectas in km/s.
  stellar_evolution_age_cut_Gyr:        0.1             # Age in Gyr above which the enrichment is downsampled.
  stellar_evolution_sampling_rate:       10	            # Number of time-steps in-between two enrichment events for a star above the age threshold.
  SNII_yield_factor_Hydrogen:           1.0	            # (Optional) Correction factor to apply to the Hydrogen yield from the SNII channel.
  SNII_yield_factor_Helium:             1.0	            # (Optional) Correction factor to apply to the Helium yield from the SNII channel.
  SNII_yield_factor_Carbon:             0.5             # (Optional) Correction factor to apply to the Carbon yield from the SNII channel.
  SNII_yield_factor_Nitrogen:           1.0             # (Optional) Correction factor to apply to the Nitrogen yield from the SNII channel.
  SNII_yield_factor_Oxygen:             1.0             # (Optional) Correction factor to apply to the Oxygen yield from the SNII channel.
  SNII_yield_factor_Neon:               1.0             # (Optional) Correction factor to apply to the Neon yield from the SNII channel.
  SNII_yield_factor_Magnesium:          2.0             # (Optional) Correction factor to apply to the Magnesium yield from the SNII channel.
  SNII_yield_factor_Silicon:            1.0             # (Optional) Correction factor to apply to the Silicon yield from the SNII channel.
  SNII_yield_factor_Iron:               0.5             # (Optional) Correction factor to apply to the Iron yield from the SNII channel.