added documentation

docs/source/Pk.rst

@@ -0,0 +1,98 @@
+Power spectra
+=============
+
+Linear power spectra
+--------------------
+
+The different folders contain both the CAMB parameter files and the matter power spectrum at z=0. In some cases transfer functions and power spectra for neutrinos, CDM, baryons, and CDM+baryons are also present. The format of the power spectrum files is
+
+- k | P(k) 
+
+where the units of k and P(k) are comoving h/Mpc and (Mpc/h)^3, respectively. For the fiducial, Om_p, Om_m, Ob_p, Ob_m, Ob2_p, Ob2_m, h_p, h_m, ns_p, ns_m, s8_p, s8_m the name of the matter power spectrum files at z=0 is 'CAMB_matterpow_0.dat'. For Mnu_p, Mnu_pp and Mnu_ppp the files are called instead 'XeV_Pm_rescaled_z0.0000.txt', where X = 0.1(Mnu_p), 0.2(Mnu_pp) and 0.4(Mnu_ppp). For the latin_hypercube simulations, the files are named 'Pk_mm_z=0.000.txt' 
+
+Notice that the matter power spectra at z=0 are not normalized (this is because the normalization is performed in the code that generates the initial conditions). The normalization factor is stored in the file Normfac.txt. One example on how to obtain the correct normalized matter power spectrum for a given cosmology is this:
+
+.. code-block:: python
+		
+    import numpy as np
+
+    f_Pk   = '/home/fvillaescusa/Quijote/Linear_Pk/ns_p/CAMB_TABLES/CAMB_matterpow_0.dat'
+    f_norm = '/home/fvillaescusa/Quijote/Linear_Pk/ns_p/Normfac.txt'
+
+    k, Pk   = np.loadtxt(f_Pk, unpack=True)
+    Normfac = np.loadtxt(f_norm)
+
+    Pk_norm = Pk*np.sqrt(Normfac)
+
+
+Non-linear power spectra
+------------------------
+
+The format of the power spectra are:
+
+- k | P(k) for power spectra in real-space
+- k | P0(k) | P2(k) | P4(k) for power spectra in redshift-space
+
+where P0(k), P2(k) and P4(k) are the monopole, quadrupole and hexadecapole, respectively. The units of k are h/Mpc, while for the power spectra are (Mpc/h)^3.
+
+In redshift-space there are three different files for each realization/redshift. These have been computed by placing the redshift-space distortions along the three different axes.
+
+In python, the files can be read as 
+
+.. code-block:: python
+		
+    import numpy as np
+
+    k, Pk = np.loadtxt('/home/fvillaescusa/Quijote/Pk/matter/fiducial/3/Pk_m_z=0.txt', unpack=True)
+    k, Pk0, Pk2, Pk4 = np.loadtxt('/home/fvillaescusa/Quijote/Pk/matter/fiducial/3/Pk_m_RS1_z=0.txt', unpack=True)
+
+
+Marked power spectra
+--------------------
+
+The files whose name starts with
+
+- Mk\_ contain marked power spectra M(k) evaluated at wavenumber k
+- Xk\_ contain the cross spectra between marked and standard density field X(k) evaluated at wavenumber k
+
+The unit of k is h/Mpc, while the one of M(k) and X(k) is (Mpc/h)^3.
+
+Files with measurements performed in the fiducial cosmology have name
+
+- Mk_fiducial0-4999\_....hdf5
+- Xk_fiducial0-4999\_....hdf5
+
+where the first numbers (in the above case 0-4999) indicate the realizations saved in the file, and the dots specify the marked model considered. 
+
+The remaining files contain measurements performed in the other cosmologies and from 500 realization per cosmology. Their name is
+
+- Mk_fTH\_....hdf5
+- Xk_fTH\_....hdf5
+
+Also in this case the dots specify the marked model considered. 
+
+In python, the files can be read as 
+
+.. code-block:: python
+		
+    import numpy as np
+
+    f = h5py.File(FILENAME, 'r')
+    k = f['k'][:]
+    # Fiducial cosmology 
+    Mk = f['i'][:]  
+    # Massive neutrino cosmologies
+    Mk = f['cosmo/i_suffix'][:]  
+    # Other cosmologies
+    Mk = f['cosmo/i'][:]  
+
+where i is the number of the realization, cosmo is the wanted cosmology and suffix can be
+
+- 'm' for the total matter field
+- 'cb' for the cold dar matter plus baryons 
+
+In order to see the name of each cosmology type
+
+.. code-block:: python
+		
+    print(list(f.keys()))

docs/source/bispectra.rst

@@ -0,0 +1,44 @@
+Bispectra
+=========
+
+The format of the individual bispectra files are:
+
+- k1/kf | k2/kf | k3/kf | P0(k1) | P0(k2) | P0(k3) | B0(k1,k2,k3) | Q(k1,k2,k3) | B_SN(k1,k2,k3) | counts 
+
+where k1, k2, k3 specify the length of the triangle sides, P0(k) is the power spectrum monopole, 
+B0(k1,k2,k3) is the bispectrum monopole, Q(k1,k2,k3) is the reduced bipsectrum, B_SN is the 
+bispectrum shot noise correction, and counts is the number of triangles in the bin. 
+B0 is already shot-noise corrected. The header specifies kf, the fundamental mode, and Nhalo, 
+the number of halos. 
+
+The individual bispectra files can be read in python as follows, 
+
+.. code-block:: python
+		
+    import numpy as np 
+
+    k1, k2, k3, p0k1, p0k2, p0k3, b123, q123, b_sn, cnts = np.loadtxt(FILENAME, skiprows=1, unpack=True, usecols=range(10))
+
+    # read header to get Nhalo 
+    hdr   = open(FILENAME).readline().rstrip()
+    Nhalo = int(hdr.split('Nhalo=')[-1])
+
+Alternatively, sets of bispectra files for a specific redshift and cosmology can easily be accessed
+
+.. code-block:: python
+		
+    import h5py 
+
+    fbk    = h5py.File(FILENAME, 'r') 
+    k1     = fbk['k1'][...]
+    k2     = fbk['k2'][...]
+    k3     = fbk['k3'][...]
+    p0k1   = fbk['p0k1'][...]
+    p0k2   = fbk['p0k2'][...]
+    p0k3   = fbk['p0k3'][...]
+    b123   = fbk['b123'][...] 
+    q123   = fbk['q123'][...]
+    b_sn   = fbk['b_sn'][...]
+    cnts   = fbk['counts'][...] # triange counts
+    Nhalos = fbk['Nhalos'][...] # number of halos 
+    files  = fbk['files'][...]  # names of individual files. 

docs/source/data.rst

@@ -0,0 +1,30 @@
+Data Access
+===========
+
+The data is stored in the three different supercomputers in San Diego, New York, and Princeton. It can be access through `globus <https://www.globus.org/>`_. 
+
+- Log in into `globus <https://www.globus.org/>`_ (create an account if you dont have one).
+- To access the data in San Diego type: Quijote_simulations (or with this `link1 <https://app.globus.org/file-manager?origin_id=feb85190-bafb-11ea-9a42-0255d23c44ef&origin_path=%2F>`_)
+- To access the data in New York type: Quijote_simulation2 (or with this `link2 <https://app.globus.org/file-manager?origin_id=e0eae0aa-5bca-11ea-9683-0e56c063f437&origin_path=%2F>`_)
+- To access the data in Princeton type: Princeton TIGRESS QUIJOTE Snapshots (or with this `link3 <https://app.globus.org/file-manager?origin_id=8ce7cdf0-7e85-11ea-97a5-0e56c063f437&origin_path=%2F>`_)
+
+Note that to download the data to your local machine (e.g. laptop) you will need to install the globus connect personal. For further details see `here <https://github.com/franciscovillaescusa/Quijote-simulations/blob/master/documentation/globus.md>`_.
+
+There are different folders:
+
+- **Snapshots**. This folder contains the snapshots of the simulations
+- **Halos**. This folder contains the halo catalogues
+- **Voids**. This folder contains the void catalogues
+- **Linear_Pk**. This folder contains the linear power spectra of each cosmological model
+- **Pk**. This folder contains the power spectra
+- **Marked_Pk**. This folder contains the marked power spectra
+- **Bk**. This folder contains the bispectra 
+- **CF**. This folder contains the correlation functions
+- **PDF**. This folder contains the pdfs
+- **density_field**. This folder contains the 3D density fields
+- **density_field_2D**. This folder contains the 2D density fields
+
+Inside each of the above folders there is the data for the different cosmologies, e.g. h_p, fiducial, Om_m. A brief description of the different cosmologies is provided in the below table. Further details can be found in the `Quijote paper <https://arxiv.org/abs/1909.05273>`_.
+
+
+.. image:: sims_table.png

docs/source/galaxies.rst

@@ -0,0 +1,6 @@
+Galaxy catalogues
+=================
+
+ChangHoon Hahn has created a set of tens of thousands of galaxy catalogues from the Quijote simulations called the Molino catalogues.
+
+You can find all the information and how to access this data `here <https://changhoonhahn.github.io/molino/>`_.

docs/source/halos.rst

@@ -0,0 +1,34 @@
+Halo catalogues
+===============
+
+The halo catalogues can be read through the `readfof.py <https://github.com/franciscovillaescusa/Pylians3/blob/master/library/readfof.py>`_ script. If you have `Pylians <https://github.com/franciscovillaescusa/Pylians3>`_ installed you already have it. An example on how to read a halo catalogue is this:
+
+.. code-block:: python
+		
+    import readfof 
+
+    # input files
+    snapdir = '/home/fvillaescusa/Quijote/Halos/s8_p/145/' #folder hosting the catalogue
+    snapnum = 4                                            #redshift 0
+
+    # determine the redshift of the catalogue
+    z_dict = {4:0.0, 3:0.5, 2:1.0, 1:2.0, 0:3.0}
+    redshift = z_dict[snapnum]
+
+    # read the halo catalogue
+    FoF = readfof.FoF_catalog(snapdir, snapnum, long_ids=False,
+		              swap=False, SFR=False, read_IDs=False)
+										
+    # get the properties of the halos
+    pos_h = FoF.GroupPos/1e3            #Halo positions in Mpc/h
+    mass  = FoF.GroupMass*1e10          #Halo masses in Msun/h
+    vel_h = FoF.GroupVel*(1.0+redshift) #Halo peculiar velocities in km/s
+    Npart = FoF.GroupLen                #Number of CDM particles in the halo
+
+The number in the name of the halo catalogue represents its redshift:
+
+- 000 ------> z=3
+- 001 ------> z=2
+- 002 ------> z=1
+- 003 ------> z=0.5
+- 004 ------> z=0

docs/source/index.rst

@@ -13,15 +13,18 @@ The Quijote simulations is a suite of 44,100 full N-body simulations designed to
    :caption: Content
 
    features
+   data
+   snapshots
+   halos
+   galaxies
+   voids
+   Pk
+   bispectra
+   xi
+   pdf
    team
    license
    help
 
 
 
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`

docs/source/pdf.rst

@@ -0,0 +1,18 @@
+*******
+PDFs
+*******
+
+The format of the PDF files is:
+
+- delta | pdf
+
+where delta is the density contrast (rho/< rho > - 1).
+
+In python, the files can be read as
+
+.. code-block:: python
+
+    import numpy as np
+    
+    delta, pdf = np.loadtxt('/home/fvillaescusa/Quijote/PDF/matter/latin_hypercube/0/PDF_m_5.0_z=0.txt', unpack=True)
+

docs/source/snapshots.rst

@@ -0,0 +1,45 @@
+Snapshots
+=========
+
+The snapshots are stored in either Gadget-II format or HDF5. They can be read using the `readgadget.py <https://github.com/franciscovillaescusa/Pylians3/blob/master/library/readgadget.py>`_ and `readsnap.py <https://github.com/franciscovillaescusa/Pylians3/blob/master/library/readsnap.py>`_ scripts. If you have `Pylians <https://github.com/franciscovillaescusa/Pylians3>`_ installed you already have them.
+
+The snapshots only contain 4 blocks:
+
+- Header: This block contains general information about the snapshot such as redshift, number of particles, box size, particle masses...etc.
+- Positions: This block contains the positions of all particles. Stored as 32-floats
+- Velocities: This block contains the velocities of all particles. Stored as 32-floats
+- IDs: This block contains the IDs of all particles. Stored as 32-integers. (This block may be removed in the future to reduce the size of the snapshots)
+
+An example on how to read a snapshot is this:
+
+.. code-block:: python
+		
+    import readgadget
+
+    # input files
+    snapshot = '/home/fvillaescusa/Quijote/Snapshots/h_p/snapdir_002/snap_002'
+    ptype    = [1] #[1](CDM), [2](neutrinos) or [1,2](CDM+neutrinos)
+
+    # read header
+    header   = readgadget.header(snapshot)
+    BoxSize  = header.boxsize/1e3  #Mpc/h
+    Nall     = header.nall         #Total number of particles
+    Masses   = header.massarr*1e10 #Masses of the particles in Msun/h
+    Omega_m  = header.omega_m      #value of Omega_m
+    Omega_l  = header.omega_l      #value of Omega_l
+    h        = header.hubble       #value of h
+    redshift = header.redshift     #redshift of the snapshot
+    Hubble   = 100.0*np.sqrt(Omega_m*(1.0+redshift)**3+Omega_l)#Value of H(z) in km/s/(Mpc/h)
+    
+    # read positions, velocities and IDs of the particles
+    pos = readgadget.read_block(snapshot, "POS ", ptype)/1e3 #positions in Mpc/h
+    vel = readgadget.read_block(snapshot, "VEL ", ptype)     #peculiar velocities in km/s
+    ids = readgadget.read_block(snapshot, "ID  ", ptype)-1   #IDs starting from 0
+    
+In the simulations with massive neutrinos it is possible to read the positions, velocities and IDs of the neutrino particles. Notice that the field should contain exactly 4 characters, that can be blank: :code:`"POS "`, :code:`"VEL "`, :code:`"ID  "`. The number in the name of the snapshot represents its redshift:
+
+- 000 ------> z=3
+- 001 ------> z=2
+- 002 ------> z=1
+- 003 ------> z=0.5
+- 004 ------> z=0

docs/source/voids.rst

@@ -0,0 +1,24 @@
+Void catalogues
+===============
+
+The void catalogues are stored as hdf5 files. They contain the following blocks:
+
+- pos:    the positions of the void centers in Mpc/h
+- radius: the sizes of the voids in in Mpc/h
+- VSF: the void size function
+- VSF_Rbins: the radii bins of the void size function
+- parameters: the values of the void finder parameters used to generate the void catalogue
+
+In python, the files can be read as
+
+.. code-block:: python
+		
+    import h5py
+
+    f = h5py.File('/home/fvillaescusa/Quijote/Voids/fiducial/0/void_catalogue_m_z=0.hdf5', 'r')
+    pos        = f['pos'][:]        #void center positions in Mpc/h
+    radius     = f['radius'][:]     #void radii in Mpc/h
+    VSF        = f['VSF'][:]        #VSF (#voids/dR/Volume)
+    VSF_Rbins  = f['VSF_Rbins'][:]  #VSF radii in Mpc/h
+    parameters = f['parameters'][:] #parameters used to run the void finder
+    f.close()

docs/source/xi.rst

@@ -0,0 +1,21 @@
+Correlation functions
+=====================
+
+The format of the correlation functions are:
+
+- R | xi(R)  for correlation functions in real-space
+- R | xi0(R) | xi2(R) | xi4(R) for correlation functions in redshift-space
+
+where xi0(R), xi2(R) and xi4(R) are the monopole, quadrupole and hexadecapole, respectively. The units of R are Mpc/h, while the different xi are dimensionless.
+
+In redshift-space there are three different files for each realization/redshift. These have been computed by placing the redshift-space distortions along the three different axes.
+
+In python, the files can be read as 
+
+.. code-block:: python
+		
+    import numpy as np
+
+    R, xi = np.loadtxt('/home/fvillaescusa/Quijote/CF/matter/fiducial/0/CF_m_1024_z=0.txt', unpack=True)
+    R, xi0, xi2, xi4 = np.loadtxt('/home/fvillaescusa/Quijote/CF/matter/fiducial/0/CF_m_RS0_1024_z=0.txt', unpack=True)
+