diff --git a/AGENTS.md b/AGENTS.md index 775695b..745bb23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,10 +8,14 @@ canonical, agent-agnostic instructions for this repo. ## Repository Structure - `scripts/` — Runnable Python tutorial scripts: - - `chapter_1_introduction/` — Grids, profiles, galaxies, ray-tracing, data, fitting - - `chapter_2_lens_modeling/` — Non-linear searches, Bayesian inference, lens modeling - - `chapter_3_search_chaining/` — Search chaining, prior passing, automated pipelines - - `chapter_4_pixelizations/` — Pixelized source reconstruction, inversions, regularization + - `chapter_1_introduction/` — Grids, profiles, galaxies, ray-tracing, point sources, the lensing + formalism, data, fitting + - `chapter_2_lens_modeling/` — Non-linear searches, Bayesian inference, lens modeling, search + chaining, prior passing, automated pipelines + - `chapter_3_pixelizations/` — Pixelized source reconstruction, inversions, regularization, the + Bayesian formalism + - `chapter_4_scaling_up_lensing/` — Extra galaxies, multi-galaxy lenses, scaling relations, group + and cluster scales, weak lensing - `chapter_optional/` — Alternative searches and advanced topics - `simulator/` — Simulator scripts that generate the tutorial datasets at runtime - `notebooks/` — Jupyter versions, generated from `scripts/` (do not edit directly) diff --git a/README.md b/README.md index 5be9417..57068c1 100644 --- a/README.md +++ b/README.md @@ -21,13 +21,14 @@ they are concise and assume the concepts taught in **HowToLens** as background. ## Chapters - `chapter_1_introduction` — An introduction to strong gravitational lensing and **PyAutoLens**: grids, light - and mass profiles, galaxies, ray-tracing, simulated imaging data, and fitting. + and mass profiles, galaxies, ray-tracing, point sources, the lensing formalism, simulated imaging data, + and fitting. - `chapter_2_lens_modeling` — Bayesian inference, non-linear searches, and how to fit a lens model to CCD - imaging data with **PyAutoLens**. -- `chapter_3_search_chaining` — Chaining multiple non-linear searches together to build automated lens - modeling pipelines for complex systems. -- `chapter_4_pixelizations` — Pixelized source reconstructions (inversions) for sources with irregular - morphologies. + imaging data with **PyAutoLens**, ending with search chaining and automated pipelines. +- `chapter_3_pixelizations` — Pixelized source reconstructions (inversions) for sources with irregular + morphologies, including the Bayesian formalism underpinning them. +- `chapter_4_scaling_up_lensing` — Scaling lens modeling up beyond a single lens galaxy: extra galaxies, + multi-galaxy lenses, scaling relations, group and cluster scales, and weak lensing. - `chapter_optional` — Optional tutorials on alternative non-linear searches and other advanced topics. **HowToLens** currently sits at four chapters. Each chapter will take around a day to work through. diff --git a/config/build/no_run.yaml b/config/build/no_run.yaml index a7b9259..88eadff 100644 --- a/config/build/no_run.yaml +++ b/config/build/no_run.yaml @@ -25,10 +25,3 @@ # permanent skips — every mega-run surfaces them with a loud warning # banner. Investigate the failure, fix the underlying bug, and remove # the NEEDS_FIX marker. - -- tutorial_5_borders # NEEDS_FIX 2026-08-04 - cap-induced, not a mask problem: - # fails only under PYAUTO_SMALL_DATASETS (IndexError: index 371 out of bounds - # for axis 0 with size 272) and passes on the same dataset files without the - # cap. Hardcoded pix/border indices outrun the capped mesh. Fix by decoupling - # the indices (or the mesh shape) from dataset resolution, as HowToGalaxy - # chapter_4 tutorial_3 did, then remove this entry. diff --git a/llms-full.txt b/llms-full.txt index 20b2f42..73ee54f 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -10,80 +10,89 @@ AUTO-GENERATED by PyAutoHands — do not edit by hand; regenerate with generate. ## chapter_1_introduction - [Tutorial 0: Visualization](scripts/chapter_1_introduction/tutorial_0_visualization.py): In this tutorial, we quickly cover visualization in **PyAutoLens** and make sure images display clearly in your Jupyter notebook and on your computer screen. - - Contents: Directories, Dataset, Subplots, Plot Customization, Overlays, Wrap Up + - Contents: Directories, Dataset, Dataset Auto-Simulation, Subplots, Plot Customization, Overlays, Wrap Up - [HowToLens: Introduction](scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py): A strong gravitational lens is a system where two (or more) galaxies align perfectly down our line of sight from Earth such that the foreground galaxy's mass curves space-time in on itself, such that the light of a background source galaxy is deflected and magnified. This means we can see the background source galaxy multiple times, as multiple arcs or rings, because multiple paths through the foreground galaxy's mass are taken by the source's light. - - Contents: Grids, Geometry, Light Profiles, One Dimension Projection, Galaxies, Units + - Contents: Grids, Geometry, Light Profiles, One Dimension Projection, Galaxies, Units, Wrap Up, Advanced Topics, Other Unit Conversion, Over Sampling - [Tutorial 2: Ray Tracing](scripts/chapter_1_introduction/tutorial_2_ray_tracing.py): Strong gravitational lensing occurs when the mass of a foreground galaxy (or galaxies) curves space-time around it, causing light rays from a background source to appear deflected. - - Contents: Grid, Mass Profiles, Ray Tracing Grids, Ray Tracing Images, Galaxies, Tracer, Mappings -- [Tutorial 5: More Ray Tracing](scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py): We'll now reinforce the ideas that we learnt about ray-tracing in the previous tutorial and introduce the following new concepts: + - Contents: Grid, Mass Profiles, Ray Tracing Grids, Ray Tracing Images, Galaxies, Tracer, Mappings, Wrap Up +- [Tutorial 3: More Ray Tracing](scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py): We'll now reinforce the ideas that we learnt about ray-tracing in the previous tutorial and introduce the following new concepts: - Contents: Initial Setup, Concise Code, Critical Curves, Caustics, Units, More Complexity, Multi Galaxy Ray Tracing, Wrap Up -- [Tutorial 4: Point Sources](scripts/chapter_1_introduction/tutorial_4_point_sources.py): This tutorial is not wrriten yet, but will explain how point source lensing works. - - Contents: Wrap Up -- [Tutorial 5: Lensing Formalism](scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py): This tutorial is not wrriten yet, but will explain what all the different lens quantities are and give a more formal description of them. - - Contents: Wrap Up +- [Tutorial 4: Point Sources](scripts/chapter_1_introduction/tutorial_4_point_sources.py): In the previous tutorials, the background sources we lensed were galaxies: extended objects whose light spreads over many thousands of light years. When lensed, their light is warped into the arcs and Einstein rings we produced with the `Tracer`, spread across many pixels of the image. + - Contents: Initial Setup, Point Sources, Point Source Tracer, The Lens Equation, Point Solver, Multiple Images and Critical Curves, Magnifications, Time Delays, Extended Versus Point Computations, Wrap Up +- [Tutorial 5: Lensing Formalism](scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py): This tutorial is the equations lecture of **HowToLens**. + - Contents: Initial Setup, Cosmological Distances, The Lens Equation, Convergence, Deflection Angles, The Lensing Potential, Shear and Magnification, Critical Curves and Caustics, Einstein Radius, Time Delays, Wrap Up - [Tutorial 6: Data](scripts/chapter_1_introduction/tutorial_6_data.py): In the last tutorials, we use tracers to create images of strong lenses. However, those images don't accurately represent what we would observe through a telescope. - - Contents: Initial Setup, Optics Blurring, Poisson Noise, Background Sky, Simulator, Output, Wrap Up -- [Tutorial 7: Fitting](scripts/chapter_1_introduction/tutorial_7_fitting.py): In previous tutorials, we used light profiles to create simulated images of tracer and visualized how these images would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope. - - Contents: Dataset & Mask, Masked Grid, Fitting, Incorrect Fit, Model Fitting, Wrap Up -- [Tutorial 9: Summary](scripts/chapter_1_introduction/tutorial_8_summary.py): In this chapter, we have learnt that: + - Contents: Initial Setup, Optics Blurring, Poisson Noise, Background Sky, Simulator, Output, Interferometer Data, Weak Lensing Data, Wrap Up +- [Tutorial 7: Fitting](scripts/chapter_1_introduction/tutorial_7_fitting.py): In previous tutorials, we used light profiles to create simulated images of a tracer and visualized how these images would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope. + - Contents: Dataset, Dataset Auto-Simulation, Mask, Masked Grid, Fitting, Incorrect Fit, Model Fitting, Wrap Up +- [Tutorial 8: Summary](scripts/chapter_1_introduction/tutorial_8_summary.py): In this chapter, we have learnt that: - Contents: Start, Object Composition, Visualization, Code Design, Source Code, Wrap Up ## chapter_2_lens_modeling -- [Tutorial 1: Non-linear Search](scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py): The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a strong lens using a telescope like the Hubble Space Telescope, and seek to learn about the lens galaxy, source galaxy and the Universe from these observations. With **PyAutoLen**, we seek to learn about the lenses may and ray-tracing, asking questions like how big is the lens galaxy and what does the unlensed source galaxy look like? - - Contents: Overview, Parameter Space, Search Types, Deeper Background, PyAutoFit, Initial Setup, Mask, Model, Priors, Analysis, Searches, Nested Sampling, Wrap Up +- [Tutorial 10: Prior Passing](scripts/chapter_2_lens_modeling/tutorial_10_prior_passing.py): In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two non-linear searches. This used an initial search to fit a simple lens model, whose results were used to tune and initialize the priors of a more complex lens model that was fitted by the second search. + - Contents: Initial Setup, Dataset Auto-Simulation, Model, Search, Result (Search 1), Prior Passing, Result, Wrap Up, Detailed Explanation Of Prior Passing, EXAMPLE +- [Tutorial 11: SLaM](scripts/chapter_2_lens_modeling/tutorial_11_slam.py): In the previous two tutorials, we learnt how search chaining breaks a lens model-fit into a sequence of simpler non-linear searches, and how prior passing carries the results of each search into the next. Together, they give us the flexibility to juggle the dimensionality, priors and settings of each search — the three drivers of run-time we met in tutorial 8 — whilst still fitting a complex and realistic lens model at the end. + - Contents: Search Chaining In The Workspace, SLaM (Source, Light and Mass), Wrap Up +- [Tutorial 1: Non-linear Search](scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py): The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a strong lens using a telescope like the Hubble Space Telescope, and seek to learn about the lens galaxy, source galaxy and the Universe from these observations. With **PyAutoLens**, we seek to learn about the lens's mass and ray-tracing, asking questions like how big is the lens galaxy and what does the unlensed source galaxy look like? + - Contents: Overview, Parameter Space, Non-Linear Search, Search Types, Deeper Background, PyAutoFit, Initial Setup, Dataset Auto-Simulation, Mask, Model, Priors, Analysis, Searches, Maximum Likelihood Estimation (MLE), Markov Chain Monte Carlo (MCMC), Nested Sampling, What is The Best Search To Use?, Wrap Up - [Tutorial 2: Practicalities](scripts/chapter_2_lens_modeling/tutorial_2_practicalities.py): In the last tutorial, we introduced foundational statistical concepts essential for model-fitting, such as parameter spaces, likelihoods, priors, and non-linear searches. Understanding these statistical concepts is crucial for performing model fits effectively. - - Contents: PyAutoFit, Initial Setup, Mask, Model, Search, Search Settings, Iterations Per Update, Analysis, VRAM Use, Run Times, Result Info, Output Folder, Unique Identifier, Output Folder Contents, Result, Other Practicalities, Wrap Up + - Contents: PyAutoFit, Initial Setup, Dataset Auto-Simulation, Mask, Model, Search, Search Settings, Iterations Per Update, Analysis, VRAM Use, Run Times, Model-Fit, Result Info, Output Folder, Unique Identifier, Output Folder Contents, Result, Other Practicalities, Wrap Up - [Tutorial 3: Realism and Complexity](scripts/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.py): In the previous two tutorials, we fitted a fairly crude and unrealistic model: the lens's mass was spherical, as was the source's light. Given most lens galaxies are literally called 'elliptical galaxies' we should probably model their mass as elliptical! Furthermore, we have completely omitted the lens galaxy's light, which in real observations outshines the source's light and therefore must be included in the lens model. - - Contents: Initial Setup, Mask, Model, Run Time, Result, Global and Local Maxima, Wrap Up + - Contents: Initial Setup, Dataset Auto-Simulation, Mask, Model, Search + Analysis, Run Time, Result, Global and Local Maxima, Wrap Up - [Tutorial 4: Dealing With Failure](scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py): In the previous tutorial we intentionally made our non-linear search infer a local maxima solution and therefore return a physically incorrect lens model. In this tutorial, we will pretend that we have modeled our lens and inferred a local maxima. We introduce three approaches one can take that changes how we fit the model, all of which have the aim of ensuring we infer the global maxima: - - Contents: Initial Setup, Mask, Prior Tuning, Run Time, Result, Discussion + - Contents: Initial Setup, Dataset Auto-Simulation, Mask, Prior Tuning, Run Time, Result, Discussion, Approach 2: Reducing Complexity, Approach 3: Look Harder, Wrap Up - [Tutorial 5: Linear Profiles](scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py): In the previous tutorial we learned how to balance model complexity with our non-linear search in order to infer accurate lens model solutions and avoid failure. We saw how in order to fit a model accurately one may have to parameterize and fit a simpler model with fewer non-linear parameters, at the expense of fitting the data less accurately. - - Contents: Initial Setup, Mask, Linear Light Profiles, Run Time, Result, Intensities, Visualization, Basis, Model Fit, Source MGE, Multi Gaussian Expansion Benefits, Disadvantage of Basis Functions, Positive Only Solver, Other Basis Functions, Wrap Up + - Contents: Initial Setup, Dataset Auto-Simulation, Mask, Linear Light Profiles, Run Time, Result, Intensities, Visualization, Basis, Model Fit, Source MGE, Multi Gaussian Expansion Benefits, Disadvantage of Basis Functions, Positive Only Solver, Other Basis Functions, Wrap Up - [Tutorial 6: Masking and Positions](scripts/chapter_2_lens_modeling/tutorial_6_masking_and_positions.py): We have learnt everything we need to know about non-linear searches to model a strong lens and infer a good lens model solution. Now, lets consider masking in more detail, something we have not given much consideration previously. We'll also learn a neat trick to improve the speed and accuracy of a non-linear search. - - Contents: Initial Setup, Mask, Run Time, Search, Discussion, Positions Thresholding, Wrap Up + - Contents: Initial Setup, Dataset Auto-Simulation, Mask, Model + Analysis, Run Time, Search, Discussion, Positions Thresholding, Wrap Up - [Tutorial 7: Results](scripts/chapter_2_lens_modeling/tutorial_7_results.py): In the previous tutorials, each search returned a `Result` object, which we used to plot the maximum log likelihood fit each model-fit. In this tutorial, we'll take a look at the result object in a little more detail. - - Contents: Initial Setup, Tracer & Fit, Samples, Workspace, Database, Wrap Up + - Contents: Initial Setup, Dataset Auto-Simulation, Tracer & Fit, Samples, Workspace, Database, Wrap Up - [Tutorial 8: Need For Speed](scripts/chapter_2_lens_modeling/tutorial_8_need_for_speed.py): In this chapter, we have learnt how to model strong lenses and how to balance complexity and realism to ensure that we infer a good lens model. - - Contents: Algorithmic Optimization, Data Quantity, Wrap Up + - Contents: Searching Non-linear Parameter Space, Cost Per Evaluation, JAX, Run Time Estimation, Data Quantity, Wrap Up +- [Tutorial 9: Search Chaining](scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py): Throughout this chapter, we have fitted the data using just one non-linear search. The previous tutorial discussed the run-time cost of doing this: with a single search, the dimensionality of the model, the priors on its parameters and the search settings must all be juggled at once, leaving us little flexibility to trade them off against one another. + - Contents: Initial Setup, Dataset Auto-Simulation, Model, Search + Analysis, Result, Prior Passing, Run Time, Model Fit, Wrap Up -## chapter_3_search_chaining +## chapter_3_pixelizations -- [Tutorial 1: Search Chaining](scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py): In chapter 2, we learnt how to perform lens modeling using a non-linear search. In all of the tutorials, we fitted the data using just one non-linear search. In this chapter, we introduce a technique called 'non-linear search chaining', fits a lens model using a sequence of non-linear searches. The initial searches fit simpler lens models whose parameter spaces can be more accurately and efficiently sampled. The results of this search are then passed to later searches which fit lens models of gradually increasing complexity. - - Contents: Initial Setup, Model, Result, Prior Passing, Run Time, Model Fit, Wrap Up -- [Tutorial 2: Prior Passing](scripts/chapter_3_search_chaining/tutorial_2_prior_passing.py): In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two non-linear searches. This used an initial search to fit a simple lens model, whose results were used to tune and initialize the priors of a more complex lens model that was fitted by the second search. - - Contents: Initial Setup, Model, Search, Prior Passing, Result, Wrap Up, Detailed Explanation Of Prior Passing, EXAMPLE -- [Tutorial 3: Lens and Source](scripts/chapter_3_search_chaining/tutorial_3_lens_and_source.py): In this tutorial, we demonstrate search chaining using three searches to fit strong lens `Imaging` which includes the lens galaxy's light. - - Contents: Dated Tutorial, Initial Setup, Paths, Notes, Wrap Up -- [Tutorial 4: Two Lens galaxies](scripts/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.py): Up to now, all the images we've fitted had one lens galaxy. However, we saw in chapter 1 that our lens plane can consist of multiple galaxies which each contribute to the strong lensing. Multi-galaxy systems are challenging to model, because they add an extra 5-10 parameters to the non-linear search and, more problematically, the degeneracies between the parameters of the mass profiles of the two galaxies can be severe. - - Contents: Initial Setup, Mask, Paths, Search Chaining Approach, Wrap Up -- [Tutorial 5: Complex Source](scripts/chapter_3_search_chaining/tutorial_5_complex_source.py): Up to now, we've not paid much attention to the source galaxy's morphology. We've assumed its a single-component exponential profile, which is a fairly crude assumption. A quick look at any image of a real galaxy reveals a wealth of different structures that could be present: bulges, disks, bars, star-forming knots and so on. Furthermore, there could be more than one source-galaxy! - - Contents: Initial Setup, Paths, Search Chaining Approach, Run Times, Wrap Up -- [Tutorial 6: SLaM](scripts/chapter_3_search_chaining/tutorial_6_slam.py): You are now familiar with pipelines, in particular how we use them to break-down the lens modeling procedure to provide more efficient and reliable model-fits. In the previous tutorials, you learnt how to write your own pipelines, which can fit whatever lens model is of particular interest to your scientific study. - -## chapter_4_pixelizations - -- [Tutorial 10: Brightness Adaption](scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py): In the previous tutorial we motivated our need to adapt the pixelization to the source's morphology, such that source pixels congregates in the source's brightest regions regardless of where the source is located in the source-plane. +- [Tutorial 10: Fit Problems](scripts/chapter_3_pixelizations/tutorial_10_fit_problems.py): To begin, make sure you have read tutorials 4 and 5 carefully, as a clear understanding of how the Bayesian evidence works is key to understanding the adaption tutorials that close this chapter! + - Contents: Initial Setup, Mask, Simulator, Fitting, Fit Problems, Discussion, Wrap Up +- [Tutorial 11: Brightness Adaption](scripts/chapter_3_pixelizations/tutorial_11_brightness_adaption.py): In the previous tutorial we motivated our need to adapt the pixelization to the source's morphology, such that source pixels congregates in the source's brightest regions regardless of where the source is located in the source-plane. - Contents: Initial Setup, Adapt Image, Adaption, Hilbert, Weight Map, Wrap Up -- [Tutorial 11: Adaptive Regularization](scripts/chapter_4_pixelizations/tutorial_11_adaptive_regularization.py): In tutorial 7, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source demand different levels of regularization, motivating a regularization scheme which adapts to the reconstructed source's surface brightness. +- [Tutorial 12: Adaptive Regularization](scripts/chapter_3_pixelizations/tutorial_12_adaptive_regularization.py): In tutorial 10, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source demand different levels of regularization, motivating a regularization scheme which adapts to the reconstructed source's surface brightness. - Contents: Initial Setup, Convenience Function, Adaptive Regularization, Wrap Up -- [Tutorial 1: pixelizations](scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py): In the previous chapters, we used light profiles to model the light of a strong lens's source galaxy, where the light profile was an analytic description of how the luminosity varies as a function of radius. In this chapter, we are instead going to reconstruct the source's light on a pixel-grid, and in this tutorial we will learn how to create a source-plane pixelization. +- [Tutorial 1: Pixelizations](scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py): In the previous chapters, we used light profiles to model the light of a strong lens's source galaxy, where the light profile was an analytic description of how the luminosity varies as a function of radius. In this chapter, we are instead going to reconstruct the source's light on a pixel-grid, and in this tutorial we will learn how to create a source-plane pixelization. - Contents: Initial Setup, Mesh, Wrap Up -- [Tutorial 2: Mappers](scripts/chapter_4_pixelizations/tutorial_2_mappers.py): In the previous tutorial, we used a pixelization to create made a `Mapper`. However, it was not clear what a `Mapper` does, why it was called a mapper and whether it was mapping anything at all! +- [Tutorial 2: Mappers](scripts/chapter_3_pixelizations/tutorial_2_mappers.py): In the previous tutorial, we used a pixelization to create a `Mapper`. However, it was not clear what a `Mapper` does, why it was called a mapper and whether it was mapping anything at all! - Contents: Initial Setup, Mappers, Mask, Wrap Up -- [Tutorial 3: Inversions](scripts/chapter_4_pixelizations/tutorial_3_inversions.py): In the previous two tutorials, we introduced: +- [Tutorial 3: Inversions](scripts/chapter_3_pixelizations/tutorial_3_inversions.py): In the previous two tutorials, we introduced: - Contents: Initial Setup, Pixelization, Positive Only Solver, Wrap Up, Detailed Explanation -- [Tutorial 4: Bayesian Regularization](scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py): So far, we have: +- [Tutorial 4: Bayesian Regularization](scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py): So far, we have: - Contents: Initial Setup, Convenience Function, Pixelization, Regularization, Bayesian Evidence, Detailed Description -- [Tutorial 5: Borders](scripts/chapter_4_pixelizations/tutorial_5_borders.py): In the previous tutorials, the source-plane pixel grid perfectly mapped over the traced image-pixel $(y,x)$ coordinates in the source plane. If these pixels mapped to a larger area in the source plane, its pixel-grid would automatically increase its size so as to cover every source-plane coordinate. +- [Tutorial 5: Bayesian Formalism](scripts/chapter_3_pixelizations/tutorial_5_bayesian_formalism.py): In tutorials 1 to 4, we built an intuition for how pixelized source reconstruction works: pixelizations place a pixel-grid in the source-plane, mappers pair source-pixels with image-pixels, inversions solve for the source-pixel fluxes that best fit the data, and regularization smooths the solution within a Bayesian framework. + - Contents: Initial Setup, Mesh Shape, Ray Tracing, Border Relocation, Source Pixel Centres, Interpolation, Mapper, Mapping Matrix, Blurred Mapping Matrix, Data Vector, Curvature Matrix, Unregularized Solve, Regularization Matrix, Source Reconstruction, Image Reconstruction, Likelihood Function, Chi Squared, Regularization Term, Complexity Terms, Noise Normalization Term, Log Evidence, Fit, Wrap Up +- [Tutorial 6: Borders](scripts/chapter_3_pixelizations/tutorial_6_borders.py): In the previous tutorials, the source-plane pixel grid perfectly mapped over the traced image-pixel $(y,x)$ coordinates in the source plane. If these pixels mapped to a larger area in the source plane, its pixel-grid would automatically increase its size so as to cover every source-plane coordinate. - Contents: Initial Setup, Borders, Wrap Up -- [Tutorial 6: Lens Modeling](scripts/chapter_4_pixelizations/tutorial_6_lens_modeling.py): When modeling complex source's with parametric profiles, we quickly entered a regime where our non-linear search was faced with a parameter space of dimensionality N=20+ parameters. This made the model-fitting inefficient and likely to infer a local maxima. +- [Tutorial 7: Lens Modeling](scripts/chapter_3_pixelizations/tutorial_7_lens_modeling.py): When modeling complex sources with parametric profiles, we quickly entered a regime where our non-linear search was faced with a parameter space of dimensionality N=20+ parameters. This made the model-fitting inefficient and likely to infer a local maxima. - Contents: Initial Setup, Unphysical Solutions, Brief Description, Light Profiles, Wrap Up -- [Tutorial 7: Adaptive Pixelization](scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py): In this tutorial we will introduce a new `Pixelization` object, which uses an `Overlay` image-mesh and a `Delaunay` mesh. - - Contents: Initial Setup, Advantages and Disadvatanges, Image Mesh, Regularization, Wrap Up -- [Tutorial 8: Model-Fit](scripts/chapter_4_pixelizations/tutorial_8_model_fit.py): You should now perform lens modeling using a pixelization, which is described fully in the example: -- [Tutorial 9: Fit Problems](scripts/chapter_4_pixelizations/tutorial_9_fit_problems.py): To begin, make sure you have read the `introduction` file carefully, as a clear understanding of how the Bayesian evidence works is key to understanding this chapter! - - Contents: Initial Setup, Mask, Simulator, Fitting, Fit Problems, Discussion, Wrap Up +- [Tutorial 8: Adaptive Pixelization](scripts/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.py): In this tutorial we will introduce a new `Pixelization` object, which uses an `Overlay` image-mesh and a `Delaunay` mesh. + - Contents: Initial Setup, Advantages and Disadvantages, Image Mesh, Regularization, Wrap Up +- [Tutorial 9: Model-Fit](scripts/chapter_3_pixelizations/tutorial_9_model_fit.py): You should now perform lens modeling using a pixelization, which is described fully in the example: + +## chapter_4_scaling_up_lensing + +- [Tutorial 1: Extra Galaxies](scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py): Welcome to chapter 4 of **HowToLens**, where we scale up lens modeling beyond a single lens galaxy. + - Contents: Initial Setup, Light Versus Mass, Mask, Approach 1 Noise Scaling, Noise Scaling Fit, Approach 2 Extra Galaxies Model, Extra Galaxy Centres, Extra Galaxies Fit, Which Approach When, Wrap Up +- [Tutorial 2: Multi-Galaxy Lenses](scripts/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.py): In the previous tutorial, we learned how to deal with extra galaxies near a strong lens — nuisance objects whose light contaminates the data but which play no meaningful role in the lensing itself. We removed their emission from the analysis, or gave them a heavily restricted model, and the single dominant lens galaxy remained the star of the show. + - Contents: Initial Setup, Mask, Over Sampling, Model, Fixing the Mass Centres, Model Fit, Result, Mass Degeneracies, No Shared Halo, Three Lens Galaxies, Wrap Up +- [Tutorial 3: Scaling Relations](scripts/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.py): The previous tutorial ended on a warning: every deflector we add to a lens model brings its own free parameters. Two galaxies were manageable, but each one cost us a mass profile's worth of dimensions, and the arithmetic only gets worse. A group-scale lens may have ten member galaxies, a cluster hundreds. If every member keeps its own free mass, the parameter space explodes — a 100-galaxy cluster with 5 free mass parameters per galaxy is a 500-dimensional model, which no non-linear search can sample and no dataset can constrain anyway. + - Contents: Mass Follows Light, Initial Setup, Mask, Measured Luminosities, The Anchor, Over Sampling, Light Via MGE, Scaling Relation, Model, Parameter Counts, Scaling To Many Galaxies, Model Fit, Results, Limitations, Lens Environments, Wrap Up +- [Tutorial 4: Group Scale](scripts/chapter_4_scaling_up_lensing/tutorial_4_group_scale.py): In the previous tutorials we took our first steps beyond the single lens galaxy: we included extra galaxies near the lens in the model, we modeled systems where two or more galaxies of comparable mass share the lensing, and we introduced scaling relations, which tie the mass of a galaxy to its light so that adding more galaxies to a model does not mean adding more free parameters. + - Contents: Initial Setup, Mask, Galaxy Centres, The dPIE Profile, Fitting a Group, A Group Halo?, Model Fit, Scaling Relation Members, The Group Scale Sweet Spot, Wrap Up +- [Tutorial 5: Cluster Scale](scripts/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.py): Throughout this chapter we have been climbing a ladder of scale: from a single lens galaxy with an extra galaxy nearby, to multi-galaxy lenses, to galaxy groups whose members share a common dark matter halo. + - Contents: Multi-Plane Ray Tracing, Dataset, Point Source Modeling, Point Datasets, The CSV Interface, Point Solver, Model, Analysis + Factor Graph, Search, Model Fit, Results, Customization, Wrap Up +- [Tutorial 6: Weak Lensing](scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py): Every tutorial in this series so far — indeed, every fit performed in all four chapters of **HowToLens** — has been a *strong* lensing analysis. Strong lensing occurs when a background galaxy lies so close (in projection) to a foreground mass that its light is bent into multiple images, arcs or a complete Einstein ring. These dramatic features are what we simulated, fitted and modeled, from the single galaxy-scale lenses of chapters 1 and 2 up to the group-scale and cluster-scale systems earlier in this chapter. + - Contents: Shear Catalogues, Mass Scales, Ray Tracing, Source Galaxy Positions, Simulate, Visualize, Mass Map, Fitting, Model Fit, Result, Joint Strong and Weak Lensing, Wrap Up ## chapter_optional @@ -92,13 +101,25 @@ AUTO-GENERATED by PyAutoHands — do not edit by hand; regenerate with generate. ## simulator +- [Simulator: Cluster](scripts/simulator/cluster.py): This script simulates a strong lens on the 'cluster' scale: 2 main lens galaxies (a brightest cluster galaxy and a satellite), 10 lower-mass cluster member galaxies on a luminosity-mass scaling relation, a cluster-scale dark matter halo not tied to any individual galaxy, and 2 multiply-imaged background source galaxies at *different* redshifts (z = 1.0 and z = 2.0) — a genuine multi-plane lens. + - Contents: Dataset Paths, Redshifts, Galaxy Centres, Grids, Main Lens Galaxies, Scaling Member Galaxies, Host Dark Matter Halo, Source Galaxies, Ray Tracing, Point Solver, Point Datasets, Combined CSV, Scaling Galaxies CSV, Model CSVs, Tracer json, Imaging, Visualize +- [Simulator: Group Scale Lens](scripts/simulator/group.py): This script simulates `Imaging` of a 'group-scale' strong lens, which is used in chapter 4 of the **HowToLens** lectures to illustrate lens modeling at the group scale. + - Contents: Dataset Paths, Grid, Galaxy Centres, Over Sampling, PSF / Simulator, Main Lens Galaxy, Member Galaxies, Source Galaxy, Ray Tracing, Output, Visualize, Tracer json, Centre JSON Files, Positions +- [Simulator: Interferometer](scripts/simulator/interferometer.py): This script simulates `Interferometer` data of a 'galaxy-scale' strong lens, as would be observed by a radio or sub-mm interferometer like ALMA or the JVLA. + - Contents: Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Tracer json +- [Simulator: Lens With Extra Galaxy](scripts/simulator/lens_extra_galaxy.py): This script simulates `Imaging` of a 'galaxy-scale' strong lens which is identical to the `lens_sersic` dataset simulated for chapter 2 (lens light + mass + lensed source), but with one extra galaxy located a few arc-seconds from the lens galaxy. + - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Extra Galaxy, Output, Mask Extra Galaxy, Visualize, Tracer json, Extra Galaxy Centre - [Simulator: Simple Sersic Lens](scripts/simulator/lens_sersic.py): This script simulates `Imaging` of a 'galaxy-scale' which is identical to the `simple` simulated in the `start_here.py` script, but where the lens galaxy's light is an `Sersic` profile. - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Tracer json - [Simulator: Lens x2](scripts/simulator/lens_x2.py): This script simulates `Imaging` of a 'galaxy-scale' lens where there are two lens galaxies, each with their own light and mass profiles. - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Tracer json +- [Simulator: Lens x3](scripts/simulator/lens_x3.py): This script simulates `Imaging` of a 'galaxy-scale' lens where there are three lens galaxies, each with their own light and mass profiles, which all contribute significantly to the lensing of a single background source. + - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Tracer json - [Simulator: No Lens Light](scripts/simulator/no_lens_light.py): This script simulates `Imaging` of a 'galaxy-scale' which is identical to the `simple` simulated in the `start_here.py` script, but where the lens galaxy's light is omitted. - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Mask Extra Galaxies, Tracer json - [Simulator: SIS](scripts/simulator/no_lens_light__mass_sis.py): This script simulates `Imaging` of a 'galaxy-scale' which is identical to the `simple` simulated in the `start_here.py` script, but where the lens galaxy's light is omitted and the lens's mass distribution is a Singular Isothermal Sphere. - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Tracer json - [Simulator: Source Complex](scripts/simulator/source_complex.py): This script simulates `Imaging` of a 'galaxy-scale' strong lens where the source galaxy's light is more complex than other examples, being composed of 4 Sersics. - Contents: Model, Dataset Paths, Simulate, Ray Tracing, Output, Visualize, Tracer json +- [Simulator: Weak Lensing](scripts/simulator/weak_lensing.py): This script simulates a weak gravitational lensing shear catalogue. Unlike the imaging simulators (which produce a 2D image of the lensed source) the weak-lensing simulator produces a *catalogue* of (gamma_2, gamma_1) shear measurements at the (y, x) positions of a population of background source galaxies. + - Contents: Dataset Paths, Ray Tracing, Source Positions, Simulator, Output, Visualize diff --git a/llms.txt b/llms.txt index 465321d..5b97710 100644 --- a/llms.txt +++ b/llms.txt @@ -26,8 +26,8 @@ analysis?" below). - **Chapter 1 — Introduction** → [scripts/chapter_1_introduction/](scripts/chapter_1_introduction/): Lensing from first principles: grids & galaxies, ray tracing, simulating data, and fitting data. The conceptual foundation everything else builds on. - **Chapter 2 — Lens Modeling** → [scripts/chapter_2_lens_modeling/](scripts/chapter_2_lens_modeling/): How to fit a lens model to data with a non-linear search — parameter spaces, priors, dealing with failure, linear light profiles, masking, and reading results. -- **Chapter 3 — Search Chaining** → [scripts/chapter_3_search_chaining/](scripts/chapter_3_search_chaining/): Breaking a hard fit into a sequence of searches (prior passing), multi-galaxy lenses, complex sources, and an introduction to the SLaM pipelines. -- **Chapter 4 — Pixelizations** → [scripts/chapter_4_pixelizations/](scripts/chapter_4_pixelizations/): Reconstructing the source on a pixel-grid instead of with light profiles — mappers, inversions, Bayesian regularization, borders, and adaptive meshes. +- **Chapter 3 — Pixelizations** → [scripts/chapter_3_pixelizations/](scripts/chapter_3_pixelizations/): Reconstructing the source on a pixel-grid instead of with light profiles — mappers, inversions, Bayesian regularization, the Bayesian formalism, borders, and adaptive meshes. +- **Chapter 4 — Scaling Up Lensing** → [scripts/chapter_4_scaling_up_lensing/](scripts/chapter_4_scaling_up_lensing/): Beyond a single lens galaxy — extra galaxies, multi-galaxy lenses, scaling relations, group and cluster scales, and weak lensing. ## I want to understand… @@ -37,8 +37,9 @@ analysis?" below). - **How does a non-linear search find a model?** → [scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py](scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py): Parameter space, priors, nested sampling with Nautilus. - **My fit found a wrong (local-maxima) solution — what do I do?** → [scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py](scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py): Prior tuning, simplifying the model, and other escapes from local maxima. - **What are linear light profiles / MGE?** → [scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py](scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py): Solving intensities linearly; the Multi-Gaussian Expansion basis. -- **How do I chain searches / what is SLaM?** → [scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py](scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py) then [scripts/chapter_3_search_chaining/tutorial_6_slam.py](scripts/chapter_3_search_chaining/tutorial_6_slam.py). -- **How do I reconstruct a complex source on a pixel grid?** → [scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py](scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py) through [tutorial_4_bayesian_regularization.py](scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py). +- **How do I chain searches / what is SLaM?** → [scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py](scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py) then [scripts/chapter_2_lens_modeling/tutorial_11_slam.py](scripts/chapter_2_lens_modeling/tutorial_11_slam.py). +- **How do I reconstruct a complex source on a pixel grid?** → [scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py](scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py) through [tutorial_4_bayesian_regularization.py](scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py). +- **How do I model lenses with many galaxies, groups, clusters or weak lensing?** → [scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py](scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py) through [tutorial_6_weak_lensing.py](scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py). ## Ready for real analysis? diff --git a/markdown/README.md b/markdown/README.md index 9811349..214f310 100644 --- a/markdown/README.md +++ b/markdown/README.md @@ -5,11 +5,11 @@ Every page below is the corresponding example script **fully executed**, rendere - [Tutorial 0: Visualization](chapter_1_introduction/tutorial_0_visualization.md) — from `scripts/chapter_1_introduction/tutorial_0_visualization.py` - [HowToLens: Introduction](chapter_1_introduction/tutorial_1_grids_and_galaxies.md) — from `scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py` - [Tutorial 2: Ray Tracing](chapter_1_introduction/tutorial_2_ray_tracing.md) — from `scripts/chapter_1_introduction/tutorial_2_ray_tracing.py` -- [Tutorial 5: More Ray Tracing](chapter_1_introduction/tutorial_3_more_ray_tracing.md) — from `scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py` +- [Tutorial 3: More Ray Tracing](chapter_1_introduction/tutorial_3_more_ray_tracing.md) — from `scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py` - [Tutorial 4: Point Sources](chapter_1_introduction/tutorial_4_point_sources.md) — from `scripts/chapter_1_introduction/tutorial_4_point_sources.py` - [Tutorial 5: Lensing Formalism](chapter_1_introduction/tutorial_5_lensing_formalism.md) — from `scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py` - [Tutorial 6: Data](chapter_1_introduction/tutorial_6_data.md) — from `scripts/chapter_1_introduction/tutorial_6_data.py` - [Tutorial 7: Fitting](chapter_1_introduction/tutorial_7_fitting.md) — from `scripts/chapter_1_introduction/tutorial_7_fitting.py` -- [Tutorial 9: Summary](chapter_1_introduction/tutorial_8_summary.md) — from `scripts/chapter_1_introduction/tutorial_8_summary.py` +- [Tutorial 8: Summary](chapter_1_introduction/tutorial_8_summary.md) — from `scripts/chapter_1_introduction/tutorial_8_summary.py` These pages are regenerated manually by PyAutoHands's `generate_markdown.py` when a curated script changes. diff --git a/markdown/chapter_1_introduction/tutorial_0_visualization.md b/markdown/chapter_1_introduction/tutorial_0_visualization.md index c00cfc4..9d2aa3b 100644 --- a/markdown/chapter_1_introduction/tutorial_0_visualization.md +++ b/markdown/chapter_1_introduction/tutorial_0_visualization.md @@ -10,8 +10,9 @@ clearly in your Jupyter notebook and on your computer screen. __Contents__ -- **Directories:** **PyAutoLens assumes** the working directory is `autolens_workspace` on your hard-disk. +- **Directories:** **HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk. - **Dataset:** Load and plot the strong lens dataset. +- **Dataset Auto-Simulation:** Create the dataset via its simulator script if it is not on your hard-disk. - **Subplots:** In addition to plotting individual figures, **PyAutoLens** can plot `subplots` which show multiple. - **Plot Customization:** Does the figure display correctly on your computer screen? - **Overlays:** Overlays such as critical curves and image positions are added using the `lines=` and `positions=`. @@ -25,16 +26,39 @@ from autolens import jax_wrapper # Sets JAX environment before other imports from autolens import setup_notebook; setup_notebook() ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + +__Directories__ -If the printed working directory does not match the workspace path on your computer, you can manually set it -as follows (the example below shows the path I would use on my laptop. The code is commented out so you do not -use this path in this tutorial! +**HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk, so that relative +paths to `dataset/` and `output/` resolve correctly. + +If your working directory does not match this path on your computer, you can manually set it as follows (the +example below shows the path I would use on my laptop. The code is commented out so you do not use this path in +this tutorial! ```python -# workspace_path = "/Users/Jammy/Code/PyAuto/autolens_workspace" +# workspace_path = "/Users/Jammy/Code/PyAuto/HowToLens" # #%cd $workspace_path # print(f"Working Directory has been set to `{workspace_path}`") ``` @@ -42,10 +66,10 @@ use this path in this tutorial! __Dataset__ The `dataset_path` specifies where the dataset is located, which is the -directory `autolens_workspace/dataset/imaging/simple__no_lens_light`. +directory `dataset/imaging/simple__no_lens_light` of the HowToLens repository. -There are many example simulated images of strong lenses in this directory that will be used throughout the -**HowToLens** lectures. +The simulated images of strong lenses used throughout the **HowToLens** lectures are written to the `dataset` +directory at runtime by the simulator scripts in `scripts/simulator/`. ```python @@ -57,6 +81,31 @@ import autolens.plot as aplt dataset_path = Path("dataset") / "imaging" / "simple__no_lens_light" ``` +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it is created by running the corresponding +simulator script. This ensures every example script can be run without manually simulating data first. + + +```python +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/no_lens_light.py"], + check=True, + ) +``` + + Figure(700x700) + .../PyAutoArray/autoarray/operators/convolver.py:1424: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution. + warnings.warn( + Figure(1800x1800) + Figure(1800x1800) + Figure(700x700) + + We now load this dataset from .fits files and create an instance of an `Imaging` object. @@ -78,7 +127,7 @@ aplt.plot_array(array=dataset.data, title="Dataset Image") -![png](tutorial_0_visualization_files/tutorial_0_visualization_9_0.png) +![png](tutorial_0_visualization_files/tutorial_0_visualization_11_0.png) @@ -96,7 +145,7 @@ aplt.subplot_imaging_dataset(dataset=dataset) -![png](tutorial_0_visualization_files/tutorial_0_visualization_11_0.png) +![png](tutorial_0_visualization_files/tutorial_0_visualization_13_0.png) @@ -106,7 +155,7 @@ Does the figure display correctly on your computer screen? If not, the default matplotlib settings can be customized via the config files in: - autolens_workspace/config/visualize/ + config/visualize/ Key config entries: @@ -126,7 +175,7 @@ aplt.plot_array(array=dataset.data, title="Dataset Image (Log10)", use_log10=Tru -![png](tutorial_0_visualization_files/tutorial_0_visualization_13_0.png) +![png](tutorial_0_visualization_files/tutorial_0_visualization_15_0.png) @@ -168,7 +217,7 @@ aplt.plot_array( -![png](tutorial_0_visualization_files/tutorial_0_visualization_15_0.png) +![png](tutorial_0_visualization_files/tutorial_0_visualization_17_0.png) @@ -186,8 +235,3 @@ The key plotting functions you'll use are: Great! Hopefully, visualization in **PyAutoLens** is displaying nicely for us to get on with the **HowToLens** lecture series. - - -```python - -``` diff --git a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_11_0.png b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_11_0.png index 158af60..b36af26 100644 Binary files a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_11_0.png and b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_11_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_13_0.png b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_13_0.png index 7ab7029..0e14191 100644 Binary files a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_13_0.png and b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_13_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_15_0.png b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_15_0.png index 6d77181..fd379f6 100644 Binary files a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_15_0.png and b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_15_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_17_0.png b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_17_0.png new file mode 100644 index 0000000..c3a8430 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_17_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_9_0.png b/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_9_0.png deleted file mode 100644 index f6e4fd0..0000000 Binary files a/markdown/chapter_1_introduction/tutorial_0_visualization_files/tutorial_0_visualization_9_0.png and /dev/null differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies.md b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies.md index 75e64eb..d959942 100644 --- a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies.md +++ b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies.md @@ -60,6 +60,10 @@ __Contents__ - **One Dimension Projection:** We often want to calculative 1D quantities of a light profile, for example to plot how its light. - **Galaxies:** Galaxies are collections of light profiles that represent a galaxy's luminous emission. - **Units:** By assuming a redshift for a galaxy we can convert its quantities from arcseconds to kiloparsecs. +- **Wrap Up:** Summary of the script and next steps. +- **Advanced Topics:** Optional reference material for expert users, not needed on a first read. +- **Other Unit Conversion:** Other unit conversions a galaxy's redshift enables (e.g. AB magnitudes). +- **Over Sampling:** The adaptive over-sampling scheme used to evaluate light profiles accurately. ```python @@ -76,13 +80,25 @@ import autolens as al import autolens.plot as aplt ``` - 2026-07-11 18:13:08,002 - matplotlib.font_manager - INFO - Failed to extract font properties from /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf: Can not load face (unknown file format; error code 0x2) - - - 2026-07-11 18:13:08,198 - matplotlib.font_manager - INFO - generated new fontManager - - + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) __Grids__ @@ -108,7 +124,7 @@ grid = al.Grid2D.uniform( shape_native=( 101, 101, - ), # The dimensions of the grid, which here is 100 x 100 pixels. + ), # The dimensions of the grid, which here is 101 x 101 pixels. pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds. ) ``` @@ -202,7 +218,7 @@ We can shift the grid to a new center, (y_c, x_c), by subtracting this center fr ```python -centre = (0.3, 0.5) # Shifting the grid to be centered at y=1.0", x=2.0". +centre = (0.3, 0.5) # Shifting the grid to be centered at y=0.3", x=0.5". grid_shifted = grid grid_shifted[:, 0] = grid_shifted[:, 0] - centre[0] # Shift in y-direction. @@ -477,9 +493,9 @@ print(image.slim[1]) ``` Intensity of pixel 0: - 0.01336649114209766 + 0.013366491142097661 Intensity of pixel 1: - 0.014020623587576737 + 0.014020623587576735 To visualize the light profile's image, we use `aplt.plot_array`. @@ -668,9 +684,9 @@ print("...") ``` Intensity of `Grid2D` pixel 0: - 0.024894917164848044 + 0.02489491716484805 Intensity of `Grid2D` pixel 1: - 0.025428546280541572 + 0.025428546280541576 Intensity of `Grid2D` pixel 2: 0.02596640780160061 ... @@ -927,9 +943,4 @@ For a new user, the details of over-sampling are not important, therefore just b adaptive over sampling scheme with high accuracy across all use cases. Once you are more experienced, you should read up on over-sampling in more detail via -the `autolens_workspace/*/guides/over_sampling.ipynb` notebook. - - -```python - -``` +the `autolens_workspace/*/guides/advanced/over_sampling.ipynb` notebook. diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_15_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_15_0.png index a51d3a4..43ab5c0 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_15_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_15_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_19_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_19_0.png index 6524496..f39ba9a 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_19_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_19_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_29_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_29_0.png index d8d2c2c..fc89e50 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_29_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_29_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_33_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_33_0.png index 1047fe1..2e1888a 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_33_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_33_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_37_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_37_0.png index 7393ae2..2cbec13 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_37_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_37_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_43_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_43_0.png index 5945e1e..b7fe2ba 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_43_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_43_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_45_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_45_0.png index 1578cbc..c8f7428 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_45_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_45_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_51_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_51_0.png index 1c45883..0ad7f72 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_51_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_51_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_53_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_53_0.png index 0acaae2..cb2b3f3 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_53_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_53_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_55_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_55_0.png index ae02510..2552728 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_55_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_55_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_57_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_57_0.png index c4da721..e9e7b6f 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_57_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_57_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_5_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_5_0.png index 8487778..e980433 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_5_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_5_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_63_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_63_0.png index 394fd23..6036f1c 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_63_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_63_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_65_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_65_0.png index 140ae5b..a97d7e6 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_65_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_65_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_67_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_67_0.png index 29c3a75..286c839 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_67_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_67_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_71_0.png b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_71_0.png index 9ca518c..7b54a47 100644 Binary files a/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_71_0.png and b/markdown/chapter_1_introduction/tutorial_1_grids_and_galaxies_files/tutorial_1_grids_and_galaxies_71_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing.md b/markdown/chapter_1_introduction/tutorial_2_ray_tracing.md index 14beb71..342ad9d 100644 --- a/markdown/chapter_1_introduction/tutorial_2_ray_tracing.md +++ b/markdown/chapter_1_introduction/tutorial_2_ray_tracing.md @@ -68,6 +68,7 @@ __Contents__ - **Galaxies:** A `Galaxy` can contain both light and mass profiles, forming realistic lens and source galaxies. - **Tracer:** The `Tracer` object automates ray-tracing for a system of galaxies at different redshifts. - **Mappings:** Every image-plane coordinate maps to a source-plane coordinate via the lens equation. +- **Wrap Up:** Summary of the script and next steps. ```python @@ -83,7 +84,25 @@ import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) __Grid__ @@ -565,8 +584,3 @@ computing the image of the entire strong lens system in a single line of code. In the next tutorial, we'll extend these ideas to more complex mass and light distributions, building towards the realistic strong lens systems we observe in real data. - - -```python - -``` diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_0.png index 2acd4b7..c44dd65 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_1.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_1.png index f980698..49e593b 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_1.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_15_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_0.png index fd284d6..7cb8eba 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_1.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_1.png index c8858a0..033aea9 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_1.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_19_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_21_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_21_0.png index c859b9f..90e2bd6 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_21_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_21_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_0.png index db585b8..94904c5 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_1.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_1.png index 74ecd97..966287b 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_1.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_23_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_0.png index 71bc1d4..eeed8b3 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_1.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_1.png index f1c6c59..86a8bcd 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_1.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_27_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_31_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_31_0.png index b48adc0..a809159 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_31_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_31_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_37_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_37_0.png index ebb5cf0..a6e9872 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_37_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_37_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_39_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_39_0.png index 96c4be1..88cd4b4 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_39_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_39_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_0.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_0.png index 71bc1d4..eeed8b3 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_0.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_1.png b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_1.png index c3c2b91..8534434 100644 Binary files a/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_1.png and b/markdown/chapter_1_introduction/tutorial_2_ray_tracing_files/tutorial_2_ray_tracing_43_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing.md b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing.md index d5c3553..e808cf6 100644 --- a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing.md +++ b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing.md @@ -2,7 +2,7 @@ > It shows the example fully executed, with its real output images. > Run it yourself via the [Python script](../../scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py) or the [Jupyter notebook](../../notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb). -Tutorial 5: More Ray Tracing +Tutorial 3: More Ray Tracing ============================ We'll now reinforce the ideas that we learnt about ray-tracing in the previous tutorial and introduce the following @@ -30,7 +30,7 @@ __Contents__ - **Initial Setup:** To begin, lets setup the grid we'll ray-trace using. - **Concise Code:** Lets set up the tracer used in the previous tutorial. -- **Critical Curves:** To end, we can finally explain what the black lines that have appeared on many of the plots. +- **Critical Curves:** We can finally explain what the black lines that have appeared on many of the plots. - **Caustics:** In the previous tutorial, we plotted the critical curves of the mass profile on the image-plane. - **Units:** Lets plot the lensing quantities again. - **More Complexity:** We now make a lens with some attributes we didn`t in the last tutorial. @@ -49,7 +49,25 @@ import autolens as al import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) __Initial Setup__ @@ -106,8 +124,8 @@ tracer = al.Tracer(galaxies=[lens, source]) __Critical Curves__ -To end, we can finally explain what the black lines that have appeared on many of the plots throughout this chapter -actually are. +We can finally explain what the black lines that have appeared on many of the plots throughout this chapter +actually are. These lines are called the 'critical curves', and they define line of infinite magnification due to a mass profile. They therefore mark where in the image-plane a mass profile perfectly `focuses` light rays such that if a source is @@ -121,7 +139,7 @@ However, a radial critical curve only appears when the lens galaxy's mass profil when its inner mass slope is less steep than a steep power-law). To make it appear below, we therefore change the mass profile of our lens galaxy to a `PowerLawSph` with a slope of 1.8. -In the next tutorial, we'll introduce 'caustics', which are where the critical curves map too in the source-plane. +In the next section, we'll introduce 'caustics', which are where the critical curves map to in the source-plane. ```python @@ -140,12 +158,12 @@ radial_critical_curves_list = al.LensCalc.from_mass_obj( __Caustics__ -In the previous tutorial, we plotted the critical curves of the mass profile on the image-plane. We will now plot the +In the previous section, we plotted the critical curves of the mass profile on the image-plane. We will now plot the 'caustics', which correspond to each critical curve ray-traced to the source-plane. This is computed by using the lens galaxy mass profile's to calculate the deflection angles at the critical curves and ray-trace them to the source-plane. -As discussed in the previous tutorial, critical curves mark regions of infinite magnification. Thus, if a source +As discussed in the previous section, critical curves mark regions of infinite magnification. Thus, if a source appears near a caustic in the source plane it will appear significantly brighter than its true luminosity. We again have to use a mass profile with a slope below 2.0 to ensure a radial critical curve and therefore radial @@ -213,9 +231,9 @@ aplt.plot_array(array=tracer.image_2d_list_from(grid=grid)[1], title="Plane 1 Im -Caustics also mark the regions in the source-plane where the multiplicity of the strong lens changes. That is, -if a source crosses a caustic, it goes from 2 images to 1 image. Try and show this yourself by changing the (y,x) -centre of the source-plane galaxy's light profile! +Caustics also mark the regions in the source-plane where the multiplicity of the strong lens changes. Each time +a source crosses a caustic, the number of images it produces changes by two. Try and show this yourself by changing +the (y,x) centre of the source-plane galaxy's light profile! ```python @@ -266,7 +284,7 @@ aplt.subplot_galaxies_images(tracer=tracer, grid=grid) -If you're too familiar with Cosmology, it will be unclear how exactly we converted the distance units from +If you're not too familiar with Cosmology, it will be unclear how exactly we converted the distance units from arcseconds to kiloparsecs. You'll need to read up on your Cosmology lecture to understand this properly. You can create a `Cosmology` object, which provides many methods for calculation different cosmological quantities, @@ -405,11 +423,6 @@ print(lens_satellite) Lets have a quick look at the appearance of our lens galaxy and its satellite. - -```python - -``` - And their deflection angles, noting that the satellite does not contribute as much to the deflections. @@ -486,18 +499,13 @@ print(source_1) Lets look at our source galaxies (before lensing) - -```python - -``` - __Multi Galaxy Ray Tracing__ Now lets pass our 4 galaxies to a `Tracer`, which means the following will occur: - - Using the galaxy redshift`s, and image-plane and source-plane will be created each with two galaxies galaxies. + - Using the galaxy redshifts, an image-plane and source-plane will be created, each with two galaxies. -We've also pass the tracer below a Planck15 cosmology, where the cosomology of the Universe describes exactly how +We also pass the tracer below a Planck15 cosmology, where the cosmology of the Universe describes exactly how ray-tracing is performed. @@ -522,7 +530,7 @@ We can now plot the tracer`s image, which now there are two galaxies in each pla 5) The image of the source galaxies is computed by summing both of their images and ray-tracing their light back to the image-plane. -This process is pretty much the same as we have single in previous tutorials when there is one galaxy per plane. We +This process is pretty much the same as we have seen in previous tutorials when there is one galaxy per plane. We are simply summing the images and deflection angles of the galaxies before using them to perform ray-tracing. @@ -532,7 +540,7 @@ aplt.plot_array(array=tracer.image_2d_from(grid=grid), title="Image") -![png](tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_35_0.png) +![png](tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_31_0.png) @@ -545,7 +553,7 @@ aplt.plot_grid(grid=tracer.traced_grid_2d_list_from(grid=grid)[1], title="Plane -![png](tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_37_0.png) +![png](tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_33_0.png) @@ -559,13 +567,13 @@ aplt.plot_grid(grid=tracer.traced_grid_2d_list_from(grid=grid)[1], title="Plane -![png](tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_39_0.png) +![png](tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_35_0.png) __Wrap Up__ -Tutorial 6 completed! Try the following: +Tutorial 3 completed! Try the following: 1) If you change the lens and source galaxy redshifts, does the tracer's image change? @@ -574,8 +582,3 @@ Tutorial 6 completed! Try the following: 3) The tracer has a small delay in being computed, whereas other tracers were almost instant. What do you think is the cause of this slow-down? - - -```python - -``` diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_0.png index 96ec34a..39d80a4 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_0.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_1.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_1.png index 7357030..3be40ee 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_1.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_11_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_13_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_13_0.png index c423256..c41da72 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_13_0.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_13_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_15_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_15_0.png index 3f1fadb..efad736 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_15_0.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_15_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_0.png index e474086..2bc8a7a 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_0.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_1.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_1.png index 194e375..47e1e7a 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_1.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_17_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_31_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_31_0.png new file mode 100644 index 0000000..5846972 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_31_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_33_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_33_0.png new file mode 100644 index 0000000..0943a6b Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_33_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_35_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_35_0.png index 076478c..0943a6b 100644 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_35_0.png and b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_35_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_37_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_37_0.png deleted file mode 100644 index 7e02c59..0000000 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_37_0.png and /dev/null differ diff --git a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_39_0.png b/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_39_0.png deleted file mode 100644 index 7e02c59..0000000 Binary files a/markdown/chapter_1_introduction/tutorial_3_more_ray_tracing_files/tutorial_3_more_ray_tracing_39_0.png and /dev/null differ diff --git a/markdown/chapter_1_introduction/tutorial_4_point_sources.md b/markdown/chapter_1_introduction/tutorial_4_point_sources.md index a2dbc3c..14cf05e 100644 --- a/markdown/chapter_1_introduction/tutorial_4_point_sources.md +++ b/markdown/chapter_1_introduction/tutorial_4_point_sources.md @@ -5,15 +5,65 @@ Tutorial 4: Point Sources ========================= -This tutorial is not wrriten yet, but will explain how point source lensing works. +In the previous tutorials, the background sources we lensed were galaxies: extended objects whose light spreads over +many thousands of light years. When lensed, their light is warped into the arcs and Einstein rings we produced with +the `Tracer`, spread across many pixels of the image. -This tutorial is not necesary for using PyAutoLens or doing strong lens analysis, so don't worry that it is not -written yet! +However, not every source is a galaxy. Some of the most scientifically valuable strong lenses have a background +source that is physically tiny: a **quasar** (the bright accretion disk around a supermassive black hole, light-days +across) or a **supernova** (an exploding star, even smaller). These are called **point sources**. -Tutorial 8 summary is written and you should check that out instead! +When a point source is strongly lensed, we do not see arcs. Instead, we see the same source appear multiple times, +as two or four (or more) distinct, unresolved points of light in the image. Modeling these systems is fundamentally +different from modeling extended sources — different enough that it deserves its own tutorial before we go any +further. + +In this tutorial, we will see why. The short version: for an extended source, we ray-trace light *forwards* from the +image-plane to the source-plane, which is computationally simple. For a point source, we must go *backwards* — we +know where the source is, and we must find every image-plane position its light appears at. That means solving the +lens equation, an inverse problem with no analytic solution, which requires a dedicated numerical tool called the +`PointSolver`. + +Only the lens equation from tutorial 2 is needed to follow this tutorial. The full lensing formalism — convergence, +potentials, and how deflection angles arise from a mass distribution — is covered in the next tutorial. + +Here is an overview of what we'll cover: + +- **Initial Setup**: Create a strong lens with a compact extended source, showing how a small source produces + distinct multiple images rather than arcs. + +- **Point Sources**: What a point source is physically, and why quasars and supernovae appear as multiple + unresolved images. + +- **Point Source Tracer**: Represent a point source in PyAutoLens using the `Point` profile and build a `Tracer`. + +- **The Lens Equation**: Why finding a point source's multiple images means solving the lens equation — an inverse + problem with no analytic solution. + +- **Point Solver**: The `PointSolver`, which solves the lens equation numerically by ray-tracing triangles. + +- **Multiple Images and Critical Curves**: Plot the solved image positions over the lensed image and the tracer's + critical curves. + +- **Magnifications**: Compute the magnification at each multiple image, and why observed fluxes are often + unreliable for modeling. + +- **Time Delays**: The arrival-time differences between multiple images, and why they matter for cosmology. + +- **Extended Versus Point Computations**: A recap of why the two regimes require fundamentally different + calculations and fitting strategies. __Contents__ +- **Initial Setup:** Create a strong lens with a compact extended source, producing distinct multiple images. +- **Point Sources:** What a point source is physically and why it appears as multiple unresolved images. +- **Point Source Tracer:** Represent a point source with the `Point` profile and build a `Tracer`. +- **The Lens Equation:** Finding multiple images means solving the lens equation, an inverse problem. +- **Point Solver:** The `PointSolver` solves the lens equation numerically via triangle ray-tracing. +- **Multiple Images and Critical Curves:** Plot the solved positions over the image and critical curves. +- **Magnifications:** The magnification of each multiple image, and why fluxes are often unreliable. +- **Time Delays:** The relative arrival times of the multiple images and their use in cosmology. +- **Extended Versus Point Computations:** Why the two regimes require fundamentally different calculations. - **Wrap Up:** Summary of the script and next steps. @@ -23,16 +73,390 @@ from autolens import jax_wrapper # Sets JAX environment before other imports from autolens import setup_notebook; setup_notebook() +import numpy as np + import autolens as al import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) -__Wrap Up__ +__Initial Setup__ + +Let's begin with a strong lens system like those of the previous tutorials: an elliptical isothermal mass profile +for the lens galaxy, and an extended light profile for the source galaxy. + +There is one deliberate difference: the source's `effective_radius` is just 0.1", making it far more compact than +the sources we lensed before. ```python +grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.05, +) + +lens_galaxy = al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=1.6, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0), + ), +) + +source_galaxy_extended = al.Galaxy( + redshift=1.0, + light=al.lp.ExponentialCore( + centre=(0.07, 0.07), intensity=0.1, effective_radius=0.1 + ), +) +tracer_extended = al.Tracer(galaxies=[lens_galaxy, source_galaxy_extended]) ``` + +When we plot the image of this system, something new happens: instead of a sweeping arc or Einstein ring, the +source appears as distinct blobs of light — the same source, imaged multiple times at different locations in +the image-plane. + +This is a general rule of strong lensing: the more compact the source, the more its lensed image breaks up into +separate multiple images. An extended galaxy blurs its multiple images together into arcs; a compact source keeps +them distinct. + + +```python +aplt.plot_array( + array=tracer_extended.image_2d_from(grid=grid), + title="Compact Source Multiple Images", +) +``` + + + +![png](tutorial_4_point_sources_files/tutorial_4_point_sources_5_0.png) + + + +__Point Sources__ + +Now imagine shrinking the source further — not to 0.01", but to the physical size of a quasar accretion disk or a +supernova photosphere. These objects are light-days to light-weeks across, which at cosmological distances +corresponds to micro-arcseconds on the sky. That is millions of times smaller than the resolution of any telescope. + +Strictly speaking, such a source still has a finite size, but the telescope cannot resolve it. Each multiple image +appears as a single unresolved point of light, with all of its flux effectively contained within one pixel (spread +only by the telescope's blurring, which we will meet in a later tutorial). + +This is what we call a **point source**. Lensed quasars and lensed supernovae are the classic examples, and they +are prized objects: the arrival-time differences between their multiple images can be used to measure the expansion +rate of the Universe (the Hubble constant), and their fluxes are sensitive probes of small-scale dark matter +structure. + +For a point source, the extended-source picture of lensing breaks down: + +- There is no arc or ring of surface brightness to fit, just a handful of image positions. + +- The observable quantities are the (y,x) **positions** of the multiple images, and optionally their **fluxes** + and **time delays**. + +- The concept of evaluating a light profile's surface brightness on a ray-traced grid — the engine of the previous + two tutorials — no longer applies, because the source has no extent to evaluate. + +__Point Source Tracer__ + +PyAutoLens represents a point source with the `Point` profile, found in the `al.ps` module (`ps` for point source). +Unlike a light profile it has no `intensity`, `effective_radius` or elliptical components — it is fully described +by its (y,x) centre in the source-plane. + +We give the point source the same centre, (0.07", 0.07"), as the compact extended source above, so its multiple +images will coincide with the bright blobs in the image we just plotted. + + +```python +point_source = al.ps.Point(centre=(0.07, 0.07)) + +source_galaxy = al.Galaxy(redshift=1.0, point_0=point_source) + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) +``` + +Note that we attached the point source to its galaxy with the name `point_0`. This name is a label that PyAutoLens +uses when fitting real data to pair each point source in the model with the dataset containing its observed image +positions. With one source the name is a formality, but group- and cluster-scale lenses can contain many point +sources, and the names keep each one matched to its own data. + +__The Lens Equation__ + +In tutorial 2, we met the **lens equation**, which relates a coordinate in the image-plane to the coordinate in +the source-plane its light ray comes from: + +$\beta = \theta - \alpha(\theta)$ + +where $\theta$ is the image-plane (y,x) coordinate, $\alpha(\theta)$ is the deflection angle at that coordinate, +and $\beta$ is the source-plane (y,x) coordinate. + +For the extended sources of tutorials 2 and 3, we always used this equation in the *forward* direction: take every +image-plane coordinate $\theta$ on our grid, compute its deflection angles, and subtract to get $\beta$. This is +easy — it is just an evaluation, one subtraction per coordinate. + +For a point source, the question is reversed. We *know* the source's position $\beta$ (the centre of the `Point` +profile), and we want to find every image-plane position $\theta$ that satisfies the lens equation for that +$\beta$. Each solution is one of the multiple images. + +This is called **solving the lens equation**, and it is much harder than evaluating it: + +- The equation is non-linear, because the deflection angles $\alpha(\theta)$ vary with position in a way set by the + mass profile. For all but the simplest mass profiles, no analytic solution exists. + +- There are multiple solutions — that is the whole point! A single $\beta$ maps to two, four or more distinct + $\theta$ positions, and we must find all of them. + +We therefore need a numerical root-finding method that searches the image-plane for every position whose ray-traced +source-plane coordinate lands on the source. (Where the deflection angles themselves come from is the subject of +the next tutorial — for now, we just need the equation.) + +__Point Solver__ + +PyAutoLens solves the lens equation with the `PointSolver`, which uses a triangle-based scheme: + +1. Overlay the image-plane with a grid of triangles. + +2. Ray-trace the corners of every triangle to the source-plane using the lens equation, giving a set of + ray-traced source-plane triangles. + +3. Keep every triangle that contains the source's (y,x) centre — somewhere inside that image-plane triangle is a + multiple image. + +4. Subdivide the retained triangles into smaller triangles and repeat, homing in on each multiple image with + progressively finer precision. + +The `pixel_scale_precision` input sets the triangle size at which the solver stops refining, and therefore the +precision of the image positions. Smaller values are more precise but cost more computation; 0.001" is a good +balance for most lens modeling. + +Mass models also predict a faint "central image" near the centre of the lens, which is usually so heavily +demagnified that real observations never detect it. The `magnification_threshold` input discards solved images +whose magnification is below the threshold, removing this central image to match what the data would contain. + + +```python +solver_grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.2, +) + +solver = al.PointSolver.for_grid( + grid=solver_grid, pixel_scale_precision=0.001, magnification_threshold=0.1 +) +``` + +We now solve the lens equation, passing the solver our tracer (which provides the mass model's deflection angles) +and the source-plane centre of the point source. + +The result is the set of image-plane (y,x) coordinates of the multiple images. + + +```python +positions = solver.solve(tracer=tracer, source_plane_coordinate=(0.07, 0.07)) + +print("Multiple Image Positions (y,x) in arc-seconds:") +print(positions) +``` + + Multiple Image Positions (y,x) in arc-seconds: + Grid2DIrregular([[-1.0390625 , -1.03832837], + [ 0.4421875 , 1.60846177], + [ 1.609375 , 0.4420338 ], + [ 1.17890625, 1.17950856]]) + + +__Multiple Images and Critical Curves__ + +For our elliptical mass profile, the solver finds four multiple images (a fifth, central image existed but was +removed by the `magnification_threshold`). The number of images depends on the mass model: + +- Spherical mass profiles produce three images (including the demagnified central image). + +- Elliptical mass profiles produce five images (again including the central image). + +- More complex systems, with multiple lens galaxies, can produce even more. + +To see that the solver got the right answer, we plot the solved positions on top of the compact extended source's +image from the start of the tutorial. The positions land on the centres of the bright multiple images. + +We also overlay the tracer's tangential critical curve (introduced in tutorial 3 — the line of infinite +magnification in the image-plane). The multiple images of a strongly lensed point source straddle the critical +curve, and images close to it are the most highly magnified. + + +```python +tangential_critical_curve_list = al.LensCalc.from_tracer( + tracer=tracer +).tangential_critical_curve_list_from(grid=grid) + +aplt.plot_array( + array=tracer_extended.image_2d_from(grid=grid), + positions=positions, + lines=tangential_critical_curve_list, + title="Multiple Images and Critical Curve", +) +``` + + + +![png](tutorial_4_point_sources_files/tutorial_4_point_sources_13_0.png) + + + +__Magnifications__ + +Lensing does not just relocate a point source's light — it magnifies it. Each multiple image has its own +magnification, set by how strongly the mass model focuses light rays at that image-plane position, and it can make +an image tens of times brighter than the unlensed source. + +For an extended source, magnification stretches the source over more pixels. For a point source, the image stays +unresolved — so the magnification instead multiplies its observed **flux**. If the source's intrinsic flux is $F$, +the flux of a multiple image with magnification $\mu$ is $|\mu| * F$. + +We compute the magnification at each solved image position below (the sign of $\mu$ encodes the image's parity — +whether it is mirror-flipped — which is why we take its absolute value for fluxes). + + +```python +magnifications = al.LensCalc.from_tracer( + tracer=tracer +).magnification_2d_via_hessian_from(grid=positions) + +print("Magnification of each Multiple Image:") +print(magnifications) + +flux = 1.0 +fluxes = [flux * np.abs(magnification) for magnification in magnifications] + +print("Flux of each Multiple Image (for a source of flux 1.0):") +print(fluxes) +``` + + Magnification of each Multiple Image: + ArrayIrregular([ -6.8235945 , 55.19635531, 53.66786018, -100.70748948]) + Flux of each Multiple Image (for a source of flux 1.0): + [np.float64(6.823594498831482), np.float64(55.1963553090608), np.float64(53.667860180048564), np.float64(100.70748948418353)] + + +In principle, comparing the observed flux ratios of the images to the model's magnification ratios is a powerful +constraint on the mass model. In practice, point-source fluxes are treated with caution, because effects the smooth +mass model does not include can change them substantially: + +- **Microlensing**: individual stars in the lens galaxy lens the point source again on micro-arcsecond scales, + magnifying or demagnifying each image by unpredictable amounts that change over time. + +- **Dark matter substructure**: small invisible clumps of dark matter near an image perturb its magnification + (this is a systematic for mass modeling, but also exactly why lensed quasars are used to *search* for + substructure). + +- Dust in the lens galaxy and the source's own intrinsic variability further distort the observed flux ratios. + +Image positions are essentially immune to all of these, which is why point-source lens modeling is built on +positions first, with fluxes included only when the science demands it and the systematics are under control. + +__Time Delays__ + +There is a third observable unique to point sources. The light of each multiple image travels a different path +through the Universe, and each path takes a different time — partly because the geometric path lengths differ, and +partly because light is slowed as it climbs through the lens galaxy's gravitational field (the Shapiro delay). + +For an extended, steady source this is unobservable. But quasars flicker and supernovae explode: when the source +varies, the variation appears in each multiple image at a different time. By monitoring the images, these +**time delays** can be measured — typically days to months apart. + +Time delays are a cosmological gold mine: their absolute scale depends on the distances between us, the lens and +the source, so a lens model plus measured delays yields the Hubble constant. This is time-delay cosmography, one +of the headline applications of point-source lensing. + +The tracer computes the arrival time at each image position from its mass model and the galaxies' redshifts. Only +the *differences* between images are observable — a delay of the same length along every path is invisible — so it +is the relative values below that matter. + + +```python +time_delays = tracer.time_delays_from(grid=positions) + +print("Time Delay of each Multiple Image (days):") +print(time_delays) + +print("Time Delays relative to first image (days):") +print(time_delays - time_delays[0]) +``` + + Time Delay of each Multiple Image (days): + ArrayIrregular([-136.98563013, -176.85048802, -177.0200605 , -176.7402858 ]) + Time Delays relative to first image (days): + ArrayIrregular([ 0. , -39.86485789, -40.03443037, -39.75465568]) + + +__Extended Versus Point Computations__ + +We can now state precisely why point-source lensing works so differently from everything in tutorials 2 and 3, on +both the computation and the data. + +**Extended sources are forward-modeled.** To predict the data, we ray-trace every image-plane pixel to the +source-plane via the lens equation (an evaluation, not a solve) and compute the source light profile's surface +brightness at each ray-traced coordinate. This is computationally cheap, requires no equation solving, and a fit +compares the predicted and observed images pixel-by-pixel. + +**Point sources require solving an inverse problem.** The source has no surface brightness to evaluate — the model +must predict the discrete positions of the multiple images, which means numerically solving the lens equation with +the `PointSolver` every time the mass model changes. A fit then compares the predicted image positions to the +observed ones (and optionally the predicted fluxes and time delays to their measurements) — a handful of numbers, +rather than thousands of pixels. + +This trade shapes the whole analysis. The point-source dataset is tiny — a few positions with uncertainties — but +each model evaluation involves root finding over the image plane rather than simple forward ray-tracing. Whole +questions that never arose for extended sources, such as how to pair predicted images with observed ones when the +model predicts too many or too few, become central to the likelihood (the workspace covers these in detail). + +__Wrap Up__ + +In this tutorial, we met point sources and the tools PyAutoLens uses to model them. Let's summarise what we've +learnt: + +- **Point Sources**: Quasars and supernovae are so much smaller than a telescope's resolution that each of their + multiple images appears as a single unresolved point of light — no arcs, no rings. + +- **The Lens Equation**: Point-source modeling means solving $\beta = \theta - \alpha(\theta)$ for the image + positions $\theta$ given the source position $\beta$ — a non-linear inverse problem with no analytic solution, + in contrast to the cheap forward evaluation used for extended sources. + +- **Point Solver**: The `PointSolver` solves the lens equation numerically, ray-tracing progressively finer + triangles until every multiple image is located to sub-pixel precision, and filtering out the demagnified + central image. + +- **Observables**: A point-source dataset consists of image positions, and optionally fluxes and time delays. + Positions are the bedrock; fluxes are often compromised by microlensing and substructure; time delays enable + measurement of the Hubble constant. + +When you are ready to model real lensed quasars and supernovae — composing mass models, fitting observed positions +with a non-linear search, and including fluxes and time delays — the `autolens_workspace/scripts/point_source` +package is the place to go, starting with its `start_here` example. + +In the next tutorial, we return to the lensing formalism itself: where deflection angles come from, and the +quantities (convergence, potential, magnification) that describe a mass distribution's lensing power. diff --git a/markdown/chapter_1_introduction/tutorial_4_point_sources_files/tutorial_4_point_sources_13_0.png b/markdown/chapter_1_introduction/tutorial_4_point_sources_files/tutorial_4_point_sources_13_0.png new file mode 100644 index 0000000..e83cad5 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_4_point_sources_files/tutorial_4_point_sources_13_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_4_point_sources_files/tutorial_4_point_sources_5_0.png b/markdown/chapter_1_introduction/tutorial_4_point_sources_files/tutorial_4_point_sources_5_0.png new file mode 100644 index 0000000..0c8c00c Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_4_point_sources_files/tutorial_4_point_sources_5_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism.md b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism.md index 5f58aa8..bb3df02 100644 --- a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism.md +++ b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism.md @@ -5,16 +5,37 @@ Tutorial 5: Lensing Formalism ============================= -This tutorial is not wrriten yet, but will explain what all the different lens quantities are and give a more -formal description of them. +This tutorial is the equations lecture of **HowToLens**. -This tutorial is not necesary for using PyAutoLens or doing strong lens analysis, so don't worry that it is not -written yet! +Every other tutorial in this series deliberately defers the formal mathematics of gravitational lensing, so that you +could first learn lensing hands-on: in the earlier tutorials we created grids, evaluated light and mass profiles, +ray-traced coordinates from the image-plane to the source-plane and looked at quantities like the convergence, +deflection angles and critical curves visually, without ever writing down their formal definitions. -Tutorial 8 summary is written and you should check that out instead! +This tutorial gathers all of that algebra in one place. Each quantity you have already computed and plotted — +deflection angles, convergence, potential, magnification, critical curves, the Einstein radius — is now given its +formal mathematical definition, explained in plain words, and tied back to the exact **PyAutoLens** method that +computes it. The goal is that after this tutorial, every symbol in a lensing paper maps to a line of code you know. + +None of this mathematics is required to use **PyAutoLens** or to analyse strong lenses — the library computes +everything for you. But understanding where each quantity comes from will make you a better lens modeler, and this +is the single place in **HowToLens** where the formalism lives, so you can return here whenever you need it. + +We will build one simple strong lens system — an elliptical isothermal (SIE) lens galaxy and a Sersic source — and +reuse it throughout, computing every quantity from the same tracer so you can see how they all fit together. __Contents__ +- **Initial Setup:** Build the grid, SIE lens, Sersic source and `Tracer` reused throughout the tutorial. +- **Cosmological Distances:** Angular diameter distances D_l, D_s, D_ls and how arcseconds convert to kiloparsecs. +- **The Lens Equation:** beta = theta - alpha(theta), the mapping from image-plane to source-plane, and multiple images. +- **Convergence:** Surface mass density Sigma, the critical surface density Sigma_cr and kappa = Sigma / Sigma_cr. +- **Deflection Angles:** The integral relating the convergence field to the deflection angles. +- **The Lensing Potential:** The scalar potential psi, with alpha = grad psi and kappa = (1/2) laplacian psi. +- **Shear and Magnification:** The Hessian, the Jacobian matrix A, shear gamma and magnification mu = 1 / det(A). +- **Critical Curves and Caustics:** Where det(A) = 0 in the image-plane, and its map to the source-plane. +- **Einstein Radius:** Its definition, the SIS and point-mass closed forms, and why it is the robust mass measurement. +- **Time Delays:** The Fermat potential and the time-delay surface, in brief. - **Wrap Up:** Summary of the script and next steps. @@ -24,16 +45,690 @@ from autolens import jax_wrapper # Sets JAX environment before other imports from autolens import setup_notebook; setup_notebook() +import matplotlib.pyplot as plt +import numpy as np import autolens as al +import autoarray as aa import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) -__Wrap Up__ +__Initial Setup__ + +We first build the strong lens system used throughout this tutorial, mirroring the objects introduced in tutorials +2 and 3: a 2D grid of (y,x) arcsecond coordinates, a lens galaxy with an elliptical isothermal (`Isothermal`) mass +profile — the singular isothermal ellipsoid, or SIE, the workhorse mass model of galaxy-scale lensing — and a +source galaxy with a Sersic light profile. + + +```python +grid = al.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05) + +mass_profile = al.mp.Isothermal( + centre=(0.0, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0), + einstein_radius=1.6, +) + +lens_galaxy = al.Galaxy(redshift=0.5, mass=mass_profile) + +source_galaxy = al.Galaxy( + redshift=1.0, + bulge=al.lp.SersicCore( + centre=(0.0, 0.1), + ell_comps=(0.1, 0.0), + intensity=0.3, + effective_radius=0.3, + sersic_index=1.0, + ), +) +``` + +We combine these into a `Tracer` with a Planck 2015 cosmology. The redshifts of the two galaxies (0.5 and 1.0) and +the cosmology together fix every distance that appears in the equations below. + + +```python +cosmology = al.cosmo.Planck15() + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy], cosmology=cosmology) +``` + +Here is the lensed image of this system, which by now should look familiar. Every equation in this tutorial is +describing some aspect of how this image forms. + + +```python +image = tracer.image_2d_from(grid=grid) +aplt.plot_array(array=image, title="Lensed Image of the Source Galaxy") +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_7_0.png) + + + +We also create a `LensCalc` object from the tracer. As we saw in tutorial 3, this is **PyAutoLens**'s calculator for +derived lensing quantities — everything it computes is derived from the tracer's deflection angles, which is a theme +we will see repeatedly below. + + +```python +lens_calc = al.LensCalc.from_tracer(tracer=tracer) +``` + +__Cosmological Distances__ + +Gravitational lensing is a geometric effect, so distances are everywhere in its equations. The distances used are +**angular diameter distances**, defined so that an object of physical size l at angular diameter distance D subtends +an angle (in radians): + + theta = l / D + +Three distances appear in every lensing formula: + +- D_l — the angular diameter distance from the observer to the lens (deflector) galaxy. +- D_s — the angular diameter distance from the observer to the source galaxy. +- D_ls — the angular diameter distance from the lens to the source. + +In an expanding Universe angular diameter distances do not add linearly, so D_ls != D_s - D_l. Each is computed by +integrating the cosmological expansion history between the two redshifts, which is why the tracer requires both +galaxy redshifts and a cosmology. + +The `Cosmology` object computes each of these directly (values below are in kiloparsecs): + + +```python +D_l = cosmology.angular_diameter_distance_to_earth_in_kpc_from(redshift=0.5) +D_s = cosmology.angular_diameter_distance_to_earth_in_kpc_from(redshift=1.0) +D_ls = cosmology.angular_diameter_distance_between_redshifts_in_kpc_from( + redshift_0=0.5, redshift_1=1.0 +) + +print(f"D_l (kpc) = {D_l:.3e}") +print(f"D_s (kpc) = {D_s:.3e}") +print(f"D_ls (kpc) = {D_ls:.3e}") +``` + + D_l (kpc) = 1.297e+06 + D_s (kpc) = 1.698e+06 + D_ls (kpc) = 7.252e+05 + + +The angular diameter distance is also what converts **PyAutoLens**'s internal angular units of arcseconds into +physical distances. An angle of 1.0" at redshift z corresponds to a physical size: + + l = D(z) * (1.0" in radians) + +The `Cosmology` object wraps this up as a `kpc_per_arcsec` conversion factor, which differs between the image-plane +(lens redshift) and source-plane (source redshift) because the two planes are at different distances. + + +```python +kpc_per_arcsec_lens = cosmology.kpc_per_arcsec_from(redshift=0.5) +kpc_per_arcsec_source = cosmology.kpc_per_arcsec_from(redshift=1.0) + +print(f"kpc per arcsec at the lens (z=0.5) = {kpc_per_arcsec_lens:.4f}") +print(f"kpc per arcsec at the source (z=1.0) = {kpc_per_arcsec_source:.4f}") +``` + + kpc per arcsec at the lens (z=0.5) = 6.2882 + kpc per arcsec at the source (z=1.0) = 8.2319 + + +So our lens's Einstein radius of 1.6" corresponds to a physical scale of roughly 1.6 * 6.3 ~ 10 kpc at the lens +redshift — a sensible size for the inner regions of a massive elliptical galaxy. + +__The Lens Equation__ + +The fundamental equation of gravitational lensing is the **lens equation**, which we met in tutorial 2: + + beta = theta - alpha(theta) + +where: + +- theta is the observed (image-plane) angular position of a light ray, in arcseconds. +- alpha(theta) is the (scaled) deflection angle — how much the ray is bent by the lens's gravity at position theta. +- beta is the true (source-plane) position — where the source would appear if there were no lens. + +In plain words: to find where a light ray we observe at theta really came from, subtract the deflection the lens +applied to it. This is exactly the ray-tracing calculation we performed in tutorial 2, and it is worth knowing that +the deflection angle here is the "scaled" deflection: the physical bending angle of the ray multiplied by the +distance ratio D_ls / D_s, which is the convention that makes the lens equation take the simple form above. + +The lens equation is trivial to evaluate one way (given theta, compute beta), but it cannot be inverted +analytically: for a strong lens, *multiple* image-plane positions theta can satisfy the equation for the *same* +source position beta. This is why strong lenses produce multiple images of a single source — and why lens modeling +works "forwards", ray-tracing image-plane grids to the source-plane rather than the other way around. + +In code, the lens equation is one line — the same `grid_2d_via_deflection_grid_from` call from tutorial 2: + + +```python +deflections = tracer.deflections_yx_2d_from(grid=grid) + +source_plane_grid = grid.grid_2d_via_deflection_grid_from(deflection_grid=deflections) + +aplt.plot_grid(grid=grid, title="Image-Plane Grid (theta)") +aplt.plot_grid(grid=source_plane_grid, title="Source-Plane Grid (beta)") +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_0.png) + + + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_1.png) + + + +__Convergence__ + +The lens galaxy's mass enters lensing via its **surface mass density** Sigma(theta): the galaxy's 3D mass density +integrated along the line of sight, giving a projected 2D mass per unit area (e.g. in solar masses per kpc^2). +Lensing only cares about this projection — two very different 3D distributions with the same projected Sigma +deflect light identically. + +Whether a lens is "strong" is set by comparing Sigma to the **critical surface density**: + + Sigma_cr = (c^2 / (4 pi G)) * (D_s / (D_l * D_ls)) + +This is a purely cosmological quantity — it depends only on the constants c and G and the three distances from the +previous section. It has a beautiful interpretation: it is the surface density a lens needs, given this observer / +lens / source geometry, to be capable of producing multiple images. + + +```python +sigma_cr_kpc = cosmology.critical_surface_density_between_redshifts_solar_mass_per_kpc2_from( + redshift_0=0.5, redshift_1=1.0 +) +sigma_cr_arcsec = cosmology.critical_surface_density_between_redshifts_from( + redshift_0=0.5, redshift_1=1.0 +) + +print(f"Sigma_cr (solar masses per kpc^2) = {sigma_cr_kpc:.3e}") +print(f"Sigma_cr (solar masses per arcsec^2) = {sigma_cr_arcsec:.3e}") +``` + + Sigma_cr (solar masses per kpc^2) = 3.002e+09 + Sigma_cr (solar masses per arcsec^2) = 1.187e+11 + + +The **convergence** kappa is the surface mass density in units of the critical density: + + kappa(theta) = Sigma(theta) / Sigma_cr + +This is the dimensionless quantity **PyAutoLens** has been plotting since tutorial 2 whenever we called +`convergence_2d_from`: + +- kappa >= 1 — the lens is super-critical at that point; multiple imaging is possible. +- kappa < 1 — the lens is sub-critical there. + +Because kappa is dimensionless, lens models can be composed and fitted without knowing the galaxy redshifts at all — +the redshifts and Sigma_cr are only needed at the end, to convert the inferred kappa back into physical masses. + + +```python +convergence = tracer.convergence_2d_from(grid=grid) + +aplt.plot_array(array=convergence, title="Convergence (kappa)", use_log10=True) +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_19_0.png) + + + +__Deflection Angles__ + +The convergence field determines the deflection angles everywhere, via a 2D integral over the whole image-plane: + + alpha(theta) = (1 / pi) * integral d^2 theta' kappa(theta') * (theta - theta') / |theta - theta'|^2 + +In plain words: every patch of mass in the lens plane pulls on every light ray. The deflection at position theta is +the sum (integral) of the pulls from all mass elements kappa(theta'), each pointing from the mass element towards +the ray and falling off as 1/distance — the 2D (projected) analogue of Newtonian gravity's inverse-square law. + +Two things follow from this integral being over *all* theta': + +- The deflection at a point depends on the mass distribution *everywhere*, not just the mass at that point. Even + mass well outside the region where images form contributes deflections (this is why external shear from + neighbouring galaxies matters, as we saw in tutorial 3). + +- For simple analytic profiles (isothermal, power-law, Sersic-like) this integral has closed-form solutions, which + is exactly what a **PyAutoLens** mass profile is: an analytic kappa(theta) paired with its analytic alpha(theta). + +The `deflections_yx_2d_from` method we have used since tutorial 2 evaluates this integral's closed-form solution. +The deflections form a 2D vector field, so we plot its y and x components separately: + + +```python +deflections = tracer.deflections_yx_2d_from(grid=grid) + +deflections_y = aa.Array2D(values=deflections.slim[:, 0], mask=grid.mask) +aplt.plot_array(array=deflections_y, title="Deflection Angles (y component)") + +deflections_x = aa.Array2D(values=deflections.slim[:, 1], mask=grid.mask) +aplt.plot_array(array=deflections_x, title="Deflection Angles (x component)") +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_0.png) + + + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_1.png) + + + +__The Lensing Potential__ + +The deflection field is not arbitrary — it is the gradient of a scalar field, the **lensing potential** psi(theta): + + alpha(theta) = grad psi(theta) + +The potential is itself an integral over the convergence: + + psi(theta) = (1 / pi) * integral d^2 theta' kappa(theta') * ln|theta - theta'| + +and taking the divergence of the gradient (the Laplacian) recovers the convergence with a factor of a half: + + kappa(theta) = (1/2) * laplacian psi(theta) + +This is the 2D Poisson equation of lensing — the projected analogue of Newtonian gravity's del^2 Phi = 4 pi G rho. +The potential is the single most economical description of a lens: one scalar field from which the deflections +(first derivatives), and the convergence, shear and magnification (second derivatives) all follow. + +The `potential_2d_from` method returns psi, which we have plotted before without defining it: ```python +potential = tracer.potential_2d_from(grid=grid) +aplt.plot_array(array=potential, title="Lensing Potential (psi)") ``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_23_0.png) + + + +We can verify alpha = grad psi numerically. Mass profiles have a `deflections_2d_via_potential_2d_from` method which +computes the deflections by numerically differentiating the potential, rather than using the profile's closed-form +deflection formula. Across the grid the two agree closely — we print the median absolute difference, since the +finite-difference derivative is inaccurate right at the isothermal profile's central cusp, where the potential is +not smooth (the closed-form deflections have no such problem): + + +```python +deflections_analytic = mass_profile.deflections_yx_2d_from(grid=grid) +deflections_via_potential = mass_profile.deflections_2d_via_potential_2d_from(grid=grid) + +difference = np.median( + np.abs(np.asarray(deflections_analytic) - np.asarray(deflections_via_potential)) +) +print(f"Median |alpha_analytic - grad psi| = {difference:.3e} arcsec") +``` + + Median |alpha_analytic - grad psi| = 1.095e-04 arcsec + + +__Shear and Magnification__ + +How a small image is distorted by lensing is governed by how the deflection angles *change* across it — the second +derivatives of the potential. These form the 2x2 **Hessian** matrix, which `LensCalc` computes by finite +differences of the deflection field (as we saw in the workspace guides, this works for any mass distribution): + + H_yy = d(alpha_y) / d(theta_y) H_xy = d(alpha_x) / d(theta_y) + H_yx = d(alpha_y) / d(theta_x) H_xx = d(alpha_x) / d(theta_x) + + +```python +hessian_yy, hessian_xy, hessian_yx, hessian_xx = lens_calc.hessian_from(grid=grid) + +print(f"Hessian components at pixel 0: H_yy = {hessian_yy[0]:.4f}, H_xx = {hessian_xx[0]:.4f}") +``` + + Hessian components at pixel 0: H_yy = 0.2032, H_xx = 0.2032 + + +Differentiating the lens equation beta = theta - alpha(theta) gives the **Jacobian matrix** A, which maps a small +displacement in the source-plane to the corresponding displacement in the image-plane: + + A = d(beta) / d(theta) = I - H = | 1 - H_yy -H_xy | + | -H_yx 1 - H_xx | + +The Jacobian decomposes into two physically distinct distortions: + +- The **convergence** kappa = (1/2) * (H_yy + H_xx) — the isotropic part, which magnifies an image uniformly + without changing its shape. (Note this is the same kappa as before: the trace of the Hessian recovers the + Poisson equation kappa = (1/2) laplacian psi.) + +- The **shear** gamma — the anisotropic part, which stretches an image along one axis and squeezes it along the + perpendicular axis. It has two components and a magnitude: + + gamma_1 = (1/2) * (H_xx - H_yy) + gamma_2 = H_xy + |gamma| = sqrt(gamma_1^2 + gamma_2^2) + +The shear is why lensed images near the lens are stretched into tangential arcs — the tidal field of the lens +elongates them around it. + + +```python +shear = lens_calc.shear_yx_2d_via_hessian_from(grid=grid) + +print(f"Shear magnitude at pixel 0: |gamma| = {shear.magnitudes[0]:.4f}") +``` + + Shear magnitude at pixel 0: |gamma| = 0.2032 + + +The **magnification** mu is the inverse of the Jacobian's determinant: + + mu = 1 / det(A) = 1 / [ (1 - kappa)^2 - |gamma|^2 ] + +Lensing conserves surface brightness, so a lensed image that covers more sky than the unlensed source appears +brighter in total by exactly the factor |mu|: + +- |mu| > 1 — the image is magnified (larger and brighter than the unlensed source). +- |mu| < 1 — the image is demagnified. +- mu < 0 — the image has negative parity: it is a mirror image of the source. + +The determinant factorises into two eigenvalues, giving the **tangential** and **radial** magnifications: + + lambda_t = 1 - kappa - |gamma| (tangential eigenvalue) + lambda_r = 1 - kappa + |gamma| (radial eigenvalue) + + mu = 1 / (lambda_t * lambda_r) + +An image is stretched by 1/lambda_t in the tangential direction (around the lens) and 1/lambda_r in the radial +direction (towards/away from the lens). Giant tangential arcs form where lambda_t is close to zero. + + +```python +magnification = lens_calc.magnification_2d_from(grid=grid) + +aplt.plot_array(array=magnification, title="Magnification (mu)") + +tangential_eigen_values = lens_calc.tangential_eigen_value_from(grid=grid) +radial_eigen_values = lens_calc.radial_eigen_value_from(grid=grid) + +print(f"Tangential eigenvalue at pixel 0: {tangential_eigen_values[0]:.4f}") +print(f"Radial eigenvalue at pixel 0: {radial_eigen_values[0]:.4f}") +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_31_0.png) + + + + Tangential eigenvalue at pixel 0: 0.5937 + Radial eigenvalue at pixel 0: 1.0000 + + +__Critical Curves and Caustics__ + +Where either eigenvalue passes through zero, det(A) = 0 and the magnification formally diverges to infinity. The +closed curves in the image-plane where this happens are the **critical curves** — the white and yellow lines that +have appeared on plots since tutorial 2, and which we explored visually in tutorial 3: + +- The **tangential critical curve** (lambda_t = 0) — roughly traces the Einstein ring; sources near its source-plane + counterpart form giant tangential arcs. +- The **radial critical curve** (lambda_r = 0) — an inner curve associated with radially stretched central images. + +Ray-tracing each critical curve through the lens equation maps it to the source-plane, where it is called a +**caustic**: + + caustic = critical_curve - alpha(critical_curve) + +Caustics divide the source-plane into regions of different image multiplicity: each time a source crosses a +caustic, the number of images it produces changes by two. For our SIE lens, a source inside the tangential caustic +produces four images (plus a faint central image); between the tangential and radial caustics, two; and outside +both caustics, just one — the source is no longer multiply imaged at all. + +`LensCalc` computes both by locating the zero-contours of the eigenvalue fields on the grid: + + +```python +tangential_critical_curve_list = lens_calc.tangential_critical_curve_list_from(grid=grid) +radial_critical_curve_list = lens_calc.radial_critical_curve_list_from(grid=grid) + +tangential_caustic_list = lens_calc.tangential_caustic_list_from(grid=grid) +radial_caustic_list = lens_calc.radial_caustic_list_from(grid=grid) + +print(f"Number of tangential critical curves: {len(tangential_critical_curve_list)}") +print(f"Number of radial critical curves: {len(radial_critical_curve_list)}") +``` + + Number of tangential critical curves: 1 + Number of radial critical curves: 0 + + +Lets plot the critical curves (image-plane) and caustics (source-plane) of our SIE side by side. Note how the +elliptical lens produces a tangential caustic with four cusps — the origin of the four-image "quad" configurations +seen in many real lenses. + + +```python +plt.figure(figsize=(10, 5)) + +plt.subplot(1, 2, 1) +for curve in tangential_critical_curve_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="black", label="Tangential") +for curve in radial_critical_curve_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="orange", label="Radial") +plt.gca().set_aspect("equal") +plt.title("Critical Curves (Image-Plane)") +plt.xlabel("x (arcsec)") +plt.ylabel("y (arcsec)") +plt.legend() + +plt.subplot(1, 2, 2) +for curve in tangential_caustic_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="black", label="Tangential") +for curve in radial_caustic_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="orange", label="Radial") +plt.gca().set_aspect("equal") +plt.title("Caustics (Source-Plane)") +plt.xlabel("x (arcsec)") +plt.ylabel("y (arcsec)") +plt.legend() + +plt.show() +plt.close() +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_35_0.png) + + + +__Einstein Radius__ + +The **Einstein radius** theta_E is the characteristic angular scale of a strong lens. For a circular lens with the +source perfectly aligned behind it, it is the radius of the Einstein ring the source forms. More generally (and +this is the definition **PyAutoLens** uses), it is the radius of the circle enclosing the same area as the +tangential critical curve: + + theta_E = sqrt(A_crit / pi) + +sometimes called the "effective" Einstein radius, since an elliptical lens's critical curve is not a circle. + +Two closed-form results are worth memorising. For a **point mass** M: + + theta_E = sqrt( (4 G M / c^2) * (D_ls / (D_l * D_s)) ) + +and for a **singular isothermal sphere** (SIS) with velocity dispersion sigma_v: + + theta_E = 4 pi * (sigma_v / c)^2 * (D_ls / D_s) + +The Einstein radius is also the radius within which the *mean* convergence equals exactly one — so measuring +theta_E directly measures the projected mass enclosed within it: + + M(< theta_E) = pi * theta_E^2 * Sigma_cr + +This is why the Einstein radius is celebrated as one of the most robust mass measurements in all of astrophysics: +the data pin down theta_E via the image separations almost independently of the assumed mass profile, so the +enclosed Einstein mass is trusted even when the profile's slope is not. (The main caveat, the mass-sheet +degeneracy, is discussed in the lens modeling chapters.) + +`LensCalc` computes the Einstein radius from the area of the tangential critical curve, and the enclosed +"Einstein mass" in angular units (pi * theta_E^2), which Sigma_cr converts to solar masses: + + +```python +einstein_radius = lens_calc.einstein_radius_from(grid=grid) + +print(f"Einstein radius (arcsec) = {einstein_radius:.4f}") +print(f"Einstein radius (kpc) = {einstein_radius * kpc_per_arcsec_lens:.4f}") + +einstein_mass_angular = lens_calc.einstein_mass_angular_from(grid=grid) +einstein_mass_solar = einstein_mass_angular * sigma_cr_arcsec + +print(f"Einstein mass (angular, arcsec^2) = {einstein_mass_angular:.4f}") +print(f"Einstein mass (solar masses) = {einstein_mass_solar:.4e}") +``` + + Einstein radius (arcsec) = 1.5901 + Einstein radius (kpc) = 9.9992 + Einstein mass (angular, arcsec^2) = 7.9437 + Einstein mass (solar masses) = 9.4290e+11 + + +Reassuringly, the Einstein radius computed from the critical curve area (~1.6") matches the `einstein_radius=1.6` +parameter we gave the `Isothermal` profile — for isothermal profiles the model parameter *is* the effective +Einstein radius, which is exactly why **PyAutoLens** parameterizes its mass profiles this way: the non-linear +search then varies the quantity the data constrain most directly. + +As a second check, an `IsothermalSph` (the SIS) has a perfectly circular critical curve, so its recovered Einstein +radius equals its input parameter even more precisely: + + +```python +sis = al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=1.6) + +sis_einstein_radius = al.LensCalc.from_mass_obj(mass_obj=sis).einstein_radius_from( + grid=grid +) + +print(f"SIS input Einstein radius = 1.6, recovered = {sis_einstein_radius:.4f}") +``` + + SIS input Einstein radius = 1.6, recovered = 1.6000 + + +__Time Delays__ + +The final piece of the formalism is time. Light rays forming different images of the same source travel different +paths and through different depths of the lens's gravitational potential, so they arrive at different times. Both +effects are captured by the **Fermat potential** (or time-delay surface): + + phi(theta) = (1/2) * |theta - beta|^2 - psi(theta) + +The first term is the **geometric delay** — the extra path length of a bent ray. The second is the **gravitational +(Shapiro) delay** — light slowing as it climbs through the lens's potential. Fermat's principle states that images +form at the stationary points (minima, maxima and saddle points) of this surface, which is a wonderfully compact +restatement of the lens equation: grad phi = 0 is exactly beta = theta - alpha(theta). + +The observable **time delay** between two images A and B is the difference in their Fermat potentials, scaled by +the cosmological distances: + + Delta t_AB = (1 + z_l) / c * (D_l * D_s / D_ls) * [ phi(theta_A) - phi(theta_B) ] + +The distance combination (1 + z_l) * D_l * D_s / D_ls is called the **time-delay distance**, and because it is +inversely proportional to the Hubble constant, measuring time delays between the images of a variable source (a +quasar or supernova) turns a strong lens into a cosmological probe. + +We only touch on this here — time delays belong to the modeling of lensed point sources, introduced in tutorial 4 +and covered in depth by the `autolens_workspace` point-source material. For now, we simply plot the Fermat +potential of our lens system: + + +```python +fermat_potential = lens_calc.fermat_potential_from(grid=grid) + +aplt.plot_array(array=fermat_potential, title="Fermat Potential (phi)") +``` + + + +![png](tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_41_0.png) + + + +__Wrap Up__ + +This was the mathematics lecture of **HowToLens** — every formal definition deferred by the other tutorials, in one +place. Lets summarise the chain of quantities, because it has a beautiful logical structure where everything flows +from the mass distribution and a handful of distances: + +- **Distances**: the angular diameter distances D_l, D_s and D_ls set the geometry, convert arcseconds to + kiloparsecs and define the critical surface density Sigma_cr. + +- **Convergence**: kappa = Sigma / Sigma_cr is the dimensionless projected mass; kappa >= 1 marks super-critical + regions capable of multiple imaging. + +- **Deflections and potential**: kappa determines the deflection field alpha (a 2D gravity integral) and the + lensing potential psi, tied together by alpha = grad psi and kappa = (1/2) laplacian psi. + +- **The lens equation**: beta = theta - alpha(theta) maps the image-plane to the source-plane; its + non-invertibility is why multiple images form. + +- **Distortion**: the Jacobian A = I - H decomposes into convergence (isotropic) and shear gamma (anisotropic); + magnification is mu = 1 / det(A) = 1 / (lambda_t * lambda_r). + +- **Critical curves and caustics**: where det(A) = 0, magnification diverges; caustics are their source-plane + images and set the image multiplicity. + +- **Einstein radius**: the area-equivalent radius of the tangential critical curve, whose enclosed mass + M(< theta_E) = pi * theta_E^2 * Sigma_cr is the most robust measurement strong lensing delivers. + +- **Time delays**: the Fermat potential phi = (1/2)|theta - beta|^2 - psi locates images at its stationary points + and its differences, scaled by the time-delay distance, give observable delays. + +You do not need to memorise any of this to continue — **PyAutoLens** computes every one of these quantities via the +methods used above — but you now know what each method is computing and can return to this tutorial whenever a +symbol needs unpacking. + +In the next tutorial, we turn from theory to observation: how telescope optics, exposure times and noise turn the +pristine images of a tracer into the CCD imaging data we actually observe, and how to simulate such data ourselves. + +Finally, a signpost for much later: chapter 3's tutorial 5 on the Bayesian formalism is this tutorial's twin — the +equivalent equations lecture for pixelized source reconstruction, deriving the linear inversion and Bayesian +evidence that underpin chapter 3 just as this tutorial derived the lensing quantities underpinning chapter 1. diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_0.png new file mode 100644 index 0000000..d222956 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_1.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_1.png new file mode 100644 index 0000000..13446cd Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_15_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_19_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_19_0.png new file mode 100644 index 0000000..83a9ce4 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_19_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_0.png new file mode 100644 index 0000000..5a3585d Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_1.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_1.png new file mode 100644 index 0000000..282c775 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_21_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_23_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_23_0.png new file mode 100644 index 0000000..4f1141b Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_23_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_31_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_31_0.png new file mode 100644 index 0000000..e05fa83 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_31_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_35_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_35_0.png new file mode 100644 index 0000000..e0a5690 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_35_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_41_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_41_0.png new file mode 100644 index 0000000..a7fce8c Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_41_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_7_0.png b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_7_0.png new file mode 100644 index 0000000..bb0aa74 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_5_lensing_formalism_files/tutorial_5_lensing_formalism_7_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data.md b/markdown/chapter_1_introduction/tutorial_6_data.md index f448cce..1910ac1 100644 --- a/markdown/chapter_1_introduction/tutorial_6_data.md +++ b/markdown/chapter_1_introduction/tutorial_6_data.md @@ -16,8 +16,8 @@ Telescope, include several factors that affect what we see: **Exposure Time:** The time the detector collects light, affecting the clarity of the image. Longer exposure times gather more light, improving the signal-to-noise ratio and creating a clearer image. -**Background Sky:** Light from the sky itself, such as distant stars or zodiacal light, adds noise to the image. -adds additional noise to the image. +**Background Sky:** Light from the sky itself, such as distant stars or zodiacal light, adds additional noise to +the image. In this tutorial, we'll simulate a strong lens image by applying these real-world effects to the light and mass profiles and images we created earlier. @@ -28,6 +28,7 @@ Here is an overview of what we'll cover in this tutorial: - **Poisson Noise:** We'll add Poisson noise to the image, simulating the randomness in the photon-to-electron conversion process on the CCD. - **Background Sky:** We'll add a background sky to the image, simulating the light from the sky that adds noise to the image. - **Simulator:** We'll use the `SimulatorImaging` object to simulate imaging data that includes all these effects. +- **Other Data Types:** We'll finish with a brief look at interferometer and weak lensing data, two other types of data used to study strong lenses. __Contents__ @@ -37,6 +38,8 @@ __Contents__ - **Background Sky:** The final effect we will consider when simulating imaging data is the background sky. - **Simulator:** The `SimulatorImaging` object lets us create simulated imaging data while including the effects of. - **Output:** We will now save these simulated data to `.fits` files, the standard format used by astronomers for. +- **Interferometer Data:** Radio / sub-mm interferometers like ALMA observe visibilities in the uv-plane rather than images. +- **Weak Lensing Data:** Weak lensing datasets are catalogues of weakly sheared background galaxy ellipticities. - **Wrap Up:** Summary of the script and next steps. @@ -53,7 +56,25 @@ import autolens as al import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) __Initial Setup__ @@ -67,7 +88,7 @@ grid = al.Grid2D.uniform( shape_native=( 101, 101, - ), # The dimensions of the grid, which here is 100 x 100 pixels. + ), # The dimensions of the grid, which here is 101 x 101 pixels. pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds. ) ``` @@ -158,7 +179,7 @@ values may significantly affect the spread and detail captured in the data. ```python -aplt.plot_array(array=psf.kernel, title="PSF 2D Kernel") +aplt.plot_array(array=psf.kernel, title="PSF 2D Kernel (Log10)", use_log10=True) ``` @@ -195,7 +216,7 @@ blurred_image = convolved_image.trimmed_after_convolution_from( ) # Trimming back to the original size. ``` - .../PyAutoArray/autoarray/operators/convolver.py:1415: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution. + .../PyAutoArray/autoarray/operators/convolver.py:1424: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution. warnings.warn( @@ -206,7 +227,7 @@ convolution affects the appearance of the galaxy, making the image appear softer ```python aplt.plot_array(array=image, title="Tracer Image Before PSF") -aplt.plot_array(array=blurred_image, title="") +aplt.plot_array(array=blurred_image, title="Tracer Image After PSF") ``` @@ -245,8 +266,8 @@ Therefore, we need to add the Poisson noise after blurring the tracer image. We also need to consider the units of our image data. Let’s assume that the tracer image is measured in units of electrons per second ($e^- s^{-1}$), which is standard for CCD imaging data. To simulate the number of electrons -actually detected in each pixel, we multiply the image by the observation’s exposure time. This conversion changes t -he units to the total number of electrons collected per pixel over the entire exposure time. +actually detected in each pixel, we multiply the image by the observation’s exposure time. This conversion changes +the units to the total number of electrons collected per pixel over the entire exposure time. Once the image is converted, we add Poisson noise, simulating the randomness in the photon-to-electron conversion process. After adding the noise, we convert the image back to units of electrons per second for analysis, as @@ -309,7 +330,7 @@ The final effect we will consider when simulating imaging data is the background In addition to light from the strong lens, the telescope also picks up light from the sky. This background sky light is primarily due to two sources: zodiacal light, which is light scattered by interplanetary dust in the solar system, -and the unresolved emission from distant stars and tracer. +and the unresolved emission from distant stars and galaxies. For our simulation, we'll assume that the background sky has a uniform brightness across the image, measured at 0.1 electrons per second per pixel. The background sky is added to the image before applying the PSF convolution @@ -387,7 +408,7 @@ simulator = al.SimulatorImaging( dataset = simulator.via_tracer_from(tracer=tracer, grid=grid) ``` - .../PyAutoArray/autoarray/operators/convolver.py:1415: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution. + .../PyAutoArray/autoarray/operators/convolver.py:1424: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution. warnings.warn( @@ -490,9 +511,8 @@ __Output__ We will now save these simulated data to `.fits` files, the standard format used by astronomers for storing images. Most imaging data from telescopes like the Hubble Space Telescope (HST) are stored in this format. -The `dataset_path` specifies where the data will be saved, in this case, in the directory -`autolens_workspace/dataset/imaging/howtolens/`, which contains many example images distributed with -the `autolens_workspace`. +The `dataset_path` specifies where the data will be saved, in this case the `dataset/imaging/howtolens/` +directory of the HowToLens repository (paths are relative to the repository root, from which scripts are run). The files are named `data.fits`, `noise_map.fits`, and `psf.fits`, and will be used in the next tutorial. @@ -513,6 +533,129 @@ aplt.fits_imaging( Dataset Path: dataset/imaging/howtolens +__Interferometer Data__ + +CCD imaging is not the only type of data used to study strong lenses. Radio and sub-mm interferometers, like the +Atacama Large Millimeter Array (ALMA) and the Jansky Very Large Array (JVLA), observe strong lenses at wavelengths +where a CCD cannot. + +An interferometer does not observe an image of the lens. Each pair of antennas in the array measures a "visibility", +a Fourier component of the sky brightness, at a point in what is called the "uv-plane" set by the separation of the +two antennas. The dataset is therefore a set of complex visibilities in Fourier space, not a 2D image, and its +noise properties are very different from those of CCD data — there is no PSF convolution, Poisson noise or +background sky; instead each visibility has Gaussian noise. + +One could Fourier transform the visibilities into an image (called a "dirty image") and fit that, but the transform +correlates the noise between pixels, making the fit statistically incorrect. **PyAutoLens** therefore fits lens +models directly in visibility space: the tracer's image is evaluated in real space on a grid (defined by a +real-space mask), Fourier transformed to the uv-plane and compared with the observed visibilities there. + +Below, we load a simulated interferometer dataset (creating it first via the `scripts/simulator/interferometer.py` +script if it does not exist on your hard-disk, using the same auto-simulation idiom as later tutorials) and plot +its dirty images — the closest an interferometer dataset comes to the CCD images we simulated above. + +HowToLens will not cover interferometry any further than this. The lecture series teaches lensing using CCD +imaging, and everything you learn transfers to visibility-space fitting. If you need to model interferometer data, +go to the `autolens_workspace/scripts/interferometer` package, which is the dedicated resource for uv-plane lens +modeling. + + +```python +dataset_path = Path("dataset") / "interferometer" / "simple" + +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/interferometer.py"], + check=True, + ) + +real_space_mask = al.Mask2D.circular( + shape_native=(100, 100), + pixel_scales=0.1, + radius=3.0, +) + +dataset = al.Interferometer.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + uv_wavelengths_path=dataset_path / "uv_wavelengths.fits", + real_space_mask=real_space_mask, + transformer_class=al.TransformerDFT, +) + +aplt.subplot_interferometer_dirty_images(dataset=dataset) +``` + + Figure(700x700) + Figure(1800x600) + + + + +![png](tutorial_6_data_files/tutorial_6_data_41_1.png) + + + +__Weak Lensing Data__ + +There is one more type of lensing data to glimpse before we move on: weak lensing. + +A weak lensing dataset looks nothing like the images above. Instead of the spectacular arcs and rings of strong +lensing, it is a *catalogue* of many background galaxies, each with a measured ellipticity — a shear — at its +(y, x) position on the sky. Each galaxy is only weakly sheared by the foreground mass, a distortion far too small +to see in any single galaxy, and the plot below shows this: a field of short line segments tracing the subtle, +coherent stretching of the background galaxy population, rather than any arc-like feature. + +We load a simulated weak lensing shear catalogue (again auto-simulating it, via `scripts/simulator/weak_lensing.py`, +if it is not on your hard-disk) and plot it, just so you have seen what this data looks like. + +That is deliberately all we will say for now — this glimpse is foreshadowing. Weak lensing gets a full treatment in +the final tutorial of chapter 4, which describes what these shear measurements are, why they are made far from the +lens centre and how they are fitted; until then, we defer all further description to that tutorial. + + +```python +dataset_path = Path("dataset") / "weak_lensing" / "simple" + +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/weak_lensing.py"], + check=True, + ) + +dataset = al.from_json(file_path=dataset_path / "dataset.json") + +aplt.subplot_weak_dataset(dataset=dataset) +``` + + name : simple + n_galaxies : 1500 + shear_yx : ShearYX2DIrregular([[-0.00525262, 0.18463656], + [ 0.13942478, -0.32888726], + [ 0.19774603, -0.02665228], + ..., + [ 0.14253715, -0.02222372], + [ 0.11760603, -0.16325793], + [ 0.64420539, -0.25627485]], shape=(1500, 2)) + noise_map : ArrayIrregular([0.25, 0.25, 0.25, ..., 0.25, 0.25, 0.25], shape=(1500,)) + redshifts : None + is_reduced : False + + Wrote dataset to dataset/weak_lensing/simple + + + + +![png](tutorial_6_data_files/tutorial_6_data_43_1.png) + + + __Wrap Up__ In this tutorial, you learned how CCD imaging data of a lens is collected using real telescopes like the @@ -520,7 +663,7 @@ Hubble Space Telescope, and how to simulate this data using the `SimulatorImagin Let's summarise what we've covered: -- **Optics Blurring**: The optics of a telescope blur the light from tracer, reducing the clarity and sharpness of +- **Optics Blurring**: The optics of a telescope blur the light of the tracer, reducing the clarity and sharpness of the images. - **Poisson Noise**: The process of converting photons to electrons on a CCD introduces Poisson noise, which is random @@ -533,8 +676,3 @@ the entire image. these effects together and contains the `data`, `psf`, and `noise_map` components. - **Output**: We saved the simulated data to `.fits` files, the standard format used by astronomers for storing images. - - -```python - -``` diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_11_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_11_0.png index eb6a698..f81029d 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_11_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_11_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_13_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_13_0.png index eb6a698..1cf3564 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_13_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_13_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_0.png index 2e8a101..6965b66 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_1.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_1.png index cacc07e..37d5bc9 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_1.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_17_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_21_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_21_0.png index 0d7e951..929c56a 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_21_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_21_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_23_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_23_0.png index 41ba8fb..240e688 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_23_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_23_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_0.png index 13a499f..4982c3d 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_1.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_1.png index bf23b9e..8a889b7 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_1.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_25_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_29_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_29_0.png index 3b6b60b..8a05120 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_29_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_29_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_31_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_31_0.png index 0f6fa08..8af9e43 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_31_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_31_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_33_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_33_0.png index 0d78811..6cf10ed 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_33_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_33_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_35_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_35_0.png index 38b6203..93f03b9 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_35_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_35_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_37_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_37_0.png index 072cc53..6319217 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_37_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_37_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_41_1.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_41_1.png new file mode 100644 index 0000000..3adb63a Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_41_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_43_1.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_43_1.png new file mode 100644 index 0000000..2e604a1 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_43_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_7_0.png b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_7_0.png index 5d76d26..8e289f0 100644 Binary files a/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_7_0.png and b/markdown/chapter_1_introduction/tutorial_6_data_files/tutorial_6_data_7_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting.md b/markdown/chapter_1_introduction/tutorial_7_fitting.md index b3c7bee..8088499 100644 --- a/markdown/chapter_1_introduction/tutorial_7_fitting.md +++ b/markdown/chapter_1_introduction/tutorial_7_fitting.md @@ -5,7 +5,7 @@ Tutorial 7: Fitting =================== -In previous tutorials, we used light profiles to create simulated images of tracer and visualized how these images +In previous tutorials, we used light profiles to create simulated images of a tracer and visualized how these images would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope. However, this simulation process is the reverse of what astronomers typically do when analyzing real data. Usually, @@ -36,7 +36,9 @@ Here is an overview of what we'll cover in this tutorial: __Contents__ -- **Dataset & Mask:** Standard set up of the dataset and mask that is fitted. +- **Dataset:** Load the imaging dataset that we previously simulated, consisting of the image, noise map, and PSF. +- **Dataset Auto-Simulation:** Create the dataset by running the tutorial 6 script if it is not on your hard-disk. +- **Mask:** Apply a mask to the data, excluding regions with low signal-to-noise ratios from the analysis. - **Masked Grid:** In tutorials 1 and 2, we emphasized that the `Grid2D` object is crucial for evaluating a lens's. - **Fitting:** Fit the lens model to the dataset and inspect the results. - **Incorrect Fit:** In the previous section, we successfully created and fitted a lens model to the image data. @@ -46,6 +48,8 @@ __Contents__ ```python +from autolens import jax_wrapper # Sets JAX environment before other imports + from autolens import setup_notebook; setup_notebook() import numpy as np @@ -54,7 +58,25 @@ import autolens as al import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) __Dataset__ @@ -62,11 +84,11 @@ __Dataset__ We begin by loading the imaging dataset that we will use for fitting in this tutorial. This dataset is identical to the one we simulated in the previous tutorial, representing how a lens would appear if captured by a CCD camera. -In the previous tutorial, we saved this dataset as .fits files in the `autolens_workspace/dataset/imaging/howtolens` -folder. The `.fits` format is commonly used in astronomy for storing image data along with metadata, making it a -standard for CCD imaging. +In the previous tutorial, we saved this dataset as .fits files in the `dataset/imaging/howtolens` folder of the +HowToLens repository. The `.fits` format is commonly used in astronomy for storing image data along with metadata, +making it a standard for CCD imaging. -The `dataset_path` below specifies where these files are located: `autolens_workspace/dataset/imaging/howtolens/`. +The `dataset_path` below specifies where these files are located: `dataset/imaging/howtolens/`. ```python @@ -75,17 +97,18 @@ dataset_path = Path("dataset") / "imaging" / "howtolens" __Dataset Auto-Simulation__ -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. +The `howtolens` dataset is the one built up and saved in tutorial 6 (`tutorial_6_data.py`). If it does +not already exist on your system, it is created by running that script. This ensures every example +script can be run without manually simulating data first. ```python -if not dataset_path.exists(): +if al.util.dataset.should_simulate(str(dataset_path)): import subprocess import sys subprocess.run( - [sys.executable, "scripts/simulator/no_lens_light__mass_sis.py"], + [sys.executable, "scripts/chapter_1_introduction/tutorial_6_data.py"], check=True, ) @@ -123,11 +146,11 @@ aplt.subplot_imaging_dataset(dataset=dataset) ``` Value of first pixel in imaging data: - 0.01999999999999999 + 0.053333333333333316 Value of first pixel in noise map: - 0.02 + 0.02260776661041756 Value of first pixel in PSF: - 0.0 + 2.210334945638401e-12 @@ -138,7 +161,7 @@ aplt.subplot_imaging_dataset(dataset=dataset) __Mask__ -The signal-to-noise map of the image highlights areas where the signal (light from the lens and source tracer) +The signal-to-noise map of the image highlights areas where the signal (light from the lens and source galaxies) is detected above the background noise. Values above 3.0 indicate regions where the light is detected with a signal-to-noise ratio of at least 3, while values below 3.0 are dominated by noise, where the light is not clearly distinguishable. @@ -166,37 +189,18 @@ print(mask) # 1 = True, meaning the pixel is masked. Edge pixels are indeed mas print(mask[48:53, 48:53]) # Central pixels are `False` and therefore unmasked. ``` - Mask2D([[False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False]]) - [] + Mask2D([[ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True], + ..., + [ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True]], shape=(101, 101)) + [[False False False False False] + [False False False False False] + [False False False False False] + [False False False False False] + [False False False False False]] We can visualize the mask over the strong lens image using an `aplt.subplot_imaging_dataset`, which helps us adjust the mask as needed. @@ -226,7 +230,7 @@ that only the unmasked regions are considered during the analysis. dataset = dataset.apply_mask(mask=mask) ``` - 2026-07-11 16:29:25,502 - autoarray.dataset.imaging.dataset - INFO - IMAGING - Data masked, contains a total of 225 image-pixels + 2026-08-06 13:37:34,866 - autoarray.dataset.imaging.dataset - INFO - IMAGING - Data masked, contains a total of 2809 image-pixels When we plot the masked imaging data again, the mask is now automatically included in the plot, even though we did @@ -255,36 +259,13 @@ print(dataset.mask) ``` Mask2D: - Mask2D([[False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False], - [False, False, False, False, False, False, False, False, False, - False, False, False, False, False, False]]) + Mask2D([[ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True], + ..., + [ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True], + [ True, True, True, ..., True, True, True]], shape=(101, 101)) In earlier tutorials, we discussed how grids and arrays have `native` and `slim` representations: @@ -312,8 +293,8 @@ print( ``` Number of unmasked pixels: - (15, 15) - (225,) + (101, 101) + (2809,) The `mask` object also has a `pixels_in_mask` attribute, which gives the number of unmasked pixels. This should @@ -324,7 +305,7 @@ match the size of the `slim` data structure. print(dataset.data.mask.pixels_in_mask) ``` - 225 + 2809 We can use the `slim` attribute to print the first unmasked values from the image and noise map: @@ -338,9 +319,9 @@ print(dataset.noise_map.slim[0]) ``` First unmasked image value: - 0.01999999999999999 + 0.4633333333333334 First unmasked noise map value: - 0.02 + 0.043333333333333335 Additionally, we can verify that the `native` data structure has zeros at the edges where the mask is applied and @@ -356,9 +337,9 @@ print(dataset.data.native[centre]) ``` Example masked pixel in the image's native representation at its edge: - 0.01999999999999999 + 0.0 Example unmasked pixel in the image's native representation at its center: - 0.6233333333333334 + 0.52 __Masked Grid__ @@ -497,14 +478,12 @@ aplt.plot_array(array=fit.model_data, title="Model Image") ``` First model image pixel: - - - 1.8009715550322956 + 0.4831484402089213 -![png](tutorial_7_fitting_files/tutorial_7_fitting_37_2.png) +![png](tutorial_7_fitting_files/tutorial_7_fitting_37_1.png) @@ -555,9 +534,9 @@ aplt.plot_array(array=fit.residual_map, title="Residual Map") ``` First residual-map pixel: - -1.7809715550322955 + -0.019815106875587907 First residual-map pixel via fit: - -1.7809715550322955 + -0.019815106875587907 @@ -593,9 +572,9 @@ aplt.plot_array(array=fit.normalized_residual_map, title="Normalized Residual Ma ``` First normalized residual-map pixel: - -89.04857775161477 + -0.45727169712895166 First normalized residual-map pixel via fit: - -89.04857775161477 + -0.45727169712895166 @@ -630,9 +609,9 @@ aplt.plot_array(array=fit.chi_squared_map, title="Chi Squared Map") ``` First chi-squared pixel: - 7929.649199585381 + 0.2090974049951917 First chi-squared pixel via fit: - 7929.649199585381 + 0.2090974049951917 @@ -662,8 +641,8 @@ print("Chi-squared = ", chi_squared) print("Chi-squared via fit = ", fit.chi_squared) ``` - Chi-squared = 645295.8813889123 - Chi-squared via fit = 645295.8813889123 + Chi-squared = 2870.509352378347 + Chi-squared via fit = 2870.509352378347 The reduced chi-squared is the `chi_squared` value divided by the number of data points (e.g., the number of pixels @@ -682,12 +661,12 @@ reduced_chi_squared = chi_squared / dataset.mask.pixels_in_mask print("Reduced Chi-squared = ", reduced_chi_squared) ``` - Reduced Chi-squared = 2867.9816950618324 + Reduced Chi-squared = 1.0218972418577241 Another quantity that contributes to our final assessment of the goodness-of-fit is the `noise_normalization`. -The `noise_normalization` is computed as the logarithm of the sum of squared noise values in our data: +The `noise_normalization` is computed by summing, over every pixel, the logarithm of 2 pi times the squared noise value: \[ \text{{noise\_normalization}} = \sum \log(2 \pi \text{{noise\_map}}^2) @@ -707,8 +686,8 @@ print("Noise Normalization = ", noise_normalization) print("Noise Normalization via fit = ", fit.noise_normalization) ``` - Noise Normalization = -1023.7670414997456 - Noise Normalization via fit = -1023.7670414997456 + Noise Normalization = -8670.33617182735 + Noise Normalization via fit = -8670.33617182735 From the `chi_squared` and `noise_normalization`, we can define a final goodness-of-fit measure known as @@ -729,8 +708,8 @@ print("Log Likelihood = ", log_likelihood) print("Log Likelihood via fit = ", fit.log_likelihood) ``` - Log Likelihood = -322136.0571737063 - Log Likelihood via fit = -322136.0571737063 + Log Likelihood = 2899.9134097245014 + Log Likelihood via fit = 2899.9134097245014 In the previous discussion, we noted that a lower \(\chi^2\) value indicates a better fit of the model to the @@ -781,7 +760,7 @@ and 'log_likelihood' before. These metrics are standard ways to quantify the quality of a model fit. They are applicable not only to 1D data but also to more complex data structures like 2D images, 3D data cubes, or any other multidimensional datasets. -__Incorrect Fit___ +__Incorrect Fit__ In the previous section, we successfully created and fitted a lens model to the image data, resulting in an excellent fit. The residual map and chi-squared map showed no significant discrepancies, indicating that the @@ -853,9 +832,9 @@ print(fit_bad.log_likelihood) ``` Previous Likelihood: - -322136.0571737063 + 2899.9134097245014 New Likelihood: - -401645.6055797826 + -64618.8911885303 As expected, we observe that the log likelihood has decreased! This decline confirms that our new model is indeed a @@ -898,7 +877,7 @@ aplt.subplot_fit_imaging(fit=fit_very_bad) -It is now evident that this model provides a terrible fit to the data. The tracer do not resemble a plausible +It is now evident that this model provides a terrible fit to the data. The tracer does not resemble a plausible representation of our simulated strong lens dataset, which we already anticipated given that we generated the data ourselves! As expected, the log likelihood has dropped dramatically with this poorly fitting model. @@ -913,10 +892,10 @@ print(fit_very_bad.log_likelihood) ``` Previous Likelihoods: - -322136.0571737063 - -401645.6055797826 + 2899.9134097245014 + -64618.8911885303 New Likelihood: - -2703441.378104259 + -1599395.1803587012 __Model Fitting__ @@ -927,8 +906,8 @@ quality, reducing the log likelihood. In practice, however, we don't know the "true" model. For example, we might have an image of a strong lens observed with the Hubble Space Telescope, but the values for parameters like its `einstein_radius` and others are -unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of -Chapter 2 of *HowToGalaxy*. +unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of +Chapter 2 of **HowToLens**. To conclude this section, let's perform a basic, hands-on model fit to develop some intuition about how we can find the best-fit model. We'll start by loading a simple dataset that was simulated without any lens light, using @@ -940,6 +919,15 @@ profiles are unknown. dataset_name = "simple__no_lens_light__mass_sis" dataset_path = Path("dataset") / "imaging" / dataset_name +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/no_lens_light__mass_sis.py"], + check=True, + ) + dataset = al.Imaging.from_fits( data_path=dataset_path / "data.fits", psf_path=dataset_path / "psf.fits", @@ -958,7 +946,13 @@ dataset = dataset.apply_mask(mask=mask) aplt.subplot_imaging_dataset(dataset=dataset) ``` - 2026-07-11 16:29:33,067 - autoarray.dataset.imaging.dataset - INFO - IMAGING - Data masked, contains a total of 225 image-pixels + Figure(700x700) + .../PyAutoArray/autoarray/operators/convolver.py:1424: UserWarning: No blurring_image provided. Only the direct image will be convolved. This may change the correctness of the PSF convolution. + warnings.warn( + Figure(1800x1800) + Figure(1800x1800) + Figure(700x700) + 2026-08-06 13:38:03,369 - autoarray.dataset.imaging.dataset - INFO - IMAGING - Data masked, contains a total of 2828 image-pixels @@ -1021,7 +1015,7 @@ print(fit.log_likelihood) Log Likelihood: - -91864.03686663607 + -468816.25824381993 Manually guessing model parameters repeatedly is a very inefficient and slow way to find the best fit. If the model @@ -1058,8 +1052,3 @@ Let's summarise what we have covered: - **Model Fitting**: We performed a basic model fit on a simple dataset, adjusting the model parameters to improve the fit quality. - - -```python - -``` diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_11_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_11_0.png index ce921f7..929a443 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_11_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_11_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_15_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_15_0.png index 2425188..346f639 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_15_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_15_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_27_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_27_0.png index e84c182..860b144 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_27_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_27_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_29_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_29_0.png index afa9d16..cbf2744 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_29_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_29_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_31_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_31_0.png index 42ec776..493846b 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_31_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_31_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_33_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_33_0.png index c12c345..40f92b8 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_33_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_33_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_37_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_37_1.png new file mode 100644 index 0000000..00a40e1 Binary files /dev/null and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_37_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_37_2.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_37_2.png deleted file mode 100644 index b51b224..0000000 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_37_2.png and /dev/null differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_0.png index e7649d6..6821f77 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_1.png index b51b224..00a40e1 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_1.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_39_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_41_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_41_1.png index 93e0681..aa448f0 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_41_1.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_41_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_43_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_43_1.png index 80ab621..a57d604 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_43_1.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_43_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_45_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_45_1.png index 52e8560..a1535a4 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_45_1.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_45_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_55_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_55_0.png index e163d68..f425a6e 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_55_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_55_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_57_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_57_0.png index c607d80..ec32cd6 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_57_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_57_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_59_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_59_0.png index 1b5f28c..c73f419 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_59_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_59_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_63_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_63_0.png index 75c9523..b910b25 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_63_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_63_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_67_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_67_1.png index 553d39a..985ddd3 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_67_1.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_67_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_69_0.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_69_0.png index 274acc1..c5b142f 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_69_0.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_69_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_7_1.png b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_7_1.png index 707c207..fcf9a90 100644 Binary files a/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_7_1.png and b/markdown/chapter_1_introduction/tutorial_7_fitting_files/tutorial_7_fitting_7_1.png differ diff --git a/markdown/chapter_1_introduction/tutorial_8_summary.md b/markdown/chapter_1_introduction/tutorial_8_summary.md index 77f7ee5..71482a1 100644 --- a/markdown/chapter_1_introduction/tutorial_8_summary.md +++ b/markdown/chapter_1_introduction/tutorial_8_summary.md @@ -2,7 +2,7 @@ > It shows the example fully executed, with its real output images. > Run it yourself via the [Python script](../../scripts/chapter_1_introduction/tutorial_8_summary.py) or the [Jupyter notebook](../../notebooks/chapter_1_introduction/tutorial_8_summary.ipynb). -Tutorial 9: Summary +Tutorial 8: Summary =================== In this chapter, we have learnt that: @@ -40,7 +40,25 @@ import autolens as al import autolens.plot as aplt ``` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) Working Directory has been set to `HowToLens` + .../PyAutoNerves/autonerves/workspace.py:206: UserWarning: Cannot verify the workspace at HowToLens/scripts/chapter_1_introduction is compatible with the installed library version (2026.7.23.1): no `version.minimum_library_version` or `version.workspace_version` key in config/general.yaml and no version.txt at the workspace root. + + If you cloned the workspace from `main` rather than a release tag, set `version.workspace_version_check: False` in config/general.yaml to silence this warning. The `main` branch updates more frequently than library releases, so version mismatches are expected and not actionable for `main`-branch users. + + You can also set the environment variable PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1 to disable temporarily. + warnings.warn(_missing_version_warning(root, library_version)) __Start__ @@ -109,7 +127,7 @@ print(tracer.planes[1][0].disk) print() ``` - + [Redshift: 0.5 Mass Profiles: @@ -277,8 +295,3 @@ repository: https://github.com/PyAutoLabs/PyAutoLens Even the smallest bit of exposure via a GitHub star can help our project grow! - - -```python - -``` diff --git a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_11_0.png b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_11_0.png index 97ce0e7..6411ec4 100644 Binary files a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_11_0.png and b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_11_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_7_0.png b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_7_0.png index 82f87ab..79a1452 100644 Binary files a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_7_0.png and b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_7_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_0.png b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_0.png index 79b5f73..f08e3a8 100644 Binary files a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_0.png and b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_0.png differ diff --git a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_1.png b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_1.png index d80e430..a8104cb 100644 Binary files a/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_1.png and b/markdown/chapter_1_introduction/tutorial_8_summary_files/tutorial_8_summary_9_1.png differ diff --git a/notebooks/README.md b/notebooks/README.md index c103a00..9717845 100644 --- a/notebooks/README.md +++ b/notebooks/README.md @@ -4,9 +4,11 @@ a strong lens. # Folders - `chapter_1_introduction`: An introduction to strong gravitational lensing and **PyAutolens**. -- `chapter_2_lens_modeling`: How to model strong lenses, including a primer on Bayesian non-linear analysis. -- `chapter_3_search_chaining`: How to fit complex lens models using non-linear search chaining. -- `chapter_4_pixelizations`: How to perform pixelized reconstructions of the source-galaxy. +- `chapter_2_lens_modeling`: How to model strong lenses, including a primer on Bayesian non-linear analysis + and non-linear search chaining. +- `chapter_3_pixelizations`: How to perform pixelized reconstructions of the source-galaxy. +- `chapter_4_scaling_up_lensing`: How to scale lens modeling up to extra galaxies, multi-galaxy lenses, + scaling relations, groups, clusters and weak lensing. - `chapter_optional`: Optional tutorials. # Full Explanation @@ -62,15 +64,15 @@ and would like to adapt it to your own code, check out the Python auto-code form # HOW TO TACKLE HowToLens -The **HowToLens** lecture series current sits at 5 chapters, and each will take a day or so to go through +The **HowToLens** lecture series currently sits at 4 chapters, and each will take a day or so to go through properly. You probably want to be modeling lenses faster than that! Furthermore, the concepts in the -later chapters are pretty challenging, and familiarity and lens modeling is desirable before you +later chapters are pretty challenging, and familiarity with lens modeling is desirable before you tackle them. Therefore, we recommend that you complete chapters 1 & 2 and then apply what you've learnt to the modeling of simulated and real strong lens data, using the scripts found in the 'autolens_workspace'. Once you're happy with the results and confident with your use of **PyAutoLens**, you can then begin to cover the advanced functionality -covered in chapters 3, 4 & 5. +covered in chapters 3 & 4. # OVERVIEW OF CHAPTER 1 (Beginner) @@ -97,49 +99,40 @@ fit CCD imaging data of a strong gravitational lens with a lens model. At the en 2. How to fit a lens model to strong lens CCD imaging via a non-linear search. 3. The trade-off between realism and complexity when choosing a lens model. 4. Why an incorrect lens model may be inferred and how to prevent this from happening. -5. The challenges that are involved in inferred a robust lens model in a computationally reasonable run-time. +5. The challenges that are involved in inferring a robust lens model in a computationally reasonable run-time. +6. How to chain non-linear searches together to build automated lens modeling pipelines, including the + Source, Light and Mass (SLaM) pipelines. **Once completed, you'll be ready to model your own strong gravitational lenses with PyAutoLens!** # OVERVIEW OF CHAPTER 3 (Intermediate) -**Automated Modeling with non-linear search chaining** +**Using an inversion to perform a pixelized source reconstruction** -In chapter 3, we'll learn how to chain multiple non-linear searches together to build automated lens modeling pipelines -which can: - -1. Break-down the fitting of a model using multiple non-linear searches and prior passing. -2. Fit CCD imaging of a strong lens where the lens light and source light are fitted separately. -3. Use a custom pipeline to fit a strong lens with multiple lens galaxies or source galaxies. -4. Know how to use advanced pipelines called the Source, Light and Mass (SLaM) pipelines. - -# OVERVIEW OF CHAPTER 4 (Intermediate) - -**Using an inverison to perform a pixelized source reconstructions** - -In chapter 4, we'll learn how to reconstruct the lensed source galaxy using a pixel-grid, ensuring that we can fit an +In chapter 3, we'll learn how to reconstruct the lensed source galaxy using a pixel-grid, ensuring that we can fit an accurate lens model to sources with complex and irregular morphologies. You'll learn how to: 1. Pixelize a source-plane into a set of source-plane pixels defined by mappings to image pixels. 2. Perform a linear inversion on this source-plane pixelization to reconstruct the source's light. 3. Apply a smoothness prior on the source reconstruction, called regularization. 4. Apply smoothing within a Bayesian framework to objectively quantify the source reconstruction's complexity. -5. Define a border in the source-plane to prevent pixels tracing outside the source reconstruction. -6. Use alternative pixelizations, for example a Voronoi mesh whose pixels adapt to the lens's mass model. -7. Use these features to fit a lens model via non-linear searches. - -# OVERVIEW OF CHAPTER 5 (Advanced) - -**Hyper-Mode** - -In hyper-mode, we introduced advanced functionality that adapts various parts of the lens modeling procedure to the -data that we are fitting. - -NOTE: Hyper-mode is conceptually quite challenging, and I advise that you make sure you are very familiar with -PyAutoLens before covering chapter 5! - -1. Adapt an inversions's `Pixelization` to the morphology of the reconstructed source galaxy. -2. Adapt the `Regularization` scheme applied to this source to its surface brightness profile. -3. Use hyper-galaxies to scale the image's noise-map during fitting, to prevent over-fitting regions of the image. -4. include aspects of the data reduction in the model fitting, for example the background sky subtraction. -5. Use these features in PyAutoLens's search chaining framework. +5. Write down the linear algebra and Bayesian evidence equations that underpin the whole framework. +6. Define a border in the source-plane to prevent pixels tracing outside the source reconstruction. +7. Use alternative pixelizations, for example a Voronoi mesh whose pixels adapt to the lens's mass model + or to the source's own brightness, alongside adaptive regularization schemes. +8. Use these features to fit a lens model via non-linear searches. + +# OVERVIEW OF CHAPTER 4 (Advanced) + +**Scaling Up Lensing** + +In chapter 4, we'll scale lens modeling up beyond a single lens galaxy, learning how to: + +1. Handle extra galaxies near a lens, by scaling their light out of the fit or modeling them explicitly. +2. Model systems with two or more co-dominant lens galaxies and understand the degeneracies this creates. +3. Use scaling relations to tie galaxy masses to their luminosities, so model complexity stops growing + with every galaxy. +4. Model group-scale lenses, using truncated dPIE mass profiles and an optional group dark matter halo. +5. Model cluster-scale lenses, using point-source positions, a CSV interface for member catalogues and + multi-plane ray tracing. +6. Fit weak-lensing shear catalogues, which probe dark matter far beyond the strong lensing region. diff --git a/notebooks/chapter_1_introduction/README.md b/notebooks/chapter_1_introduction/README.md index f4be8f1..f3514ba 100644 --- a/notebooks/chapter_1_introduction/README.md +++ b/notebooks/chapter_1_introduction/README.md @@ -12,7 +12,7 @@ In chapter 1, we introduce you to strong gravitational lensing and the core **Py - [Tutorial 3: More Ray Tracing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb) — Advanced strong lens ray-tracing. -- [Tutorial 4: Point Sources](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb) — How lensing calculations when the source galaxy is a point-source (e.g. a quasar). +- [Tutorial 4: Point Sources](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb) — How lensing calculations change when the source galaxy is a point source (e.g. a quasar). - [Tutorial 5: Lensing Formalism](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb) — The algebraic lensing formalism used to describe strong lensing. diff --git a/notebooks/chapter_1_introduction/tutorial_0_visualization.ipynb b/notebooks/chapter_1_introduction/tutorial_0_visualization.ipynb index a8a9203..a215a69 100644 --- a/notebooks/chapter_1_introduction/tutorial_0_visualization.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_0_visualization.ipynb @@ -12,8 +12,9 @@ "\n", "__Contents__\n", "\n", - "- **Directories:** **PyAutoLens assumes** the working directory is `autolens_workspace` on your hard-disk.\n", + "- **Directories:** **HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk.\n", "- **Dataset:** Load and plot the strong lens dataset.\n", + "- **Dataset Auto-Simulation:** Create the dataset via its simulator script if it is not on your hard-disk.\n", "- **Subplots:** In addition to plotting individual figures, **PyAutoLens** can plot `subplots` which show multiple.\n", "- **Plot Customization:** Does the figure display correctly on your computer screen?\n", "- **Overlays:** Overlays such as critical curves and image positions are added using the `lines=` and `positions=`.\n", @@ -65,7 +66,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()" + "from autolens import setup_notebook; setup_notebook()" ], "outputs": [], "execution_count": null @@ -74,16 +75,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If the printed working directory does not match the workspace path on your computer, you can manually set it\n", - "as follows (the example below shows the path I would use on my laptop. The code is commented out so you do not\n", - "use this path in this tutorial!" + "__Directories__\n", + "\n", + "**HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk, so that relative\n", + "paths to `dataset/` and `output/` resolve correctly.\n", + "\n", + "If your working directory does not match this path on your computer, you can manually set it as follows (the\n", + "example below shows the path I would use on my laptop. The code is commented out so you do not use this path in\n", + "this tutorial!" ] }, { "cell_type": "code", "metadata": {}, "source": [ - "# workspace_path = \"/Users/Jammy/Code/PyAuto/autolens_workspace\"\n", + "# workspace_path = \"/Users/Jammy/Code/PyAuto/HowToLens\"\n", "# #%cd $workspace_path\n", "# print(f\"Working Directory has been set to `{workspace_path}`\")" ], @@ -97,10 +103,10 @@ "__Dataset__\n", "\n", "The `dataset_path` specifies where the dataset is located, which is the\n", - "directory `autolens_workspace/dataset/imaging/simple__no_lens_light`.\n", + "directory `dataset/imaging/simple__no_lens_light` of the HowToLens repository.\n", "\n", - "There are many example simulated images of strong lenses in this directory that will be used throughout the\n", - "**HowToLens** lectures." + "The simulated images of strong lenses used throughout the **HowToLens** lectures are written to the `dataset`\n", + "directory at runtime by the simulator scripts in `scripts/simulator/`." ] }, { @@ -211,7 +217,7 @@ "\n", "If not, the default matplotlib settings can be customized via the config files in:\n", "\n", - " autolens_workspace/config/visualize/\n", + " config/visualize/\n", "\n", "Key config entries:\n", "\n", diff --git a/notebooks/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb b/notebooks/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb index ba30f7d..4efb4dc 100644 --- a/notebooks/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_1_grids_and_galaxies.ipynb @@ -61,7 +61,11 @@ "- **Light Profiles:** Galaxies are collections of stars, gas, dust, and other astronomical objects that emit light.\n", "- **One Dimension Projection:** We often want to calculative 1D quantities of a light profile, for example to plot how its light.\n", "- **Galaxies:** Galaxies are collections of light profiles that represent a galaxy's luminous emission.\n", - "- **Units:** By assuming a redshift for a galaxy we can convert its quantities from arcseconds to kiloparsecs." + "- **Units:** By assuming a redshift for a galaxy we can convert its quantities from arcseconds to kiloparsecs.\n", + "- **Wrap Up:** Summary of the script and next steps.\n", + "- **Advanced Topics:** Optional reference material for expert users, not needed on a first read.\n", + "- **Other Unit Conversion:** Other unit conversions a galaxy's redshift enables (e.g. AB magnitudes).\n", + "- **Over Sampling:** The adaptive over-sampling scheme used to evaluate light profiles accurately." ] }, { @@ -109,7 +113,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", @@ -151,7 +155,7 @@ " shape_native=(\n", " 101,\n", " 101,\n", - " ), # The dimensions of the grid, which here is 100 x 100 pixels.\n", + " ), # The dimensions of the grid, which here is 101 x 101 pixels.\n", " pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds.\n", ")" ], @@ -263,7 +267,7 @@ "cell_type": "code", "metadata": {}, "source": [ - "centre = (0.3, 0.5) # Shifting the grid to be centered at y=1.0\", x=2.0\".\n", + "centre = (0.3, 0.5) # Shifting the grid to be centered at y=0.3\", x=0.5\".\n", "\n", "grid_shifted = grid\n", "grid_shifted[:, 0] = grid_shifted[:, 0] - centre[0] # Shift in y-direction.\n", diff --git a/notebooks/chapter_1_introduction/tutorial_2_ray_tracing.ipynb b/notebooks/chapter_1_introduction/tutorial_2_ray_tracing.ipynb index 101f594..35b0ead 100644 --- a/notebooks/chapter_1_introduction/tutorial_2_ray_tracing.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_2_ray_tracing.ipynb @@ -69,7 +69,8 @@ "- **Ray Tracing Images:** Evaluating a source's light on the ray-traced grid produces its lensed image.\n", "- **Galaxies:** A `Galaxy` can contain both light and mass profiles, forming realistic lens and source galaxies.\n", "- **Tracer:** The `Tracer` object automates ray-tracing for a system of galaxies at different redshifts.\n", - "- **Mappings:** Every image-plane coordinate maps to a source-plane coordinate via the lens equation." + "- **Mappings:** Every image-plane coordinate maps to a source-plane coordinate via the lens equation.\n", + "- **Wrap Up:** Summary of the script and next steps." ] }, { @@ -117,7 +118,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import matplotlib.pyplot as plt\n", "import autolens as al\n", diff --git a/notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb b/notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb index 5477a41..df37b39 100644 --- a/notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 5: More Ray Tracing\n", + "Tutorial 3: More Ray Tracing\n", "============================\n", "\n", "We'll now reinforce the ideas that we learnt about ray-tracing in the previous tutorial and introduce the following\n", @@ -32,7 +32,7 @@ "\n", "- **Initial Setup:** To begin, lets setup the grid we'll ray-trace using.\n", "- **Concise Code:** Lets set up the tracer used in the previous tutorial.\n", - "- **Critical Curves:** To end, we can finally explain what the black lines that have appeared on many of the plots.\n", + "- **Critical Curves:** We can finally explain what the black lines that have appeared on many of the plots.\n", "- **Caustics:** In the previous tutorial, we plotted the critical curves of the mass profile on the image-plane.\n", "- **Units:** Lets plot the lensing quantities again.\n", "- **More Complexity:** We now make a lens with some attributes we didn`t in the last tutorial.\n", @@ -85,7 +85,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "import autolens as al\n", @@ -175,8 +175,8 @@ "source": [ "__Critical Curves__\n", "\n", - "To end, we can finally explain what the black lines that have appeared on many of the plots throughout this chapter \n", - "actually are. \n", + "We can finally explain what the black lines that have appeared on many of the plots throughout this chapter\n", + "actually are.\n", "\n", "These lines are called the 'critical curves', and they define line of infinite magnification due to a mass profile. \n", "They therefore mark where in the image-plane a mass profile perfectly `focuses` light rays such that if a source is \n", @@ -190,7 +190,7 @@ "when its inner mass slope is less steep than a steep power-law). To make it appear below, we therefore change\n", "the mass profile of our lens galaxy to a `PowerLawSph` with a slope of 1.8.\n", "\n", - "In the next tutorial, we'll introduce 'caustics', which are where the critical curves map too in the source-plane." + "In the next section, we'll introduce 'caustics', which are where the critical curves map to in the source-plane." ] }, { @@ -217,12 +217,12 @@ "source": [ "__Caustics__\n", "\n", - "In the previous tutorial, we plotted the critical curves of the mass profile on the image-plane. We will now plot the\n", + "In the previous section, we plotted the critical curves of the mass profile on the image-plane. We will now plot the\n", "'caustics', which correspond to each critical curve ray-traced to the source-plane. This is computed by using the \n", "lens galaxy mass profile's to calculate the deflection angles at the critical curves and ray-trace them to the \n", "source-plane.\n", "\n", - "As discussed in the previous tutorial, critical curves mark regions of infinite magnification. Thus, if a source\n", + "As discussed in the previous section, critical curves mark regions of infinite magnification. Thus, if a source\n", "appears near a caustic in the source plane it will appear significantly brighter than its true luminosity. \n", "\n", "We again have to use a mass profile with a slope below 2.0 to ensure a radial critical curve and therefore radial\n", @@ -289,9 +289,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Caustics also mark the regions in the source-plane where the multiplicity of the strong lens changes. That is,\n", - "if a source crosses a caustic, it goes from 2 images to 1 image. Try and show this yourself by changing the (y,x) \n", - "centre of the source-plane galaxy's light profile!" + "Caustics also mark the regions in the source-plane where the multiplicity of the strong lens changes. Each time\n", + "a source crosses a caustic, the number of images it produces changes by two. Try and show this yourself by changing\n", + "the (y,x) centre of the source-plane galaxy's light profile!" ] }, { @@ -342,7 +342,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If you're too familiar with Cosmology, it will be unclear how exactly we converted the distance units from \n", + "If you're not too familiar with Cosmology, it will be unclear how exactly we converted the distance units from\n", "arcseconds to kiloparsecs. You'll need to read up on your Cosmology lecture to understand this properly.\n", "\n", "You can create a `Cosmology` object, which provides many methods for calculation different cosmological quantities, \n", @@ -518,9 +518,9 @@ "\n", "Now lets pass our 4 galaxies to a `Tracer`, which means the following will occur:\n", "\n", - " - Using the galaxy redshift`s, and image-plane and source-plane will be created each with two galaxies galaxies.\n", + " - Using the galaxy redshifts, an image-plane and source-plane will be created, each with two galaxies.\n", "\n", - "We've also pass the tracer below a Planck15 cosmology, where the cosomology of the Universe describes exactly how \n", + "We also pass the tracer below a Planck15 cosmology, where the cosmology of the Universe describes exactly how\n", "ray-tracing is performed." ] }, @@ -554,7 +554,7 @@ " 5) The image of the source galaxies is computed by summing both of their images and ray-tracing their light back to \n", " the image-plane.\n", " \n", - "This process is pretty much the same as we have single in previous tutorials when there is one galaxy per plane. We\n", + "This process is pretty much the same as we have seen in previous tutorials when there is one galaxy per plane. We\n", "are simply summing the images and deflection angles of the galaxies before using them to perform ray-tracing." ] }, @@ -606,7 +606,7 @@ "source": [ "__Wrap Up__\n", "\n", - "Tutorial 6 completed! Try the following:\n", + "Tutorial 3 completed! Try the following:\n", "\n", " 1) If you change the lens and source galaxy redshifts, does the tracer's image change?\n", "\n", diff --git a/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb b/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb index 754a4b0..46aad80 100644 --- a/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb @@ -7,15 +7,65 @@ "Tutorial 4: Point Sources\n", "=========================\n", "\n", - "This tutorial is not wrriten yet, but will explain how point source lensing works.\n", + "In the previous tutorials, the background sources we lensed were galaxies: extended objects whose light spreads over\n", + "many thousands of light years. When lensed, their light is warped into the arcs and Einstein rings we produced with\n", + "the `Tracer`, spread across many pixels of the image.\n", "\n", - "This tutorial is not necesary for using PyAutoLens or doing strong lens analysis, so don't worry that it is not\n", - "written yet!\n", + "However, not every source is a galaxy. Some of the most scientifically valuable strong lenses have a background\n", + "source that is physically tiny: a **quasar** (the bright accretion disk around a supermassive black hole, light-days\n", + "across) or a **supernova** (an exploding star, even smaller). These are called **point sources**.\n", "\n", - "Tutorial 8 summary is written and you should check that out instead!\n", + "When a point source is strongly lensed, we do not see arcs. Instead, we see the same source appear multiple times,\n", + "as two or four (or more) distinct, unresolved points of light in the image. Modeling these systems is fundamentally\n", + "different from modeling extended sources \u2014 different enough that it deserves its own tutorial before we go any\n", + "further.\n", + "\n", + "In this tutorial, we will see why. The short version: for an extended source, we ray-trace light *forwards* from the\n", + "image-plane to the source-plane, which is computationally simple. For a point source, we must go *backwards* \u2014 we\n", + "know where the source is, and we must find every image-plane position its light appears at. That means solving the\n", + "lens equation, an inverse problem with no analytic solution, which requires a dedicated numerical tool called the\n", + "`PointSolver`.\n", + "\n", + "Only the lens equation from tutorial 2 is needed to follow this tutorial. The full lensing formalism \u2014 convergence,\n", + "potentials, and how deflection angles arise from a mass distribution \u2014 is covered in the next tutorial.\n", + "\n", + "Here is an overview of what we'll cover:\n", + "\n", + "- **Initial Setup**: Create a strong lens with a compact extended source, showing how a small source produces\n", + " distinct multiple images rather than arcs.\n", + "\n", + "- **Point Sources**: What a point source is physically, and why quasars and supernovae appear as multiple\n", + " unresolved images.\n", + "\n", + "- **Point Source Tracer**: Represent a point source in PyAutoLens using the `Point` profile and build a `Tracer`.\n", + "\n", + "- **The Lens Equation**: Why finding a point source's multiple images means solving the lens equation \u2014 an inverse\n", + " problem with no analytic solution.\n", + "\n", + "- **Point Solver**: The `PointSolver`, which solves the lens equation numerically by ray-tracing triangles.\n", + "\n", + "- **Multiple Images and Critical Curves**: Plot the solved image positions over the lensed image and the tracer's\n", + " critical curves.\n", + "\n", + "- **Magnifications**: Compute the magnification at each multiple image, and why observed fluxes are often\n", + " unreliable for modeling.\n", + "\n", + "- **Time Delays**: The arrival-time differences between multiple images, and why they matter for cosmology.\n", + "\n", + "- **Extended Versus Point Computations**: A recap of why the two regimes require fundamentally different\n", + " calculations and fitting strategies.\n", "\n", "__Contents__\n", "\n", + "- **Initial Setup:** Create a strong lens with a compact extended source, producing distinct multiple images.\n", + "- **Point Sources:** What a point source is physically and why it appears as multiple unresolved images.\n", + "- **Point Source Tracer:** Represent a point source with the `Point` profile and build a `Tracer`.\n", + "- **The Lens Equation:** Finding multiple images means solving the lens equation, an inverse problem.\n", + "- **Point Solver:** The `PointSolver` solves the lens equation numerically via triangle ray-tracing.\n", + "- **Multiple Images and Critical Curves:** Plot the solved positions over the image and critical curves.\n", + "- **Magnifications:** The magnification of each multiple image, and why fluxes are often unreliable.\n", + "- **Time Delays:** The relative arrival times of the multiple images and their use in cosmology.\n", + "- **Extended Versus Point Computations:** Why the two regimes require fundamentally different calculations.\n", "- **Wrap Up:** Summary of the script and next steps." ] }, @@ -64,7 +114,9 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import numpy as np\n", "\n", "import autolens as al\n", "import autolens.plot as aplt" @@ -76,7 +128,407 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "__Wrap Up__" + "__Initial Setup__\n", + "\n", + "Let's begin with a strong lens system like those of the previous tutorials: an elliptical isothermal mass profile\n", + "for the lens galaxy, and an extended light profile for the source galaxy.\n", + "\n", + "There is one deliberate difference: the source's `effective_radius` is just 0.1\", making it far more compact than\n", + "the sources we lensed before." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = al.Grid2D.uniform(\n", + " shape_native=(100, 100),\n", + " pixel_scales=0.05,\n", + ")\n", + "\n", + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=1.6,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", + " ),\n", + ")\n", + "\n", + "source_galaxy_extended = al.Galaxy(\n", + " redshift=1.0,\n", + " light=al.lp.ExponentialCore(\n", + " centre=(0.07, 0.07), intensity=0.1, effective_radius=0.1\n", + " ),\n", + ")\n", + "\n", + "tracer_extended = al.Tracer(galaxies=[lens_galaxy, source_galaxy_extended])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When we plot the image of this system, something new happens: instead of a sweeping arc or Einstein ring, the\n", + "source appears as distinct blobs of light \u2014 the same source, imaged multiple times at different locations in\n", + "the image-plane.\n", + "\n", + "This is a general rule of strong lensing: the more compact the source, the more its lensed image breaks up into\n", + "separate multiple images. An extended galaxy blurs its multiple images together into arcs; a compact source keeps\n", + "them distinct." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(\n", + " array=tracer_extended.image_2d_from(grid=grid),\n", + " title=\"Compact Source Multiple Images\",\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Point Sources__\n", + "\n", + "Now imagine shrinking the source further \u2014 not to 0.01\", but to the physical size of a quasar accretion disk or a\n", + "supernova photosphere. These objects are light-days to light-weeks across, which at cosmological distances\n", + "corresponds to micro-arcseconds on the sky. That is millions of times smaller than the resolution of any telescope.\n", + "\n", + "Strictly speaking, such a source still has a finite size, but the telescope cannot resolve it. Each multiple image\n", + "appears as a single unresolved point of light, with all of its flux effectively contained within one pixel (spread\n", + "only by the telescope's blurring, which we will meet in a later tutorial).\n", + "\n", + "This is what we call a **point source**. Lensed quasars and lensed supernovae are the classic examples, and they\n", + "are prized objects: the arrival-time differences between their multiple images can be used to measure the expansion\n", + "rate of the Universe (the Hubble constant), and their fluxes are sensitive probes of small-scale dark matter\n", + "structure.\n", + "\n", + "For a point source, the extended-source picture of lensing breaks down:\n", + "\n", + "- There is no arc or ring of surface brightness to fit, just a handful of image positions.\n", + "\n", + "- The observable quantities are the (y,x) **positions** of the multiple images, and optionally their **fluxes**\n", + " and **time delays**.\n", + "\n", + "- The concept of evaluating a light profile's surface brightness on a ray-traced grid \u2014 the engine of the previous\n", + " two tutorials \u2014 no longer applies, because the source has no extent to evaluate.\n", + "\n", + "__Point Source Tracer__\n", + "\n", + "PyAutoLens represents a point source with the `Point` profile, found in the `al.ps` module (`ps` for point source).\n", + "Unlike a light profile it has no `intensity`, `effective_radius` or elliptical components \u2014 it is fully described\n", + "by its (y,x) centre in the source-plane.\n", + "\n", + "We give the point source the same centre, (0.07\", 0.07\"), as the compact extended source above, so its multiple\n", + "images will coincide with the bright blobs in the image we just plotted." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "point_source = al.ps.Point(centre=(0.07, 0.07))\n", + "\n", + "source_galaxy = al.Galaxy(redshift=1.0, point_0=point_source)\n", + "\n", + "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that we attached the point source to its galaxy with the name `point_0`. This name is a label that PyAutoLens\n", + "uses when fitting real data to pair each point source in the model with the dataset containing its observed image\n", + "positions. With one source the name is a formality, but group- and cluster-scale lenses can contain many point\n", + "sources, and the names keep each one matched to its own data.\n", + "\n", + "__The Lens Equation__\n", + "\n", + "In tutorial 2, we met the **lens equation**, which relates a coordinate in the image-plane to the coordinate in\n", + "the source-plane its light ray comes from:\n", + "\n", + "$\\beta = \\theta - \\alpha(\\theta)$\n", + "\n", + "where $\\theta$ is the image-plane (y,x) coordinate, $\\alpha(\\theta)$ is the deflection angle at that coordinate,\n", + "and $\\beta$ is the source-plane (y,x) coordinate.\n", + "\n", + "For the extended sources of tutorials 2 and 3, we always used this equation in the *forward* direction: take every\n", + "image-plane coordinate $\\theta$ on our grid, compute its deflection angles, and subtract to get $\\beta$. This is\n", + "easy \u2014 it is just an evaluation, one subtraction per coordinate.\n", + "\n", + "For a point source, the question is reversed. We *know* the source's position $\\beta$ (the centre of the `Point`\n", + "profile), and we want to find every image-plane position $\\theta$ that satisfies the lens equation for that\n", + "$\\beta$. Each solution is one of the multiple images.\n", + "\n", + "This is called **solving the lens equation**, and it is much harder than evaluating it:\n", + "\n", + "- The equation is non-linear, because the deflection angles $\\alpha(\\theta)$ vary with position in a way set by the\n", + " mass profile. For all but the simplest mass profiles, no analytic solution exists.\n", + "\n", + "- There are multiple solutions \u2014 that is the whole point! A single $\\beta$ maps to two, four or more distinct\n", + " $\\theta$ positions, and we must find all of them.\n", + "\n", + "We therefore need a numerical root-finding method that searches the image-plane for every position whose ray-traced\n", + "source-plane coordinate lands on the source. (Where the deflection angles themselves come from is the subject of\n", + "the next tutorial \u2014 for now, we just need the equation.)\n", + "\n", + "__Point Solver__\n", + "\n", + "PyAutoLens solves the lens equation with the `PointSolver`, which uses a triangle-based scheme:\n", + "\n", + "1. Overlay the image-plane with a grid of triangles.\n", + "\n", + "2. Ray-trace the corners of every triangle to the source-plane using the lens equation, giving a set of\n", + " ray-traced source-plane triangles.\n", + "\n", + "3. Keep every triangle that contains the source's (y,x) centre \u2014 somewhere inside that image-plane triangle is a\n", + " multiple image.\n", + "\n", + "4. Subdivide the retained triangles into smaller triangles and repeat, homing in on each multiple image with\n", + " progressively finer precision.\n", + "\n", + "The `pixel_scale_precision` input sets the triangle size at which the solver stops refining, and therefore the\n", + "precision of the image positions. Smaller values are more precise but cost more computation; 0.001\" is a good\n", + "balance for most lens modeling.\n", + "\n", + "Mass models also predict a faint \"central image\" near the centre of the lens, which is usually so heavily\n", + "demagnified that real observations never detect it. The `magnification_threshold` input discards solved images\n", + "whose magnification is below the threshold, removing this central image to match what the data would contain." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "solver_grid = al.Grid2D.uniform(\n", + " shape_native=(100, 100),\n", + " pixel_scales=0.2,\n", + ")\n", + "\n", + "solver = al.PointSolver.for_grid(\n", + " grid=solver_grid, pixel_scale_precision=0.001, magnification_threshold=0.1\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We now solve the lens equation, passing the solver our tracer (which provides the mass model's deflection angles)\n", + "and the source-plane centre of the point source.\n", + "\n", + "The result is the set of image-plane (y,x) coordinates of the multiple images." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "positions = solver.solve(tracer=tracer, source_plane_coordinate=(0.07, 0.07))\n", + "\n", + "print(\"Multiple Image Positions (y,x) in arc-seconds:\")\n", + "print(positions)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Multiple Images and Critical Curves__\n", + "\n", + "For our elliptical mass profile, the solver finds four multiple images (a fifth, central image existed but was\n", + "removed by the `magnification_threshold`). The number of images depends on the mass model:\n", + "\n", + "- Spherical mass profiles produce three images (including the demagnified central image).\n", + "\n", + "- Elliptical mass profiles produce five images (again including the central image).\n", + "\n", + "- More complex systems, with multiple lens galaxies, can produce even more.\n", + "\n", + "To see that the solver got the right answer, we plot the solved positions on top of the compact extended source's\n", + "image from the start of the tutorial. The positions land on the centres of the bright multiple images.\n", + "\n", + "We also overlay the tracer's tangential critical curve (introduced in tutorial 3 \u2014 the line of infinite\n", + "magnification in the image-plane). The multiple images of a strongly lensed point source straddle the critical\n", + "curve, and images close to it are the most highly magnified." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tangential_critical_curve_list = al.LensCalc.from_tracer(\n", + " tracer=tracer\n", + ").tangential_critical_curve_list_from(grid=grid)\n", + "\n", + "aplt.plot_array(\n", + " array=tracer_extended.image_2d_from(grid=grid),\n", + " positions=positions,\n", + " lines=tangential_critical_curve_list,\n", + " title=\"Multiple Images and Critical Curve\",\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Magnifications__\n", + "\n", + "Lensing does not just relocate a point source's light \u2014 it magnifies it. Each multiple image has its own\n", + "magnification, set by how strongly the mass model focuses light rays at that image-plane position, and it can make\n", + "an image tens of times brighter than the unlensed source.\n", + "\n", + "For an extended source, magnification stretches the source over more pixels. For a point source, the image stays\n", + "unresolved \u2014 so the magnification instead multiplies its observed **flux**. If the source's intrinsic flux is $F$,\n", + "the flux of a multiple image with magnification $\\mu$ is $|\\mu| * F$.\n", + "\n", + "We compute the magnification at each solved image position below (the sign of $\\mu$ encodes the image's parity \u2014\n", + "whether it is mirror-flipped \u2014 which is why we take its absolute value for fluxes)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "magnifications = al.LensCalc.from_tracer(\n", + " tracer=tracer\n", + ").magnification_2d_via_hessian_from(grid=positions)\n", + "\n", + "print(\"Magnification of each Multiple Image:\")\n", + "print(magnifications)\n", + "\n", + "flux = 1.0\n", + "fluxes = [flux * np.abs(magnification) for magnification in magnifications]\n", + "\n", + "print(\"Flux of each Multiple Image (for a source of flux 1.0):\")\n", + "print(fluxes)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In principle, comparing the observed flux ratios of the images to the model's magnification ratios is a powerful\n", + "constraint on the mass model. In practice, point-source fluxes are treated with caution, because effects the smooth\n", + "mass model does not include can change them substantially:\n", + "\n", + "- **Microlensing**: individual stars in the lens galaxy lens the point source again on micro-arcsecond scales,\n", + " magnifying or demagnifying each image by unpredictable amounts that change over time.\n", + "\n", + "- **Dark matter substructure**: small invisible clumps of dark matter near an image perturb its magnification\n", + " (this is a systematic for mass modeling, but also exactly why lensed quasars are used to *search* for\n", + " substructure).\n", + "\n", + "- Dust in the lens galaxy and the source's own intrinsic variability further distort the observed flux ratios.\n", + "\n", + "Image positions are essentially immune to all of these, which is why point-source lens modeling is built on\n", + "positions first, with fluxes included only when the science demands it and the systematics are under control.\n", + "\n", + "__Time Delays__\n", + "\n", + "There is a third observable unique to point sources. The light of each multiple image travels a different path\n", + "through the Universe, and each path takes a different time \u2014 partly because the geometric path lengths differ, and\n", + "partly because light is slowed as it climbs through the lens galaxy's gravitational field (the Shapiro delay).\n", + "\n", + "For an extended, steady source this is unobservable. But quasars flicker and supernovae explode: when the source\n", + "varies, the variation appears in each multiple image at a different time. By monitoring the images, these\n", + "**time delays** can be measured \u2014 typically days to months apart.\n", + "\n", + "Time delays are a cosmological gold mine: their absolute scale depends on the distances between us, the lens and\n", + "the source, so a lens model plus measured delays yields the Hubble constant. This is time-delay cosmography, one\n", + "of the headline applications of point-source lensing.\n", + "\n", + "The tracer computes the arrival time at each image position from its mass model and the galaxies' redshifts. Only\n", + "the *differences* between images are observable \u2014 a delay of the same length along every path is invisible \u2014 so it\n", + "is the relative values below that matter." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "time_delays = tracer.time_delays_from(grid=positions)\n", + "\n", + "print(\"Time Delay of each Multiple Image (days):\")\n", + "print(time_delays)\n", + "\n", + "print(\"Time Delays relative to first image (days):\")\n", + "print(time_delays - time_delays[0])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Extended Versus Point Computations__\n", + "\n", + "We can now state precisely why point-source lensing works so differently from everything in tutorials 2 and 3, on\n", + "both the computation and the data.\n", + "\n", + "**Extended sources are forward-modeled.** To predict the data, we ray-trace every image-plane pixel to the\n", + "source-plane via the lens equation (an evaluation, not a solve) and compute the source light profile's surface\n", + "brightness at each ray-traced coordinate. This is computationally cheap, requires no equation solving, and a fit\n", + "compares the predicted and observed images pixel-by-pixel.\n", + "\n", + "**Point sources require solving an inverse problem.** The source has no surface brightness to evaluate \u2014 the model\n", + "must predict the discrete positions of the multiple images, which means numerically solving the lens equation with\n", + "the `PointSolver` every time the mass model changes. A fit then compares the predicted image positions to the\n", + "observed ones (and optionally the predicted fluxes and time delays to their measurements) \u2014 a handful of numbers,\n", + "rather than thousands of pixels.\n", + "\n", + "This trade shapes the whole analysis. The point-source dataset is tiny \u2014 a few positions with uncertainties \u2014 but\n", + "each model evaluation involves root finding over the image plane rather than simple forward ray-tracing. Whole\n", + "questions that never arose for extended sources, such as how to pair predicted images with observed ones when the\n", + "model predicts too many or too few, become central to the likelihood (the workspace covers these in detail).\n", + "\n", + "__Wrap Up__\n", + "\n", + "In this tutorial, we met point sources and the tools PyAutoLens uses to model them. Let's summarise what we've\n", + "learnt:\n", + "\n", + "- **Point Sources**: Quasars and supernovae are so much smaller than a telescope's resolution that each of their\n", + " multiple images appears as a single unresolved point of light \u2014 no arcs, no rings.\n", + "\n", + "- **The Lens Equation**: Point-source modeling means solving $\\beta = \\theta - \\alpha(\\theta)$ for the image\n", + " positions $\\theta$ given the source position $\\beta$ \u2014 a non-linear inverse problem with no analytic solution,\n", + " in contrast to the cheap forward evaluation used for extended sources.\n", + "\n", + "- **Point Solver**: The `PointSolver` solves the lens equation numerically, ray-tracing progressively finer\n", + " triangles until every multiple image is located to sub-pixel precision, and filtering out the demagnified\n", + " central image.\n", + "\n", + "- **Observables**: A point-source dataset consists of image positions, and optionally fluxes and time delays.\n", + " Positions are the bedrock; fluxes are often compromised by microlensing and substructure; time delays enable\n", + " measurement of the Hubble constant.\n", + "\n", + "When you are ready to model real lensed quasars and supernovae \u2014 composing mass models, fitting observed positions\n", + "with a non-linear search, and including fluxes and time delays \u2014 the `autolens_workspace/scripts/point_source`\n", + "package is the place to go, starting with its `start_here` example.\n", + "\n", + "In the next tutorial, we return to the lensing formalism itself: where deflection angles come from, and the\n", + "quantities (convergence, potential, magnification) that describe a mass distribution's lensing power." ] } ], diff --git a/notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb b/notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb index bf6d4cd..7955791 100644 --- a/notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb @@ -7,16 +7,37 @@ "Tutorial 5: Lensing Formalism\n", "=============================\n", "\n", - "This tutorial is not wrriten yet, but will explain what all the different lens quantities are and give a more\n", - "formal description of them.\n", + "This tutorial is the equations lecture of **HowToLens**.\n", "\n", - "This tutorial is not necesary for using PyAutoLens or doing strong lens analysis, so don't worry that it is not\n", - "written yet!\n", + "Every other tutorial in this series deliberately defers the formal mathematics of gravitational lensing, so that you\n", + "could first learn lensing hands-on: in the earlier tutorials we created grids, evaluated light and mass profiles,\n", + "ray-traced coordinates from the image-plane to the source-plane and looked at quantities like the convergence,\n", + "deflection angles and critical curves visually, without ever writing down their formal definitions.\n", "\n", - "Tutorial 8 summary is written and you should check that out instead!\n", + "This tutorial gathers all of that algebra in one place. Each quantity you have already computed and plotted \u2014\n", + "deflection angles, convergence, potential, magnification, critical curves, the Einstein radius \u2014 is now given its\n", + "formal mathematical definition, explained in plain words, and tied back to the exact **PyAutoLens** method that\n", + "computes it. The goal is that after this tutorial, every symbol in a lensing paper maps to a line of code you know.\n", + "\n", + "None of this mathematics is required to use **PyAutoLens** or to analyse strong lenses \u2014 the library computes\n", + "everything for you. But understanding where each quantity comes from will make you a better lens modeler, and this\n", + "is the single place in **HowToLens** where the formalism lives, so you can return here whenever you need it.\n", + "\n", + "We will build one simple strong lens system \u2014 an elliptical isothermal (SIE) lens galaxy and a Sersic source \u2014 and\n", + "reuse it throughout, computing every quantity from the same tracer so you can see how they all fit together.\n", "\n", "__Contents__\n", "\n", + "- **Initial Setup:** Build the grid, SIE lens, Sersic source and `Tracer` reused throughout the tutorial.\n", + "- **Cosmological Distances:** Angular diameter distances D_l, D_s, D_ls and how arcseconds convert to kiloparsecs.\n", + "- **The Lens Equation:** beta = theta - alpha(theta), the mapping from image-plane to source-plane, and multiple images.\n", + "- **Convergence:** Surface mass density Sigma, the critical surface density Sigma_cr and kappa = Sigma / Sigma_cr.\n", + "- **Deflection Angles:** The integral relating the convergence field to the deflection angles.\n", + "- **The Lensing Potential:** The scalar potential psi, with alpha = grad psi and kappa = (1/2) laplacian psi.\n", + "- **Shear and Magnification:** The Hessian, the Jacobian matrix A, shear gamma and magnification mu = 1 / det(A).\n", + "- **Critical Curves and Caustics:** Where det(A) = 0 in the image-plane, and its map to the source-plane.\n", + "- **Einstein Radius:** Its definition, the SIS and point-mass closed forms, and why it is the robust mass measurement.\n", + "- **Time Delays:** The Fermat potential and the time-delay surface, in brief.\n", "- **Wrap Up:** Summary of the script and next steps." ] }, @@ -65,9 +86,12 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", "import autolens as al\n", + "import autoarray as aa\n", "import autolens.plot as aplt" ], "outputs": [], @@ -77,7 +101,746 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "__Wrap Up__" + "__Initial Setup__\n", + "\n", + "We first build the strong lens system used throughout this tutorial, mirroring the objects introduced in tutorials\n", + "2 and 3: a 2D grid of (y,x) arcsecond coordinates, a lens galaxy with an elliptical isothermal (`Isothermal`) mass\n", + "profile \u2014 the singular isothermal ellipsoid, or SIE, the workhorse mass model of galaxy-scale lensing \u2014 and a\n", + "source galaxy with a Sersic light profile." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = al.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05)\n", + "\n", + "mass_profile = al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0),\n", + " einstein_radius=1.6,\n", + ")\n", + "\n", + "lens_galaxy = al.Galaxy(redshift=0.5, mass=mass_profile)\n", + "\n", + "source_galaxy = al.Galaxy(\n", + " redshift=1.0,\n", + " bulge=al.lp.SersicCore(\n", + " centre=(0.0, 0.1),\n", + " ell_comps=(0.1, 0.0),\n", + " intensity=0.3,\n", + " effective_radius=0.3,\n", + " sersic_index=1.0,\n", + " ),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We combine these into a `Tracer` with a Planck 2015 cosmology. The redshifts of the two galaxies (0.5 and 1.0) and\n", + "the cosmology together fix every distance that appears in the equations below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "cosmology = al.cosmo.Planck15()\n", + "\n", + "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy], cosmology=cosmology)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here is the lensed image of this system, which by now should look familiar. Every equation in this tutorial is\n", + "describing some aspect of how this image forms." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "image = tracer.image_2d_from(grid=grid)\n", + "aplt.plot_array(array=image, title=\"Lensed Image of the Source Galaxy\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We also create a `LensCalc` object from the tracer. As we saw in tutorial 3, this is **PyAutoLens**'s calculator for\n", + "derived lensing quantities \u2014 everything it computes is derived from the tracer's deflection angles, which is a theme\n", + "we will see repeatedly below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_calc = al.LensCalc.from_tracer(tracer=tracer)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Cosmological Distances__\n", + "\n", + "Gravitational lensing is a geometric effect, so distances are everywhere in its equations. The distances used are\n", + "**angular diameter distances**, defined so that an object of physical size l at angular diameter distance D subtends\n", + "an angle (in radians):\n", + "\n", + " theta = l / D\n", + "\n", + "Three distances appear in every lensing formula:\n", + "\n", + "- D_l \u2014 the angular diameter distance from the observer to the lens (deflector) galaxy.\n", + "- D_s \u2014 the angular diameter distance from the observer to the source galaxy.\n", + "- D_ls \u2014 the angular diameter distance from the lens to the source.\n", + "\n", + "In an expanding Universe angular diameter distances do not add linearly, so D_ls != D_s - D_l. Each is computed by\n", + "integrating the cosmological expansion history between the two redshifts, which is why the tracer requires both\n", + "galaxy redshifts and a cosmology.\n", + "\n", + "The `Cosmology` object computes each of these directly (values below are in kiloparsecs):" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "D_l = cosmology.angular_diameter_distance_to_earth_in_kpc_from(redshift=0.5)\n", + "D_s = cosmology.angular_diameter_distance_to_earth_in_kpc_from(redshift=1.0)\n", + "D_ls = cosmology.angular_diameter_distance_between_redshifts_in_kpc_from(\n", + " redshift_0=0.5, redshift_1=1.0\n", + ")\n", + "\n", + "print(f\"D_l (kpc) = {D_l:.3e}\")\n", + "print(f\"D_s (kpc) = {D_s:.3e}\")\n", + "print(f\"D_ls (kpc) = {D_ls:.3e}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The angular diameter distance is also what converts **PyAutoLens**'s internal angular units of arcseconds into\n", + "physical distances. An angle of 1.0\" at redshift z corresponds to a physical size:\n", + "\n", + " l = D(z) * (1.0\" in radians)\n", + "\n", + "The `Cosmology` object wraps this up as a `kpc_per_arcsec` conversion factor, which differs between the image-plane\n", + "(lens redshift) and source-plane (source redshift) because the two planes are at different distances." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "kpc_per_arcsec_lens = cosmology.kpc_per_arcsec_from(redshift=0.5)\n", + "kpc_per_arcsec_source = cosmology.kpc_per_arcsec_from(redshift=1.0)\n", + "\n", + "print(f\"kpc per arcsec at the lens (z=0.5) = {kpc_per_arcsec_lens:.4f}\")\n", + "print(f\"kpc per arcsec at the source (z=1.0) = {kpc_per_arcsec_source:.4f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So our lens's Einstein radius of 1.6\" corresponds to a physical scale of roughly 1.6 * 6.3 ~ 10 kpc at the lens\n", + "redshift \u2014 a sensible size for the inner regions of a massive elliptical galaxy.\n", + "\n", + "__The Lens Equation__\n", + "\n", + "The fundamental equation of gravitational lensing is the **lens equation**, which we met in tutorial 2:\n", + "\n", + " beta = theta - alpha(theta)\n", + "\n", + "where:\n", + "\n", + "- theta is the observed (image-plane) angular position of a light ray, in arcseconds.\n", + "- alpha(theta) is the (scaled) deflection angle \u2014 how much the ray is bent by the lens's gravity at position theta.\n", + "- beta is the true (source-plane) position \u2014 where the source would appear if there were no lens.\n", + "\n", + "In plain words: to find where a light ray we observe at theta really came from, subtract the deflection the lens\n", + "applied to it. This is exactly the ray-tracing calculation we performed in tutorial 2, and it is worth knowing that\n", + "the deflection angle here is the \"scaled\" deflection: the physical bending angle of the ray multiplied by the\n", + "distance ratio D_ls / D_s, which is the convention that makes the lens equation take the simple form above.\n", + "\n", + "The lens equation is trivial to evaluate one way (given theta, compute beta), but it cannot be inverted\n", + "analytically: for a strong lens, *multiple* image-plane positions theta can satisfy the equation for the *same*\n", + "source position beta. This is why strong lenses produce multiple images of a single source \u2014 and why lens modeling\n", + "works \"forwards\", ray-tracing image-plane grids to the source-plane rather than the other way around.\n", + "\n", + "In code, the lens equation is one line \u2014 the same `grid_2d_via_deflection_grid_from` call from tutorial 2:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "deflections = tracer.deflections_yx_2d_from(grid=grid)\n", + "\n", + "source_plane_grid = grid.grid_2d_via_deflection_grid_from(deflection_grid=deflections)\n", + "\n", + "aplt.plot_grid(grid=grid, title=\"Image-Plane Grid (theta)\")\n", + "aplt.plot_grid(grid=source_plane_grid, title=\"Source-Plane Grid (beta)\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Convergence__\n", + "\n", + "The lens galaxy's mass enters lensing via its **surface mass density** Sigma(theta): the galaxy's 3D mass density\n", + "integrated along the line of sight, giving a projected 2D mass per unit area (e.g. in solar masses per kpc^2).\n", + "Lensing only cares about this projection \u2014 two very different 3D distributions with the same projected Sigma\n", + "deflect light identically.\n", + "\n", + "Whether a lens is \"strong\" is set by comparing Sigma to the **critical surface density**:\n", + "\n", + " Sigma_cr = (c^2 / (4 pi G)) * (D_s / (D_l * D_ls))\n", + "\n", + "This is a purely cosmological quantity \u2014 it depends only on the constants c and G and the three distances from the\n", + "previous section. It has a beautiful interpretation: it is the surface density a lens needs, given this observer /\n", + "lens / source geometry, to be capable of producing multiple images." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "sigma_cr_kpc = cosmology.critical_surface_density_between_redshifts_solar_mass_per_kpc2_from(\n", + " redshift_0=0.5, redshift_1=1.0\n", + ")\n", + "sigma_cr_arcsec = cosmology.critical_surface_density_between_redshifts_from(\n", + " redshift_0=0.5, redshift_1=1.0\n", + ")\n", + "\n", + "print(f\"Sigma_cr (solar masses per kpc^2) = {sigma_cr_kpc:.3e}\")\n", + "print(f\"Sigma_cr (solar masses per arcsec^2) = {sigma_cr_arcsec:.3e}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The **convergence** kappa is the surface mass density in units of the critical density:\n", + "\n", + " kappa(theta) = Sigma(theta) / Sigma_cr\n", + "\n", + "This is the dimensionless quantity **PyAutoLens** has been plotting since tutorial 2 whenever we called\n", + "`convergence_2d_from`:\n", + "\n", + "- kappa >= 1 \u2014 the lens is super-critical at that point; multiple imaging is possible.\n", + "- kappa < 1 \u2014 the lens is sub-critical there.\n", + "\n", + "Because kappa is dimensionless, lens models can be composed and fitted without knowing the galaxy redshifts at all \u2014\n", + "the redshifts and Sigma_cr are only needed at the end, to convert the inferred kappa back into physical masses." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "convergence = tracer.convergence_2d_from(grid=grid)\n", + "\n", + "aplt.plot_array(array=convergence, title=\"Convergence (kappa)\", use_log10=True)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Deflection Angles__\n", + "\n", + "The convergence field determines the deflection angles everywhere, via a 2D integral over the whole image-plane:\n", + "\n", + " alpha(theta) = (1 / pi) * integral d^2 theta' kappa(theta') * (theta - theta') / |theta - theta'|^2\n", + "\n", + "In plain words: every patch of mass in the lens plane pulls on every light ray. The deflection at position theta is\n", + "the sum (integral) of the pulls from all mass elements kappa(theta'), each pointing from the mass element towards\n", + "the ray and falling off as 1/distance \u2014 the 2D (projected) analogue of Newtonian gravity's inverse-square law.\n", + "\n", + "Two things follow from this integral being over *all* theta':\n", + "\n", + "- The deflection at a point depends on the mass distribution *everywhere*, not just the mass at that point. Even\n", + " mass well outside the region where images form contributes deflections (this is why external shear from\n", + " neighbouring galaxies matters, as we saw in tutorial 3).\n", + "\n", + "- For simple analytic profiles (isothermal, power-law, Sersic-like) this integral has closed-form solutions, which\n", + " is exactly what a **PyAutoLens** mass profile is: an analytic kappa(theta) paired with its analytic alpha(theta).\n", + "\n", + "The `deflections_yx_2d_from` method we have used since tutorial 2 evaluates this integral's closed-form solution.\n", + "The deflections form a 2D vector field, so we plot its y and x components separately:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "deflections = tracer.deflections_yx_2d_from(grid=grid)\n", + "\n", + "deflections_y = aa.Array2D(values=deflections.slim[:, 0], mask=grid.mask)\n", + "aplt.plot_array(array=deflections_y, title=\"Deflection Angles (y component)\")\n", + "\n", + "deflections_x = aa.Array2D(values=deflections.slim[:, 1], mask=grid.mask)\n", + "aplt.plot_array(array=deflections_x, title=\"Deflection Angles (x component)\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__The Lensing Potential__\n", + "\n", + "The deflection field is not arbitrary \u2014 it is the gradient of a scalar field, the **lensing potential** psi(theta):\n", + "\n", + " alpha(theta) = grad psi(theta)\n", + "\n", + "The potential is itself an integral over the convergence:\n", + "\n", + " psi(theta) = (1 / pi) * integral d^2 theta' kappa(theta') * ln|theta - theta'|\n", + "\n", + "and taking the divergence of the gradient (the Laplacian) recovers the convergence with a factor of a half:\n", + "\n", + " kappa(theta) = (1/2) * laplacian psi(theta)\n", + "\n", + "This is the 2D Poisson equation of lensing \u2014 the projected analogue of Newtonian gravity's del^2 Phi = 4 pi G rho.\n", + "The potential is the single most economical description of a lens: one scalar field from which the deflections\n", + "(first derivatives), and the convergence, shear and magnification (second derivatives) all follow.\n", + "\n", + "The `potential_2d_from` method returns psi, which we have plotted before without defining it:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "potential = tracer.potential_2d_from(grid=grid)\n", + "\n", + "aplt.plot_array(array=potential, title=\"Lensing Potential (psi)\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can verify alpha = grad psi numerically. Mass profiles have a `deflections_2d_via_potential_2d_from` method which\n", + "computes the deflections by numerically differentiating the potential, rather than using the profile's closed-form\n", + "deflection formula. Across the grid the two agree closely \u2014 we print the median absolute difference, since the\n", + "finite-difference derivative is inaccurate right at the isothermal profile's central cusp, where the potential is\n", + "not smooth (the closed-form deflections have no such problem):" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "deflections_analytic = mass_profile.deflections_yx_2d_from(grid=grid)\n", + "deflections_via_potential = mass_profile.deflections_2d_via_potential_2d_from(grid=grid)\n", + "\n", + "difference = np.median(\n", + " np.abs(np.asarray(deflections_analytic) - np.asarray(deflections_via_potential))\n", + ")\n", + "print(f\"Median |alpha_analytic - grad psi| = {difference:.3e} arcsec\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Shear and Magnification__\n", + "\n", + "How a small image is distorted by lensing is governed by how the deflection angles *change* across it \u2014 the second\n", + "derivatives of the potential. These form the 2x2 **Hessian** matrix, which `LensCalc` computes by finite\n", + "differences of the deflection field (as we saw in the workspace guides, this works for any mass distribution):\n", + "\n", + " H_yy = d(alpha_y) / d(theta_y) H_xy = d(alpha_x) / d(theta_y)\n", + " H_yx = d(alpha_y) / d(theta_x) H_xx = d(alpha_x) / d(theta_x)" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "hessian_yy, hessian_xy, hessian_yx, hessian_xx = lens_calc.hessian_from(grid=grid)\n", + "\n", + "print(f\"Hessian components at pixel 0: H_yy = {hessian_yy[0]:.4f}, H_xx = {hessian_xx[0]:.4f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Differentiating the lens equation beta = theta - alpha(theta) gives the **Jacobian matrix** A, which maps a small\n", + "displacement in the source-plane to the corresponding displacement in the image-plane:\n", + "\n", + " A = d(beta) / d(theta) = I - H = | 1 - H_yy -H_xy |\n", + " | -H_yx 1 - H_xx |\n", + "\n", + "The Jacobian decomposes into two physically distinct distortions:\n", + "\n", + "- The **convergence** kappa = (1/2) * (H_yy + H_xx) \u2014 the isotropic part, which magnifies an image uniformly\n", + " without changing its shape. (Note this is the same kappa as before: the trace of the Hessian recovers the\n", + " Poisson equation kappa = (1/2) laplacian psi.)\n", + "\n", + "- The **shear** gamma \u2014 the anisotropic part, which stretches an image along one axis and squeezes it along the\n", + " perpendicular axis. It has two components and a magnitude:\n", + "\n", + " gamma_1 = (1/2) * (H_xx - H_yy)\n", + " gamma_2 = H_xy\n", + " |gamma| = sqrt(gamma_1^2 + gamma_2^2)\n", + "\n", + "The shear is why lensed images near the lens are stretched into tangential arcs \u2014 the tidal field of the lens\n", + "elongates them around it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "shear = lens_calc.shear_yx_2d_via_hessian_from(grid=grid)\n", + "\n", + "print(f\"Shear magnitude at pixel 0: |gamma| = {shear.magnitudes[0]:.4f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The **magnification** mu is the inverse of the Jacobian's determinant:\n", + "\n", + " mu = 1 / det(A) = 1 / [ (1 - kappa)^2 - |gamma|^2 ]\n", + "\n", + "Lensing conserves surface brightness, so a lensed image that covers more sky than the unlensed source appears\n", + "brighter in total by exactly the factor |mu|:\n", + "\n", + "- |mu| > 1 \u2014 the image is magnified (larger and brighter than the unlensed source).\n", + "- |mu| < 1 \u2014 the image is demagnified.\n", + "- mu < 0 \u2014 the image has negative parity: it is a mirror image of the source.\n", + "\n", + "The determinant factorises into two eigenvalues, giving the **tangential** and **radial** magnifications:\n", + "\n", + " lambda_t = 1 - kappa - |gamma| (tangential eigenvalue)\n", + " lambda_r = 1 - kappa + |gamma| (radial eigenvalue)\n", + "\n", + " mu = 1 / (lambda_t * lambda_r)\n", + "\n", + "An image is stretched by 1/lambda_t in the tangential direction (around the lens) and 1/lambda_r in the radial\n", + "direction (towards/away from the lens). Giant tangential arcs form where lambda_t is close to zero." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "magnification = lens_calc.magnification_2d_from(grid=grid)\n", + "\n", + "aplt.plot_array(array=magnification, title=\"Magnification (mu)\")\n", + "\n", + "tangential_eigen_values = lens_calc.tangential_eigen_value_from(grid=grid)\n", + "radial_eigen_values = lens_calc.radial_eigen_value_from(grid=grid)\n", + "\n", + "print(f\"Tangential eigenvalue at pixel 0: {tangential_eigen_values[0]:.4f}\")\n", + "print(f\"Radial eigenvalue at pixel 0: {radial_eigen_values[0]:.4f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Critical Curves and Caustics__\n", + "\n", + "Where either eigenvalue passes through zero, det(A) = 0 and the magnification formally diverges to infinity. The\n", + "closed curves in the image-plane where this happens are the **critical curves** \u2014 the white and yellow lines that\n", + "have appeared on plots since tutorial 2, and which we explored visually in tutorial 3:\n", + "\n", + "- The **tangential critical curve** (lambda_t = 0) \u2014 roughly traces the Einstein ring; sources near its source-plane\n", + " counterpart form giant tangential arcs.\n", + "- The **radial critical curve** (lambda_r = 0) \u2014 an inner curve associated with radially stretched central images.\n", + "\n", + "Ray-tracing each critical curve through the lens equation maps it to the source-plane, where it is called a\n", + "**caustic**:\n", + "\n", + " caustic = critical_curve - alpha(critical_curve)\n", + "\n", + "Caustics divide the source-plane into regions of different image multiplicity: each time a source crosses a\n", + "caustic, the number of images it produces changes by two. For our SIE lens, a source inside the tangential caustic\n", + "produces four images (plus a faint central image); between the tangential and radial caustics, two; and outside\n", + "both caustics, just one \u2014 the source is no longer multiply imaged at all.\n", + "\n", + "`LensCalc` computes both by locating the zero-contours of the eigenvalue fields on the grid:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tangential_critical_curve_list = lens_calc.tangential_critical_curve_list_from(grid=grid)\n", + "radial_critical_curve_list = lens_calc.radial_critical_curve_list_from(grid=grid)\n", + "\n", + "tangential_caustic_list = lens_calc.tangential_caustic_list_from(grid=grid)\n", + "radial_caustic_list = lens_calc.radial_caustic_list_from(grid=grid)\n", + "\n", + "print(f\"Number of tangential critical curves: {len(tangential_critical_curve_list)}\")\n", + "print(f\"Number of radial critical curves: {len(radial_critical_curve_list)}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lets plot the critical curves (image-plane) and caustics (source-plane) of our SIE side by side. Note how the\n", + "elliptical lens produces a tangential caustic with four cusps \u2014 the origin of the four-image \"quad\" configurations\n", + "seen in many real lenses." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "plt.figure(figsize=(10, 5))\n", + "\n", + "plt.subplot(1, 2, 1)\n", + "for curve in tangential_critical_curve_list:\n", + " curve = np.asarray(curve)\n", + " plt.plot(curve[:, 1], curve[:, 0], color=\"black\", label=\"Tangential\")\n", + "for curve in radial_critical_curve_list:\n", + " curve = np.asarray(curve)\n", + " plt.plot(curve[:, 1], curve[:, 0], color=\"orange\", label=\"Radial\")\n", + "plt.gca().set_aspect(\"equal\")\n", + "plt.title(\"Critical Curves (Image-Plane)\")\n", + "plt.xlabel(\"x (arcsec)\")\n", + "plt.ylabel(\"y (arcsec)\")\n", + "plt.legend()\n", + "\n", + "plt.subplot(1, 2, 2)\n", + "for curve in tangential_caustic_list:\n", + " curve = np.asarray(curve)\n", + " plt.plot(curve[:, 1], curve[:, 0], color=\"black\", label=\"Tangential\")\n", + "for curve in radial_caustic_list:\n", + " curve = np.asarray(curve)\n", + " plt.plot(curve[:, 1], curve[:, 0], color=\"orange\", label=\"Radial\")\n", + "plt.gca().set_aspect(\"equal\")\n", + "plt.title(\"Caustics (Source-Plane)\")\n", + "plt.xlabel(\"x (arcsec)\")\n", + "plt.ylabel(\"y (arcsec)\")\n", + "plt.legend()\n", + "\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Einstein Radius__\n", + "\n", + "The **Einstein radius** theta_E is the characteristic angular scale of a strong lens. For a circular lens with the\n", + "source perfectly aligned behind it, it is the radius of the Einstein ring the source forms. More generally (and\n", + "this is the definition **PyAutoLens** uses), it is the radius of the circle enclosing the same area as the\n", + "tangential critical curve:\n", + "\n", + " theta_E = sqrt(A_crit / pi)\n", + "\n", + "sometimes called the \"effective\" Einstein radius, since an elliptical lens's critical curve is not a circle.\n", + "\n", + "Two closed-form results are worth memorising. For a **point mass** M:\n", + "\n", + " theta_E = sqrt( (4 G M / c^2) * (D_ls / (D_l * D_s)) )\n", + "\n", + "and for a **singular isothermal sphere** (SIS) with velocity dispersion sigma_v:\n", + "\n", + " theta_E = 4 pi * (sigma_v / c)^2 * (D_ls / D_s)\n", + "\n", + "The Einstein radius is also the radius within which the *mean* convergence equals exactly one \u2014 so measuring\n", + "theta_E directly measures the projected mass enclosed within it:\n", + "\n", + " M(< theta_E) = pi * theta_E^2 * Sigma_cr\n", + "\n", + "This is why the Einstein radius is celebrated as one of the most robust mass measurements in all of astrophysics:\n", + "the data pin down theta_E via the image separations almost independently of the assumed mass profile, so the\n", + "enclosed Einstein mass is trusted even when the profile's slope is not. (The main caveat, the mass-sheet\n", + "degeneracy, is discussed in the lens modeling chapters.)\n", + "\n", + "`LensCalc` computes the Einstein radius from the area of the tangential critical curve, and the enclosed\n", + "\"Einstein mass\" in angular units (pi * theta_E^2), which Sigma_cr converts to solar masses:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "einstein_radius = lens_calc.einstein_radius_from(grid=grid)\n", + "\n", + "print(f\"Einstein radius (arcsec) = {einstein_radius:.4f}\")\n", + "print(f\"Einstein radius (kpc) = {einstein_radius * kpc_per_arcsec_lens:.4f}\")\n", + "\n", + "einstein_mass_angular = lens_calc.einstein_mass_angular_from(grid=grid)\n", + "einstein_mass_solar = einstein_mass_angular * sigma_cr_arcsec\n", + "\n", + "print(f\"Einstein mass (angular, arcsec^2) = {einstein_mass_angular:.4f}\")\n", + "print(f\"Einstein mass (solar masses) = {einstein_mass_solar:.4e}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Reassuringly, the Einstein radius computed from the critical curve area (~1.6\") matches the `einstein_radius=1.6`\n", + "parameter we gave the `Isothermal` profile \u2014 for isothermal profiles the model parameter *is* the effective\n", + "Einstein radius, which is exactly why **PyAutoLens** parameterizes its mass profiles this way: the non-linear\n", + "search then varies the quantity the data constrain most directly.\n", + "\n", + "As a second check, an `IsothermalSph` (the SIS) has a perfectly circular critical curve, so its recovered Einstein\n", + "radius equals its input parameter even more precisely:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "sis = al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=1.6)\n", + "\n", + "sis_einstein_radius = al.LensCalc.from_mass_obj(mass_obj=sis).einstein_radius_from(\n", + " grid=grid\n", + ")\n", + "\n", + "print(f\"SIS input Einstein radius = 1.6, recovered = {sis_einstein_radius:.4f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Time Delays__\n", + "\n", + "The final piece of the formalism is time. Light rays forming different images of the same source travel different\n", + "paths and through different depths of the lens's gravitational potential, so they arrive at different times. Both\n", + "effects are captured by the **Fermat potential** (or time-delay surface):\n", + "\n", + " phi(theta) = (1/2) * |theta - beta|^2 - psi(theta)\n", + "\n", + "The first term is the **geometric delay** \u2014 the extra path length of a bent ray. The second is the **gravitational\n", + "(Shapiro) delay** \u2014 light slowing as it climbs through the lens's potential. Fermat's principle states that images\n", + "form at the stationary points (minima, maxima and saddle points) of this surface, which is a wonderfully compact\n", + "restatement of the lens equation: grad phi = 0 is exactly beta = theta - alpha(theta).\n", + "\n", + "The observable **time delay** between two images A and B is the difference in their Fermat potentials, scaled by\n", + "the cosmological distances:\n", + "\n", + " Delta t_AB = (1 + z_l) / c * (D_l * D_s / D_ls) * [ phi(theta_A) - phi(theta_B) ]\n", + "\n", + "The distance combination (1 + z_l) * D_l * D_s / D_ls is called the **time-delay distance**, and because it is\n", + "inversely proportional to the Hubble constant, measuring time delays between the images of a variable source (a\n", + "quasar or supernova) turns a strong lens into a cosmological probe.\n", + "\n", + "We only touch on this here \u2014 time delays belong to the modeling of lensed point sources, introduced in tutorial 4\n", + "and covered in depth by the `autolens_workspace` point-source material. For now, we simply plot the Fermat\n", + "potential of our lens system:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "fermat_potential = lens_calc.fermat_potential_from(grid=grid)\n", + "\n", + "aplt.plot_array(array=fermat_potential, title=\"Fermat Potential (phi)\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Wrap Up__\n", + "\n", + "This was the mathematics lecture of **HowToLens** \u2014 every formal definition deferred by the other tutorials, in one\n", + "place. Lets summarise the chain of quantities, because it has a beautiful logical structure where everything flows\n", + "from the mass distribution and a handful of distances:\n", + "\n", + "- **Distances**: the angular diameter distances D_l, D_s and D_ls set the geometry, convert arcseconds to\n", + " kiloparsecs and define the critical surface density Sigma_cr.\n", + "\n", + "- **Convergence**: kappa = Sigma / Sigma_cr is the dimensionless projected mass; kappa >= 1 marks super-critical\n", + " regions capable of multiple imaging.\n", + "\n", + "- **Deflections and potential**: kappa determines the deflection field alpha (a 2D gravity integral) and the\n", + " lensing potential psi, tied together by alpha = grad psi and kappa = (1/2) laplacian psi.\n", + "\n", + "- **The lens equation**: beta = theta - alpha(theta) maps the image-plane to the source-plane; its\n", + " non-invertibility is why multiple images form.\n", + "\n", + "- **Distortion**: the Jacobian A = I - H decomposes into convergence (isotropic) and shear gamma (anisotropic);\n", + " magnification is mu = 1 / det(A) = 1 / (lambda_t * lambda_r).\n", + "\n", + "- **Critical curves and caustics**: where det(A) = 0, magnification diverges; caustics are their source-plane\n", + " images and set the image multiplicity.\n", + "\n", + "- **Einstein radius**: the area-equivalent radius of the tangential critical curve, whose enclosed mass\n", + " M(< theta_E) = pi * theta_E^2 * Sigma_cr is the most robust measurement strong lensing delivers.\n", + "\n", + "- **Time delays**: the Fermat potential phi = (1/2)|theta - beta|^2 - psi locates images at its stationary points\n", + " and its differences, scaled by the time-delay distance, give observable delays.\n", + "\n", + "You do not need to memorise any of this to continue \u2014 **PyAutoLens** computes every one of these quantities via the\n", + "methods used above \u2014 but you now know what each method is computing and can return to this tutorial whenever a\n", + "symbol needs unpacking.\n", + "\n", + "In the next tutorial, we turn from theory to observation: how telescope optics, exposure times and noise turn the\n", + "pristine images of a tracer into the CCD imaging data we actually observe, and how to simulate such data ourselves.\n", + "\n", + "Finally, a signpost for much later: chapter 3's tutorial 5 on the Bayesian formalism is this tutorial's twin \u2014 the\n", + "equivalent equations lecture for pixelized source reconstruction, deriving the linear inversion and Bayesian\n", + "evidence that underpin chapter 3 just as this tutorial derived the lensing quantities underpinning chapter 1." ] } ], diff --git a/notebooks/chapter_1_introduction/tutorial_6_data.ipynb b/notebooks/chapter_1_introduction/tutorial_6_data.ipynb index 4710193..e0a945d 100644 --- a/notebooks/chapter_1_introduction/tutorial_6_data.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_6_data.ipynb @@ -18,8 +18,8 @@ "**Exposure Time:** The time the detector collects light, affecting the clarity of the image. Longer exposure times\n", "gather more light, improving the signal-to-noise ratio and creating a clearer image.\n", "\n", - "**Background Sky:** Light from the sky itself, such as distant stars or zodiacal light, adds noise to the image.\n", - "adds additional noise to the image.\n", + "**Background Sky:** Light from the sky itself, such as distant stars or zodiacal light, adds additional noise to\n", + "the image.\n", "\n", "In this tutorial, we'll simulate a strong lens image by applying these real-world effects to the light and mass\n", "profiles and images we created earlier.\n", @@ -30,6 +30,7 @@ "- **Poisson Noise:** We'll add Poisson noise to the image, simulating the randomness in the photon-to-electron conversion process on the CCD.\n", "- **Background Sky:** We'll add a background sky to the image, simulating the light from the sky that adds noise to the image.\n", "- **Simulator:** We'll use the `SimulatorImaging` object to simulate imaging data that includes all these effects.\n", + "- **Other Data Types:** We'll finish with a brief look at interferometer and weak lensing data, two other types of data used to study strong lenses.\n", "\n", "__Contents__\n", "\n", @@ -39,6 +40,8 @@ "- **Background Sky:** The final effect we will consider when simulating imaging data is the background sky.\n", "- **Simulator:** The `SimulatorImaging` object lets us create simulated imaging data while including the effects of.\n", "- **Output:** We will now save these simulated data to `.fits` files, the standard format used by astronomers for.\n", + "- **Interferometer Data:** Radio / sub-mm interferometers like ALMA observe visibilities in the uv-plane rather than images.\n", + "- **Weak Lensing Data:** Weak lensing datasets are catalogues of weakly sheared background galaxy ellipticities.\n", "- **Wrap Up:** Summary of the script and next steps." ] }, @@ -87,7 +90,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "from pathlib import Path\n", @@ -116,7 +119,7 @@ " shape_native=(\n", " 101,\n", " 101,\n", - " ), # The dimensions of the grid, which here is 100 x 100 pixels.\n", + " ), # The dimensions of the grid, which here is 101 x 101 pixels.\n", " pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds.\n", ")" ], @@ -240,7 +243,7 @@ "cell_type": "code", "metadata": {}, "source": [ - "aplt.plot_array(array=psf.kernel, title=\"PSF 2D Kernel\")" + "aplt.plot_array(array=psf.kernel, title=\"PSF 2D Kernel (Log10)\", use_log10=True)" ], "outputs": [], "execution_count": null @@ -296,7 +299,7 @@ "source": [ "aplt.plot_array(array=image, title=\"Tracer Image Before PSF\")\n", "\n", - "aplt.plot_array(array=blurred_image, title=\"\")\n" + "aplt.plot_array(array=blurred_image, title=\"Tracer Image After PSF\")\n" ], "outputs": [], "execution_count": null @@ -328,8 +331,8 @@ "\n", "We also need to consider the units of our image data. Let\u2019s assume that the tracer image is measured in units of \n", "electrons per second ($e^- s^{-1}$), which is standard for CCD imaging data. To simulate the number of electrons \n", - "actually detected in each pixel, we multiply the image by the observation\u2019s exposure time. This conversion changes t\n", - "he units to the total number of electrons collected per pixel over the entire exposure time.\n", + "actually detected in each pixel, we multiply the image by the observation\u2019s exposure time. This conversion changes\n", + "the units to the total number of electrons collected per pixel over the entire exposure time.\n", "\n", "Once the image is converted, we add Poisson noise, simulating the randomness in the photon-to-electron conversion \n", "process. After adding the noise, we convert the image back to units of electrons per second for analysis, as \n", @@ -407,7 +410,7 @@ "\n", "In addition to light from the strong lens, the telescope also picks up light from the sky. This background sky light is \n", "primarily due to two sources: zodiacal light, which is light scattered by interplanetary dust in the solar system, \n", - "and the unresolved emission from distant stars and tracer.\n", + "and the unresolved emission from distant stars and galaxies.\n", "\n", "For our simulation, we'll assume that the background sky has a uniform brightness across the image, measured at \n", "0.1 electrons per second per pixel. The background sky is added to the image before applying the PSF convolution \n", @@ -605,9 +608,8 @@ "We will now save these simulated data to `.fits` files, the standard format used by astronomers for storing images.\n", "Most imaging data from telescopes like the Hubble Space Telescope (HST) are stored in this format.\n", "\n", - "The `dataset_path` specifies where the data will be saved, in this case, in the directory \n", - "`autolens_workspace/dataset/imaging/howtolens/`, which contains many example images distributed with \n", - "the `autolens_workspace`.\n", + "The `dataset_path` specifies where the data will be saved, in this case the `dataset/imaging/howtolens/`\n", + "directory of the HowToLens repository (paths are relative to the repository root, from which scripts are run).\n", "\n", "The files are named `data.fits`, `noise_map.fits`, and `psf.fits`, and will be used in the next tutorial." ] @@ -630,6 +632,115 @@ "outputs": [], "execution_count": null }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Interferometer Data__\n", + "\n", + "CCD imaging is not the only type of data used to study strong lenses. Radio and sub-mm interferometers, like the\n", + "Atacama Large Millimeter Array (ALMA) and the Jansky Very Large Array (JVLA), observe strong lenses at wavelengths\n", + "where a CCD cannot.\n", + "\n", + "An interferometer does not observe an image of the lens. Each pair of antennas in the array measures a \"visibility\",\n", + "a Fourier component of the sky brightness, at a point in what is called the \"uv-plane\" set by the separation of the\n", + "two antennas. The dataset is therefore a set of complex visibilities in Fourier space, not a 2D image, and its\n", + "noise properties are very different from those of CCD data \u2014 there is no PSF convolution, Poisson noise or\n", + "background sky; instead each visibility has Gaussian noise.\n", + "\n", + "One could Fourier transform the visibilities into an image (called a \"dirty image\") and fit that, but the transform\n", + "correlates the noise between pixels, making the fit statistically incorrect. **PyAutoLens** therefore fits lens\n", + "models directly in visibility space: the tracer's image is evaluated in real space on a grid (defined by a\n", + "real-space mask), Fourier transformed to the uv-plane and compared with the observed visibilities there.\n", + "\n", + "Below, we load a simulated interferometer dataset (creating it first via the `scripts/simulator/interferometer.py`\n", + "script if it does not exist on your hard-disk, using the same auto-simulation idiom as later tutorials) and plot\n", + "its dirty images \u2014 the closest an interferometer dataset comes to the CCD images we simulated above.\n", + "\n", + "HowToLens will not cover interferometry any further than this. The lecture series teaches lensing using CCD\n", + "imaging, and everything you learn transfers to visibility-space fitting. If you need to model interferometer data,\n", + "go to the `autolens_workspace/scripts/interferometer` package, which is the dedicated resource for uv-plane lens\n", + "modeling." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_path = Path(\"dataset\") / \"interferometer\" / \"simple\"\n", + "\n", + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/interferometer.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "real_space_mask = al.Mask2D.circular(\n", + " shape_native=(100, 100),\n", + " pixel_scales=0.1,\n", + " radius=3.0,\n", + ")\n", + "\n", + "dataset = al.Interferometer.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " uv_wavelengths_path=dataset_path / \"uv_wavelengths.fits\",\n", + " real_space_mask=real_space_mask,\n", + " transformer_class=al.TransformerDFT,\n", + ")\n", + "\n", + "aplt.subplot_interferometer_dirty_images(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Weak Lensing Data__\n", + "\n", + "There is one more type of lensing data to glimpse before we move on: weak lensing.\n", + "\n", + "A weak lensing dataset looks nothing like the images above. Instead of the spectacular arcs and rings of strong\n", + "lensing, it is a *catalogue* of many background galaxies, each with a measured ellipticity \u2014 a shear \u2014 at its\n", + "(y, x) position on the sky. Each galaxy is only weakly sheared by the foreground mass, a distortion far too small\n", + "to see in any single galaxy, and the plot below shows this: a field of short line segments tracing the subtle,\n", + "coherent stretching of the background galaxy population, rather than any arc-like feature.\n", + "\n", + "We load a simulated weak lensing shear catalogue (again auto-simulating it, via `scripts/simulator/weak_lensing.py`,\n", + "if it is not on your hard-disk) and plot it, just so you have seen what this data looks like.\n", + "\n", + "That is deliberately all we will say for now \u2014 this glimpse is foreshadowing. Weak lensing gets a full treatment in\n", + "the final tutorial of chapter 4, which describes what these shear measurements are, why they are made far from the\n", + "lens centre and how they are fitted; until then, we defer all further description to that tutorial." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_path = Path(\"dataset\") / \"weak_lensing\" / \"simple\"\n", + "\n", + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/weak_lensing.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.from_json(file_path=dataset_path / \"dataset.json\")\n", + "\n", + "aplt.subplot_weak_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, { "cell_type": "markdown", "metadata": {}, @@ -641,7 +752,7 @@ "\n", "Let's summarise what we've covered:\n", "\n", - "- **Optics Blurring**: The optics of a telescope blur the light from tracer, reducing the clarity and sharpness of \n", + "- **Optics Blurring**: The optics of a telescope blur the light of the tracer, reducing the clarity and sharpness of\n", "the images.\n", "\n", "- **Poisson Noise**: The process of converting photons to electrons on a CCD introduces Poisson noise, which is random \n", diff --git a/notebooks/chapter_1_introduction/tutorial_7_fitting.ipynb b/notebooks/chapter_1_introduction/tutorial_7_fitting.ipynb index d0e98d4..2e12b36 100644 --- a/notebooks/chapter_1_introduction/tutorial_7_fitting.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_7_fitting.ipynb @@ -7,7 +7,7 @@ "Tutorial 7: Fitting\n", "===================\n", "\n", - "In previous tutorials, we used light profiles to create simulated images of tracer and visualized how these images\n", + "In previous tutorials, we used light profiles to create simulated images of a tracer and visualized how these images\n", "would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope.\n", "\n", "However, this simulation process is the reverse of what astronomers typically do when analyzing real data. Usually,\n", @@ -38,7 +38,9 @@ "\n", "__Contents__\n", "\n", - "- **Dataset & Mask:** Standard set up of the dataset and mask that is fitted.\n", + "- **Dataset:** Load the imaging dataset that we previously simulated, consisting of the image, noise map, and PSF.\n", + "- **Dataset Auto-Simulation:** Create the dataset by running the tutorial 6 script if it is not on your hard-disk.\n", + "- **Mask:** Apply a mask to the data, excluding regions with low signal-to-noise ratios from the analysis.\n", "- **Masked Grid:** In tutorials 1 and 2, we emphasized that the `Grid2D` object is crucial for evaluating a lens's.\n", "- **Fitting:** Fit the lens model to the dataset and inspect the results.\n", "- **Incorrect Fit:** In the previous section, we successfully created and fitted a lens model to the image data.\n", @@ -89,7 +91,9 @@ "metadata": {}, "source": [ "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "from pathlib import Path\n", @@ -108,11 +112,11 @@ "We begin by loading the imaging dataset that we will use for fitting in this tutorial. This dataset is identical to the \n", "one we simulated in the previous tutorial, representing how a lens would appear if captured by a CCD camera.\n", "\n", - "In the previous tutorial, we saved this dataset as .fits files in the `autolens_workspace/dataset/imaging/howtolens` \n", - "folder. The `.fits` format is commonly used in astronomy for storing image data along with metadata, making it a\n", - "standard for CCD imaging.\n", + "In the previous tutorial, we saved this dataset as .fits files in the `dataset/imaging/howtolens` folder of the\n", + "HowToLens repository. The `.fits` format is commonly used in astronomy for storing image data along with metadata,\n", + "making it a standard for CCD imaging.\n", "\n", - "The `dataset_path` below specifies where these files are located: `autolens_workspace/dataset/imaging/howtolens/`." + "The `dataset_path` below specifies where these files are located: `dataset/imaging/howtolens/`." ] }, { @@ -198,7 +202,7 @@ "source": [ "__Mask__\n", "\n", - "The signal-to-noise map of the image highlights areas where the signal (light from the lens and source tracer) \n", + "The signal-to-noise map of the image highlights areas where the signal (light from the lens and source galaxies)\n", "is detected above the background noise. Values above 3.0 indicate regions where the light is detected with a \n", "signal-to-noise ratio of at least 3, while values below 3.0 are dominated by noise, where the light is not \n", "clearly distinguishable.\n", @@ -744,7 +748,7 @@ "source": [ "Another quantity that contributes to our final assessment of the goodness-of-fit is the `noise_normalization`.\n", "\n", - "The `noise_normalization` is computed as the logarithm of the sum of squared noise values in our data: \n", + "The `noise_normalization` is computed by summing, over every pixel, the logarithm of 2 pi times the squared noise value:\n", "\n", "\\[\n", "\\text{{noise\\_normalization}} = \\sum \\log(2 \\pi \\text{{noise\\_map}}^2)\n", @@ -851,7 +855,7 @@ "These metrics are standard ways to quantify the quality of a model fit. They are applicable not only to 1D data but \n", "also to more complex data structures like 2D images, 3D data cubes, or any other multidimensional datasets.\n", "\n", - "__Incorrect Fit___\n", + "__Incorrect Fit__\n", "\n", "In the previous section, we successfully created and fitted a lens model to the image data, resulting in an \n", "excellent fit. The residual map and chi-squared map showed no significant discrepancies, indicating that the \n", @@ -980,7 +984,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It is now evident that this model provides a terrible fit to the data. The tracer do not resemble a plausible \n", + "It is now evident that this model provides a terrible fit to the data. The tracer does not resemble a plausible\n", "representation of our simulated strong lens dataset, which we already anticipated given that we generated the data ourselves!\n", "\n", "As expected, the log likelihood has dropped dramatically with this poorly fitting model." @@ -1011,8 +1015,8 @@ "\n", "In practice, however, we don't know the \"true\" model. For example, we might have an image of a strong lens observed with \n", "the Hubble Space Telescope, but the values for parameters like its `einstein_radius` and others are \n", - "unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of \n", - "Chapter 2 of *HowToGalaxy*.\n", + "unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of\n", + "Chapter 2 of **HowToLens**.\n", "\n", "To conclude this section, let's perform a basic, hands-on model fit to develop some intuition about how we can find \n", "the best-fit model. We'll start by loading a simple dataset that was simulated without any lens light, using \n", diff --git a/notebooks/chapter_1_introduction/tutorial_8_summary.ipynb b/notebooks/chapter_1_introduction/tutorial_8_summary.ipynb index f777e5e..f8e9322 100644 --- a/notebooks/chapter_1_introduction/tutorial_8_summary.ipynb +++ b/notebooks/chapter_1_introduction/tutorial_8_summary.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 9: Summary\n", + "Tutorial 8: Summary\n", "===================\n", "\n", "In this chapter, we have learnt that:\n", @@ -76,7 +76,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", diff --git a/notebooks/chapter_2_lens_modeling/README.md b/notebooks/chapter_2_lens_modeling/README.md index 8d51818..16f952f 100644 --- a/notebooks/chapter_2_lens_modeling/README.md +++ b/notebooks/chapter_2_lens_modeling/README.md @@ -1,4 +1,5 @@ -In chapter 2, we'll take you through how to model strong lenses using a non-linear search. +In chapter 2, we'll take you through how to model strong lenses using a non-linear search, ending with how +searches are chained together into automated lens modeling pipelines. **Colab** links to every tutorial are included. @@ -19,3 +20,9 @@ In chapter 2, we'll take you through how to model strong lenses using a non-line - [Tutorial 7: Results](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_7_results.ipynb) — Overview of the results available after successfully fitting a lens model. - [Tutorial 8: Need for Speed](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb) — How to fit complex models whilst balancing efficiency and run-time. + +- [Tutorial 9: Search Chaining](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_9_search_chaining.ipynb) — Breaking the lens modeling procedure into a chained sequence of model-fits. + +- [Tutorial 10: Prior Passing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_10_prior_passing.ipynb) — How the results of earlier searches are passed to later searches. + +- [Tutorial 11: SLaM](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_11_slam.ipynb) — Template pipelines for fitting lens models in standardized ways. diff --git a/notebooks/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_10_prior_passing.ipynb similarity index 89% rename from notebooks/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb rename to notebooks/chapter_2_lens_modeling/tutorial_10_prior_passing.ipynb index cafc329..7ac69d5 100644 --- a/notebooks/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_10_prior_passing.ipynb @@ -4,8 +4,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 2: Prior Passing\n", - "=========================\n", + "Tutorial 10: Prior Passing\n", + "==========================\n", "\n", "In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two\n", "non-linear searches. This used an initial search to fit a simple lens model, whose results were used to tune and\n", @@ -17,13 +17,15 @@ "\n", "__Contents__\n", "\n", - "- **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where.\n", + "- **Initial Setup:** Load the same strong lensing data as the previous tutorial.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Model:** Compose the lens model fitted to the data.\n", "- **Search:** Configure the non-linear search used to fit the model.\n", - "- **Prior Passing:** We are now going to use the prior passing API to pass these results, in a way which does not.\n", + "- **Result (Search 1):** The results of search 1, which are used for prior passing.\n", + "- **Prior Passing:** Use the prior passing API to pass the results of search 1 without writing out values manually.\n", "- **Result:** Overview of the results of the model-fit.\n", "- **Wrap Up:** Summary of the script and next steps.\n", - "- **Detailed Explanation Of Prior Passing:** To end, I provide a detailed overview of how prior passing works and illustrate tools that can be.\n", + "- **Detailed Explanation Of Prior Passing:** A detailed overview of how prior passing works and tools to customize it.\n", "- **EXAMPLE:** Lets go through an example using a real parameter." ] }, @@ -72,7 +74,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "from pathlib import Path\n", @@ -221,11 +223,11 @@ "metadata": {}, "source": [ "search_1 = af.Nautilus(\n", - " path_prefix=Path(\"howtolens\", \"chapter_3\"),\n", - " name=\"tutorial_1_search_chaining_1\",\n", + " path_prefix=Path(\"howtolens\", \"chapter_2\"),\n", + " name=\"tutorial_9_search_chaining_1\",\n", " unique_tag=dataset_name,\n", " n_live=100,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis_1 = al.AnalysisImaging(dataset=dataset)\n", @@ -319,8 +321,8 @@ "\n", "We can use a special prior passing method to do this, called `take_attributes`. This scans the `Exponential`\n", "passed to the `take_attributes` method for all parameters which have the same name as the `Sersic` model,\n", - "and if their names are the same it passes their prior as a `model` (like we did above). Thus, it will locate all 6\n", - "parameters in common between the two profiles (centre_, ell_comps, intensity, effective_radius) and pass those,\n", + "and if their names are the same it passes their prior as a `model` (like we did above). Thus, it will locate the\n", + "parameters in common between the two profiles (`centre`, `ell_comps`, `effective_radius`) and pass those,\n", "leaving the `sersic_index`'s priors as the default values.\n", "\n", "The `take_attributes` method is used in many examples of prior passing, when we pass a simpler parameterization of a\n", @@ -389,17 +391,17 @@ "metadata": {}, "source": [ "search_2 = af.Nautilus(\n", - " path_prefix=Path(\"howtolens\", \"chapter_3\"),\n", - " name=\"tutorial_2_search_chaining_2\",\n", + " path_prefix=Path(\"howtolens\", \"chapter_2\"),\n", + " name=\"tutorial_10_prior_passing_2\",\n", " unique_tag=dataset_name,\n", " n_live=100,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis_2 = al.AnalysisImaging(dataset=dataset)\n", "\n", "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/5_chaining_searches\"\n", + " \"The non-linear search has begun running - checkout the output/howtolens/chapter_2\"\n", " \" folder for live output of the results, images and lens model.\"\n", " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", ")\n", @@ -451,18 +453,19 @@ "source": [ "__Wrap Up__\n", "\n", - "We will expand on the prior passing API in the following tutorials. The main thing to note is that we can pass \n", - "entire profiles or galaxies using prior passing, if their model does not change (which for the bulge, mass and \n", - "source_bulge above, was not true). The API to pass a whole profile or galaxy is as follows:\n", - " \n", + "This tutorial has covered the core of the prior passing API; the chaining examples in the `autolens_workspace`\n", + "(e.g. `autolens_workspace/scripts/guides/modeling/chaining.py`) expand on it further. The main thing to note is that\n", + "we can pass entire profiles or galaxies using prior passing, if their model does not change (which for the bulge,\n", + "mass and source_bulge above, was not true). The API to pass a whole profile or galaxy is as follows:\n", + "\n", " bulge = result_1.model.galaxies.lens.bulge\n", " lens = result_1.model.galaxies.lens\n", " source = result_1.model.galaxies.source\n", - " \n", + "\n", "We can also pass priors using an `instance` instead of a `model`. When an `instance` is used, the maximum likelihood\n", - "parameter values are passed as fixed values that are therefore not fitted for nby the non-linear search (reducing its\n", - "dimensionality). We will use this in the next tutorial to fit the lens light, fix it to the best-fit model in a second\n", - "search, and then go on to fit it as a model in the final search.\n", + "parameter values are passed as fixed values that are therefore not fitted for by the non-linear search (reducing its\n", + "dimensionality). Chained fits often use this to fix the lens light to the best-fit model of an early search, reducing\n", + "the dimensionality of the searches that follow \u2014 the workspace's chaining examples show this in action.\n", " \n", "Lets now think about how priors are passed. Checkout the `model.info` file of the second search of this tutorial. The \n", "parameters do not use the default priors we saw in search 1 (which are typically broad UniformPriors). Instead, \n", @@ -472,7 +475,7 @@ " - The sigma values are specified in the `width_modifier` field of the profile's entry in the `priors.yaml' config \n", " file (we will discuss why this is used in a moment).\n", "\n", - "Like the manual `GaussianPrior`'s that were used in tutorial 1, the prior passing API sets up the prior on each \n", + "Like the manual `GaussianPrior`'s that were used in the previous tutorial, the prior passing API sets up the prior on each\n", "parameter with a `GaussianPrior` centred on the high likelihood regions of parameter space!\n", "\n", "__Detailed Explanation Of Prior Passing__\n", diff --git a/notebooks/chapter_2_lens_modeling/tutorial_11_slam.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_11_slam.ipynb new file mode 100644 index 0000000..2d92fdd --- /dev/null +++ b/notebooks/chapter_2_lens_modeling/tutorial_11_slam.ipynb @@ -0,0 +1,135 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 11: SLaM\n", + "=================\n", + "\n", + "In the previous two tutorials, we learnt how search chaining breaks a lens model-fit into a sequence of simpler\n", + "non-linear searches, and how prior passing carries the results of each search into the next. Together, they give us\n", + "the flexibility to juggle the dimensionality, priors and settings of each search \u2014 the three drivers of run-time we\n", + "met in tutorial 8 \u2014 whilst still fitting a complex and realistic lens model at the end.\n", + "\n", + "You could now write your own chained sequences of searches, tailored to whatever lens model is of particular interest\n", + "to your scientific study. However, for most lens models there are standardized approaches one can take to fitting\n", + "them. For example, an effective approach is to first fit a model for the lens's light, then a model for its mass and\n", + "the source. It would be wasteful for all **PyAutoLens** users to have to write their own chained pipelines to perform\n", + "the same tasks.\n", + "\n", + "For this reason, the `autolens_workspace` comes with standardized chained pipelines, which fit common lens models in\n", + "ways we have tested are efficient and robust. This closing tutorial does not run any code; it is a pointer to where\n", + "that material lives, so you know where to go once you want to scale your lens modeling up.\n", + "\n", + "__Contents__\n", + "\n", + "- **Search Chaining In The Workspace:** Where the workspace documents the chaining API introduced in tutorials 9 and 10.\n", + "- **SLaM (Source, Light and Mass):** The standardized pipelines that chain searches for you.\n", + "- **Wrap Up:** Choosing between single searches, your own chained fits and the SLaM pipelines.\n", + "\n", + "__Search Chaining In The Workspace__\n", + "\n", + "The chaining and prior passing API we introduced in tutorials 9 and 10 is documented in full in the workspace guide\n", + "`autolens_workspace/scripts/guides/modeling/chaining.py`. It covers the concise model composition API used in chained\n", + "fits, prior passing and the tools for customizing it, and how chaining is used in practice for automated lens\n", + "modeling.\n", + "\n", + "__SLaM (Source, Light and Mass)__\n", + "\n", + "The workspace's standardized pipelines are called the **SLaM** (Source, Light and Mass) pipelines. They are composed\n", + "of a specific sequence of chained searches:\n", + "\n", + " - `Source`: A pipeline that focuses on producing a robust model for the source's light, using simpler models for the\n", + " lens's light and mass.\n", + "\n", + " - `Light`: A pipeline that fits a complex lens light model (e.g. one with many components), using the initialized\n", + " source model to cleanly deblend the lens and source light.\n", + "\n", + " - `Mass`: A pipeline that fits a complex lens mass model, benefitting from the good models for the lens's light and\n", + " source.\n", + "\n", + "For fitting very complex lens models, for example ones which decompose the mass into its stellar and dark components,\n", + "the **SLaM** pipelines have been carefully crafted to do this in a reliable and automated way that is still efficient.\n", + "They also make fitting many different models to a single dataset efficient, as they reuse the results of earlier\n", + "searches (e.g. in the Source pipeline) to fit different models in the `Light` and `Mass` pipelines for the lens's\n", + "light and mass.\n", + "\n", + "The canonical introduction is `autolens_workspace/scripts/guides/modeling/slam_start_here.py`, which walks through the\n", + "pipeline structure end-to-end. Runnable SLaM examples then live as `slam.py` scripts inside the feature folders of\n", + "each topic, for example `autolens_workspace/scripts/imaging/features/pixelization/slam.py` and\n", + "`autolens_workspace/scripts/imaging/features/multi_gaussian_expansion/slam.py`.\n", + "\n", + "__Wrap Up__\n", + "\n", + "Whether you should use individual searches, your own chained fits, the SLaM pipelines or write your own model-fitting\n", + "script depends on the scope of your scientific analysis. I would advise you begin by trying to adapt the scripts in\n", + "the `autolens_workspace` to fit your data, and also try using the SLaM pipelines once you are a confident\n", + "**PyAutoLens** user.\n", + "\n", + "This tutorial concludes the lens modeling chapter. In the next chapter, we introduce pixelizations, which reconstruct\n", + "the source galaxy on a pixel-grid rather than with light profiles." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_2_lens_modeling/tutorial_1_non_linear_search.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_1_non_linear_search.ipynb index c4c2fb4..d4bc315 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_1_non_linear_search.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_1_non_linear_search.ipynb @@ -9,7 +9,7 @@ "\n", "The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a strong lens\n", "using a telescope like the Hubble Space Telescope, and seek to learn about the lens galaxy, source galaxy and the\n", - "Universe from these observations. With **PyAutoLen**, we seek to learn about the lenses may and ray-tracing, asking questions like\n", + "Universe from these observations. With **PyAutoLens**, we seek to learn about the lens's mass and ray-tracing, asking questions like\n", "how big is the lens galaxy and what does the unlensed source galaxy look like?\n", "\n", "To answer these questions, we must therefore fit the dataset with a lens model, where the lens model defines the\n", @@ -32,7 +32,7 @@ "and begin learning about real galaxies in the Universe.\n", "\n", "This first tutorial introduces a number of key statistical concepts that are fundamental to understanding how\n", - "model-fitting works, both for **PyAutoLen** and in general.\n", + "model-fitting works, both for **PyAutoLens** and in general.\n", "\n", "__Overview__\n", "\n", @@ -45,22 +45,27 @@ " model instance to the data.\n", "\n", "- Fit datasets with different non-linear searches, including a maximum likelihood estimator (MLE),\n", - " Markok Chain Monte Carlo (MCMC) and nested sampling.\n", + " Markov Chain Monte Carlo (MCMC) and nested sampling.\n", "\n", "__Contents__\n", "\n", "- **Overview:** In this tutorial, we will use a non-linear search to fit a lens model to simulated imaging of.\n", "- **Parameter Space:** In mathematics, a function is defined by its parameters, which map inputs to outputs.\n", + "- **Non-Linear Search:** The method used to fit the model to the data, by sampling the non-linear parameter space.\n", "- **Search Types:** There are different types of non-linear searches, each of which explores parameter space in a.\n", - "- **Deeper Background:** **The descriptions of how searches work in this example are simplfied and phoenomenological and do.\n", + "- **Deeper Background:** **The descriptions of how searches work in this example are simplified and phenomenological and do.\n", "- **PyAutoFit:** Modeling uses the probabilistic programming language.\n", "- **Initial Setup:** Let's first load the `Imaging` dataset, which we will use to fit a model with a non-linear search.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", "- **Model:** Compose the lens model fitted to the data.\n", "- **Priors:** When we examine the `.info` of our model, we notice that each parameter (like `centre`.\n", "- **Analysis:** Create the Analysis object that defines how the model is fitted to the data.\n", "- **Searches:** To perform a non-linear search, we create an instance of a `NonLinearSearch` object.\n", + "- **Maximum Likelihood Estimation (MLE):** The simplest search type, which moves \"up\" the likelihood gradient.\n", + "- **Markov Chain Monte Carlo (MCMC):** Walkers explore parameter space, mapping out the likelihood surface.\n", "- **Nested Sampling:** **Nested Sampling** is an advanced method for model-fitting that excels in handling complex models.\n", + "- **What is The Best Search To Use?:** Choosing between MLE, MCMC and nested sampling for lens modeling.\n", "- **Wrap Up:** Summary of the script and next steps.\n", "\n", "__Parameter Space__\n", @@ -83,8 +88,8 @@ "This defines a parameter space in three dimensions, representing the relationships between \\(x\\), \\(y\\), \\(z\\),\n", "and the output \\(f(x, y, z)\\).\n", "\n", - "This concept of parameter space is closely related to how we approach model-fitting. For instance, in chapter 1, w\n", - "e created instances of `Galaxy` objects with\n", + "This concept of parameter space is closely related to how we approach model-fitting. For instance, in chapter 1,\n", + "we created instances of `Galaxy` objects with\n", "parameters like \\( (\\text{`centre_0`}, \\text{`centre_1`}, \\text{`ell_comps_0`}, \\text{`ell_comps_1`}, \\text{`intensity`}, \\text{`effective_radius`}, \\text{`sersic_index`}) \\).\n", "These parameters were used to fit data and compute a log likelihood.\n", "\n", @@ -149,7 +154,7 @@ "\n", "__Deeper Background__\n", "\n", - "**The descriptions of how searches work in this example are simplfied and phoenomenological and do not give a full\n", + "**The descriptions of how searches work in this example are simplified and phenomenological and do not give a full\n", "description of how they work at a deep statistical level. The goal is to provide you with an intuition for how to use\n", "them and when different searches are appropriate for different problems. Later tutorials will provide a more formal\n", "description of how these searches work.**\n", @@ -207,7 +212,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -226,11 +231,11 @@ "[PyAutoFit](https://github.com/PyAutoLabs/PyAutoFit), an open-source project that allows complex model\n", "fitting techniques to be straightforwardly integrated into scientific modeling software. \n", "\n", - "**PyAutoFit** is actually a spin-off project of **PyAutoLen**. whereby we found that the statistic techniques and\n", + "**PyAutoFit** is actually a spin-off project of **PyAutoLens**, whereby we found that the statistical techniques and\n", "methods we applied to model galaxies could be used in a more general setting to many different scientific \n", "problems. Check it out if you are interested in developing your own software to perform advanced model-fitting!\n", "\n", - "We import this library separately from **PyAutoLen**." + "We import this library separately from **PyAutoLens**." ] }, { @@ -250,12 +255,12 @@ "\n", "Let's first load the `Imaging` dataset, which we will use to fit a model with a non-linear search.\n", "\n", - "The strong lens in this image was generated using an `Isothermal` lens mass profile and `SersicCore` source light profile, \n", + "The strong lens in this image was generated using an `IsothermalSph` lens mass profile and `ExponentialCoreSph` source light profile,\n", "which we'll also use in our model fitting in this tutorial. This means the model we are going to fit is identical to \n", "the one used to simulate the data, allowing us to assess the fitting process under controlled conditions.\n", "\n", - "The dataset, as well as all subsequent datasets used in future tutorials, is stored in \n", - "the `autolens_workspace/dataset/imaging` folder. " + "The dataset, as well as all subsequent datasets used in future tutorials, is stored in the HowToLens\n", + "repository's `dataset/imaging` folder (simulated at runtime by the simulator scripts, as shown below)." ] }, { @@ -442,10 +447,10 @@ "been chosen to be broad, and contain the breadth of plausible solutions one should expect when fitting light and mass\n", "profiles of a real galaxy.\n", "\n", - "For instance, consider the `centre` parameter of our `IsothermalSph` light profile. In theory, it could take on any value from \n", - "negative to positive infinity. However, imaging datasets are typically reduced such that the galaxy centre is close \n", - "to (0.0\", 0.0\"). Therefore, a `TruncatedGaussianPrior` with `mean=0.0` and `sigma=0.1` is a good description of where the\n", - "galaxy `centre` is. \n", + "For instance, consider the `centre` parameter of our `IsothermalSph` mass profile. In theory, it could take on any value from\n", + "negative to positive infinity. However, imaging datasets are typically reduced such that the galaxy centre is close\n", + "to (0.0\", 0.0\"). Therefore, a `GaussianPrior` with `mean=0.0` and `sigma=0.1` is a good description of where the\n", + "galaxy `centre` is.\n", "\n", "If the galaxy had a different centre in the dataset, we would change the mean of the prior to reflect this.\n", "However, in general, we advise all galaxy images are reduced such that the galaxy is at (0.0\", 0.0\").\n", @@ -764,7 +769,7 @@ "one, other walkers will explore other regions of parameter space and encourage the walker in local maxima to move \n", "away from it and head towards higher likelihood regions.\n", "\n", - "In the example below, we use the `Emcee` MCMC search to fit the strong lens The search starts with walkers \n", + "In the example below, we use the `Emcee` MCMC search to fit the strong lens. The search starts with walkers\n", "initialized in a \"ball\" around the center of the model\u2019s priors, similar to the MLE search that failed earlier." ] }, @@ -807,7 +812,7 @@ "\n", "A major advantage of MCMC is that it provides estimates of parameter uncertainties by \"mapping out\" the likelihood \n", "surface, unlike MLE, which only finds the maximum likelihood solution. These error estimates are accessible in \n", - "the `result.info` string and through the `result.samples` object, which is explained fully in tutorial 5.\n", + "the `result.info` string and through the `result.samples` object, which is explained fully in tutorial 7.\n", "\n", "While a good starting point wasn't necessary for this simple model, it becomes essential for efficiently mapping the \n", "likelihood surface in more complex models with many parameters. The code below shows an MCMC fit using a good starting \n", @@ -902,7 +907,7 @@ "search = af.Nautilus(\n", " n_live=100,\n", " n_batch=50, # Explained in next tutorial\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")" ], "outputs": [], @@ -979,12 +984,12 @@ "When approaching a model-fitting problem, it's usually advisable to try various search methods to find the most \n", "effective one. \n", "\n", - "However, after extensive testing within **PyAutoLen**, a clear recommendation has emerged for fitting strong lens models. \n", + "However, after extensive testing within **PyAutoLens**, a clear recommendation has emerged for fitting strong lens models.\n", "The nested sampling method `Nautilus` consistently proves to be the most effective. It requires fewer iterations than \n", "MCMC or MLE methods (when no starting point is used), provides robust sampling even for complex models, includes a \n", "built-in stopping criterion, and delivers reliable error estimates.\n", "\n", - "All examples in the `autolens_workspace` use the `Nautilus` search, and future tutorials in the **HowToGalaxy** \n", + "All examples in the `autolens_workspace` use the `Nautilus` search, and future tutorials in the **HowToLens**\n", "series will also use it. We strongly recommend using `Nautilus` from now on.\n", "\n", "That said, MLE and MCMC searches can still be effective, and you're encouraged to experiment with them. If you have \n", diff --git a/notebooks/chapter_2_lens_modeling/tutorial_2_practicalities.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_2_practicalities.ipynb index 8952443..c889a7e 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_2_practicalities.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_2_practicalities.ipynb @@ -24,6 +24,7 @@ "\n", "- **PyAutoFit:** Modeling uses the probabilistic programming language.\n", "- **Initial Setup:** Lets first load the `Imaging` dataset we'll fit a model with using a non-linear search.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", "- **Model:** Compose the lens model fitted to the data.\n", "- **Search:** Configure the non-linear search used to fit the model.\n", @@ -32,6 +33,7 @@ "- **Analysis:** Create the Analysis object that defines how the model is fitted to the data.\n", "- **VRAM Use:** When running AutoLens with JAX on a GPU, the analysis must fit within the GPU\u2019s available VRAM.\n", "- **Run Times:** Profiling the expected run time of the model-fit.\n", + "- **Model-Fit:** Begin the model-fit by passing the model and analysis to the search.\n", "- **Result Info:** A concise readable summary of the results is given by printing its `info` attribute.\n", "- **Output Folder:** Now checkout the `autolens_workspace/output` folder.\n", "- **Unique Identifier:** In the output folder, you will note that results are in a folder which is a collection of random.\n", @@ -52,7 +54,7 @@ "\n", "The ideal number of live points depends on model complexity. More parameters generally require more live points, but\n", "the default of 200 is sufficient for most lens models. Lower values can still yield reliable results, particularly\n", - "for simpler models. For this example (7 parameters), we reduce the live points to 100 to speed up runtime without\n", + "for simpler models. For this example (6 parameters), we reduce the live points to 100 to speed up runtime without\n", "compromising accuracy.\n", "\n", "Tuning non-linear search settings (e.g., the number of live points) to match model complexity is essential. We aim\n", @@ -60,8 +62,8 @@ "is excessive.\n", "\n", "In practice, the optimal number of live points is often found through trial and error, guided by summary statistics\n", - "on how well the search is performing, which we\u2019ll cover below. For this single Sersic model with a linear light\n", - "profile, 80 live points is sufficient to achieve reliable results.\n", + "on how well the search is performing, which we\u2019ll cover below. For the simple model fitted here, with a linear light\n", + "profile, 100 live points is sufficient to achieve reliable results.\n", "\n", "__Iterations Per Update__\n", "\n", @@ -71,8 +73,8 @@ "Depending on how long it takes for the model to be fitted to the data (see discussion about run times below),\n", "this can take up a large fraction of the run-time of the non-linear search.\n", "\n", - "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates\n", - "so not slow down the overall speed of the model-fit.\n", + "For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=2500` to ensure these updates\n", + "do not slow down the overall speed of the model-fit.\n", "\n", "**If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to\n", "output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it\n", @@ -124,7 +126,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -273,15 +275,31 @@ "\n", "model = af.Collection(galaxies=af.Collection(lens=lens, source=source))\n", "\n", - "print(model.info)\n", + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Search__\n", "\n", + "We again use the `Nautilus` non-linear search, with settings chosen following the Search Settings discussion above." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ "search = af.Nautilus(\n", " path_prefix=Path(\"howtolens\") / \"chapter_2\",\n", " name=\"tutorial_2_practicalities\",\n", " unique_tag=dataset_name,\n", " n_live=100,\n", " n_batch=50, # GPU batching and VRAM use explained in VRAM section below.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")" ], "outputs": [], @@ -331,7 +349,7 @@ "The method below prints the VRAM usage estimate for the analysis and model with the specified batch size,\n", "it takes about 20-30 seconds to run so you may want to comment it out once you are familiar with your GPU's VRAM limits.\n", "\n", - "For a MGE model with the low resolution dataset fitted in this example VRAM use is relatively low (~0.027GB) For other \n", + "For the simple model and low resolution dataset fitted in this example VRAM use is relatively low (~0.027GB). For other\n", "models (e.g. pixelized sources) and higher resolution datasets it can be much higher (> 1GB going beyond 10GB)." ] }, @@ -468,9 +486,9 @@ "\n", "__Output Folder Contents__\n", "\n", - "Now this is running you should checkout the `autolens_workspace/output` folder. This is where the results of the \n", - "search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json,\n", - ".csv or text files).\n", + "Now this is running you should checkout the `output` folder. This is where the results of the\n", + "search are written to hard-disk (in the `tutorial_2_practicalities` folder), where all outputs are human readable\n", + "(e.g. as .json, .csv or text files).\n", "\n", "As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found\n", "by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to\n", @@ -553,7 +571,7 @@ "The following are examples of other practicalities which I will document fully in this example script in the future,\n", "but so far have no found the time:\n", "\n", - "- `config`: The files in `autogalaxy_workspace/config` which control many aspects of how PyAutoGalaxy runs,\n", + "- `config`: The files in the `config` folder which control many aspects of how **PyAutoLens** runs,\n", " including visualization, the non-linear search settings.\n", "\n", "- `config/priors`: Folder containing the default priors on all model components.\n", diff --git a/notebooks/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.ipynb index fd5c482..70d24c2 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.ipynb @@ -30,8 +30,10 @@ "__Contents__\n", "\n", "- **Initial Setup:** we'll use new strong lensing data, where.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", "- **Model:** Compose the lens model fitted to the data.\n", + "- **Search + Analysis:** Create the non-linear search and analysis used to fit the model.\n", "- **Run Time:** Profiling the expected run time of the model-fit.\n", "- **Result:** Overview of the results of the model-fit.\n", "- **Global and Local Maxima:** Up to now, all our non-linear searches have successfully found lens models that provide visibly.\n", @@ -83,7 +85,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -233,7 +235,7 @@ "__Search + Analysis__\n", "\n", "We set up `Nautilus` as we did in the previous tutorial, however given the increase in model complexity we'll use\n", - "a higher `n_live` value of 150 to ensure we sample the complex parameter space efficiently." + "a higher `n_live` value of 200 to ensure we sample the complex parameter space efficiently." ] }, { @@ -246,7 +248,7 @@ " unique_tag=dataset_name,\n", " n_live=200,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis = al.AnalysisImaging(dataset=dataset)" @@ -355,7 +357,7 @@ " unique_tag=dataset_name,\n", " n_live=75,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "print(\n", @@ -458,13 +460,13 @@ "inferring a local maxima is still very real, especially as we make our lens model more complex.\n", "\n", "Lets think about *complexity*. As we make our lens model more realistic, we also made it more complex. For this \n", - "tutorial, our non-linear parameter space went from 7 dimensions to 18. This means there was a much larger *volume* of \n", + "tutorial, our non-linear parameter space went from 6 dimensions to 20. This means there was a much larger *volume* of\n", "parameter space to search. As this volume grows, there becomes a higher chance that our non-linear search gets lost \n", "and infers a local maxima, especially if we don't set it up with enough live points!\n", "\n", "At its core, lens modeling is all about learning how to get a non-linear search to find the global maxima region of \n", - "parameter space, even when the lens model is complex. This will be the main theme throughout the rest of this chapter\n", - "and is the main subject of chapter 3.\n", + "parameter space, even when the lens model is complex. This will be the main theme throughout the rest of this chapter,\n", + "culminating in the search chaining technique introduced in tutorials 9 and 10.\n", "\n", "In the next exercise, we'll learn how to deal with failure and begin thinking about how we can ensure our non-linear \n", "search finds the global-maximum log likelihood solution. First, think about the following:\n", @@ -475,7 +477,7 @@ " 2) The non-linear search failed because parameter space was too complex. Could we make it less complex, whilst \n", " still keeping our lens model fairly realistic?\n", " \n", - " 3) The source galaxy in this example had only 7 non-linear parameters. Real source galaxies may have multiple \n", + " 3) The source galaxy in this example had only 6 non-linear parameters. Real source galaxies may have multiple\n", " components (e.g. a disk, bulge, bar, star-forming knot) and there may even be more than 1 source galaxy! Do you \n", " think there is any hope of us navigating a parameter space if the source contributes 20+ parameters by itself?" ] diff --git a/notebooks/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.ipynb index 69c6694..c94aac4 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.ipynb @@ -26,11 +26,15 @@ "__Contents__\n", "\n", "- **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", "- **Prior Tuning:** First, we will try to assist our non-linear search by tuning our priors.\n", "- **Run Time:** Profiling the expected run time of the model-fit.\n", "- **Result:** Overview of the results of the model-fit.\n", - "- **Discussion:** By tuning our priors to the specific lens model we are fitting, we increase the chances of finding." + "- **Discussion:** By tuning our priors to the specific lens model we are fitting, we increase the chances of finding.\n", + "- **Approach 2: Reducing Complexity:** Simplify the lens model, for example by assuming light-traces-mass.\n", + "- **Approach 3: Look Harder:** Brute force the global maxima with more thorough non-linear search settings.\n", + "- **Wrap Up:** Summary of the script and next steps." ] }, { @@ -78,7 +82,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "from pathlib import Path\n", @@ -441,7 +445,7 @@ " unique_tag=dataset_name,\n", " n_live=150,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis = al.AnalysisImaging(dataset=dataset)" @@ -625,7 +629,7 @@ " unique_tag=dataset_name,\n", " n_live=200,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")" ], "outputs": [], @@ -707,11 +711,8 @@ "number of live points used by `Nautilus`. The more thoroughly the search examines the space, the more likely it is to \n", "find the global maximum lens model. However, this also means the search will take longer!\n", "\n", - "Below, we configure a more thorough Nautilus search with `n_live=200`. These settings and what they change are \n", - "discussed in the optional tutorial `chapter_optional/tutorial_searches.ipynb`.\n", - "\n", - "Due to the long run times of this search, the output is commented out below. Feel free to uncomment it to run the \n", - "script faster." + "Below, we configure a more thorough Nautilus search with `n_live=300`. These settings and what they change are\n", + "discussed in the optional tutorial `chapter_optional/tutorial_searches.ipynb`." ] }, { @@ -735,7 +736,7 @@ " name=\"tutorial_4_look_harder\",\n", " unique_tag=dataset_name,\n", " n_live=300,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")" ], "outputs": [], @@ -796,8 +797,8 @@ "weeks, or even months!\n", "\n", "In conclusion, we can now fit strong lenses effectively. When the process fails, we know how to make it work.\n", - "In chapter 3 of **HowToLens**, we will introduce a technique called 'non-linear search chaining', which performs a model \n", - "fit by chaining together multiple searches back-to-back . This allows us to combine the 3 different approaches \n", + "In tutorials 9 and 10 of this chapter, we will introduce a technique called 'non-linear search chaining', which performs\n", + "a model fit by chaining together multiple searches back-to-back. This allows us to combine the 3 different approaches\n", "discussed and exploit the advantages of each, whilst not being hindered by their disadvantages.\n", "\n", "With search chaining, we can:\n", @@ -814,8 +815,8 @@ " \n", "__Wrap Up__\n", "\n", - "To wrap up chapter 2, we have a few more tutorials, where we will discuss masking in more detail, the `Result` object\n", - "and how to make **PyAutoLens** run faster." + "Before we get to search chaining, we have a few more tutorials, where we will discuss masking in more detail, the\n", + "`Result` object and how to make **PyAutoLens** run faster." ] } ], diff --git a/notebooks/chapter_2_lens_modeling/tutorial_5_linear_profiles.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_5_linear_profiles.ipynb index 272ff41..abc7096 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_5_linear_profiles.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_5_linear_profiles.ipynb @@ -31,6 +31,7 @@ "__Contents__\n", "\n", "- **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", "- **Linear Light Profiles:** We use a variant of a light profile discussed called a \"linear light profile\", which is accessed.\n", "- **Run Time:** Profiling the expected run time of the model-fit.\n", @@ -92,7 +93,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "from pathlib import Path\n", @@ -287,7 +288,7 @@ " unique_tag=dataset_name,\n", " n_live=100,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis = al.AnalysisImaging(dataset=dataset)" @@ -714,7 +715,7 @@ " unique_tag=dataset_name,\n", " n_live=100,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "print(\n", @@ -841,7 +842,7 @@ "lens modeling problems. \n", "\n", "If you have a desire to fit sources with even more complex morphologies we recommend you look at how to reconstruct \n", - "sources using pixelizations in the `modeling/features` section or chapter 4 of **HowToLens**.\n", + "sources using pixelizations in the `modeling/features` section or chapter 3 of **HowToLens**.\n", " \n", "__Wrap Up__\n", "\n", @@ -854,7 +855,7 @@ "galaxies which may have multiple components with multiple centres (e.g. many star forming knots) which such a \n", "basis cannot capture.\n", "\n", - "In chapter 4 of **HowToLens** we introduce non-parametric pixelizations, which reconstruct the data in way\n", + "In chapter 3 of **HowToLens** we introduce non-parametric pixelizations, which reconstruct the data in way\n", "that does not make assumptions like a centre and can thus reconstruct even more complex, asymmetric and irregular\n", "galaxy morphologies." ] diff --git a/notebooks/chapter_2_lens_modeling/tutorial_6_masking_and_positions.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_6_masking_and_positions.ipynb index 5e188ec..dd7d721 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_6_masking_and_positions.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_6_masking_and_positions.ipynb @@ -14,7 +14,9 @@ "__Contents__\n", "\n", "- **Initial Setup:** we'll use the same strong lensing data as tutorials 1 & 2, where.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", + "- **Model + Analysis:** Compose the lens model and create the analysis that fits it using the chosen mask.\n", "- **Run Time:** Profiling the expected run time of the model-fit.\n", "- **Search:** Configure the non-linear search used to fit the model.\n", "- **Discussion:** So, we can choose the mask we use in a model-fit.\n", @@ -67,7 +69,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -86,7 +88,7 @@ "we'll use the same strong lensing data as tutorials 1 & 2, where:\n", "\n", " - The lens galaxy's total mass distribution is a `IsothermalSph`.\n", - " - The source galaxy's light is a `ExponentialSph`." + " - The source galaxy's light is a `ExponentialCoreSph`." ] }, { @@ -228,7 +230,7 @@ " unique_tag=dataset_name,\n", " n_live=80,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis = al.AnalysisImaging(dataset=dataset)" @@ -412,7 +414,7 @@ " unique_tag=dataset_name,\n", " n_live=80,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "print(\n", diff --git a/notebooks/chapter_2_lens_modeling/tutorial_7_results.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_7_results.ipynb index fd3c7dd..bfdd67f 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_7_results.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_7_results.ipynb @@ -13,6 +13,7 @@ "__Contents__\n", "\n", "- **Initial Setup:** Lets use the model-fit performed in tutorial 1 to get a `Result` object.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Tracer & Fit:** In the previous tutorials, we saw that this result contains the maximum log likelihood fit, which.\n", "- **Samples:** The result contains a lot more information about the model-fit.\n", "- **Workspace:** We are not going into any more detail on the result variable in this tutorial, or in the.\n", @@ -65,7 +66,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -147,7 +148,7 @@ " unique_tag=dataset_name,\n", " n_live=80,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis = al.AnalysisImaging(dataset=dataset)\n", diff --git a/notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb index dec2e8c..08a77f8 100644 --- a/notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb @@ -19,53 +19,126 @@ "fit performed in this chapter. Lets quickly remind ourselves of the main factors that drive the run-time, and\n", "how we might reduce it.\n", "\n", + "The overall run-time of a model-fit is, to a good approximation, a simple product of two numbers:\n", + "\n", + " - The number of log likelihood evaluations the non-linear search performs before it converges.\n", + "\n", + " - The time each individual log likelihood evaluation takes.\n", + "\n", + "Everything we discuss below reduces one (or both) of these numbers.\n", + "\n", + "__Contents__\n", + "\n", + "- **Searching Non-linear Parameter Space:** What drives the number of likelihood evaluations a search performs.\n", + "- **Cost Per Evaluation:** What drives the time a single log likelihood evaluation takes.\n", + "- **JAX:** How **PyAutoLens** compiles the likelihood function with JAX and runs it on CPUs and GPUs.\n", + "- **Run Time Estimation:** How to estimate the overall run-time of a fit before committing to it.\n", + "- **Data Quantity:** How the amount of data fitted, and the mask applied to it, drives run-times.\n", + "- **Wrap Up:** Summary of the script and next steps.\n", + "\n", "__Searching Non-linear Parameter Space__\n", "\n", "The time it takes for the non-linear search to sample parameter space and find the high likelihood models is driven by:\n", "\n", " - Dimensionality: A more complex parameter space (e.g. more parameters) takes longer to search.\n", " - Priors: The broader the priors on each parameter the longer the search.\n", - " - Settings: Non-linear search settings which sample parameter space more thoroughly lead to longer run-times.\n", + " - Settings: Non-linear search settings which sample parameter space more thoroughly (e.g. a higher `n_live`) lead to\n", + " longer run-times.\n", "\n", "When we use only one search to fit a lens model, we are somewhat restricted in how we can try to achieve faster run\n", - "times by changing these 3 aspects of the search.\n", + "times by changing these 3 aspects of the search. We have already seen one powerful exception: the linear light\n", + "profiles of tutorial 5 remove every `intensity` parameter from the non-linear parameter space, reducing its\n", + "dimensionality and removing some of its most difficult degeneracies at the same time.\n", "\n", - "In the next chapter, we introduce 'non-linear search chaining', which fits a lens model using multiple searches that\n", - "are performed back-to-back. A key motivation for this is that it gives us a lot more flexibility in juggling the\n", - "dimensionality, priors and settings so as to perform faster and more efficient lens modeling.\n", + "The final three tutorials of this chapter (tutorials 9, 10 and 11) introduce 'non-linear search chaining', which fits\n", + "a lens model using multiple searches that are performed back-to-back, passing the results of earlier searches to\n", + "later ones. A key motivation for this is that it gives us a lot more flexibility in juggling the dimensionality,\n", + "priors and settings so as to perform faster and more efficient lens modeling, culminating in the fully automated\n", + "SLaM pipelines of tutorial 11.\n", "\n", - "In the optional **HowToGalaxy** tutorial `chapter_optional/tutorial_searches.ipynb` we discuss other non-linear\n", - "searches supported by **HowToGalaxy** which use a different approach to sample parameter sample than `Nautilus`. For\n", - "those familiar with statistical inference, this includes maximum likelihood estimators and MCMC algorithms.\n", + "In the optional **PyAutoLens** tutorial `chapter_optional/tutorial_searches.py` we discuss other non-linear searches\n", + "supported by **PyAutoLens** which use a different approach to sample parameter space than `Nautilus`. For those\n", + "familiar with statistical inference, this includes maximum likelihood estimators and MCMC algorithms.\n", "\n", "For lens modeling, we have not found another non-linear search that provides as robust and efficient results as\n", - "Nautilus. We therefore recommend uses stick to `Nautilus`.\n", + "Nautilus. We therefore recommend users stick to `Nautilus`.\n", "\n", - "__Contents__\n", + "__Cost Per Evaluation__\n", "\n", - "- **Algorithmic Optimization:** Every operation **PyAutoLens** performs to fit strong lens data with a model takes time, for.\n", - "- **Data Quantity:** The final factor driving run-speed is the quantity of data that is fitted.\n", - "- **Wrap Up:** Summary of the script and next steps.\n", + "The second number is the time a single log likelihood evaluation takes, which is set by every operation\n", + "**PyAutoLens** performs to fit the strong lens data with a model, for example:\n", "\n", - "__Algorithmic Optimization__\n", + " - Computing the deflection angles of the mass model and ray-tracing the image-plane grid to the source-plane.\n", + " - Computing the intensity values of every light profile on those grids.\n", + " - Over-sampling: sub-dividing image pixels into finer sub-pixels where light profiles change rapidly (e.g. the\n", + " centre of the lens galaxy), which multiplies the number of intensity evaluations performed.\n", + " - Convolving the image that comes from the tracer with the telescope's PSF to compare it to the data. The larger\n", + " the PSF, the more expensive this convolution becomes.\n", + " - For linear light profiles, basis functions like the MGE of tutorial 5 and the pixelizations of chapter 3, solving\n", + " the linear algebra of the inversion, which computes the `intensity` values that best fit the data.\n", "\n", - "Every operation **PyAutoLens** performs to fit strong lens data with a model takes time, for example:\n", + "More complex fitting techniques therefore cost more per evaluation: a couple of standard light profiles are cheap,\n", + "an MGE costs more (its inversion solves for many Gaussians), and pixelized source reconstructions cost the most.\n", + "As we saw in tutorial 5, this trade often still favours the more advanced technique, because the simpler parameter\n", + "space it produces reduces the number of evaluations the search needs.\n", "\n", - " - Computing the intensity values from a light profile.\n", - " - Convolving the image that comes from a tracer with the PSF to compare it to the data.\n", + "__JAX__\n", "\n", - "One can therefore in principle make **PyAutoLens** run faster by using more efficient algorithms. However, I am\n", - "confident that for many tasks and operations we have written code that is already very fast!\n", + "How are these operations made fast? **PyAutoLens** uses JAX (https://github.com/jax-ml/jax), Google's numerical\n", + "computing library, which just-in-time (JIT) compiles the entire log likelihood function into optimized machine code.\n", "\n", - "I often get asked, given that **PyAutoLens** is written in Python (a synonymously slow programming language), is it not\n", - "really slow? **PyAutoLens** uses a library called `numba` to ensure that it runs fast, which recompiles Python\n", - "functions into C functions before **PyAutoLens** runs. This gives us C-like speed, but in Python code. If you`ve got\n", - "your own code that needs speeding up, I strongly recommend that you look up Numba:\n", + "JIT compilation means the first time the likelihood function is called, JAX traces through every operation it\n", + "performs (ray-tracing, light profile evaluation, PSF convolution, the linear algebra of an inversion) and compiles\n", + "them into a single optimized program. This compilation is a one-off overhead, typically taking seconds to a couple\n", + "of minutes depending on model complexity. Every one of the tens of thousands of likelihood evaluations that follow\n", + "then reuses the compiled program and runs dramatically faster than ordinary Python. When a search begins you will\n", + "see a log message like `JAX jit compiling vectorized (vmap) likelihood function, could take seconds or minutes...`\n", + "-- that is the compilation happening.\n", "\n", - "http://numba.pydata.org/\n", + "JAX's second superpower is that the same compiled code runs on either a CPU or a GPU. On a GPU, likelihood\n", + "evaluations are not only faster individually but can be batched, with many evaluated simultaneously (the\n", + "`n_batch` search input and its VRAM implications were discussed in tutorial 2). GPU speed-ups over CPU\n", + "of an order of magnitude or more are common, and they grow with the size of the dataset and model. If you have\n", + "access to a GPU (even a modest laptop GPU), it is the single biggest speed-up available to you.\n", "\n", - "Therefore, **PyAutoLens** is pretty well optimized and there are no 'low hanging fruit' speed ups available by writing\n", - "the code in a different language.\n", + "You do not need to do anything to switch this on. If JAX is installed (`pip install autolens[jax]`), every\n", + "`AnalysisImaging` object we have created in this chapter defaults to `use_jax=True` and the search compiles and\n", + "batches the likelihood function automatically. You can force the plain NumPy path with\n", + "`al.AnalysisImaging(dataset=dataset, use_jax=False)` (or by setting the environment variable\n", + "`PYAUTO_DISABLE_JAX=1`), which is mainly useful for debugging, as NumPy errors and stack traces are much easier\n", + "to read than JAX ones.\n", + "\n", + "The one trade-off to keep in mind is the compilation overhead itself. For a full model-fit performing tens of\n", + "thousands of likelihood evaluations it is negligible. For a quick one-off calculation (e.g. fitting a single\n", + "tracer to data once, as we did in chapter 1) compilation can take longer than the calculation it speeds up, which\n", + "is why short interactive computations sometimes feel slower than you might expect the first time they run.\n", + "\n", + "Long-time users may remember that **PyAutoLens** previously used `numba`, which recompiles individual Python\n", + "functions into fast machine code. JAX supersedes it: rather than accelerating functions one at a time, it compiles\n", + "and optimizes the likelihood function end-to-end, and adds GPU support and batching on top.\n", + "\n", + "Therefore, **PyAutoLens** is pretty well optimized and there are no 'low hanging fruit' speed ups available by\n", + "writing the code in a different language.\n", + "\n", + "__Run Time Estimation__\n", + "\n", + "Because run-time is the product of the two numbers above, we can estimate it before committing to a fit:\n", + "\n", + " - Time the log likelihood evaluation: perform a fit of the model to the data once (e.g. via a `FitImaging`) and\n", + " time it, remembering to discard the first call, which includes JIT compilation.\n", + "\n", + " - Estimate the number of evaluations: for `Nautilus` a reasonable rule of thumb is of order 10000 evaluations per\n", + " free parameter, with simpler parameter spaces (e.g. linear light profiles, chained searches with tight priors)\n", + " converging in fewer.\n", + "\n", + "Multiplying the two gives a ballpark overall run-time. In tutorial 2 we used exactly this arithmetic: a log\n", + "likelihood evaluation time below 0.01 seconds on CPU (below 0.001 seconds on GPU) and 50000-100000 evaluations gave\n", + "expected run-times of roughly 30 minutes on CPU and 10 minutes on GPU. If your own estimate comes out at days or\n", + "weeks, that is the moment to simplify the model, tighten the search settings, reach for search chaining or find\n", + "a GPU -- before starting the fit, not after.\n", + "\n", + "On a GPU, tutorial 2 also showed the other pre-flight check: `analysis.print_vram_use` estimates whether the fit\n", + "will fit within the GPU's memory for your chosen batch size.\n", "\n", "__Data Quantity__\n", "\n", @@ -80,11 +153,17 @@ "\n", "There are ways that we can reduce the number of image-pixels we fit, via masking. If we mask out more of the image,\n", "we will fit fewer pixels and **PyAutoLens** will run faster. If you want the best, most perfect lens model possible,\n", - "aggressive masking and cutting the data in this way is a bad idea, as discussed in tutorial 5.\n", + "aggressive masking and cutting the data in this way is a bad idea, as discussed in tutorial 6.\n", "\n", "__Wrap Up__\n", "\n", - "This tutorial simply wanted to get you thinking about *why* a lens model takes as long to fit as it does." + "This tutorial simply wanted to get you thinking about *why* a lens model takes as long to fit as it does: the number\n", + "of likelihood evaluations the search performs, times the cost of each evaluation, with JAX's compiled likelihood\n", + "function (and a GPU, if you have one) driving the latter down as far as it will go.\n", + "\n", + "The remaining tutorials of this chapter attack the other half of the product: search chaining (tutorial 9) and prior\n", + "passing (tutorial 10) reduce the number of evaluations needed to fit complex models, and the SLaM pipelines\n", + "(tutorial 11) package these ideas into automated pipelines for fitting large samples of lenses." ] }, { diff --git a/notebooks/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb b/notebooks/chapter_2_lens_modeling/tutorial_9_search_chaining.ipynb similarity index 90% rename from notebooks/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb rename to notebooks/chapter_2_lens_modeling/tutorial_9_search_chaining.ipynb index 1612c78..1d4987a 100644 --- a/notebooks/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb +++ b/notebooks/chapter_2_lens_modeling/tutorial_9_search_chaining.ipynb @@ -4,17 +4,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 1: Search Chaining\n", + "Tutorial 9: Search Chaining\n", "===========================\n", "\n", - "In chapter 2, we learnt how to perform lens modeling using a non-linear search. In all of the tutorials, we fitted the\n", - "data using just one non-linear search. In this chapter, we introduce a technique called 'non-linear search chaining',\n", - "fits a lens model using a sequence of non-linear searches. The initial searches fit simpler lens models whose parameter\n", - "spaces can be more accurately and efficiently sampled. The results of this search are then passed to later searches\n", - "which fit lens models of gradually increasing complexity.\n", + "Throughout this chapter, we have fitted the data using just one non-linear search. The previous tutorial discussed\n", + "the run-time cost of doing this: with a single search, the dimensionality of the model, the priors on its parameters\n", + "and the search settings must all be juggled at once, leaving us little flexibility to trade them off against one\n", + "another.\n", "\n", - "Lets think back to tutorial 4 of chapter 2. We learnt there were three approaches one could take fitting a lens model\n", - "accurately if we found that a model fit failed. These were:\n", + "In this closing part of the chapter, we introduce a technique called 'non-linear search chaining', which fits a lens\n", + "model using a sequence of non-linear searches. The initial searches fit simpler lens models whose parameter spaces can\n", + "be more accurately and efficiently sampled. The results of this search are then passed to later searches which fit\n", + "lens models of gradually increasing complexity.\n", + "\n", + "Lets think back to tutorial 4 of this chapter. We learnt there were three approaches one could take fitting a lens\n", + "model accurately if we found that a model fit failed. These were:\n", "\n", " 1) Tuning our priors to the strong lens we're fitting.\n", " 2) Making our lens model less complex.\n", @@ -29,7 +33,7 @@ "we'll run 2 searches, and chain the lens model inferred in the first search to the priors of the second search`s lens\n", "model.\n", "\n", - "Our first search will make the same light-traces-mass assumption we made in the previous tutorial. We saw that this\n", + "Our first search will make the same light-traces-mass assumption we made in tutorial 4. We saw that this\n", "gives a reasonable lens model. However, we'll make a couple of extra simplifying assumptions, to really try and bring\n", "our lens model complexity down and get the non-linear search running fast.\n", "\n", @@ -45,10 +49,12 @@ "\n", "__Contents__\n", "\n", - "- **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where.\n", + "- **Initial Setup:** Load the same strong lensing data fitted throughout this chapter.\n", + "- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk.\n", "- **Model:** Compose the lens model fitted to the data.\n", + "- **Search + Analysis:** Create the non-linear search and analysis used to fit the first model.\n", "- **Result:** Overview of the results of the model-fit.\n", - "- **Prior Passing:** Now all we need to do is look at the results of search 1 and pass the results as priors for search.\n", + "- **Prior Passing:** Use the results of search 1 to set up the priors of search 2.\n", "- **Run Time:** Profiling the expected run time of the model-fit.\n", "- **Model Fit:** Perform the model-fit using the search and analysis.\n", "- **Wrap Up:** Summary of the script and next steps." @@ -99,7 +105,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import numpy as np\n", "from pathlib import Path\n", @@ -116,7 +122,7 @@ "source": [ "__Initial Setup__\n", "\n", - "we'll use the same strong lensing data as the previous tutorial, where:\n", + "we'll use the same strong lensing data as tutorials 3, 4 and 5 of this chapter, where:\n", "\n", " - The lens galaxy's light is an `Sersic`.\n", " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", @@ -313,11 +319,11 @@ "metadata": {}, "source": [ "search_1 = af.Nautilus(\n", - " path_prefix=Path(\"howtolens\", \"chapter_3\"),\n", - " name=\"tutorial_1_search_chaining_1\",\n", + " path_prefix=Path(\"howtolens\", \"chapter_2\"),\n", + " name=\"tutorial_9_search_chaining_1\",\n", " unique_tag=dataset_name,\n", " n_live=100,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis_1 = al.AnalysisImaging(dataset=dataset)" @@ -338,7 +344,7 @@ "metadata": {}, "source": [ "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/5_chaining_searches\"\n", + " \"The non-linear search has begun running - checkout the output/howtolens/chapter_2\"\n", " \" folder for live output of the results, images and lens model.\"\n", " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", ")\n", @@ -567,11 +573,11 @@ "metadata": {}, "source": [ "search_2 = af.Nautilus(\n", - " path_prefix=Path(\"howtolens\", \"chapter_3\"),\n", - " name=\"tutorial_1_search_chaining_2\",\n", + " path_prefix=Path(\"howtolens\", \"chapter_2\"),\n", + " name=\"tutorial_9_search_chaining_2\",\n", " unique_tag=dataset_name,\n", " n_live=150,\n", - " iterations_per_quick_update=25000, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=25000, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis_2 = al.AnalysisImaging(dataset=dataset)" @@ -601,7 +607,7 @@ "metadata": {}, "source": [ "print(\n", - " \"The non-linear search has begun running - checkout the workspace/output/5_chaining_searches\"\n", + " \"The non-linear search has begun running - checkout the output/howtolens/chapter_2\"\n", " \" folder for live output of the results, images and lens model.\"\n", " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", ")\n", diff --git a/notebooks/chapter_3_pixelizations/README.md b/notebooks/chapter_3_pixelizations/README.md new file mode 100644 index 0000000..c5d9288 --- /dev/null +++ b/notebooks/chapter_3_pixelizations/README.md @@ -0,0 +1,29 @@ +In chapter 3, we use **Pixelizations** to reconstruct complex source galaxies on pixelized grids. + +**Colab** links to every tutorial are included. + +# Files + +- [Tutorial 1: Pixelizations](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_1_pixelizations.ipynb) — Creating a pixel-grid in the source-plane. + +- [Tutorial 2: Mappers](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_2_mappers.ipynb) — How a pixelization maps source-pixels to image-pixels. + +- [Tutorial 3: Inversions](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_3_inversions.ipynb) — Inverting the mappings to reconstruct the source's light. + +- [Tutorial 4: Bayesian Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_4_bayesian_regularization.ipynb) — Smoothing the source within a Bayesian framework. + +- [Tutorial 5: Bayesian Formalism](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_5_bayesian_formalism.ipynb) — The linear algebra and Bayesian evidence equations behind pixelized source reconstruction. + +- [Tutorial 6: Borders](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_6_borders.ipynb) — Preventing highly demagnified image-pixels ruining the inversion. + +- [Tutorial 7: Lens Modeling](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_7_lens_modeling.ipynb) — How to use inversions to fit a lens model. + +- [Tutorial 8: Adaptive Pixelization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.ipynb) — A Delaunay mesh which adapts to the mass model's magnification. + +- [Tutorial 9: Model Fit](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_9_model_fit.ipynb) — An example lens modeling pipeline which uses an inversion. + +- [Tutorial 10: Fit Problems](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_10_fit_problems.ipynb) — The shortcomings of our lens models and inversions. + +- [Tutorial 11: Brightness Adaption](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_11_brightness_adaption.ipynb) — Adapting the pixelization to the source's morphology. + +- [Tutorial 12: Adaptive Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_12_adaptive_regularization.ipynb) — Adapting the regularization to the source's morphology. diff --git a/notebooks/chapter_4_pixelizations/tutorial_9_fit_problems.ipynb b/notebooks/chapter_3_pixelizations/tutorial_10_fit_problems.ipynb similarity index 91% rename from notebooks/chapter_4_pixelizations/tutorial_9_fit_problems.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_10_fit_problems.ipynb index bdf8717..a297fcf 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_9_fit_problems.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_10_fit_problems.ipynb @@ -4,20 +4,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 9: Fit Problems\n", - "========================\n", + "Tutorial 10: Fit Problems\n", + "=========================\n", "\n", - "To begin, make sure you have read the `introduction` file carefully, as a clear understanding of how the Bayesian\n", - "evidence works is key to understanding this chapter!\n", + "To begin, make sure you have read tutorials 4 and 5 carefully, as a clear understanding of how the Bayesian\n", + "evidence works is key to understanding the adaption tutorials that close this chapter!\n", "\n", - "In the previous chapter we investigated two pixelization's: `RectangularAdaptDensity` and `RectangularAdaptDensity`. We argued that the\n", - "latter was better than the former, because it dedicated more source-pixels to the regions of the source-plane where we\n", - "had more data, e.g, the high-magnification regions. Therefore, we could fit the data using fewer source pixels,\n", - "which improved computational efficiency and increased the Bayesian evidence.\n", + "Earlier in this chapter (tutorial 8) we investigated two pixelizations: the `RectangularAdaptDensity` mesh and the\n", + "`Delaunay` mesh with an `Overlay` image-mesh. We argued that the latter was better than the former, because it\n", + "dedicated more source-pixels to the regions of the source-plane where we had more data, e.g. the high-magnification\n", + "regions. Therefore, we could fit the data using fewer source pixels, which improved computational efficiency and\n", + "increased the Bayesian evidence.\n", "\n", "So far, we've used just one regularization scheme; `Constant`. As the name suggests, this scheme applies just one\n", "regularization coefficient when comparing source pixel fluxes to apply smoothing. Here is a recap of our discussion\n", - "about regularization from chapter 4:\n", + "about regularization from tutorial 4:\n", "\n", "--------------------------------------------\n", "\n", @@ -105,7 +106,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import autolens as al\n", "import autolens.plot as aplt" @@ -253,8 +254,8 @@ "source": [ "__Fitting__\n", "\n", - "we'll make one more convenience function which fits the simulated imaging data with an `Overlay` image-mesh, \n", - "`RectangularAdaptDensity` mesh and `Constant` regularization scheme pixelization.\n", + "we'll make one more convenience function which fits the simulated imaging data with a\n", + "`RectangularAdaptDensity` mesh and `Constant` regularization scheme pixelization.\n", "\n", "We'll input the `coefficient` of each fit, so that for each simulated source we regularize it at an appropriate level. \n", "There is nothing new in this function you haven't seen before." @@ -395,14 +396,15 @@ "\n", "*Image-Mesh / Mesh*:\n", "\n", - "The problem is the same one we discussed when we compared the `RectangularAdaptDensity` and `RectangularAdaptDensity` meshes in tutorial 7. \n", + "The problem is the same one we discussed in tutorial 8, when we motivated pixelizations which adapt to where the\n", + "data constrains the source.\n", "\n", - "We are simply not dedicating enough source-pixels to the central regions of the source reconstruction, \n", - "e.g. where it`s brightest. As the source becomes more compact, the source reconstruction no longer has enough \n", + "We are simply not dedicating enough source-pixels to the central regions of the source reconstruction,\n", + "e.g. where it's brightest. As the source becomes more compact, the source reconstruction no longer has enough\n", "resolution to resolve its fine-detailed central structure, causing the fit to the image to degrade.\n", "\n", - "As we made our sources more compact we go from reconstructing them using ~100 source pixels, to ~20 source pixels \n", - "to ~ 10 source pixels. This is why we advocated not using the `RectangularAdaptDensity` mesh previously!\n", + "As we made our sources more compact we went from reconstructing them using ~100 source pixels, to ~20 source pixels\n", + "to ~10 source pixels.\n", "\n", "Adapting to the mass model magnification is not the best approach. As we simulated more compact sources the \n", "magnification (which is determined via the mass model) does not change. We therefore reconstructed each source\n", @@ -443,7 +445,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As you can see, all pixels are regularized with our input regularization_coefficient value of 3.6.\n", + "As you can see, all pixels are regularized with our single input regularization coefficient value.\n", "\n", "This is not the best approach to regularizing the source. In fact, different regions of the source prefer different \n", "levels of regularization:\n", @@ -465,7 +467,7 @@ "By using a single regularization coefficient, we infer an intermediate regularization coefficient that over-smooths \n", "the source's central regions whilst failing to fully correlate exterior pixels. \n", "\n", - "An adaptive regularization scheme, where the regularization coefficient varies from the outskirts to the centrel \n", + "An adaptive regularization scheme, where the regularization coefficient varies from the outskirts to the central\n", "regions, will produce solutions that further increase the Bayesian evidence.\n", "\n", "__Wrap Up__\n", diff --git a/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb b/notebooks/chapter_3_pixelizations/tutorial_11_brightness_adaption.ipynb similarity index 98% rename from notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_11_brightness_adaption.ipynb index ee250e1..fdade77 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_11_brightness_adaption.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 10: Brightness Adaption\n", + "Tutorial 11: Brightness Adaption\n", "================================\n", "\n", "In the previous tutorial we motivated our need to adapt the pixelization to the source's morphology, such that source\n", @@ -86,7 +86,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -202,7 +202,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Lets have a quick look to make sure it has the same residuals we saw in tutorial 1." + "Lets have a quick look to make sure it has the same residuals we saw in the previous tutorial." ] }, { @@ -542,9 +542,9 @@ "source": [ "__Wrap Up__\n", "\n", - "To end, lets think about the Bayesian evidence, which we saw now goes to significantly higher values than for a \n", - "magnification-based grid. At this point, it might be worth reminding yourself how the Bayesian evidence works by \n", - "going back to description in this chapters `introduction` text file.\n", + "To end, lets think about the Bayesian evidence, which we saw now goes to significantly higher values than for a\n", + "magnification-based grid. At this point, it might be worth reminding yourself how the Bayesian evidence works by\n", + "going back to tutorials 4 and 5 of this chapter.\n", "\n", "So, why do you think why adapting to the source's brightness increases the log evidence?\n", "\n", @@ -558,7 +558,7 @@ "**PyAutoLens** can now access those solutions, this means that adapt-mode will run much faster than the magnification \n", "based grid! Put simply, fewer source-pixels means lower computational overheads. YAY!\n", "\n", - "Tutorial 2 done, next up, adaptive regularization!" + "Tutorial 11 done, next up, adaptive regularization!" ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_11_adaptive_regularization.ipynb b/notebooks/chapter_3_pixelizations/tutorial_12_adaptive_regularization.ipynb similarity index 91% rename from notebooks/chapter_4_pixelizations/tutorial_11_adaptive_regularization.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_12_adaptive_regularization.ipynb index e59c4ce..d95f186 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_11_adaptive_regularization.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_12_adaptive_regularization.ipynb @@ -4,10 +4,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 11: Adaptive Regularization\n", + "Tutorial 12: Adaptive Regularization\n", "====================================\n", "\n", - "In tutorial 7, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source\n", + "In tutorial 10, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source\n", "demand different levels of regularization, motivating a regularization scheme which adapts to the reconstructed\n", "source's surface brightness.\n", "\n", @@ -68,7 +68,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -303,8 +303,8 @@ "\n", "Combining the adaptive mesh and regularization will only further benefit lens modeling!\n", "\n", - "However, as shown below, we don't fit the source as well as the morphology based mesh did in the last chapter. \n", - "This is because although the adaptive regularization scheme improves the fit, the magnification based \n", + "However, as shown below, we don't fit the source as well as the morphology based mesh did in the previous tutorial.\n", + "This is because although the adaptive regularization scheme improves the fit, the magnification based\n", "mesh simply does not have sufficient resolution to resolve the source's cuspy central light." ] }, @@ -418,22 +418,27 @@ "__Wrap Up__\n", "\n", "To end, lets consider what this adaptive regularization scheme means in the context of maximizing the Bayesian\n", - "evidence. In the previous tutorial, we noted that by using a brightness-based adaptive pixelization we increased \n", - "the Bayesian evidence by allowing for new solutions which fit the data user fewer source pixels; the key criteria \n", + "evidence. In the previous tutorial, we noted that by using a brightness-based adaptive pixelization we increased\n", + "the Bayesian evidence by allowing for new solutions which fit the data using fewer source pixels; the key criteria\n", "in making a source reconstruction 'more simple' and 'less complex'.\n", "\n", - "As you might of guessed, adaptive regularization increases the Bayesian log evidence by making the source \n", + "As you might have guessed, adaptive regularization increases the Bayesian log evidence by making the source\n", "reconstruction simpler:\n", "\n", - " 1) Reducing regularization in the source's brightest regions produces a `simpler` solution in that we are not \n", + " 1) Reducing regularization in the source's brightest regions produces a `simpler` solution in that we are not\n", " over-smoothing our reconstruction of its brightest regions.\n", - " \n", - " 2) Increasing regularization in the outskirts produces a simpler solution by correlating more source-pixels, \n", + "\n", + " 2) Increasing regularization in the outskirts produces a simpler solution by correlating more source-pixels,\n", " effectively reducing the number of pixels used by the reconstruction.\n", "\n", - "Together, brightness based pixelization's and regularization allow us to find the objectively `simplest` source \n", - "solution possible and therefore ensure that our Bayesian evidence has a well defined maximum value. This was not the \n", - "case for magnification based pixelization's and constant regularization schemes." + "Together, brightness based pixelizations and regularization allow us to find the objectively `simplest` source\n", + "solution possible and therefore ensure that our Bayesian evidence has a well defined maximum value. This was not the\n", + "case for magnification based pixelizations and constant regularization schemes.\n", + "\n", + "That completes this chapter on pixelized source reconstruction. You can now reconstruct sources of arbitrary\n", + "complexity and adapt every part of the analysis to the data itself. In chapter 4, `Scaling Up Lensing`, we take these\n", + "tools to bigger systems: lenses with extra galaxies along the line of sight, multi-galaxy lenses, scaling relations,\n", + "group and cluster scale lenses and weak lensing." ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb b/notebooks/chapter_3_pixelizations/tutorial_1_pixelizations.ipynb similarity index 89% rename from notebooks/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_1_pixelizations.ipynb index d6bbb36..10df8e6 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_1_pixelizations.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 1: pixelizations\n", + "Tutorial 1: Pixelizations\n", "=========================\n", "\n", "In the previous chapters, we used light profiles to model the light of a strong lens's source galaxy, where the light\n", @@ -64,7 +64,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "import autolens as al\n", "import autolens.plot as aplt" @@ -158,10 +158,10 @@ "metadata": {}, "source": [ "By itself, a pixelization does not tell us much. It has no grid of $(y,x)$ coordinates, no image, and no information\n", - "about the lens we are fitting. \n", + "about the lens we are fitting.\n", "\n", - "This information comes when we use the pixelization to create up a `Mapper`, which we\n", - "perform below using the (traced) source-plane grid that we created above." + "This information comes when we use the pixelization to create a `Mapper`, which we\n", + "do below using the (traced) source-plane grid that we created above." ] }, { @@ -181,7 +181,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This `Mapper` is a `RectangularMapper`, every `Pixelization` generates it owns mapper." + "This `Mapper` is a `RectangularMapper` -- every `Pixelization` generates its own mapper." ] }, { @@ -213,8 +213,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "However, the `Mapper` does contain lots of interesting information about our `Pixelization`, for example its \n", - "pixelization_grid tells us where the pixel centers are located." + "The `Mapper` contains lots of interesting information about our `Pixelization`, for example its\n", + "`source_plane_mesh_grid` tells us where the pixel centres are located." ] }, { @@ -253,7 +253,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The `Mapper` also has the (source-plane) grid that we passed when we set it up. Lets check they`re the same." + "The `Mapper` also has the (source-plane) grid that we passed when we set it up. Lets check they're the same." ] }, { @@ -295,7 +295,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Finally, the mapper`s `mesh_grid` has lots of information about the pixelization, for example, the arc-second \n", + "Finally, the mapper's mesh grid has lots of information about the pixelization, for example, its arc-second\n", "size and dimensions." ] }, @@ -319,12 +319,12 @@ "This was a relatively gentle overview of pixelizations, but one that was hopefully easy to follow. Think about the \n", "following questions before moving on to the next tutorial:\n", "\n", - " 1) Look at how the source-grid coordinates are distributed over the rectangular pixel-grid. Are these points \n", - " distributed evenly over the rectangular grid`s pixels? Do some pixels have a lot more grid-points inside of them? \n", + " 1) Look at how the source-grid coordinates are distributed over the rectangular pixel-grid. Are these points\n", + " distributed evenly over the rectangular grid's pixels? Do some pixels have a lot more grid-points inside of them?\n", " Do some pixels have no grid-points in them?\n", "\n", - " 2) The rectangular pixelization`s edges are aligned with the most exterior coordinates of the source-grid. This is \n", - " intentional, why do you think this is?" + " 2) The rectangular pixelization's edges are aligned with the most exterior coordinates of the source-grid. This is\n", + " intentional -- why do you think this is?" ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_2_mappers.ipynb b/notebooks/chapter_3_pixelizations/tutorial_2_mappers.ipynb similarity index 93% rename from notebooks/chapter_4_pixelizations/tutorial_2_mappers.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_2_mappers.ipynb index 1e9e7cd..c2ee7ef 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_2_mappers.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_2_mappers.ipynb @@ -7,12 +7,12 @@ "Tutorial 2: Mappers\n", "===================\n", "\n", - "In the previous tutorial, we used a pixelization to create made a `Mapper`. However, it was not clear what a `Mapper`\n", + "In the previous tutorial, we used a pixelization to create a `Mapper`. However, it was not clear what a `Mapper`\n", "does, why it was called a mapper and whether it was mapping anything at all!\n", "\n", "Therefore, in this tutorial, we'll cover mappers in more detail.\n", "\n", - "WARNING: THHIS TUTORIAL VISUALS ARE SLIGHTLY BUGGY CURRENTLY AND WILL BE FIXED IN THE FUTURE.\n", + "WARNING: THIS TUTORIAL'S VISUALS ARE SLIGHTLY BUGGY CURRENTLY AND WILL BE FIXED IN THE FUTURE.\n", "\n", "__Contents__\n", "\n", @@ -67,7 +67,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -157,8 +157,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Our `Tracer` will use the same lens galaxy and source galaxy that we used to Simulate the imaging data (although, \n", - "becuase we're modeling the source with a pixel-grid, we do not pass it any light profiles." + "Our `Tracer` will use the same lens galaxy and source galaxy that we used to simulate the imaging data (although,\n", + "because we're modeling the source with a pixel-grid, we do not pass the source any light profiles)." ] }, { @@ -234,8 +234,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Using the `lines=`/`positions=` overlays object we are also going to highlight specific grid coordinates certain colors, such that we\n", - "can see how they map from the image-plane to source-plane and visa versa.\n", + "Using the `positions=` overlay input we are also going to highlight specific grid coordinates in certain colors, such\n", + "that we can see how they map from the image-plane to the source-plane and vice versa.\n", "\n", "We do this by specifying their integer indexes, corresponding to the index of each data point in the image and source\n", "plane grids. These indexes are used to highlight the grid coordinates in the image and source-plane grids that map\n", @@ -289,8 +289,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can now make these mappings appear the other way round. That is, we can input a source-pixel index (of our \n", - "rectangular grid) and highlight how all of the (sub-)image-pixels that it contains map to the image-plane. \n", + "We can now make these mappings appear the other way round. That is, we can input a source-pixel index (of our\n", + "rectangular grid) and highlight how all of the image-pixels that it contains map to the image-plane.\n", "\n", "To make the indexes appear in the image-plane, we have to convert them from their source-plane pixel indexes\n", "to image plane image-pixel indexes using the mapper.\n", @@ -408,7 +408,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "To create the mapper, we need to set up the masked imaging's grid as the source-plane gird via the tracer." + "To create the mapper, we need to trace the masked imaging's grid to the source-plane via the tracer." ] }, { @@ -508,9 +508,9 @@ " radius deviates from 1.6\" (the input value of the simulated lens), what do you notice about where the points map \n", " from the centre of the source-plane (where the source-galaxy is simulated, e.g. (0.0\", 0.0\"))?\n", " \n", - " 2) Think about how this could help us actually model lenses. We have said we're going to reconstruct our source \n", - " galaxies on the pixel-grid. So, how does knowing how each pixel maps to the image actually help us? If you`ve not got \n", - " any bright ideas, then worry not, that exactly what we're going to cover in the next tutorial." + " 2) Think about how this could help us actually model lenses. We have said we're going to reconstruct our source\n", + " galaxies on the pixel-grid. So, how does knowing how each pixel maps to the image actually help us? If you've not got\n", + " any bright ideas, then worry not, that's exactly what we're going to cover in the next tutorial." ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_3_inversions.ipynb b/notebooks/chapter_3_pixelizations/tutorial_3_inversions.ipynb similarity index 91% rename from notebooks/chapter_4_pixelizations/tutorial_3_inversions.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_3_inversions.ipynb index 78a6080..afb3bd5 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_3_inversions.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_3_inversions.ipynb @@ -12,7 +12,7 @@ " - `Pixelization`'s: which place a pixel-grid in the source-plane.\n", " - `Mappers`'s: which describe how each source-pixel maps to one or more image pixels.\n", "\n", - "However, non of this has actually helped us fit strong lens data or reconstruct the source galaxy. This is the subject\n", + "However, none of this has actually helped us fit strong lens data or reconstruct the source galaxy. This is the subject\n", "of this tutorial, where the process of reconstructing the source's light on the pixelization is called an `Inversion`.\n", "\n", "__Contents__\n", @@ -69,7 +69,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -138,7 +138,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Lets create an annular mask which traces the stongly lensed source's ring of light." + "Lets create an annular mask which traces the strongly lensed source's ring of light." ] }, { @@ -407,9 +407,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Pretty great, huh? If you ran the complex source pipeline in chapter 3, you'll remember that getting a model image \n", - "that looked this good simply *was not possible*. With an inversion, we can do this with ease and without having to \n", - "perform model-fitting with 20+ parameters for the source's light!\n", + "Pretty great, huh? If you fitted complex sources using light profiles in the search chaining tutorials at the end of\n", + "chapter 2 (tutorials 9-11), you'll remember that getting a model image that looked this good simply *was not possible*.\n", + "With an inversion, we can do this with ease and without having to perform model-fitting with 20+ parameters for the\n", + "source's light!\n", "\n", "We will now briefly discuss how an inversion actually works, however the explanation I give in this tutorial will be \n", "overly-simplified. To be good at lens modeling you do not need to understand the details of how an inversion works, you \n", @@ -445,11 +446,16 @@ " 1) The mappings between every source-pixel and sets of image-pixels.\n", " 2) The flux values in every observed image-pixel, which are the values we want to fit successfully.\n", "\n", - "It turns out that with these two pieces of information we can linearly solve for the set of source-pixel fluxes that \n", - "best-fit (e.g. maximize the log likelihood) our observed image. Essentially, we set up the mappings between source and \n", - "image pixels as a large matrix and solve for the source-pixel fluxes in an analogous fashion to how you would solve a \n", + "It turns out that with these two pieces of information we can linearly solve for the set of source-pixel fluxes that\n", + "best-fit (e.g. maximize the log likelihood) our observed image. Essentially, we set up the mappings between source and\n", + "image pixels as a large matrix and solve for the source-pixel fluxes in an analogous fashion to how you would solve a\n", "set of simultaneous linear equations. This process is called a `linear inversion`.\n", "\n", + "In the language of linear algebra: the mappings form a `mapping matrix` $f$, which together with the data and\n", + "noise-map defines a linear system whose solution is the vector of source-pixel fluxes $s$. Tutorial 5 of this\n", + "chapter writes this system out in full and computes it step-by-step in code -- for now, the intuition above is all\n", + "you need.\n", + "\n", "There are three more things about a linear inversion that are worth knowing:\n", "\n", " 1) When performing fits using light profiles, we discussed how a `model_image` was generated by convolving the light\n", @@ -461,7 +467,7 @@ " the mapping between every sub-pixel and source-pixel that is computed and used to perform the inversion. This prevents \n", " aliasing effects degrading the image reconstruction. By default **PyAutoLens** uses sub-gridding of degree 4x4.\n", "\n", - " 3) The inversion`s solution is regularized. But wait, that`s what we'll cover in the next tutorial!\n", + " 3) The inversion's solution is regularized. But wait, that's what we'll cover in the next tutorial!\n", "\n", "Finally, let me show you how easy it is to fit an image with an `Inversion` using a `FitImaging` object. Instead of \n", "giving the source galaxy a light profile, we simply pass it a `Pixelization` and regularization, and pass it to a \n", @@ -537,9 +543,10 @@ " \n", "__Detailed Explanation__\n", "\n", - "If you are interested in a more detailed description of how inversions work, then checkout the file\n", - "`autolens_workspace/*/imaging/features/pixelization/likelihood_function.ipynb` which gives a visual step-by-step\n", - "guide of the process alongside equations and references to literature on the subject." + "If you are interested in a more detailed description of how inversions work, tutorial 5 of this chapter collects the\n", + "full linear algebra in one place. The file\n", + "`autolens_workspace/*/imaging/features/pixelization/likelihood_function.ipynb` gives a complementary visual\n", + "step-by-step guide of the process alongside equations and references to literature on the subject." ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb b/notebooks/chapter_3_pixelizations/tutorial_4_bayesian_regularization.ipynb similarity index 91% rename from notebooks/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_4_bayesian_regularization.ipynb index ae8d885..24c021c 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_4_bayesian_regularization.ipynb @@ -72,7 +72,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -143,11 +143,11 @@ "source": [ "__Convenience Function__\n", "\n", - "we're going to perform a lot of fits using an `Inversion` this tutorial. This would create a lot of code, so to keep \n", + "we're going to perform a lot of fits using an `Inversion` this tutorial. This would create a lot of code, so to keep\n", "things tidy, I've setup this function which handles it all for us.\n", "\n", - "(You may notice we include an option to `use_pixelization_border, ignore this for now, as we'll be covering borders in \n", - "the next tutorial)" + "(Borders, which relocate demagnified image pixels in the source-plane, are used behind the scenes here -- we'll be\n", + "covering them in tutorial 6.)" ] }, { @@ -249,12 +249,15 @@ "So, what is happening here? Why does reducing the `coefficient` do this to our source reconstruction? First, we need\n", "to understand what regularization actually does!\n", "\n", - "When the inversion reconstructs the source, it does not *only* compute the set of source-pixel fluxes that best-fit \n", - "the image. It also regularizes this solution, whereby it goes to every pixel on the rectangular source-plane grid \n", - "and computes the different between the reconstructed flux values of every source pixel with its 4 neighboring pixels. \n", - "If the difference in flux is large the solution is penalized, reducing its log likelihood. You can think of this as \n", + "When the inversion reconstructs the source, it does not *only* compute the set of source-pixel fluxes that best-fit\n", + "the image. It also regularizes this solution, whereby it goes to every pixel on the rectangular source-plane grid\n", + "and computes the difference between the reconstructed flux values of every source pixel with its 4 neighboring pixels.\n", + "If the difference in flux is large the solution is penalized, reducing its log likelihood. You can think of this as\n", "us applying a 'smoothness prior' on the reconstructed source galaxy's light.\n", "\n", + "(In the language of linear algebra, this smoothness prior is encoded in a `regularization matrix` $H$ which is\n", + "added to the linear system the inversion solves -- tutorial 5 writes this out in full.)\n", + "\n", "This smoothing adds a 'penalty term' to the log likelihood of an inversion which is the summed difference between the \n", "reconstructed fluxes of every source-pixel pair multiplied by the `coefficient`. By setting the regularization \n", "coefficient to zero, we set this penalty term to zero, meaning that regularization is completely omitted.\n", @@ -441,14 +444,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - " 2) Can you think of any other ways we might increase the Bayesian evidence even further? In future tutorials we will \n", - " learn how **PyAutoLens** can adapts the source reconstructions to the properties of the image so as to maximize the \n", + " 2) Can you think of any other ways we might increase the Bayesian evidence even further? In future tutorials we will\n", + " learn how **PyAutoLens** can adapt the source reconstruction to the properties of the image so as to maximize the\n", " Bayesian evidence!\n", - " \n", + "\n", "__Detailed Description__\n", "\n", "Below, I provide a more detailed discussion of the Bayesian evidence. It is not paramount that you understand this to\n", - "use **PyAutoLens**, but I recommend you give it a read to get an intuition for how the evidence works.\n", + "use **PyAutoLens**, but I recommend you give it a read to get an intuition for how the evidence works. The evidence\n", + "itself is a sum of chi-squared, regularization and matrix log-determinant terms -- tutorial 5 derives the exact\n", + "expression and computes every term in code.\n", "\n", "The Bayesian log evidence quantifies the following 3 aspects of a fit to strong lens imaging data:\n", "\n", @@ -459,10 +464,10 @@ "\n", " However, this raises the question of what constitutes a \u2018good\u2019 solution? The Bayesian evidence defines this by\n", " assuming that the image data consists of independent Gaussian noise in every image pixel. A \u2018good\u2019 solution is one\n", - " whose chi-squared residuals are consistent with Gaussian noise, producing a reduced chi-squared near 1.0 .Solutions\n", - " which give a reduced chi squared below 1 are penalized for being overly complex and fitting the image\u2019s noise, whereas\n", - " solutions with a reduced chi-squared above are penalized for not invoking a more complex source model when the data it\n", - " is necessary to fit the data bettter. In both circumstances, these penalties reduce the inferred Bayesian evidence!\n", + " whose chi-squared residuals are consistent with Gaussian noise, producing a reduced chi-squared near 1.0. Solutions\n", + " which give a reduced chi-squared below 1 are penalized for being overly complex and fitting the image\u2019s noise, whereas\n", + " solutions with a reduced chi-squared above 1 are penalized for not invoking a more complex source model when the data\n", + " necessitates it. In both circumstances, these penalties reduce the inferred Bayesian evidence!\n", "\n", "2) *The complexity of the source reconstruction:* The log evidence estimates the number of source pixels that are used \n", " to reconstruct the image, after accounting for their correlation with one another due to regularization. Solutions that\n", @@ -470,9 +475,9 @@ " reconstructions are favoured.\n", "\n", "3) *The signal-to-noise (S/N) of the image that is fitted:* The Bayesian evidence favours models which fit higher S/N\n", - " realizations of the observed data (where the S/N is determined using the image-pixel variances, e.g. the noise-map). Up \n", - " to now, all **PyAutoLens** fits assumed fixed variances, meaning that this aspect of the Bayeisan evidence has no impact \n", - " on the inferred evidence values. However, in hyper-mode we will invoke functionality that increases the variances \n", + " realizations of the observed data (where the S/N is determined using the image-pixel variances, e.g. the noise-map). Up\n", + " to now, all **PyAutoLens** fits assumed fixed variances, meaning that this aspect of the Bayesian evidence has no impact\n", + " on the inferred evidence values. However, adaptive features can invoke functionality that increases the variances\n", " of image-pixels where the lens model fits the data poorly.\n", " \n", " The premise is that whilst increasing the variances of image pixels lowers their S/N values and therefore also\n", diff --git a/notebooks/chapter_3_pixelizations/tutorial_5_bayesian_formalism.ipynb b/notebooks/chapter_3_pixelizations/tutorial_5_bayesian_formalism.ipynb new file mode 100644 index 0000000..852839e --- /dev/null +++ b/notebooks/chapter_3_pixelizations/tutorial_5_bayesian_formalism.ipynb @@ -0,0 +1,1009 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 5: Bayesian Formalism\n", + "==============================\n", + "\n", + "In tutorials 1 to 4, we built an intuition for how pixelized source reconstruction works: pixelizations place a\n", + "pixel-grid in the source-plane, mappers pair source-pixels with image-pixels, inversions solve for the source-pixel\n", + "fluxes that best fit the data, and regularization smooths the solution within a Bayesian framework.\n", + "\n", + "This tutorial collects the linear algebra behind all of that in one place. It is the counterpart of chapter 1's\n", + "tutorial 5 on the lensing formalism: the hands-on tutorials built intuition, and now we write down the equations.\n", + "We will construct, step-by-step and in code, every matrix and vector the inversion uses, solve for the source\n", + "reconstruction ourselves and compute the Bayesian evidence by hand, comparing our answer at the end to the\n", + "`FitImaging` object which performs this calculation internally.\n", + "\n", + "None of this is required to *use* **PyAutoLens** for lens modeling. However, if you publish results which use a\n", + "pixelization, this is the calculation your paper's likelihood function section will describe, and understanding it\n", + "removes any sense that the source reconstruction is a \"black box\".\n", + "\n", + "The formalism follows Warren & Dye 2003 (https://arxiv.org/abs/astro-ph/0302587), hereafter WD03, with the data\n", + "vector notation of Nightingale & Dye 2015 (https://arxiv.org/abs/1412.7436), hereafter N15. The Bayesian evidence\n", + "was derived by Suyu et al. 2006 (https://arxiv.org/abs/astro-ph/0601493) and translated to **PyAutoLens** notation\n", + "in Dye et al. 2008 (https://arxiv.org/abs/0804.4002).\n", + "\n", + "__Contents__\n", + "\n", + "- **Initial Setup:** Load the dataset, mask it and disable over sampling so the algebra stays simple.\n", + "- **Mesh Shape:** Fix the rectangular mesh's shape and note why edge pixels are zeroed.\n", + "- **Ray Tracing:** Trace the masked image-pixel coordinates to the source-plane via the lens equation.\n", + "- **Border Relocation:** Relocate demagnified coordinates that trace far outside the source-plane border.\n", + "- **Source Pixel Centres:** Overlay the rectangular mesh over the traced coordinates.\n", + "- **Interpolation:** Pair every traced image-pixel with source pixels via bilinear interpolation.\n", + "- **Mapper:** Package the interpolation into a `Mapper` describing all image-to-source mappings.\n", + "- **Mapping Matrix:** Express the mappings as the 2D matrix $f$.\n", + "- **Blurred Mapping Matrix:** Convolve every column of $f$ with the imaging PSF.\n", + "- **Data Vector:** Compute the data vector $D$ from the blurred mapping matrix, data and noise-map.\n", + "- **Curvature Matrix:** Compute the curvature matrix $F$.\n", + "- **Unregularized Solve:** Solve $s = F^{-1} D$ and see the over-fitted mess this produces.\n", + "- **Regularization Matrix:** Compute the regularization matrix $H$ encoding the smoothness prior.\n", + "- **Source Reconstruction:** Solve the regularized system $s = [F + H]^{-1} D$.\n", + "- **Image Reconstruction:** Map the reconstruction back to the image-plane via the blurred mapping matrix.\n", + "- **Likelihood Function:** The five terms which combine into the log evidence.\n", + "- **Chi Squared:** The goodness-of-fit of the reconstructed image to the data.\n", + "- **Regularization Term:** The penalty $s^{T} H s$ applied by the smoothness prior.\n", + "- **Complexity Terms:** The log determinant terms which penalize complex source reconstructions.\n", + "- **Noise Normalization Term:** The Gaussian noise normalization.\n", + "- **Log Evidence:** Combine all five terms into the log evidence.\n", + "- **Fit:** Compare our by-hand log evidence to the `FitImaging` object's internal calculation.\n", + "- **Wrap Up:** Summary and next steps." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "from pathlib import Path\n", + "\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Initial Setup__\n", + "\n", + "we'll use the same strong lensing data as the previous tutorials, where:\n", + "\n", + " - The lens galaxy's light is omitted.\n", + " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", + " - The source galaxy's light is an `Sersic`.\n", + "\n", + "Because the lens galaxy's light is omitted, the data is already \"lens subtracted\". WD03 make the same assumption,\n", + "so in the equations below the lens light model $b_{j}$ is zero everywhere. For a lens with light, one simply computes\n", + "the lens galaxy's PSF-convolved image first and subtracts it from the data before the steps below (the workspace\n", + "guide referenced at the end shows this in full)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"simple__no_lens_light\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/no_lens_light.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "The likelihood is only evaluated within a mask, which we define as a 3.0\" circle containing all of the lensed\n", + "source's light." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=3.0,\n", + ")\n", + "\n", + "masked_dataset = dataset.apply_mask(mask=mask)\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=masked_dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Over Sampling__\n", + "\n", + "Over sampling splits every image pixel into a sub-grid of sub-pixels, each of which is traced to the source-plane\n", + "and paired with source pixels individually. It improves accuracy, but it also multiplies the number of rows in every\n", + "matrix below by the number of sub-pixels.\n", + "\n", + "To keep the algebra as easy to follow as possible, we disable over sampling by setting both sub-grid sizes to 1, so\n", + "each image pixel is represented by the single coordinate at its centre." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "masked_dataset = masked_dataset.apply_over_sampling(\n", + " over_sample_size_lp=1,\n", + " over_sample_size_pixelization=1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mesh Shape__\n", + "\n", + "The `mesh_shape` defines the number of pixels in the rectangular mesh used to reconstruct the source, set below\n", + "to 20 x 20 = 400 source pixels.\n", + "\n", + "We use the `RectangularUniform` mesh, where all rectangular source pixels have the same size, rather than the\n", + "`RectangularAdaptDensity` mesh used in the previous tutorials. The uniform mesh keeps the geometry simple, and every\n", + "equation below applies unchanged to the adaptive meshes -- only the source pixel centres move.\n", + "\n", + "By default, source pixels at the edge of the mesh are forced to solutions of zero flux by the linear algebra solver.\n", + "This prevents unphysical solutions where the mesh edge lights up to fit residuals, and does not change any of the\n", + "formalism below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mesh_pixels_yx = 20\n", + "mesh_shape = (mesh_pixels_yx, mesh_pixels_yx)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Tracer__\n", + "\n", + "We use the same lens galaxy mass model as the previous tutorials (an `Isothermal` plus `ExternalShear`, the true\n", + "model of the simulated data) and a source galaxy whose `Pixelization` pairs the `RectangularUniform` mesh with\n", + "`Constant` regularization (whose role appears later, when we reach the matrix $H$)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=1.6,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", + " ),\n", + " shear=al.mp.ExternalShear(gamma_1=0.05, gamma_2=0.05),\n", + ")\n", + "\n", + "pixelization = al.Pixelization(\n", + " mesh=al.mesh.RectangularUniform(shape=mesh_shape),\n", + " regularization=al.reg.Constant(coefficient=1.0),\n", + ")\n", + "\n", + "source_galaxy = al.Galaxy(redshift=1.0, pixelization=pixelization)\n", + "\n", + "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "Every 2D (y,x) image-plane coordinate $\\theta$ is ray-traced to its source-plane coordinate $\\beta$ by subtracting\n", + "the deflection angles $\\alpha$ of the mass profiles, via the lens equation we met in chapter 1:\n", + "\n", + " $\\beta = \\theta - \\alpha(\\theta)$\n", + "\n", + "The pixelization uses its own grid of coordinates, `masked_dataset.grids.pixelization`, with one coordinate at the\n", + "centre of every masked image pixel (because we disabled over sampling above). We trace this grid to the source-plane,\n", + "taking the final entry of the traced grid list (the source-plane)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "traced_grid_pixelization = tracer.traced_grid_2d_list_from(\n", + " grid=masked_dataset.grids.pixelization\n", + ")[-1]\n", + "\n", + "aplt.plot_grid(grid=traced_grid_pixelization, title=\"Traced Source-Plane Grid\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Border Relocation__\n", + "\n", + "Coordinates which pass close to the mass profile's centre are heavily demagnified and can trace to the far outskirts\n", + "of the source-plane.\n", + "\n", + "We relocate these coordinates to the edge of the source-plane border (defined via the border of the image-plane\n", + "mask), exactly as tutorial 6 explains in detail (see also figure 2 of https://arxiv.org/abs/1708.07377). This stops\n", + "a handful of demagnified coordinates stretching the mesh over a huge, empty region of the source-plane." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "from autoarray.inversion.mesh.border_relocator import BorderRelocator\n", + "\n", + "border_relocator = BorderRelocator(mask=masked_dataset.mask, sub_size=1)\n", + "\n", + "relocated_grid = border_relocator.relocated_grid_from(grid=traced_grid_pixelization)\n", + "\n", + "aplt.plot_grid(grid=relocated_grid, title=\"Relocated Source-Plane Grid\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Source Pixel Centres__\n", + "\n", + "To reconstruct the source on a `RectangularUniform` mesh, we need the centres of its rectangular source pixels.\n", + "\n", + "We compute these by overlaying a uniform rectangular grid over the relocated traced grid, sized so the mesh fully\n", + "contains the region of the source-plane the traced image-pixels cover, without wasted edge pixels beyond it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "from autoarray.inversion.mesh.mesh.rectangular_adapt_density import overlay_grid_from\n", + "\n", + "mesh_grid = overlay_grid_from(\n", + " shape_native=mesh_shape, grid=al.Grid2DIrregular(relocated_grid)\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Interpolation__\n", + "\n", + "We now combine the two grids to create an `Interpolator`, which describes how every traced image-pixel coordinate\n", + "maps to the rectangular mesh pixels.\n", + "\n", + "For a rectangular mesh the scheme is bilinear interpolation: every image pixel maps to the rectangular pixel it\n", + "lands in *and* its three nearest neighbours, with weights set by how close the coordinate is to each pixel centre.\n", + "Interpolation is what lets the mesh reconstruct smooth source morphologies. We can print the mappings and weights of\n", + "the first image pixel to confirm it pairs with four source pixels." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "interpolator = pixelization.mesh.interpolator_from(\n", + " source_plane_data_grid=relocated_grid,\n", + " source_plane_mesh_grid=mesh_grid,\n", + ")\n", + "\n", + "print(interpolator.mappings[0])\n", + "print(interpolator.weights[0])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mapper__\n", + "\n", + "The interpolator creates a `Mapper`, the object tutorials 1 and 2 introduced. It packages up the mapping between\n", + "every image pixel and every rectangular source pixel.\n", + "\n", + "The key attribute is `pix_indexes_for_sub_slim_index`, mapping every image-pixel index (`sub_slim_index`) to the\n", + "source-pixel indexes (`pix_indexes`) it interpolates onto, alongside the number of mappings per image pixel and\n", + "their interpolation weights." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mapper = al.Mapper(interpolator=interpolator)\n", + "\n", + "pix_indexes_for_sub_slim_index = mapper.pix_indexes_for_sub_slim_index\n", + "\n", + "print(pix_indexes_for_sub_slim_index[0:9])\n", + "print(mapper.pix_sizes_for_sub_slim_index[0:9])\n", + "print(mapper.pix_weights_for_sub_slim_index[0:9])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mapping Matrix__\n", + "\n", + "The `mapping_matrix` expresses these image-pixel to source-pixel mappings as a single 2D matrix, with\n", + "dimensions `(total_image_pixels, total_source_pixels)`.\n", + "\n", + "Each column is the \"image\" of one source pixel: entries are the interpolation weights for image pixels which map to\n", + "that source pixel and zero everywhere else." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mapping_matrix = al.util.mapper.mapping_matrix_from(\n", + " pix_indexes_for_sub_slim_index=pix_indexes_for_sub_slim_index,\n", + " pix_size_for_sub_slim_index=mapper.pix_sizes_for_sub_slim_index,\n", + " pix_weights_for_sub_slim_index=mapper.pix_weights_for_sub_slim_index,\n", + " pixels=mapper.pixels,\n", + " total_mask_pixels=mapper.source_plane_data_grid.mask.pixels_in_mask,\n", + " slim_index_for_sub_slim_index=mapper.slim_index_for_sub_slim_index,\n", + " sub_fraction=mapper.over_sampler.sub_fraction,\n", + ")\n", + "\n", + "plt.imshow(mapping_matrix, aspect=(mapping_matrix.shape[1] / mapping_matrix.shape[0]))\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Because each column is an image of zeros and interpolation weights, we can plot any column as a 2D image showing all\n", + "image pixels its source pixel maps to. For a source pixel near the mesh centre these image pixels trace the multiple\n", + "images of that patch of the source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "source_pixel_index = 210\n", + "\n", + "array_2d = al.Array2D(\n", + " values=mapping_matrix[:, source_pixel_index], mask=masked_dataset.mask\n", + ")\n", + "\n", + "aplt.plot_array(array=array_2d, title=\"Image of Source Pixel 210\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Blurred Mapping Matrix__\n", + "\n", + "The imaging data is blurred by the telescope's PSF, so the model must be too. Since each column of the mapping\n", + "matrix is an image, we simply convolve each column with the PSF via 2D convolution.\n", + "\n", + "This produces the `blurred_mapping_matrix`, with the same dimensions `(total_image_pixels, total_source_pixels)`.\n", + "\n", + "In WD03 this matrix is denoted $f_{ij}$, where $i$ runs over all $I$ source pixels and $j$ over all $J$ image\n", + "pixels. For example:\n", + "\n", + " - $f_{0, 2} = 0.3$ indicates that image-pixel $2$ maps to source-pixel $0$ with a weight of $0.3$ after PSF\n", + " convolution.\n", + " - $f_{4, 8} = 0$ indicates that image-pixel $8$ does not map to source-pixel $4$, even after PSF convolution.\n", + "\n", + "(The indexing of the code's `mapping_matrix` is transposed relative to WD03's $f$: image pixels are the first index\n", + "in the code, but the second index in the equations.)\n", + "\n", + "PSF blurring has an important consequence: whereas before, no two source pixels shared an image pixel, blurring\n", + "correlates neighbouring columns, so the images of nearby source pixels now overlap." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "blurred_mapping_matrix = masked_dataset.psf.convolved_mapping_matrix_from(\n", + " mapping_matrix=mapping_matrix, mask=masked_dataset.mask\n", + ")\n", + "\n", + "plt.imshow(\n", + " blurred_mapping_matrix,\n", + " aspect=(blurred_mapping_matrix.shape[1] / blurred_mapping_matrix.shape[0]),\n", + ")\n", + "plt.colorbar()\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Data Vector (D)__\n", + "\n", + "We now pose the reconstruction as a linear inversion, converting the blurred mapping matrix, data and noise-map\n", + "into two objects: the data vector $D$ and the curvature matrix $F$.\n", + "\n", + "The data vector has dimensions `(total_source_pixels,)` and is given by (WD03 / N15):\n", + "\n", + " $\\vec{D}_{i} = \\sum_{j=1}^{J} f_{ij} (d_{j} - b_{j}) / \\sigma_{j}^2 \\, \\, .$\n", + "\n", + "Where:\n", + "\n", + " - $d_{j}$ are the image-pixel data values.\n", + " - $b_{j}$ are the lens light model values ($d_{j} - b_{j}$ is the lens-subtracted image; zero for this dataset).\n", + " - $\\sigma_{j}^2$ are the statistical uncertainties of each image pixel.\n", + "\n", + "Each entry of $D$ is therefore the noise-weighted overlap between one source pixel's blurred image and the data:\n", + "it measures how much evidence the data provides for flux in that source pixel, with the PSF fully accounted for." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "data_vector = al.util.inversion_imaging.data_vector_via_blurred_mapping_matrix_from(\n", + " blurred_mapping_matrix=blurred_mapping_matrix,\n", + " image=np.array(masked_dataset.data),\n", + " noise_map=np.array(masked_dataset.noise_map),\n", + ")\n", + "\n", + "plt.imshow(\n", + " data_vector.reshape(data_vector.shape[0], 1), aspect=10.0 / data_vector.shape[0]\n", + ")\n", + "plt.colorbar()\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Curvature Matrix (F)__\n", + "\n", + "The curvature matrix has dimensions `(total_source_pixels, total_source_pixels)` and is given by (WD03):\n", + "\n", + " ${F}_{ik} = \\sum_{j=1}^{J} f_{ij} f_{kj} / \\sigma_{j}^2 \\, \\, .$\n", + "\n", + "Every entry of $F$ is the noise-weighted overlap between the blurred images of two source pixels: $F_{ik}$ sums the\n", + "product of columns $i$ and $k$ of $f$ over all image pixels. For $F_{ik}$ to be non-zero, source pixels $i$ and $k$\n", + "must share at least one image pixel, which (interpolation aside) only happens because of PSF blurring.\n", + "\n", + "$F$ describes how degenerate pairs of source pixels are with one another: two source pixels whose blurred images\n", + "overlap heavily can trade flux between themselves whilst fitting the data almost equally well." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "curvature_matrix = al.util.inversion.curvature_matrix_via_mapping_matrix_from(\n", + " mapping_matrix=blurred_mapping_matrix, noise_map=masked_dataset.noise_map\n", + ")\n", + "\n", + "plt.imshow(curvature_matrix)\n", + "plt.colorbar()\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Unregularized Solve__\n", + "\n", + "The inversion seeks the source-pixel fluxes $s$ (a vector with one entry per source pixel) that minimize the\n", + "chi-squared:\n", + "\n", + " $\\chi^2 = \\sum_{j=1}^{J} \\bigg[ \\frac{(\\sum_{i=1}^{I} s_{i} f_{ij}) + b_{j} - d_{j}}{\\sigma_{j}} \\bigg]^2$\n", + "\n", + "Setting the derivative of $\\chi^2$ with respect to each $s_{i}$ to zero gives the linear system whose solution is\n", + "(equation 5 of WD03):\n", + "\n", + " $s = F^{-1} D$\n", + "\n", + "We can solve this directly with NumPy. (Without regularization the curvature matrix is often singular, so the loop\n", + "below adds a tiny value to its diagonal to avoid a `LinAlgError` -- it is a numerical crutch, not part of the\n", + "formalism.)" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for i in range(curvature_matrix.shape[0]):\n", + " curvature_matrix[i, i] += 1e-8\n", + "\n", + "reconstruction = np.linalg.solve(curvature_matrix, data_vector)\n", + "\n", + "print(reconstruction)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The reconstructed source-pixel fluxes are a noisy, unsmooth mess -- exactly the over-fitting we saw in tutorial 4\n", + "when we set the regularization coefficient to zero. The linear inversion is fitting the noise in the data, because\n", + "this system of equations is ill-posed: we need a smoothness prior.\n", + "\n", + "__Regularization Matrix (H)__\n", + "\n", + "Regularization adds a linear regularization term $G_{L}$ to the merit function we minimize (equation 11 of WD03):\n", + "\n", + " $G = \\chi^2 + \\lambda \\, G_{L}$\n", + "\n", + "where $\\lambda$ is the `regularization_coefficient` controlling the degree of smoothing. The `Constant` scheme uses\n", + "gradient regularization (equation 14 of WD03):\n", + "\n", + " $G_{L} = \\sum_{i}^{I} \\sum_{n=1}^{N} [s_{i} - s_{i, n}]^2$\n", + "\n", + "In words: for every source pixel, compare its flux with each of its $N$ neighbours $n$, and penalize solutions where\n", + "the differences are large. This is precisely the \"smoothness prior\" of tutorial 4, now written as an equation.\n", + "\n", + "To fold this into the linear algebra we define the regularization matrix $H$, with\n", + "dimensions `(total_source_pixels, total_source_pixels)` (equation 13 of WD03):\n", + "\n", + " $H_{ik} = \\frac{1}{2} \\frac{\\partial^{2} G_{L}}{\\partial s_{i} \\partial s_{k}}$\n", + "\n", + "$H$ has the coefficient $\\lambda$ folded into it. Its non-zero off-diagonal entries mark pairs of source pixels\n", + "which are neighbours and therefore regularized with one another; most entries are zero because most source pixels\n", + "are not neighbours." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "regularization_matrix = al.util.regularization.constant_regularization_matrix_from(\n", + " coefficient=source_galaxy.pixelization.regularization.coefficient,\n", + " neighbors=mapper.neighbors,\n", + " neighbors_sizes=mapper.neighbors.sizes,\n", + ")\n", + "\n", + "plt.imshow(regularization_matrix)\n", + "plt.colorbar()\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Source Reconstruction (s)__\n", + "\n", + "$H$ enters the linear system as follows (equation 12 of WD03):\n", + "\n", + " $s = [F + H]^{-1} D$\n", + "\n", + "We add the two matrices and solve again. The diagonal jitter used above is no longer needed, because $H$ makes the\n", + "system well-posed." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "curvature_reg_matrix = np.add(curvature_matrix, regularization_matrix)\n", + "\n", + "reconstruction = np.linalg.solve(curvature_reg_matrix, data_vector)\n", + "\n", + "print(reconstruction)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The reconstructed fluxes are now smooth and physical: regularization has suppressed the noisy solution and the\n", + "reconstruction actually looks like a galaxy, without over-fitting the noise.\n", + "\n", + "__Image Reconstruction__\n", + "\n", + "Using the reconstructed source-pixel fluxes, we map the source back to the image-plane via the blurred mapping\n", + "matrix (so the reconstructed image includes PSF blurring) to produce the model image of the lensed source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mapped_reconstructed_data = (\n", + " al.util.inversion.mapped_reconstructed_data_via_mapping_matrix_from(\n", + " mapping_matrix=blurred_mapping_matrix, reconstruction=reconstruction\n", + " )\n", + ")\n", + "\n", + "mapped_reconstructed_data = al.Array2D(\n", + " values=mapped_reconstructed_data, mask=masked_dataset.mask\n", + ")\n", + "\n", + "aplt.plot_array(array=mapped_reconstructed_data, title=\"Reconstructed Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Likelihood Function__\n", + "\n", + "We now quantify the goodness-of-fit of the source reconstruction, computing the quantity tutorial 4 called the\n", + "Bayesian evidence. The log evidence consists of five terms:\n", + "\n", + " $-2 \\, \\mathrm{ln} \\, \\epsilon = \\chi^2 + s^{T} H s + \\mathrm{ln} \\, [ \\mathrm{det} (F + H) ] - \\mathrm{ln} \\, [ \\mathrm{det} (H) ] + \\sum_{j=1}^{J} \\mathrm{ln} \\, [2 \\pi (\\sigma_{j})^2 ] \\, .$\n", + "\n", + "This expression was first derived by Suyu et al. 2006 (https://arxiv.org/abs/astro-ph/0601493), equation (19), and\n", + "is given in **PyAutoLens** notation by Dye et al. 2008 (https://arxiv.org/abs/0804.4002), equation (5).\n", + "\n", + "We now compute each term in turn.\n", + "\n", + "__Chi Squared__\n", + "\n", + "The first term is the $\\chi^2$ statistic from the merit function above, computed as:\n", + "\n", + " - `model_data` = the reconstructed image of the lensed source (plus the lens light model, zero here).\n", + " - `residual_map` = (`data` - `model_data`)\n", + " - `normalized_residual_map` = (`data` - `model_data`) / `noise_map`\n", + " - `chi_squared_map` = (`normalized_residual_map`) ** 2.0\n", + " - `chi_squared` = sum(`chi_squared_map`)\n", + "\n", + "High chi-squared values indicate image pixels the reconstruction fits poorly, lowering the likelihood." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "model_image = mapped_reconstructed_data\n", + "\n", + "residual_map = masked_dataset.data - model_image\n", + "normalized_residual_map = residual_map / masked_dataset.noise_map\n", + "chi_squared_map = normalized_residual_map**2.0\n", + "\n", + "chi_squared = np.sum(chi_squared_map)\n", + "\n", + "print(chi_squared)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Regularization Term__\n", + "\n", + "The second term, $s^{T} H s$, is the $\\lambda \\, G_{L}$ regularization penalty evaluated at the solution: the summed\n", + "difference in flux between all neighbouring source pixels, weighted by the regularization coefficient (which is\n", + "already folded into $H$).\n", + "\n", + "Less smooth solutions have larger values of this term and therefore lower likelihoods." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "regularization_term = np.matmul(\n", + " reconstruction.T, np.matmul(regularization_matrix, reconstruction)\n", + ")\n", + "\n", + "print(regularization_term)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Complexity Terms__\n", + "\n", + "Up to this point, nothing has justified our choice of `regularization_coefficient=1.0`. We cannot choose it using\n", + "the two terms above, because increasing the coefficient smooths the solution more, which *both* worsens the\n", + "chi-squared *and* (for a fixed solution) raises the regularization penalty. Optimizing those two terms alone would\n", + "drive the coefficient to zero and put us right back at the over-fitted mess.\n", + "\n", + "The two log determinant terms, $\\mathrm{ln} \\, [ \\mathrm{det} (F + H) ]$ and $- \\mathrm{ln} \\, [ \\mathrm{det} (H) ]$,\n", + "fix this. Together they measure how *complex* the source reconstruction is -- roughly, how many effective degrees\n", + "of freedom the source uses after regularization correlates its pixels -- and penalize more complex solutions.\n", + "Lowering the regularization coefficient frees the source to use more of its flexibility, increasing this complexity\n", + "penalty.\n", + "\n", + "These terms therefore counteract the chi-squared and regularization terms, so the highest evidence goes to solutions\n", + "which fit the data well with the *simplest* source reconstruction. This is the Occam's razor behaviour that\n", + "tutorial 4 demonstrated empirically." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "log_curvature_reg_matrix_term = np.linalg.slogdet(curvature_reg_matrix)[1]\n", + "log_regularization_matrix_term = np.linalg.slogdet(regularization_matrix)[1]\n", + "\n", + "print(log_curvature_reg_matrix_term)\n", + "print(log_regularization_matrix_term)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Noise Normalization Term__\n", + "\n", + "The likelihood function assumes the imaging data consists of independent Gaussian noise in every image pixel, and\n", + "the final term is the normalization of those Gaussians: the sum of the log of every noise-map value squared.\n", + "\n", + "Because the noise-map is fixed, this term is constant throughout lens modeling and has no impact on the model\n", + "we infer -- it simply normalizes the likelihood." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "noise_normalization = float(np.sum(np.log(2 * np.pi * masked_dataset.noise_map**2.0)))\n", + "\n", + "print(noise_normalization)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Log Evidence__\n", + "\n", + "We can now combine the five terms into the log evidence of the source reconstruction." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "log_evidence = float(\n", + " -0.5\n", + " * (\n", + " chi_squared\n", + " + regularization_term\n", + " + log_curvature_reg_matrix_term\n", + " - log_regularization_matrix_term\n", + " + noise_normalization\n", + " )\n", + ")\n", + "\n", + "print(log_evidence)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Fit__\n", + "\n", + "Everything above is what the `FitImaging` object does internally when it fits a tracer whose source galaxy has a\n", + "pixelization. We can see this by performing the fit and comparing its `log_evidence` to ours.\n", + "\n", + "The two values are close but not identical, because the real fit improves on our simplified solve in two ways\n", + "mentioned along the way: it uses the positive-only solver (tutorial 3), which forbids the negative source-pixel\n", + "fluxes our unconstrained `np.linalg.solve` permits, and it zeroes the pixels at the edge of the mesh. Neither\n", + "changes the formalism -- the same $f$, $D$, $F$ and $H$ feed a solver with extra constraints." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "fit = al.FitImaging(\n", + " dataset=masked_dataset,\n", + " tracer=tracer,\n", + " settings=al.Settings(use_border_relocator=True),\n", + ")\n", + "\n", + "print(fit.log_evidence)\n", + "\n", + "aplt.subplot_fit_imaging(fit=fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Wrap Up__\n", + "\n", + "We have walked through the complete linear algebra of a pixelized source reconstruction:\n", + "\n", + " - The `mapping_matrix` and PSF-blurred mapping matrix $f$, whose columns are the blurred images of each\n", + " source pixel.\n", + "\n", + " - The data vector $D$ and curvature matrix $F$, the noise-weighted overlaps of those images with the data and with\n", + " each other.\n", + "\n", + " - The regularization matrix $H$, which encodes the smoothness prior, and the linear solve $s = [F + H]^{-1} D$ for\n", + " the source reconstruction.\n", + "\n", + " - The five terms of the Suyu et al. 2006 log evidence -- chi-squared, the regularization penalty, the two log\n", + " determinant complexity terms and the noise normalization -- and their Bayesian interpretation as an Occam's\n", + " razor which favours the simplest source reconstruction the data allows.\n", + "\n", + "During lens modeling, this whole calculation is one likelihood evaluation: the non-linear search varies the mass\n", + "model (and, later in this chapter, the pixelization and regularization parameters), and each sample triggers the\n", + "full ray-trace, solve and evidence computation above.\n", + "\n", + "Two simplifications are worth remembering: real fits use over sampling (each image pixel contributes several\n", + "sub-pixel rows to $f$) and **PyAutoLens** uses a positive-only solver for $s$ rather than the unconstrained\n", + "`np.linalg.solve` used here (see tutorial 3). The workspace\n", + "guide `autolens_workspace/*/imaging/features/pixelization/likelihood_function.ipynb` repeats this walk-through with\n", + "lens light included and additional visualization of every step.\n", + "\n", + "In the next tutorial, we return to hands-on territory and look at borders, which deal with the demagnified traced\n", + "coordinates whose relocation we performed in a single line above." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_4_pixelizations/tutorial_5_borders.ipynb b/notebooks/chapter_3_pixelizations/tutorial_6_borders.ipynb similarity index 87% rename from notebooks/chapter_4_pixelizations/tutorial_5_borders.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_6_borders.ipynb index 440b2bb..6c1a7f4 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_5_borders.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_6_borders.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 5: Borders\n", + "Tutorial 6: Borders\n", "===================\n", "\n", "In the previous tutorials, the source-plane pixel grid perfectly mapped over the traced image-pixel $(y,x)$ coordinates\n", @@ -12,7 +12,8 @@ "increase its size so as to cover every source-plane coordinate.\n", "\n", "In this tutorial, we will consider how the size of the pixelization grid is chosen and introduce the concept of a\n", - "border.\n", + "border. The previous tutorial applied border relocation as a single step of the likelihood algebra -- here we see\n", + "why that step is needed.\n", "\n", "__Contents__\n", "\n", @@ -66,7 +67,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -143,7 +144,7 @@ "\n", "So, what is a border? In the image-plane, a border is the set of exterior pixels in a mask that are at its border.\n", "\n", - "Lets plot the image with a circular circular and tell our `aplt.subplot_imaging_dataset` to plot the border." + "Lets apply a circular mask to the image and plot the dataset -- the border is the ring of pixels at the mask's edge." ] }, { @@ -169,8 +170,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As you can see, for a circular mask the border *is* the edge of the mask (the ring of black dots we're used to \n", - "seeing whenever we plot a mask. \n", + "As you can see, for a circular mask the border *is* the edge of the mask (the ring of black dots we're used to\n", + "seeing whenever we plot a mask).\n", "\n", "For an annular mask, pixels on its inner edge are not a part of the border, whereas those on its outer edge are." ] @@ -240,10 +241,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Okay, so lets first look at the mapper without using a border and using annular mask.\n", + "Okay, so lets first look at the mapper without using a border, using the annular mask.\n", "\n", - "First, note how we set up the border, using a `al.Settings` object. This behaves analogously to the \n", - "`SettingsLens` objects we have used in previous tutorials." + "Note how we control the border via the `al.Settings` object's `use_border_relocator` input." ] }, { @@ -331,12 +331,12 @@ "So, why is this happening? What is the mass profile physically doing to create these source plane coordinates at \n", "extremely large radial values? \n", "\n", - "Towards the centre of th elliptical isothermal mass profile, the density begins to rise very sharply, it becomes \n", - "extremely steep or 'cuspy'. This cuspy behaviour towards its centre can cause extremely large deflection angles to be \n", - "calculated:\n", + "Towards the centre of the elliptical isothermal mass profile, the density begins to rise very sharply, it becomes\n", + "extremely steep or 'cuspy'. This cuspy behaviour towards its centre causes extremely large deflection angles to be\n", + "calculated.\n", "\n", - "Central image pixel can therefore be subjected to 'demagnification', whereby they trace to extremely large values in \n", - "the source plane! \n", + "Central image pixels are therefore subjected to 'demagnification', whereby they trace to extremely large values in\n", + "the source plane!\n", "\n", "Physically, this is not a problem, and it is the reason we do not see a 'central image' in most strong lenses, as the \n", "light-rays which take this path through the centre of the lens are demagnified. However, if the lens galaxy had a less\n", @@ -345,15 +345,15 @@ "Demagnification is a problem for the pixelization and inversion though, which reconstruct the flux of these \n", "demagnified pixels just like the other pixels in the image-pixel. There are two negative consequences:\n", "\n", - " 1) The rectangular pixel-grid that we 'lay over' the source-plane is very larger because it expands to include the \n", - " demagnified image-pixels. As a result, larger source-pixels are used to reconstruct the central regions of the \n", - " source-plane (where the source galaxy is actually located), meaning we reconstruct the source-galaxy at a lower \n", + " 1) The rectangular pixel-grid that we 'lay over' the source-plane becomes very large, because it expands to include\n", + " the demagnified image-pixels. As a result, larger source-pixels are used to reconstruct the central regions of the\n", + " source-plane (where the source galaxy is actually located), meaning we reconstruct the source-galaxy at a lower\n", " effective resolution.\n", - " \n", - " 2) The inversion reconstructs the flux of the demanigified image pixels using source-pixels which contain *only* \n", - " demagnified image pixels (these are the source pixels at the edge of the source plane). These source-pixels *should* \n", - " have had other image-pixels traced within them via image-pixels at even larger radii from the centre of the lens \n", - " galaxy. However, these image-pixels are at radii above 3.0\", meaning the circular mask removed them from the inversion.\n", + "\n", + " 2) The inversion reconstructs the flux of the demagnified image pixels using source-pixels which contain *only*\n", + " demagnified image pixels (these are the source pixels at the edge of the source plane). These source-pixels *should*\n", + " have had other image-pixels traced within them via image-pixels at even larger radii from the centre of the lens\n", + " galaxy. However, these image-pixels are at radii above 2.5\", meaning the circular mask removed them from the inversion.\n", "\n", "Lets quickly use a large circular mask to confirm that these pixels exist when we don't mask them." ] @@ -371,7 +371,7 @@ "fit = perform_fit_with_source_galaxy_mask_and_border(\n", " dataset=dataset_unmasked,\n", " source_galaxy=source_galaxy,\n", - " mask=mask_circular,\n", + " mask=mask_circular_large,\n", " settings=al.Settings(use_border_relocator=False),\n", ")\n", "\n", @@ -432,7 +432,7 @@ "and lensing configurations produce nasty edge effects where image pixels not just in the centre of mask, but anywhere \n", "in the mask, trace beyond the source-plane border.\n", "\n", - "we'll use new strong lensing data as the previous tutorial, where:\n", + "we'll use new strong lensing data, where:\n", "\n", " - The lens galaxy's light is omitted.\n", " - There are two lens galaxies whose `MassProfile`'s are `Isothermal`.\n", @@ -476,7 +476,7 @@ " data_path=dataset_path / \"data.fits\",\n", " noise_map_path=dataset_path / \"noise_map.fits\",\n", " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.05,\n", + " pixel_scales=0.1,\n", ")" ], "outputs": [], @@ -486,7 +486,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We again must define a mask around this image, lets start with a 2.8\" mask. we'll use larger masks to illustrate the\n", + "We again must define a mask around this image, lets start with a 2.8\" mask. we'll vary the mask size to illustrate the\n", "effects of the border in a moment." ] }, @@ -540,7 +540,7 @@ " sersic_index=2.5,\n", " ),\n", " mass=al.mp.Isothermal(\n", - " centre=(1.1, 0.51), ell_comps=(0.0, 0.15), einstein_radius=1.07\n", + " centre=(0.0, -1.0), ell_comps=(0.17647, 0.0), einstein_radius=1.0\n", " ),\n", " )\n", "\n", @@ -554,7 +554,7 @@ " sersic_index=3.0,\n", " ),\n", " mass=al.mp.Isothermal(\n", - " centre=(-0.20, -0.35), ell_comps=(0.06, 0.1053), einstein_radius=0.71\n", + " centre=(0.0, 1.0), ell_comps=(0.0, -0.111111), einstein_radius=0.8\n", " ),\n", " )\n", "\n", @@ -586,10 +586,6 @@ "\n", "mapper = fit.inversion.cls_list_from(al.Mapper)[0]\n", "\n", - "border = mapper.source_plane_data_grid.over_sampled[\n", - " fit.dataset.grids.border_relocator.sub_border_slim\n", - "]\n", - "\n", "\n", "aplt.plot_array(array=fit.model_data, title=\"Plane 1 Image\")" ], @@ -617,10 +613,6 @@ "\n", "mapper = fit.inversion.cls_list_from(al.Mapper)[0]\n", "\n", - "border = mapper.source_plane_data_grid.over_sampled[\n", - " fit.dataset.grids.border_relocator.sub_border_slim\n", - "]\n", - "\n", "aplt.plot_array(array=fit.model_data, title=\"Plane 1 Image\")" ], "outputs": [], @@ -630,15 +622,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Multi-galaxy modeling is rife for border effects and if you have multiple lens galaxies I heartily recommend you pay \n", - "a close eye to your source-plane borders!\n", + "Multi-galaxy modeling is rife with border effects, and if you have multiple lens galaxies I heartily recommend you\n", + "keep a close eye on your source-plane borders!\n", "\n", - "Care must also be taken when choosing the size of your mask. If you don't choose a big enough mask, the border won't \n", - "be able to relocate all of the demanigified image pixels to the border edge.\n", + "Care must also be taken when choosing the size of your mask. If you don't choose a big enough mask, the border won't\n", + "be able to relocate all of the demagnified image pixels to the border edge.\n", "\n", - "(The figures below look pretty horrible, because every ray-traced image coordinate is being plotted in the \n", - "source plane. Therefore, there are many black dots which overwhelm the figure. The point to focus on are the\n", - "edges of the grid, where one can see the relocations of these coordinates." + "The fits below repeat the analysis with progressively larger circular masks, so you can see how the mask size\n", + "changes the border and the resulting model image." ] }, { @@ -665,10 +656,6 @@ "\n", "mapper = fit.inversion.cls_list_from(al.Mapper)[0]\n", "\n", - "border = mapper.source_plane_data_grid.over_sampled[\n", - " fit.dataset.grids.border_relocator.sub_border_slim\n", - "]\n", - "\n", "\n", "aplt.plot_array(array=fit.model_data, title=\"Plane 1 Image\")\n", "\n", @@ -686,10 +673,6 @@ "\n", "mapper = fit.inversion.cls_list_from(al.Mapper)[0]\n", "\n", - "border = mapper.source_plane_data_grid.over_sampled[\n", - " fit.dataset.grids.border_relocator.sub_border_slim\n", - "]\n", - "\n", "\n", "aplt.plot_array(array=fit.model_data, title=\"Plane 1 Image\")\n", "\n", @@ -714,10 +697,6 @@ "\n", "mapper = fit.inversion.cls_list_from(al.Mapper)[0]\n", "\n", - "border = mapper.source_plane_data_grid.over_sampled[\n", - " fit.dataset.grids.border_relocator.sub_border_slim\n", - "]\n", - "\n", "\n", "aplt.plot_array(array=fit.model_data, title=\"Plane 1 Image\")\n", "\n", @@ -743,10 +722,6 @@ "\n", "mapper = fit.inversion.cls_list_from(al.Mapper)[0]\n", "\n", - "border = mapper.source_plane_data_grid.over_sampled[\n", - " fit.dataset.grids.border_relocator.sub_border_slim\n", - "]\n", - "\n", "\n", "aplt.plot_array(array=fit.model_data, title=\"Plane 1 Image\")" ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_6_lens_modeling.ipynb b/notebooks/chapter_3_pixelizations/tutorial_7_lens_modeling.ipynb similarity index 88% rename from notebooks/chapter_4_pixelizations/tutorial_6_lens_modeling.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_7_lens_modeling.ipynb index 2489d0e..ccebcce 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_6_lens_modeling.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_7_lens_modeling.ipynb @@ -4,19 +4,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 6: Lens Modeling\n", + "Tutorial 7: Lens Modeling\n", "=========================\n", "\n", - "When modeling complex source's with parametric profiles, we quickly entered a regime where our non-linear search was\n", + "When modeling complex sources with parametric profiles, we quickly entered a regime where our non-linear search was\n", "faced with a parameter space of dimensionality N=20+ parameters. This made the model-fitting inefficient and likely to\n", "infer a local maxima.\n", "\n", - "Inversions do not suffer this problem, meaning they are a very a powerful tool for modeling strong lenses. Furthermore,\n", - "they have *more* freemdom than parametric light profiles because they do not relying on specific analytic light\n", + "Inversions do not suffer this problem, meaning they are a very powerful tool for modeling strong lenses. Furthermore,\n", + "they have *more* freedom than parametric light profiles because they do not rely on specific analytic light\n", "distributions and a symmetric profile shape. This will allow us to fit more complex mass models and ask ever more\n", "interesting scientific questions!\n", "\n", - "However, inversion do have some short comings that we need to be aware of before we use them for lens modeling. That`s\n", + "However, inversions do have some shortcomings that we need to be aware of before we use them for lens modeling. That's\n", "what we cover in this tutorial.\n", "\n", "__Contents__\n", @@ -73,7 +73,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -185,8 +185,8 @@ "The code below illustrates a systematic set of solutions called demagnified solutions, which negatively impact\n", "lens modeling using source pixelizations.\n", "\n", - "Since writing the code below, I have wrote a full readthedocs page illustrating the issue, which is linked too below.\n", - "I recommend you read this page first, to understand what a demagnified solution is, why its a problem and how we\n", + "Since writing the code below, I have written a full readthedocs page illustrating the issue, which is linked to below.\n", + "I recommend you read this page first, to understand what a demagnified solution is, why it is a problem and how we\n", "fix it. The code below should then build on this.\n", "\n", " https://pyautolens.readthedocs.io/en/latest/general/demagnified_solutions.html\n", @@ -236,9 +236,9 @@ "What happened!? This incorrect mass-model provides a really good fit to the image! The residuals and chi-squared-map \n", "are as good as the ones we saw in the previous tutorials.\n", "\n", - "How can an incorrect lens model provide such a fit? Well, as I'm sure you noticed, the source has been reconstructed \n", - "as a demagnified version of the image. Clearly, this is not a physical solution or a solution that we want our \n", - "non-linear search to find, but for inversion's the reality is these solutions eixst.\n", + "How can an incorrect lens model provide such a fit? Well, as I'm sure you noticed, the source has been reconstructed\n", + "as a demagnified version of the image. Clearly, this is not a physical solution or a solution that we want our\n", + "non-linear search to find, but for inversions the reality is these solutions exist.\n", "\n", "This is not necessarily problematic for lens modeling. Afterall, the source reconstruction above is extremely complex, \n", "it requires a lot of source pixels to fit the image accurately and its lack of smoothness will be heavily penalized\n", @@ -286,19 +286,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The `log_evidence` *is* lower. However, the difference in `log_evidence` is not *that large*. This could be a problem \n", - "for the non-linear search, as it will see many solutions in parameter space with high `log_evidence` values. Furthermore, \n", - "these solutions occupy a *large volumne* of parameter space (e.g. everywhere the lens model that is wrong). This makes \n", - "it easy for the non-linear search to get lost searching through these unphysical solutions and, unfortunately, inferring \n", + "The `log_evidence` *is* lower. However, the difference in `log_evidence` is not *that large*. This could be a problem\n", + "for the non-linear search, as it will see many solutions in parameter space with high `log_evidence` values. Furthermore,\n", + "these solutions occupy a *large volume* of parameter space (e.g. everywhere the lens model is wrong). This makes\n", + "it easy for the non-linear search to get lost searching through these unphysical solutions and, unfortunately, inferring\n", "an incorrect lens model (e.g. a local maxima).\n", "\n", - "There is no simple fix for this, and it is the price we pay for making the inversion has so much flexibility in how it\n", - "reconstructs the source's light. The solution to this problem? Search chaining. In fact, this is the problem that lead\n", - "us to initially conceive of search chaining! \n", + "There is no simple fix for this, and it is the price we pay for giving the inversion so much flexibility in how it\n", + "reconstructs the source's light. The solution to this problem? Search chaining, which we covered at the end of\n", + "chapter 2 (tutorials 9-11). In fact, this is the problem that led us to initially conceive of search chaining!\n", "\n", "The idea is simple, we write a pipeline that begins by modeling the source galaxy's light using a light profile, thereby\n", - "initializing the priors for the lens galaxy's light and mass. Then, when we switch to an `Inversion` in the next \n", - "search, the mass model starts in the correct regions of parameter space and does not get lost sampling these \n", + "initializing the priors for the lens galaxy's light and mass. Then, when we switch to an `Inversion` in the next\n", + "search, the mass model starts in the correct regions of parameter space and does not get lost sampling these\n", "incorrect solutions.\n", "\n", "The following paper discusses these solutions in more detail (https://arxiv.org/abs/2012.04665).\n", @@ -309,8 +309,8 @@ "simultaneously fit and subtract the lens galaxy's light using a light profile whilst reconstructing the source's\n", "light using an inversion. \n", "\n", - "To do this, all we have to do is give the lens galaxy a light profile and use the tracer and fit objects we are used \n", - "too:." + "To do this, all we have to do is give the lens galaxy a light profile and use the tracer and fit objects we are\n", + "used to." ] }, { @@ -342,9 +342,7 @@ " radius=2.5,\n", ")\n", "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)\n", - "\n", - "aplt.subplot_fit_imaging_of_planes(fit=fit)" + "aplt.subplot_imaging_dataset(dataset=dataset)" ], "outputs": [], "execution_count": null @@ -468,13 +466,13 @@ "source": [ "__Wrap Up__\n", "\n", - "And with that, we're done. I'll end by pointing out a few things about what we've covered to get you thinking about \n", - "the next tutorial on adaption.\n", - " \n", - " - When the lens galaxy's light is subtracted perfectly it leaves no residuals. However, if it isn't subtracted \n", - " perfectly it does leave residuals, which will be fitted by the inversion. If the residual are significant this is \n", - " going to impact the source reconstruction negatively and can lead to some pretty nasty systematics. In the next \n", - " chapter, we'll learn how our adaptive analysis can prevent this residual fitting." + "And with that, we're done. I'll end by pointing out a few things about what we've covered to get you thinking about\n", + "the adaption tutorials later in this chapter.\n", + "\n", + " - When the lens galaxy's light is subtracted perfectly it leaves no residuals. However, if it isn't subtracted\n", + " perfectly it does leave residuals, which will be fitted by the inversion. If the residuals are significant this is\n", + " going to impact the source reconstruction negatively and can lead to some pretty nasty systematics. In the adaption\n", + " tutorials at the end of this chapter, we'll learn how our adaptive analysis can prevent this residual fitting." ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb b/notebooks/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.ipynb similarity index 89% rename from notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.ipynb index 88a136f..3cce15b 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 7: Adaptive Pixelization\n", + "Tutorial 8: Adaptive Pixelization\n", "=================================\n", "\n", "In this tutorial we will introduce a new `Pixelization` object, which uses an `Overlay` image-mesh and a `Delaunay`\n", @@ -17,7 +17,7 @@ "__Contents__\n", "\n", "- **Initial Setup:** We'll use the same strong lensing data as the previous tutorial, where.\n", - "- **Advantages and Disadvatanges:** Lets think about the rectangular pixelization.\n", + "- **Advantages and Disadvantages:** Lets think about the rectangular pixelization.\n", "- **Image Mesh:** The Delaunay mesh is an irregular grid of pixels (or triangles) in the source-plane.\n", "- **Regularization:** On the rectangular grid, we regularized each source pixel with its 4 neighbors.\n", "- **Wrap Up:** Summary of the script and next steps." @@ -78,7 +78,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -200,12 +200,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "__Advantages and Disadvatanges__\n", + "__Advantages and Disadvantages__\n", "\n", - "Lets think about the rectangular pixelization. Is this the optimal way to reconstruct our source? Are there features \n", - "in the source-plane that arn`t ideal? How do you think we could do a better job?\n", + "Lets think about the rectangular pixelization. Is this the optimal way to reconstruct our source? Are there features\n", + "in the source-plane that aren't ideal? How do you think we could do a better job?\n", "\n", - "There are a number of reasons the rectangular pixelization is not optimal, and is infact a pretty poor method to \n", + "There are a number of reasons the rectangular pixelization is not optimal, and it is in fact a pretty poor method to\n", "model strong lenses!\n", "\n", "So what is wrong with the grid? Well, lets think about the source reconstruction." @@ -227,10 +227,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "There is one clear problem, we are using only a small number of the total source pixels to reconstruct the source. The \n", - "majority of source pixels are located away from the source. By my estimate, we are using just 16 pixels (the central \n", - "4x4 grid) out of the 1600 pixels to actually fit the data! The remaining ~1500 pixels are doing nothing but fitting \n", - "noise. \n", + "There is one clear problem, we are using only a small number of the total source pixels to reconstruct the source. The\n", + "majority of source pixels are located away from the source. By my estimate, we are using just 16 pixels (the central\n", + "4x4 grid) out of the 784 pixels to actually fit the data! The remaining ~750 pixels are doing nothing but fitting\n", + "noise.\n", "\n", "This means that regularization is sub-optimal. In tutorial 4, we discussed how the Bayesian evidence of the \n", "regularization favours the simplest source solution. That is, the solution which fits the data using the fewest source \n", @@ -246,7 +246,7 @@ "__Image Mesh__\n", "\n", "The Delaunay mesh is an irregular grid of pixels (or triangles) in the source-plane. We must first therefore determine\n", - "a set of (y,x) source-plane coordinates defining this grid, specifically where each triangle vertex is loated.\n", + "a set of (y,x) source-plane coordinates defining this grid, specifically where each triangle vertex is located.\n", "\n", "We do this using an `image_mesh`, which defines a method to determine a set of coordinates in the image-plane \n", "which are ray-traced to the source-plane. These traced coordinates are the triangle vertexes of our source-pixel mesh. \n", @@ -272,7 +272,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can plot this grid over the image, to see that it is a coarse grid of (y,x) coordinates laid ove the image." + "We can plot this grid over the image, to see that it is a coarse grid of (y,x) coordinates laid over the image." ] }, { @@ -342,7 +342,7 @@ "source": [ "A closer inspection of the pixelization shows the improvement. \n", "\n", - "We are using fewer pixels than the rectangular grid (400, instead of 1600) and reconstructing the source is far \n", + "We are using fewer pixels than the rectangular grid (400, instead of 784) and reconstructing the source in far\n", "greater detail!" ] }, @@ -369,15 +369,15 @@ "\n", "__Wrap Up__\n", "\n", - "The `Overlay` image-mesh and `Delaunay` mesh is still far from optimal. There are lots of source-pixels effectively f\n", - "itting just noise. We can achieve even better solutions if the central regions of the source were reconstructed using \n", - "more pixels and fewer source pixels are used in the outskirts of the source plane. \n", + "The `Overlay` image-mesh and `Delaunay` mesh is still far from optimal. There are lots of source-pixels effectively\n", + "fitting just noise. We could achieve even better solutions if the central regions of the source were reconstructed\n", + "using more pixels, and fewer source pixels were used in the outskirts of the source plane.\n", "\n", - "Tutorials 9, 10 and 11 show even more advanced and adaptive pixelizations which do just this, by adapting to the\n", + "Tutorials 10, 11 and 12 show even more advanced and adaptive pixelizations which do just this, by adapting to the\n", "source galaxy's morphology rather than the mass model magnification.\n", "\n", - "In the mean time, you may wish to experiment with using both Delaunay and Delaunay grids to fit \n", - "lenses which can be easily achieve by changing the input pixelization given to a pipeline." + "In the mean time, you may wish to experiment with using both rectangular and Delaunay meshes to fit\n", + "lenses, which is easily achieved by changing the input pixelization given to a pipeline." ] } ], diff --git a/notebooks/chapter_4_pixelizations/tutorial_8_model_fit.ipynb b/notebooks/chapter_3_pixelizations/tutorial_9_model_fit.ipynb similarity index 98% rename from notebooks/chapter_4_pixelizations/tutorial_8_model_fit.ipynb rename to notebooks/chapter_3_pixelizations/tutorial_9_model_fit.ipynb index dc54f84..0e4f6ed 100644 --- a/notebooks/chapter_4_pixelizations/tutorial_8_model_fit.ipynb +++ b/notebooks/chapter_3_pixelizations/tutorial_9_model_fit.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Tutorial 8: Model-Fit\n", + "Tutorial 9: Model-Fit\n", "=====================\n", "\n", "You should now perform lens modeling using a pixelization, which is described fully in the example:\n", diff --git a/notebooks/chapter_3_search_chaining/README.md b/notebooks/chapter_3_search_chaining/README.md deleted file mode 100644 index 2e2091f..0000000 --- a/notebooks/chapter_3_search_chaining/README.md +++ /dev/null @@ -1,18 +0,0 @@ -In chapter 3, we introduce non-linear search chaining, whereby lens modeling pipelines are composed which each fit a -different lens model. - -**Colab** links to every tutorial are included. - -# Files - -- [Tutorial 1: Search Chaining](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb) — Breaking the lens modeling procedure into a chained sequence of model-fits. - -- [Tutorial 2: Prior Passing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb) — How the results of earlier searches are passed to later searches. - -- [Tutorial 3: Lens and Source](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_3_lens_and_source.ipynb) — Fitting the lens's light followed by its mass using chained searches. - -- [Tutorial 4: Two Lens galaxies](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.ipynb) — Modeling a strong lens with two lens galaxies using chained searches. - -- [Tutorial 5: Complex Source](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_5_complex_source.ipynb) — Using multiple light profiles to fit a complex and irregular source using chained searches. - -- [Tutorial 6: SLaM](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_6_slam.ipynb) — Template pipelines for fitting lens model is standardized ways. diff --git a/notebooks/chapter_3_search_chaining/tutorial_3_lens_and_source.ipynb b/notebooks/chapter_3_search_chaining/tutorial_3_lens_and_source.ipynb deleted file mode 100644 index 00e1ab9..0000000 --- a/notebooks/chapter_3_search_chaining/tutorial_3_lens_and_source.ipynb +++ /dev/null @@ -1,592 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 3: Lens and Source\n", - "===========================\n", - "\n", - "In this tutorial, we demonstrate search chaining using three searches to fit strong lens `Imaging` which includes the\n", - "lens galaxy's light.\n", - "\n", - "The crucial point to note is that for many lenses the lens galaxy's light can be fitted and subtracted reasonably\n", - "well before we attempt to fit the source galaxy. This makes sense, as fitting the lens's light (which is an elliptical\n", - "blob of light in the centre of the imaging) looks nothing like the source's light (which is a ring of light)! Formally,\n", - "we would say that these two model components (the lens's light and source's light) are not covariate.\n", - "\n", - "So, as a newly trained lens modeler, what does the lack of covariance between these parameters make you think?\n", - "Hopefully, you're thinking, why should I bother fitting the lens and source galaxy simultaneously? Surely we can\n", - "find the right regions of non-linear parameter space by fitting each separately first? This is what we're going to do\n", - "in this tutorial, using a pipeline composed of a modest 3 searches:\n", - "\n", - " 1) Fit the lens galaxy's light, ignoring the source.\n", - " 2) Fit the source-galaxy's light (and therefore lens galaxy's mass), ignoring the len`s light.\n", - " 3) Fit both simultaneously, using these results to initialize our starting location in parameter space.\n", - "\n", - "Of course, given that we do not care for the errors in searches 1 and 2, we will set up our non-linear search to\n", - "perform sampling as fast as possible!\n", - "\n", - "__Contents__\n", - "\n", - "- **Dated Tutorial:** This example tutorial was written ~4 years ago, when **PyAutoLens** was in its infancy and had a.\n", - "- **Initial Setup:** we'll use strong lensing data, where.\n", - "- **Paths:** All three searches will use the same `path_prefix`, so we write it here to avoid repetition.\n", - "- **Notes:** We use linear light profiles througout this script, given that the model is quite complex and this.\n", - "- **Wrap Up:** Summary of the script and next steps.\n", - "\n", - "__Dated Tutorial__\n", - "\n", - "This example tutorial was written ~4 years ago, when **PyAutoLens** was in its infancy and had a number of limitations:\n", - "\n", - " - The non-linear search used MultiNest or dynesty, which were less reliable (e.g. more likely to infer a local maxima\n", - " for complex lens models) and less efficient than Nautilus.\n", - "\n", - " - Linear light profiles and techniques like a Multi-Gaussian Expansion were not available.\n", - "\n", - "With all the new features added to **PyAutoLens** since, we no longer recommend that one breaks down the fitting of\n", - "the lens and source galaxy's light into separate searches, as perform in this search chaining example. Instead, we\n", - "would recommend you fit the lens and source simultaneously, using linear light profiles to make the model simpler\n", - "or a Multi-Gaussian Expansion.\n", - "\n", - "However, the example is still useful for demonstrating the core concepts of search chaining, which is still vital\n", - "for fitting complex lens model. Therefore, we recommend you still read through this tutorial and try to get a good\n", - "understanding of how search chaining works, but bear in mind that the example is a little dated and we now recommend\n", - "you fit the lens and source simultaneously!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Google Colab Setup__\n", - "\n", - "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", - "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", - "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", - "your own installation) it does nothing, and you can run it safely.\n", - "\n", - "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", - "type\" -> \"Hardware accelerator\" before running the notebook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " import google.colab\n", - "except ImportError:\n", - " from autolens import setup_colab as _setup_colab\n", - "else:\n", - " import importlib\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.check_call(\n", - " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", - " )\n", - " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", - "\n", - "_setup_colab.setup(\"howtolens\")" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autolens import jax_wrapper # Sets JAX environment before other imports\n", - "\n", - "# from autolens import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autolens as al\n", - "import autolens.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use strong lensing data, where:\n", - "\n", - " - The lens galaxy's light is an `Sersic`.\n", - " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", - " - The source galaxy's light is an `Exponential`.\n", - " \n", - "This image was fitted throughout chapter 2." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"lens_sersic\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if al.util.dataset.should_simulate(str(dataset_path)):\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/simulator/lens_sersic.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = al.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Paths__\n", - "\n", - "All three searches will use the same `path_prefix`, so we write it here to avoid repetition." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "path_prefix = Path(\"howtolens\") / \"chapter_3\" / \"tutorial_3_lens_and_source\"" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Masking (Search 1)__\n", - "\n", - "We need to choose our mask for the analysis. We use an annular mask that does not remove the source's light \n", - "from the fit, assuming the lens light model will still be sufficiently accurate to reveal the source in the \n", - "second search." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask = al.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 1)__\n", - "\n", - "Search 1 fits a lens model where:\n", - "\n", - " - The lens galaxy's light is a linear `Sersic` bulge [6 parameters].\n", - " \n", - " - The lens galaxy's mass and source galaxy are omitted.\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=6.\n", - "\n", - "__Notes__\n", - "\n", - "We use linear light profiles througout this script, given that the model is quite complex and this helps\n", - "simplify it." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model_1 = af.Collection(\n", - " galaxies=af.Collection(lens=af.Model(al.Galaxy, redshift=0.5, bulge=al.lp.Sersic)),\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search + Analysis + Model-Fit (Search 1)__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_1 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_1 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[1]_light[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=75,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 1)__\n", - "\n", - "The results which are used for prior passing are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Masking (Search 2)__\n", - "\n", - "Search 2 we are only fitting the source's light, thus we can apply an annular mask that removes regions of the\n", - "image that contained only the lens's light." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = al.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask = al.Mask2D.circular_annular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " inner_radius=0.6,\n", - " outer_radius=2.4,\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 2)__\n", - "\n", - "Search 2 fits a lens model where:\n", - "\n", - " - The lens galaxy's light is a linear `Sersic` bulge [Parameters fixed to results of search 1].\n", - " \n", - " - The lens galaxy's total mass distribution is an `Isothermal` with `ExternalShear` [7 parameters].\n", - " \n", - " - The source galaxy's light is a linear `Sersic` [6 parameters].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=13.\n", - "\n", - "Search 2, we fit the source-`galaxy's light and fix the lens light model to the model inferred in search 1, \n", - "ensuring the image we has the foreground lens subtracted. We do this below by passing the lens light as an `instance` \n", - "object.\n", - "\n", - "By passing an `instance`, we are telling **PyAutoLens** that we want it to pass the maximum log likelihood result of \n", - "that search and use those parameters as fixed values in the model. The model parameters passed as an `instance` are not \n", - "free parameters fitted for by the non-linear search, thus this reduces the dimensionality of the non-linear search \n", - "making model-fitting faster and more reliable. \n", - " \n", - "Thus, search 2 includes the lens light model from search 1, but it is completely fixed during the model-fit!\n", - "\n", - "We also use the centre of the `bulge` to initialize the priors on the lens's `mass`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mass = af.Model(al.mp.Isothermal)\n", - "mass.centre_0 = result_1.model.galaxies.lens.bulge.centre_0\n", - "mass.centre_1 = result_1.model.galaxies.lens.bulge.centre_1\n", - "\n", - "model_2 = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(\n", - " al.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_1.instance.galaxies.lens.bulge,\n", - " mass=mass,\n", - " shear=al.mp.ExternalShear,\n", - " ),\n", - " source=af.Model(al.Galaxy, redshift=1.0, bulge=al.lp.Sersic),\n", - " ),\n", - ")\n", - "\n", - "analysis_2 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[2]_mass[sie]_source[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 2)__\n", - "\n", - "The results which are used for prior passing are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_2.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Masking (Search 3)__\n", - "\n", - "Search 3 we fit the lens and source, therefore we will use a large circular mask." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = al.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.1,\n", - ")\n", - "\n", - "mask_radius = 3.0\n", - "\n", - "mask = al.Mask2D.circular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " radius=mask_radius,\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 3)__\n", - "\n", - "Search 3 fits a lens model where:\n", - "\n", - " - The lens galaxy's light is a linear `Sersic` bulge [6 Parameters: priors initialized from search 1].\n", - " \n", - " - The lens galaxy's total mass distribution is an `Isothermal` with `ExternalShear` [7 parameters: priors\n", - " initialized from search 2].\n", - " \n", - " - The source galaxy's light is a linear `Sersic` [6 parameters: priors initialized from search 2].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=23.\n", - "\n", - "There isn't a huge amount to say about this search, we have initialized the priors on all of our models parameters\n", - "and the only thing that is left to do is fit for all model components simultaneously, with slower Nautilus settings\n", - "that will give us more accurate parameter values and errors." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model_3 = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(\n", - " al.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_1.model.galaxies.lens.bulge,\n", - " mass=result_2.model.galaxies.lens.mass,\n", - " ),\n", - " source=af.Model(\n", - " al.Galaxy, redshift=1.0, bulge=result_2.model.galaxies.source.bulge\n", - " ),\n", - " ),\n", - ")\n", - "\n", - "analysis_3 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_3 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[3]_light[bulge]_mass[sie]_source[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=150,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_3 = search_3.fit(model=model_3, analysis=analysis_3)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Result (Search 3)__\n", - "\n", - "The final results are summarised in the `info` attribute." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(result_3.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "And there we have it, a sequence of searches that breaks modeling the lens and source galaxy into 3 simple searches. \n", - "This approach is much faster than fitting the lens and source simultaneously from the beginning. Instead of asking you \n", - "questions at the end of this chapter`s tutorials, I'm going to ask questions which I then answer. This will hopefully \n", - "get you thinking about how to approach pipeline writing.\n", - "\n", - " 1) Can this pipeline really be generalized to any lens? Surely the radii of the masks depends on the lens and source \n", - " galaxies?\n", - "\n", - "Whilst this is true, we chose mask radii above that are `excessive` and masks out a lot more of the image than just \n", - "the source (which, in terms of run-time, is desirable). Thus, provided you know the Einstein radius distribution of \n", - "your lens sample, you can choose mask radii that will masks out every source in your sample adequately (and even if \n", - "some of the source is still there, who cares? The fit to the lens galaxy will be okay).\n", - "\n", - "However, the template pipelines provided on the `autolens_workspace` simply use circular masks for every search and do\n", - "not attempt to use different masks for the lens light fit and source fit. This is to keep things simple (at the expense\n", - "of slower run times). It is up to you if you want to adapt these scripts to try and use more specific masking strategies.\n", - "\n", - "__Dated Tutorial__\n", - "\n", - "In fact, we now strongly recommend that you do not change masks between each search when using search chaining. \n", - "This is because it is very fiddly, and can waste a lot of your time refining masks to ensure they are suitable for\n", - "each lens. We recommend you always just use a large circular mask which is big enough to include the entire lens and \n", - "source of all lenses in your sample. This will save you a lot of time and means lens modeling can be automated much\n", - "easier.\n", - "\n", - "Building on the discussion above, a known limitation of using a pipeline which fits the lens light first, then the\n", - "source, is that it will do a poor job deblending the lens and source light if the Einstein radius is low. This often\n", - "leads the mass model to infer incorrect solutions which fit residuals from the lens light subtraction.\n", - "\n", - "This is why, given all the improvements to autolens, we now recommend that you do not use this pipeline and instead\n", - "always begin by fitting the lens and source simultaneously. This can use linear light profiles of a Multi-Gaussian\n", - "Expansion. " - ] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.ipynb b/notebooks/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.ipynb deleted file mode 100644 index 01f53e5..0000000 --- a/notebooks/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.ipynb +++ /dev/null @@ -1,515 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 4: Two Lens galaxies\n", - "=============================\n", - "\n", - "Up to now, all the images we've fitted had one lens galaxy. However, we saw in chapter 1 that our lens plane can\n", - "consist of multiple galaxies which each contribute to the strong lensing. Multi-galaxy systems are challenging to\n", - "model, because they add an extra 5-10 parameters to the non-linear search and, more problematically, the degeneracies\n", - "between the parameters of the mass profiles of the two galaxies can be severe.\n", - "\n", - "However, we can still break their analysis down using multiple searches and give ourselves a shot at getting a good\n", - "lens model. Here, we're going to fit a double lens system, fitting as much about each individual lens galaxy before\n", - "fitting them simultaneously.\n", - "\n", - "Up to now, I've put a focus on an analysis being general. The script we write in this example is going to be the\n", - "opposite, specific to the image we're modeling. Fitting multiple lens galaxies is really difficult and writing a\n", - "pipeline that we can generalize to many lenses isn't currently possible.\n", - "\n", - "__Contents__\n", - "\n", - "- **Initial Setup:** we'll use new strong lensing data, where.\n", - "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", - "- **Paths:** All four searches will use the same `path_prefix`, so we write it here to avoid repetition.\n", - "- **Search Chaining Approach:** Looking at the image, there are two blobs of light corresponding to the two lens galaxies.\n", - "- **Wrap Up:** Summary of the script and next steps." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Google Colab Setup__\n", - "\n", - "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", - "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", - "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", - "your own installation) it does nothing, and you can run it safely.\n", - "\n", - "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", - "type\" -> \"Hardware accelerator\" before running the notebook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " import google.colab\n", - "except ImportError:\n", - " from autolens import setup_colab as _setup_colab\n", - "else:\n", - " import importlib\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.check_call(\n", - " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", - " )\n", - " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", - "\n", - "_setup_colab.setup(\"howtolens\")" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autolens import jax_wrapper # Sets JAX environment before other imports\n", - "\n", - "# from autolens import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autolens as al\n", - "import autolens.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use new strong lensing data, where:\n", - "\n", - " - There are two lens galaxy's whose `LightProfile`'s are both linear `Sersic`'s.\n", - " - There are two lens galaxy's whose `MassProfile`'s are both `Isothermal`'s\n", - " - The source galaxy's light is an `Exponential`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"x2_lens_galaxies\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if al.util.dataset.should_simulate(str(dataset_path)):\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/simulator/lens_x2.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = al.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.05,\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Mask__\n", - "\n", - "We need to choose our mask for the analysis. Given the lens light is present in the image we'll need to include all \n", - "of its light in the central regions of the image, so lets use a circular mask. \n", - "\n", - "We'll use this mask in all four searches." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "mask_radius = 3.0\n", - "\n", - "mask = al.Mask2D.circular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " radius=mask_radius,\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Paths__\n", - "\n", - "All four searches will use the same `path_prefix`, so we write it here to avoid repetition." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "path_prefix = Path(\"howtolens\") / \"chapter_3\" / \"tutorial_4_x2_lens_galaxies\"" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search Chaining Approach__\n", - "\n", - "Looking at the image, there are two blobs of light corresponding to the two lens galaxies. The source's light is also \n", - "pretty complex and the arcs don't posses the rotational symmetry we're used to seeing up to now. \n", - "Multi-galaxy ray-tracing is just a lot more complicated, which means so is modeling it!\n", - "\n", - "So, how can we break the lens modeling up? As follows:\n", - "\n", - " 1) Fit and subtract the light of each lens galaxy individually.\n", - " 2) Use these results to initialize each lens galaxy's total mass distribution.\n", - "\n", - "So, with this in mind, we'll perform an analysis using searches:\n", - "\n", - " 1) Fit the light of the lens galaxy on the left of the image, at coordinates (0.0\", -1.0\").\n", - " 2) Fit the light of the lens galaxy on the right of the image, at coordinates (0.0\", 1.0\").\n", - " 3) Use this lens-subtracted image to fit the source's light. The mass profile centres of the two lens galaxies are \n", - " fixed to (0.0\", -1.0\") and (0.0\", 1.0\").\n", - " 4) Fit all relevant parameters simultaneously, using priors from searches 1, 2 and 3.\n", - "\n", - "__Model + Search + Analysis + Model-Fit (Search 1)__\n", - "\n", - "Search 1 fits a lens model where:\n", - "\n", - " - The left lens galaxy's light is a linear `Sersic` bulge with a fixed centre [4 parameters].\n", - "\n", - " - The lens galaxy's mass and source galaxy are omitted.\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=4.\n", - "\n", - "We fix the centre of its light to (0.0, -1.0), the pixel we know the left galaxy's light centre peaks." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "left_lens = af.Model(al.Galaxy, redshift=0.5, bulge=al.lp_linear.Sersic)\n", - "left_lens.bulge.centre_0 = 0.0\n", - "left_lens.bulge.centre_1 = -1.0\n", - "\n", - "model_1 = af.Collection(galaxies=af.Collection(left_lens=left_lens))" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search + Analysis + Model-Fit (Search 1)__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_1 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_1 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[1]__left_lens_light[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=75,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)\n" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 2)__\n", - "\n", - "Search 2 fits a lens model where:\n", - "\n", - " - The left lens galaxy's light is a linear `Sersic` bulge [0 parameters: fixed from search 1].\n", - "\n", - " - The right lens galaxy's light is a linear `Sersic` bulge with a fixec centre [4 parameters].\n", - "\n", - " - The lens galaxy's mass and source galaxy are omitted.\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=4.\n", - "\n", - "We fix the centre of the right lens's light to (0.0, 1.0), the pixel we know the right galaxy's light centre peaks.\n", - "\n", - "We also pass the result of the `left_lens` from search ` as an `instance`, which should improve the fitting of the\n", - "right lens." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "right_lens = af.Model(al.Galaxy, redshift=0.5, bulge=al.lp_linear.Sersic)\n", - "right_lens.bulge.centre_0 = 0.0\n", - "right_lens.bulge.centre_1 = 1.0\n", - "\n", - "model_2 = af.Collection(\n", - " galaxies=af.Collection(\n", - " left_lens=result_1.instance.galaxies.left_lens, right_lens=right_lens\n", - " ),\n", - ")\n", - "\n", - "analysis_2 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[2]__right_lens_light[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=75,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 3)__\n", - "\n", - "Search 3 fits a lens model where:\n", - "\n", - " - The left lens galaxy's light is a linear `Sersic` bulge [0 parameters: fixed from search 1].\n", - "\n", - " - The right lens galaxy's light is a linear `Sersic` bulge [0 parameters: fixed from search 2].\n", - "\n", - " - The lens galaxy's mass is modeled using two `Isothermal` profiles whose centres are fixed to (0.0, -1.0)\n", - " and (0.0, 1.0) [6 parameters].\n", - " \n", - " - The source galaxy's light is a linear `Exponential` [5 parameters].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=11." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "left_lens = af.Model(\n", - " al.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_1.instance.galaxies.left_lens.bulge,\n", - " mass=al.mp.Isothermal,\n", - ")\n", - "\n", - "right_lens = af.Model(\n", - " al.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_2.instance.galaxies.right_lens.bulge,\n", - " mass=al.mp.Isothermal,\n", - ")\n", - "\n", - "left_lens.mass.centre_0 = 0.0\n", - "left_lens.mass.centre_1 = -1.0\n", - "right_lens.mass.centre_0 = 0.0\n", - "right_lens.mass.centre_1 = 1.0\n", - "\n", - "model_3 = af.Collection(\n", - " galaxies=af.Collection(\n", - " left_lens=left_lens,\n", - " right_lens=right_lens,\n", - " source=af.Model(al.Galaxy, redshift=1.0, bulge=al.lp_linear.ExponentialCore),\n", - " ),\n", - ")\n", - "\n", - "analysis_3 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_3 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[3]__mass_x2[sie]__source[exp]\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_3 = search_3.fit(model=model_3, analysis=analysis_3)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 4)__\n", - "\n", - "Search 4 fits a lens model where:\n", - "\n", - " - The left lens galaxy's light is a linear `Sersic` bulge [6 parameters: priors initialized from search 1].\n", - "\n", - " - The right lens galaxy's light is a linear `Sersic` bulge [6 parameters: priors initialized from search 2].\n", - "\n", - " - The lens galaxy's mass is modeled using two `Isothermal` profiles whose centres are fixed to (0.0, -1.0)\n", - " and (0.0, 1.0) [6 parameters: priors initialized from search 3].\n", - "\n", - " - The source galaxy's light is a linear `Sersic` [6 parameters: priors initialized from search 3].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=24.\n", - "\n", - "We don't explicitly check the run-time, but the same principles discussed above apply here." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "left_lens = af.Model(\n", - " al.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_1.model.galaxies.left_lens.bulge,\n", - " mass=result_3.model.galaxies.left_lens.mass,\n", - ")\n", - "\n", - "right_lens = af.Model(\n", - " al.Galaxy,\n", - " redshift=0.5,\n", - " bulge=result_2.model.galaxies.right_lens.bulge,\n", - " mass=result_3.model.galaxies.right_lens.mass,\n", - ")\n", - "\n", - "source_bulge = af.Model(al.lp_linear.Sersic)\n", - "\n", - "source_bulge.take_attributes(result_3.model.galaxies.source.bulge)\n", - "\n", - "model_4 = af.Collection(\n", - " galaxies=af.Collection(\n", - " left_lens=left_lens,\n", - " right_lens=right_lens,\n", - " source=af.Model(al.Galaxy, redshift=1.0, bulge=source_bulge),\n", - " ),\n", - ")\n", - "\n", - "analysis_4 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_4 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[4]_light_x2[bulge]_mass_x2[sie]_source[exp]\",\n", - " unique_tag=dataset_name,\n", - " n_live=100,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_4 = search_4.fit(model=model_4, analysis=analysis_4)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "This pipeline takes a while to run, as is the nature of multi-galaxy modeling. Nevertheless, the techniques we've \n", - "learnt above can be applied to systems with even more galaxies, albeit the increases in parameters will slow down the \n", - "non-linear search. Here are some more Q&A`s\n", - "\n", - " 1) This system had two very similar lens galaxy's with comparable amounts of light and mass. How common is this? \n", - " Does it make it harder to model them?\n", - "\n", - "Typically, lenses with 2 galaxies have one massive galaxy (that makes up some 80%-90% of the overall light and mass), \n", - "accompanied by a smaller satellite. The satellite cannotbe ignored, it impacts the ray-tracing in a measureable way, \n", - "but it is a lot less degenerate with the main lens galaxy. This means we can often model the satellite with much \n", - "simpler profiles (e.g. spherical profiles). So yes, multi-galaxy systems can often be easier to model.\n", - "\n", - " 2) It got pretty confusing passing all those priors towards the end of the pipeline there, didn`t it?\n", - "\n", - "It does get confusing. This is why we made galaxies named objects, so that we could call them the `left_lens` and \n", - "`right_lens`. It still requires caution when writing the pipeline, but goes to show that if you name your galaxies \n", - "sensibly you should be able to avoid errors, or spot them quickly when you make them." - ] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/chapter_3_search_chaining/tutorial_5_complex_source.ipynb b/notebooks/chapter_3_search_chaining/tutorial_5_complex_source.ipynb deleted file mode 100644 index be9fe14..0000000 --- a/notebooks/chapter_3_search_chaining/tutorial_5_complex_source.ipynb +++ /dev/null @@ -1,511 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 5: Complex Source\n", - "==========================\n", - "\n", - "Up to now, we've not paid much attention to the source galaxy's morphology. We've assumed its a single-component\n", - "exponential profile, which is a fairly crude assumption. A quick look at any image of a real galaxy reveals a\n", - "wealth of different structures that could be present: bulges, disks, bars, star-forming knots and so on. Furthermore,\n", - "there could be more than one source-galaxy!\n", - "\n", - "In this example, we'll explore how far we get fitting a complex source using a pipeline. Fitting complex source's is\n", - "an exercise in diminishing returns. Each light profile we add to our source model brings with it an extra 5-7,\n", - "parameters. If there are 4 components, or multiple galaxies, we are quickly entering the somewhat nasty regime of\n", - "30-40+ parameters in our non-linear search. Even with a pipeline, that is a lot of parameters to fit!\n", - "\n", - "__Contents__\n", - "\n", - "- **Initial Setup:** we'll use new strong lensing data, where.\n", - "- **Paths:** All four searches will use the same `path_prefix`, so we write it here to avoid repetition.\n", - "- **Search Chaining Approach:** The source is clearly complex, with more than 4 peaks of light.\n", - "- **Run Times:** Profiling the expected run time of the model-fit.\n", - "- **Wrap Up:** Summary of the script and next steps." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Google Colab Setup__\n", - "\n", - "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", - "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", - "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", - "your own installation) it does nothing, and you can run it safely.\n", - "\n", - "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", - "type\" -> \"Hardware accelerator\" before running the notebook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " import google.colab\n", - "except ImportError:\n", - " from autolens import setup_colab as _setup_colab\n", - "else:\n", - " import importlib\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.check_call(\n", - " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", - " )\n", - " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", - "\n", - "_setup_colab.setup(\"howtolens\")" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "\n", - "from autolens import jax_wrapper # Sets JAX environment before other imports\n", - "\n", - "# from autolens import setup_notebook; setup_notebook()\n", - "\n", - "from pathlib import Path\n", - "import autofit as af\n", - "import autolens as al\n", - "import autolens.plot as aplt" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Initial Setup__\n", - "\n", - "we'll use new strong lensing data, where:\n", - "\n", - " - The lens galaxy's light is omitted.\n", - " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", - " - The source galaxy's light is four linear `Sersic`." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset_name = \"source_complex\"\n", - "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Dataset Auto-Simulation__\n", - "\n", - "If the dataset does not already exist on your system, it will be created by running the corresponding\n", - "simulator script. This ensures that all example scripts can be run without manually simulating data first." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "if al.util.dataset.should_simulate(str(dataset_path)):\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.run(\n", - " [sys.executable, \"scripts/simulator/source_complex.py\"],\n", - " check=True,\n", - " )\n", - "\n", - "dataset = al.Imaging.from_fits(\n", - " data_path=dataset_path / \"data.fits\",\n", - " noise_map_path=dataset_path / \"noise_map.fits\",\n", - " psf_path=dataset_path / \"psf.fits\",\n", - " pixel_scales=0.05,\n", - ")\n", - "\n", - "mask_radius = 3.0\n", - "\n", - "mask = al.Mask2D.circular(\n", - " shape_native=dataset.shape_native,\n", - " pixel_scales=dataset.pixel_scales,\n", - " radius=mask_radius,\n", - ")\n", - "\n", - "dataset = dataset.apply_mask(mask=mask)\n", - "\n", - "aplt.subplot_imaging_dataset(dataset=dataset)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Paths__\n", - "\n", - "All four searches will use the same `path_prefix`, so we write it here to avoid repetition." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "path_prefix = Path(\"howtolens\") / \"chapter_3\" / \"tutorial_4_complex_source\"" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search Chaining Approach__\n", - "\n", - "The source is clearly complex, with more than 4 peaks of light. Through visual inspection of this image, we cannot state\n", - "with confidence how many sources of light there truly is! The data also omits he lens galaxy's light. This keep the \n", - "number of parameters down and therefore makes the searches faster, however we would not get such a luxury for a real \n", - "galaxy.\n", - "\n", - "To fit this lens with a complex source model, our approach is simply to fit the lens galaxy mass and source using\n", - "one light profile in the first search, and then add an additional light profile to each search. The mass model and\n", - "light profiles inferred in the previous search are then used to pass priors.\n", - "\n", - "__Run Times__\n", - "\n", - "In this example we don't explicitly check run-times, for brevity. However, the same rules of thumb we discussed in the\n", - "previous tutorial still apply. \n", - "\n", - "For example, as we add more light profiles to the source model, the likelihood evaluation time will increase. As the\n", - "model becomes more complex, search chaining is key to ensuring run times stay lower.\n", - "\n", - "__Model + Search + Analysis + Model-Fit (Search 1)__\n", - "\n", - " - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters].\n", - " \n", - " - The source galaxy's light is a linear `Sersic` [6 parameters].\n", - " \n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=11." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model_1 = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(al.Galaxy, redshift=0.5, mass=al.mp.Isothermal),\n", - " source=af.Model(al.Galaxy, redshift=1.0, bulge_0=al.lp_linear.Sersic),\n", - " ),\n", - ")" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `info` attribute shows the model in a readable format." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "print(model_1.info)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Search + Analysis + Model-Fit (Search 1)__" - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "analysis_1 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_1 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[1]__mass[sie]__source_x1[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=120,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_1 = search_1.fit(model=model_1, analysis=analysis_1)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 2)__\n", - "\n", - " - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters: priors initialized from \n", - " search 1].\n", - "\n", - " - The source galaxy's light is two linear `Sersic` [12 parameters: first Sersic initialized from \n", - " search 1].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=17." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model_2 = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(al.Galaxy, redshift=0.5, mass=result_1.model.galaxies.lens.mass),\n", - " source=af.Model(\n", - " al.Galaxy,\n", - " redshift=1.0,\n", - " bulge_0=result_1.model.galaxies.source.bulge_0,\n", - " bulge_1=al.lp_linear.Sersic,\n", - " ),\n", - " ),\n", - ")\n", - "\n", - "analysis_2 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_2 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[2]_mass[sie]_source_x2[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=120,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_2 = search_2.fit(model=model_2, analysis=analysis_2)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 3)__\n", - "\n", - " - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters: priors initialized from \n", - " search 2].\n", - "\n", - " - The source galaxy's light is three linear `Sersic` [18 parameters: first two Sersic's initialized from \n", - " search 2].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=21." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model_3 = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(al.Galaxy, redshift=0.5, mass=result_2.model.galaxies.lens.mass),\n", - " source=af.Model(\n", - " al.Galaxy,\n", - " redshift=1.0,\n", - " bulge_0=result_2.model.galaxies.source.bulge_0,\n", - " bulge_1=result_2.model.galaxies.source.bulge_1,\n", - " bulge_2=al.lp_linear.Sersic,\n", - " ),\n", - " ),\n", - ")\n", - "\n", - "analysis_3 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_3 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[3]_mass[sie]_source_x3[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=140,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_3 = search_3.fit(model=model_3, analysis=analysis_3)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Model + Search + Analysis + Model-Fit (Search 4)__\n", - "\n", - " - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters: priors initialized from \n", - " search 4].\n", - "\n", - " - The source galaxy's light is four linear `Sersic` [24 parameters: first three Sersic's initialized from \n", - " search 2].\n", - "\n", - "The number of free parameters and therefore the dimensionality of non-linear parameter space is N=29." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "model_4 = af.Collection(\n", - " galaxies=af.Collection(\n", - " lens=af.Model(al.Galaxy, redshift=0.5, mass=result_3.model.galaxies.lens.mass),\n", - " source=af.Model(\n", - " al.Galaxy,\n", - " redshift=1.0,\n", - " bulge_0=result_3.model.galaxies.source.bulge_0,\n", - " bulge_1=result_3.model.galaxies.source.bulge_1,\n", - " bulge_2=result_3.model.galaxies.source.bulge_2,\n", - " bulge_3=al.lp_linear.Sersic,\n", - " ),\n", - " ),\n", - ")\n", - "\n", - "analysis_4 = al.AnalysisImaging(dataset=dataset)\n", - "\n", - "search_4 = af.Nautilus(\n", - " path_prefix=path_prefix,\n", - " name=\"search[4]_mass[sie]_source_x4[bulge]\",\n", - " unique_tag=dataset_name,\n", - " n_live=160,\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", - ")\n", - "\n", - "result_4 = search_4.fit(model=model_4, analysis=analysis_4)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Wrap Up__\n", - "\n", - "With four light profiles, we were still unable to produce a fit to the source that did not leave residuals. However, I \n", - "actually simulated the lens using a source with four light profiles. A `perfect fit` was therefore somewhere in \n", - "parameter space, but our search unfortunately was unable to locate this.\n", - "\n", - "Lets confirm this, by manually fitting the imaging data with the true input model.\n", - "\n", - "We cannot apply a mask to a dataset that was already masked, so we first reload the imaging from .fits." - ] - }, - { - "cell_type": "code", - "metadata": {}, - "source": [ - "dataset = dataset.apply_mask(\n", - " mask=al.Mask2D.circular(\n", - " shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0\n", - " )\n", - ")\n", - "\n", - "lens_galaxy = al.Galaxy(\n", - " redshift=0.5,\n", - " mass=al.mp.Isothermal(\n", - " centre=(0.0, 0.0), einstein_radius=1.6, ell_comps=(0.17647, 0.0)\n", - " ),\n", - ")\n", - "\n", - "source_galaxy = al.Galaxy(\n", - " redshift=1.0,\n", - " light_0=al.lp.Sersic(\n", - " centre=(0.1, 0.1),\n", - " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0),\n", - " intensity=0.1,\n", - " effective_radius=1.0,\n", - " sersic_index=2.5,\n", - " ),\n", - " light_1=al.lp.Sersic(\n", - " centre=(0.8, 0.6),\n", - " ell_comps=al.convert.ell_comps_from(axis_ratio=0.5, angle=30.0),\n", - " intensity=0.2,\n", - " effective_radius=0.3,\n", - " sersic_index=3.0,\n", - " ),\n", - " light_2=al.lp.Sersic(\n", - " centre=(-0.3, 0.6),\n", - " ell_comps=al.convert.ell_comps_from(axis_ratio=0.3, angle=120.0),\n", - " intensity=0.6,\n", - " effective_radius=0.5,\n", - " sersic_index=1.5,\n", - " ),\n", - " light_3=al.lp.Sersic(\n", - " centre=(-0.3, -0.3),\n", - " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=85.0),\n", - " intensity=0.4,\n", - " effective_radius=0.1,\n", - " sersic_index=2.0,\n", - " ),\n", - ")\n", - "\n", - "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy])\n", - "\n", - "true_fit = al.FitImaging(dataset=dataset, tracer=tracer)\n", - "\n", - "aplt.subplot_fit_imaging(fit=true_fit)\n", - "aplt.subplot_fit_imaging_of_planes(fit=true_fit)" - ], - "outputs": [], - "execution_count": null - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And indeed, we see an improved residual-map, chi-squared-map, and so forth.\n", - "\n", - "If the source morphology is complex, there is no way we chain searches to fit it perfectly. The non-linear parameter \n", - "space simply becomes too complex. For this tutorial, this was true even though our source model could actually fit \n", - "the data perfectly. For real lenses, the source may be *even more complex* giving us even less hope of getting a \n", - "good fit.\n", - "\n", - "But fear not, **PyAutoLens** has you covered. In chapter 4, we'll introduce a completely new way to model the source \n", - "galaxy, which addresses the problem faced here." - ] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/chapter_3_search_chaining/tutorial_6_slam.ipynb b/notebooks/chapter_3_search_chaining/tutorial_6_slam.ipynb deleted file mode 100644 index 496fe86..0000000 --- a/notebooks/chapter_3_search_chaining/tutorial_6_slam.ipynb +++ /dev/null @@ -1,112 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tutorial 6: SLaM\n", - "================\n", - "\n", - "You are now familiar with pipelines, in particular how we use them to break-down the lens modeling procedure\n", - "to provide more efficient and reliable model-fits. In the previous tutorials, you learnt how to write your own\n", - "pipelines, which can fit whatever lens model is of particular interest to your scientific study.\n", - "\n", - "However, for most lens models there are standardized approaches one can take to fitting them. For example, as we saw in\n", - "tutorial 3 of this chapter, an effective approach is to fit a model for the lens's light followed by a model for its\n", - "mass and the source. It would be wasteful for all **PyAutoLens** users to have to write their own pipelines to\n", - "perform the same tasks.\n", - "\n", - "For this reason, the `autolens_workspace` comes with a number of standardized pipelines, which fit common lens models\n", - "in ways we have tested are efficient and robust. These pipelines can be found the `chaining/pipelines` packages and\n", - "can be easily adapted to fit lens models with a range of different parameterizations and assumptions.\n", - "\n", - "__SLaM (Source, Light and Mass)__\n", - "\n", - "A second set of template pipelines, called the **SLaM** (Source, Light and Mass) pipelines can be found in the folder\n", - "`autolens_workspace/slam`. These are similar in design to the pipelines, but are composed of a specific sequence of\n", - "pipelines:\n", - "\n", - " - `Source`: A pipeline that focuses on producing a robust model for the source's light, using simpler models for the\n", - " lens's light and mass.\n", - "\n", - " - `Light`: A pipeline that fits a complex lens light model (e.g. one with many components), using the initialized\n", - " source model to cleanly deblend the lens and source light.\n", - "\n", - " - `Mass`: A pipeline that fits a complex lens mass model, benefitting from the good models for the lens's light and\n", - " source.\n", - "\n", - "For fitting very complex lens models, for example ones which decompose its mass into its stellar and dark components,\n", - "the **SLaM** pipelines have been carefully crafted to do this in a reliable and automated way that is still efficient.\n", - "The **SLaM** pipelines also make fitting many different models to a single dataset efficient, as they reuse the results\n", - "of earlier searches (e.g. in the Source pipeline) to fit different models in the `Light` and `Mass` pipelines for the\n", - "lens's light and mass.\n", - "\n", - "__Wrap Up__\n", - "\n", - "Whether you should use individual searches, pipelines, The SLaM pipelines or write your own model-fitting script\n", - "depends on the scope of your scientific analysis. I would advise you begin by trying to adapting the scripts in the\n", - "`autolens_workspace`to fit your data, and also try using the SLaM pipelines once you are a confident **PyAutoLens** user." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "__Google Colab Setup__\n", - "\n", - "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", - "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", - "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", - "your own installation) it does nothing, and you can run it safely.\n", - "\n", - "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", - "type\" -> \"Hardware accelerator\" before running the notebook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " import google.colab\n", - "except ImportError:\n", - " from autolens import setup_colab as _setup_colab\n", - "else:\n", - " import importlib\n", - " import subprocess\n", - " import sys\n", - "\n", - " subprocess.check_call(\n", - " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", - " )\n", - " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", - "\n", - "_setup_colab.setup(\"howtolens\")" - ] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} \ No newline at end of file diff --git a/notebooks/chapter_4_pixelizations/README.md b/notebooks/chapter_4_pixelizations/README.md deleted file mode 100644 index 8dea793..0000000 --- a/notebooks/chapter_4_pixelizations/README.md +++ /dev/null @@ -1,27 +0,0 @@ -In chapter 4, we use **Pixelizations** to reconstruct complex source galaxies on pixelized grids. - -**Colab** links to every tutorial are included. - -# Files - -- [Tutorial 1: Pixelizations](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb) — Creating a pixel-grid in the source-plane. - -- [Tutorial 2: Mappers](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_2_mappers.ipynb) — How a pixelization maps source-pixels to image-pixels. - -- [Tutorial 3: Inversions](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_3_inversions.ipynb) — Inverting the mappings to reconstruct the source's light. - -- [Tutorial 4: Bayesian Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb) — Smoothing the source within a Bayesian framework. - -- [Tutorial 5: Borders](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_5_borders.ipynb) — Preventing highly demagnified image-pixels ruining the inversion. - -- [Tutorial 6: Lens Modeling](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_6_lens_modeling.ipynb) — How to use inversions to fit a lens model. - -- [Tutorial 7: Adaptive Pixelization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb) — A Voronoi mesh which adapts to the mass model's magnification. - -- [Tutorial 8: Model Fit](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_8_model_fit.ipynb) — An example lens modeling pipeline which uses an inversion. - -- [Tutorial 9: Fit Problems](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_9_fit_problems.ipynb) — The shortcomings of our lens models and inversions. - -- [Tutorial 10: Brightness Adaption](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb) — Adapting the pixelization to the source's morphology. - -- [Tutorial 11: Adaptive Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_11_adaptive_regularization.ipynb) — Adapting the regularization to the source's morphology. diff --git a/notebooks/chapter_4_scaling_up_lensing/README.md b/notebooks/chapter_4_scaling_up_lensing/README.md new file mode 100644 index 0000000..f059478 --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/README.md @@ -0,0 +1,18 @@ +In chapter 4, we scale lens modeling up beyond a single lens galaxy: extra galaxies, multi-galaxy lenses, +scaling relations, group and cluster scales, and weak lensing. + +**Colab** links to every tutorial are included. + +# Files + +- [Tutorial 1: Extra Galaxies](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.ipynb) — Handling nearby galaxies by scaling their light out of the fit or modeling them explicitly. + +- [Tutorial 2: Multi Galaxy](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.ipynb) — Modeling lenses with two or more co-dominant lens galaxies and the degeneracies this creates. + +- [Tutorial 3: Scaling Relation](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.ipynb) — Tying galaxy masses to their luminosities so model complexity stops growing with each galaxy. + +- [Tutorial 4: Group Scale](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_4_group_scale.ipynb) — Modeling group-scale lenses with truncated dPIE members and an optional group dark matter halo. + +- [Tutorial 5: Cluster Scale](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.ipynb) — Modeling cluster-scale lenses with point-source positions, the CSV interface and multi-plane ray tracing. + +- [Tutorial 6: Weak Lensing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.ipynb) — Fitting weak-lensing shear catalogues, which probe mass far beyond the strong lensing region. diff --git a/notebooks/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.ipynb b/notebooks/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.ipynb new file mode 100644 index 0000000..79ba77e --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.ipynb @@ -0,0 +1,744 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 1: Extra Galaxies\n", + "==========================\n", + "\n", + "Welcome to chapter 4 of **HowToLens**, where we scale up lens modeling beyond a single lens galaxy.\n", + "\n", + "In chapters 1 to 3, every strong lens we studied had the same simple anatomy: one lens galaxy, whose light and mass\n", + "we modeled, and one background source galaxy, whose lensed emission we fitted. This is the cleanest possible lensing\n", + "configuration, and it was the right place to learn the fundamentals of ray-tracing, non-linear searches and\n", + "pixelized source reconstructions.\n", + "\n", + "Real strong lenses are rarely this tidy. Galaxies live in crowded fields: a lens galaxy may have a faint companion a\n", + "few arc-seconds away, sit in a small group of comparably massive galaxies, or be embedded in a cluster containing\n", + "hundreds of members and a vast dark matter halo. The science of strong lensing scales up through this hierarchy, and\n", + "so must our modeling:\n", + "\n", + "- **Extra galaxies (this tutorial)**: a single dominant lens galaxy, with one or more nearby interloper galaxies\n", + " whose light and mass may contaminate the analysis.\n", + "\n", + "- **Multi-galaxy lenses (next tutorial)**: systems where two or more galaxies are co-equal lenses, and no single\n", + " galaxy can be called \"the\" lens.\n", + "\n", + "- **Scaling relations**: modeling many galaxies at once by tying their properties to their luminosity, so that\n", + " adding galaxies does not add free parameters.\n", + "\n", + "- **Group and cluster scales**: lenses whose deflectors are entire groups or clusters of galaxies, including a\n", + " shared dark matter halo.\n", + "\n", + "- **Weak lensing**: the statistical distortion of many background galaxies by foreground mass, extending lensing\n", + " beyond the strong regime entirely.\n", + "\n", + "Much of the methodology in this chapter is not specific to lensing. Fitting the light of extra galaxies, modeling\n", + "blended multi-galaxy fields and composing models via scaling relations are equally important for non-lensing galaxy\n", + "studies, and **PyAutoGalaxy** provides the same API for these tasks applied to galaxy morphology (without the\n", + "ray-tracing).\n", + "\n", + "In this first tutorial, we take the first step up in scale: a galaxy-scale lens with one extra galaxy nearby. We\n", + "will learn how to decide whether the extra galaxy matters, and the two approaches to dealing with it when it does:\n", + "removing its light from the data, or including it in the lens model.\n", + "\n", + "__Contents__\n", + "\n", + "- **Initial Setup:** Load the imaging dataset of a lens with an extra galaxy and inspect the interloper.\n", + "- **Light Versus Mass:** The core decision: do the extra galaxy's light and / or mass affect the analysis?\n", + "- **Mask:** Define a circular mask large enough to include the extra galaxy's emission.\n", + "- **Approach 1 Noise Scaling:** Remove the extra galaxy's light from the fit by scaling its data and noise values.\n", + "- **Noise Scaling Fit:** Fit a lens model to the noise-scaled data, without the extra galaxy in the model.\n", + "- **Approach 2 Extra Galaxies Model:** Include the extra galaxy's light and mass in the lens model explicitly.\n", + "- **Extra Galaxy Centres:** Why the extra galaxy's centre is fixed to its observed light centre.\n", + "- **Extra Galaxies Fit:** Fit the lens model which includes the extra galaxy.\n", + "- **Which Approach When:** Guidance on choosing between noise scaling and explicit modeling.\n", + "- **Wrap Up:** Summary of the script and next steps." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "import autolens as al\n", + "import autolens.plot as aplt\n", + "import autofit as af" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Initial Setup__\n", + "\n", + "Lets load the `Imaging` dataset we'll fit in this tutorial. It is similar to the `lens_sersic` dataset used\n", + "throughout chapter 2, where:\n", + "\n", + " - The lens galaxy's light is an `Sersic`.\n", + " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", + " - The source galaxy's light is an `SersicCore`.\n", + "\n", + "However, there is one addition: an extra galaxy, with its own light (an `ExponentialSph`) and its own\n", + "mass (an `IsothermalSph`), located a few arc-seconds from the lens galaxy." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"lens_extra_galaxy\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/lens_extra_galaxy.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When we plot the dataset, the familiar sight of a lens galaxy surrounded by a lensed source's ring of emission is\n", + "joined by a blob of light in the upper-right of the image.\n", + "\n", + "This is the extra galaxy. It is not part of the strong lens system we are studying: it is an interloper, a galaxy\n", + "that happens to lie close to the lens on the sky. Interlopers like this are extremely common in real imaging of\n", + "strong lenses, and every lens modeler has to decide what to do about them." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Light Versus Mass__\n", + "\n", + "The decision of what to do about an extra galaxy hinges on one question, asked twice:\n", + "\n", + "**Does its light affect the analysis?** The extra galaxy's emission may extend into the region of the image we fit.\n", + "If it does, and we fit a model containing only the lens and source, the model has no component that can produce\n", + "this emission. The fit will respond by distorting the lens light, source light or mass model to soak it up,\n", + "biasing every parameter we infer.\n", + "\n", + "**Does its mass affect the analysis?** The extra galaxy's mass deflects light rays, just like the lens galaxy's\n", + "mass does. If the galaxy is close (in projection) to the lensed source's light, its deflections measurably perturb\n", + "the ray-tracing. A model without this mass will compensate by biasing the main lens galaxy's mass parameters, for\n", + "example its `einstein_radius` or `ell_comps`.\n", + "\n", + "These two effects are dealt with separately, which gives us two approaches:\n", + "\n", + "- **Approach 1 (remove its light)**: If only the light matters, we can remove the extra galaxy's emission from the\n", + " fit entirely, without adding anything to the model. The mass is ignored.\n", + "\n", + "- **Approach 2 (model it explicitly)**: If the mass matters (or the light blends too closely with the lensed source\n", + " to cleanly remove), we include the extra galaxy in the lens model, with its own light and / or mass profiles.\n", + "\n", + "We will now perform both, and at the end of the tutorial discuss when each is appropriate.\n", + "\n", + "__Mask__\n", + "\n", + "We first define the circular mask used to fit the data. In chapter 2 we typically used a 2.6\" - 3.0\" mask, which\n", + "tightly contained the lens and lensed source.\n", + "\n", + "Here, we use a larger 4.0\" mask, so that the region containing the extra galaxy is included in the fit. If we\n", + "simply shrank the mask to exclude the extra galaxy, we would also throw away pixels containing lensed source\n", + "emission, and the mask's hard edge could still cut through the extra galaxy's light." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=4.0,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting the masked dataset confirms the extra galaxy's emission is inside the mask, and will therefore impact the\n", + "model-fit unless we do something about it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Approach 1 Noise Scaling__\n", + "\n", + "Our first approach removes the extra galaxy's light from the fit.\n", + "\n", + "The most obvious way to do this would be to mask the extra galaxy's pixels, removing them from the fit entirely.\n", + "However, removing pixels changes the fit in subtle ways: their coordinates are no longer used in the ray-tracing,\n", + "and for certain models (e.g. the pixelized source reconstructions of chapter 3) removing interior pixels creates\n", + "discontinuities in the pixelization that introduce unexpected systematics.\n", + "\n", + "Instead, we use **noise scaling**: the pixels stay in the fit, but their data values are set to zero and their\n", + "noise-map values are increased to very large values. A pixel with enormous noise contributes negligibly to the\n", + "likelihood, so the extra galaxy's light cannot influence the model, while the pixels themselves remain part of the\n", + "fit's geometry.\n", + "\n", + "To do this we need a mask of the extra galaxy's region. For real data, you would create this yourself by\n", + "inspecting the image (the workspace's `data_preparation` package includes a GUI for drawing it); for this simulated\n", + "dataset the simulator script has already output a `mask_extra_galaxies.fits` circle covering the extra galaxy.\n", + "\n", + "We reload the dataset first, because noise scaling must be applied before the circular mask." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "mask_extra_galaxies = al.Mask2D.from_fits(\n", + " file_path=dataset_path / \"mask_extra_galaxies.fits\",\n", + " pixel_scales=0.1,\n", + " invert=True, # Note that we invert the mask here as `True` means a pixel is scaled.\n", + ")\n", + "\n", + "dataset = dataset.apply_noise_scaling(mask=mask_extra_galaxies)\n", + "\n", + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=4.0,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting the dataset shows the extra galaxy's emission has vanished: its data values are zero and the\n", + "signal-to-noise of its pixels is effectively zero, so the fit will simply ignore that region of the image." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Noise Scaling Fit__\n", + "\n", + "We now fit this noise-scaled dataset with a lens model that does **not** include the extra galaxy. The model is the\n", + "same one we used in chapter 2's linear profiles tutorial: a linear `Sersic` lens bulge, an `Isothermal` plus\n", + "`ExternalShear` mass model and a linear `SersicCore` source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Lens:\n", + "\n", + "bulge = af.Model(al.lp_linear.Sersic)\n", + "mass = af.Model(al.mp.Isothermal)\n", + "\n", + "lens = af.Model(\n", + " al.Galaxy, redshift=0.5, bulge=bulge, mass=mass, shear=al.mp.ExternalShear\n", + ")\n", + "\n", + "# Source:\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0, bulge=al.lp_linear.SersicCore)\n", + "\n", + "# Overall Lens Model:\n", + "\n", + "model = af.Collection(galaxies=af.Collection(lens=lens, source=source))\n", + "\n", + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\", \"chapter_4\"),\n", + " name=\"tutorial_1_extra_galaxies_noise_scaling\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", + ")\n", + "\n", + "analysis = al.AnalysisImaging(dataset=dataset)\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_1_extra_galaxies_noise_scaling\"\n", + " \" folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result_noise_scaling = search.fit(model=model, analysis=analysis)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting the maximum log likelihood fit shows the lens and source are fitted well, with the noise-scaled region\n", + "contributing nothing to the residuals.\n", + "\n", + "The extra galaxy's light has been dealt with. However, note what this approach did **not** do: the extra galaxy's\n", + "mass is completely absent from the model. Its deflection of the source's light rays is unaccounted for, and the\n", + "inferred lens mass model will have absorbed that perturbation as best it can. For an extra galaxy that is small\n", + "and / or far from the lensed emission this bias is negligible; for one that is massive and close, it is not." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_fit_imaging(fit=result_noise_scaling.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Approach 2 Extra Galaxies Model__\n", + "\n", + "Our second approach includes the extra galaxy in the lens model, fitting its light (so we no longer need to remove\n", + "it from the data) and its mass (so its deflections are included in the ray-tracing).\n", + "\n", + "We reload the dataset and apply the 4.0\" circular mask, but this time we do **not** apply noise scaling, because\n", + "the extra galaxy's emission is now something the model itself will fit." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=4.0,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Extra Galaxy Centres__\n", + "\n", + "To include the extra galaxy in the model, we input the centre of its light, as observed in the image.\n", + "\n", + "In principle, we could add the extra galaxy to the model with a completely free centre, and let the non-linear\n", + "search figure out where it is. In practice this is a bad idea: the extra parameters make parameter space more\n", + "complex, and fits commonly go wrong in characteristic ways. For example, the extra galaxy's light profile may\n", + "wander off and fit one of the lensed source's multiple images instead of the interloper, or its mass profile may\n", + "recentre itself onto the main lens galaxy and act as part of its mass distribution.\n", + "\n", + "Fixing each extra galaxy's light and mass centres to its observed light centre removes these failure modes and\n", + "keeps the model as simple as possible. The observed centre is an excellent estimate of the true centre, because\n", + "(unlike the lensed source) the extra galaxy's light is not distorted by lensing.\n", + "\n", + "For real data you would measure these centres from the image (the workspace's `data_preparation` package shows\n", + "how, including a GUI for marking them); for this simulated dataset the simulator has output them to a .json file,\n", + "which we load below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "extra_galaxies_centres = al.Grid2DIrregular(\n", + " al.from_json(file_path=dataset_path / \"extra_galaxies_centres.json\")\n", + ")\n", + "\n", + "print(extra_galaxies_centres)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Extra Galaxies Model Composition__\n", + "\n", + "We compose the main lens and source model exactly as before." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Lens:\n", + "\n", + "bulge = af.Model(al.lp_linear.Sersic)\n", + "mass = af.Model(al.mp.Isothermal)\n", + "\n", + "lens = af.Model(\n", + " al.Galaxy, redshift=0.5, bulge=bulge, mass=mass, shear=al.mp.ExternalShear\n", + ")\n", + "\n", + "# Source:\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0, bulge=al.lp_linear.SersicCore)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We now compose the model of the extra galaxy, using the dedicated `extra_galaxies` modeling API.\n", + "\n", + "For each extra galaxy centre (there is only one here, but the loop below scales to any number) we create a `Galaxy`\n", + "model with:\n", + "\n", + " - A linear `ExponentialSph` light profile, with its `centre` fixed to the observed centre [1 free\n", + " parameter: `effective_radius`, as the `intensity` is solved for by the linear inversion].\n", + "\n", + " - An `IsothermalSph` mass profile, with its `centre` fixed to the observed centre [1 free\n", + " parameter: `einstein_radius`].\n", + "\n", + "Extra galaxy mass profiles can wander to unphysically high `einstein_radius` values, degrading the fit, so we set\n", + "a `UniformPrior` with an upper limit of 0.5\" to prevent this. The extra galaxy is much less massive than the main\n", + "lens (whose `einstein_radius` is around 1.6\"), so this prior comfortably contains all plausible solutions.\n", + "\n", + "The extra galaxies are grouped into their own `af.Collection`, which is passed to the overall model via its\n", + "`extra_galaxies` input, alongside the `galaxies` collection containing the lens and source. This is the same\n", + "API used throughout the `autolens_workspace` for extra galaxies, and it is how **PyAutoLens** knows to include\n", + "these galaxies in the ray-tracing without treating them as the main lens or source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# Extra Galaxies:\n", + "\n", + "extra_galaxies_list = []\n", + "\n", + "for extra_galaxy_centre in extra_galaxies_centres:\n", + "\n", + " # Extra Galaxy Light\n", + "\n", + " light = af.Model(al.lp_linear.ExponentialSph)\n", + " light.centre = extra_galaxy_centre\n", + "\n", + " # Extra Galaxy Mass\n", + "\n", + " mass = af.Model(al.mp.IsothermalSph)\n", + " mass.centre = extra_galaxy_centre\n", + " mass.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=0.5)\n", + "\n", + " # Extra Galaxy\n", + "\n", + " extra_galaxy = af.Model(al.Galaxy, redshift=0.5, light=light, mass=mass)\n", + "\n", + " extra_galaxies_list.append(extra_galaxy)\n", + "\n", + "extra_galaxies = af.Collection(extra_galaxies_list)\n", + "\n", + "# Overall Lens Model:\n", + "\n", + "model = af.Collection(\n", + " galaxies=af.Collection(lens=lens, source=source), extra_galaxies=extra_galaxies\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `info` attribute confirms the model includes the extra galaxy, with its fixed centres and its\n", + "free `effective_radius` and `einstein_radius` parameters." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Extra Galaxies Fit__\n", + "\n", + "We fit this model with the same search set up as before. The model has only two more free parameters than the\n", + "noise-scaling fit, thanks to the fixed centres and linear light profile, so the fit remains fast." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\", \"chapter_4\"),\n", + " name=\"tutorial_1_extra_galaxies_model\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", + ")\n", + "\n", + "analysis = al.AnalysisImaging(dataset=dataset)\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_1_extra_galaxies_model\"\n", + " \" folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result_extra_galaxies = search.fit(model=model, analysis=analysis)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting the maximum log likelihood fit shows the extra galaxy's emission is now fitted by its own light profile,\n", + "and its mass has contributed deflections to the ray-tracing of the source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_fit_imaging(fit=result_extra_galaxies.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The result's `info` shows the inferred `einstein_radius` of the extra galaxy, quantifying how much lensing power\n", + "the interloper contributes to the system." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result_extra_galaxies.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Which Approach When__\n", + "\n", + "We have seen the two extremes of dealing with an extra galaxy. Choosing between them comes back to the\n", + "light-versus-mass question at the start of this tutorial, which in practice is settled by three properties of the\n", + "interloper:\n", + "\n", + "- **Distance from the lensed emission**: An extra galaxy whose light is well separated from the lensed source (as\n", + " in this tutorial) can be cleanly noise-scaled away. If its light blends into the source's arcs, noise scaling\n", + " would also delete source emission we need, and the light must be modeled instead.\n", + "\n", + "- **Brightness**: A faint interloper whose emission barely rises above the noise can often be ignored entirely, or\n", + " noise-scaled with no consequence. A bright one must be removed or modeled, or it will bias the fit.\n", + "\n", + "- **Mass and proximity to the lens**: The mass matters when the extra galaxy is massive enough, and close enough\n", + " (in projection) to the lensed source's light, for its deflections to measurably perturb the ray-tracing. A rough\n", + " rule of thumb is to compare its expected Einstein radius to the astrometric precision of the data: a small galaxy\n", + " several arc-seconds from the arcs can have its mass safely ignored, whereas one abutting the Einstein ring\n", + " cannot. When in doubt, fit both approaches and compare the inferred lens models: if the main lens's mass\n", + " parameters shift appreciably, the extra galaxy's mass matters.\n", + "\n", + "Intermediate options also exist and are fully supported by the API: you can noise-scale the light but still include\n", + "the mass profile in the model, or model the light but omit the mass. The `extra_galaxies` collection simply\n", + "contains whatever profiles you give it.\n", + "\n", + "__Wrap Up__\n", + "\n", + "In this tutorial, we took the first step up in scale from the single lens galaxy of chapters 1 to 3, and learnt:\n", + "\n", + "1. Real lenses live in crowded fields, and interloping extra galaxies are the first complication real data throws\n", + " at us.\n", + "\n", + "2. Whether an extra galaxy matters hinges on two separate questions: does its **light** contaminate the region of\n", + " the image we fit, and does its **mass** perturb the ray-tracing of the source?\n", + "\n", + "3. Noise scaling removes an extra galaxy's light from the fit without adding model complexity, by zeroing its data\n", + " and inflating its noise, but leaves its mass unaccounted for.\n", + "\n", + "4. The `extra_galaxies` modeling API includes extra galaxies in the model with their own light and mass profiles,\n", + " with their centres fixed to the observed light centres to keep parameter space simple and well behaved.\n", + "\n", + "5. Which approach is appropriate depends on the interloper's distance from the lensed emission, its brightness,\n", + " and its mass's proximity to the lens.\n", + "\n", + "Throughout, the extra galaxy was a nuisance: something to remove or account for, so that our analysis of the\n", + "main lens remained accurate. In the next tutorial we meet systems where that framing breaks down entirely, because\n", + "a second galaxy is not a nuisance but a co-equal lens, with light and mass comparable to the first. There, no\n", + "single galaxy is \"the\" lens, and the model must treat them all on an equal footing." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.ipynb b/notebooks/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.ipynb new file mode 100644 index 0000000..685708e --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.ipynb @@ -0,0 +1,721 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 2: Multi-Galaxy Lenses\n", + "===============================\n", + "\n", + "In the previous tutorial, we learned how to deal with extra galaxies near a strong lens \u2014 nuisance objects whose\n", + "light contaminates the data but which play no meaningful role in the lensing itself. We removed their emission\n", + "from the analysis, or gave them a heavily restricted model, and the single dominant lens galaxy remained the\n", + "star of the show.\n", + "\n", + "In this tutorial, we meet systems where that picture breaks down entirely: **multi-galaxy lenses**, where two\n", + "(or more) galaxies of comparable mass both contribute significantly to the lensing of a single background source.\n", + "Neither galaxy is a minor perturber we can mask away or simplify \u2014 they are **co-dominant deflectors**, and every\n", + "one of them needs its own free light and mass model.\n", + "\n", + "How do such systems arise physically? There are two main channels:\n", + "\n", + "- **Merging galaxies**: two galaxies at the same redshift caught in the act of merging, or orbiting one another\n", + " in close proximity. Their projected separation is small compared to the Einstein radius of their combined mass,\n", + " so the source's light rays feel both galaxies' gravity at once. Because both deflectors are at one redshift,\n", + " ray tracing is single-plane: their deflection angles simply add.\n", + "\n", + "- **Chance line-of-sight alignments**: two physically unrelated galaxies at *different* redshifts that happen to\n", + " lie along the same line of sight. Light from the source is deflected first by the more distant galaxy, and the\n", + " already-deflected rays are deflected again by the nearer one. This is compound, multi-plane ray tracing \u2014 the\n", + " `Tracer` handles it natively by simply assigning each galaxy its redshift, and we will defer its details to\n", + " later in this chapter, where multi-plane lensing becomes the default at cluster scale.\n", + "\n", + "In this tutorial, both lens galaxies are at the same redshift, so we stay in the simpler single-plane regime and\n", + "focus on the modeling challenge that defines multi-galaxy lensing: the growth of the model's parameter space.\n", + "\n", + "__Contents__\n", + "\n", + "- **Initial Setup:** Load the double lens galaxy dataset, simulating it first if it is not on disk.\n", + "- **Mask:** Define a mask which encloses the combined Einstein ring of both lens galaxies.\n", + "- **Over Sampling:** Centre the adaptive over sampling grid on every deflector, not just one.\n", + "- **Model:** Compose a lens model with a free light and mass model per deflector, and count its parameters.\n", + "- **Fixing the Mass Centres:** Fix each galaxy's mass centre to its observed light centre, and why this is standard.\n", + "- **Model Fit:** Fit the two-deflector model to the data with a non-linear search.\n", + "- **Result:** Inspect the combined critical curve and the fit to the data.\n", + "- **Mass Degeneracies:** The total mass is well constrained, but its split between the galaxies is not.\n", + "- **No Shared Halo:** What we are deliberately not yet assuming, and why that changes at group scale.\n", + "- **Three Lens Galaxies:** A triple-deflector system, and how the parameter count keeps growing.\n", + "- **Wrap Up:** Summary and the road to scaling relations." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "import autofit as af\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Initial Setup__\n", + "\n", + "We begin with `Imaging` of a strong lens where **two** lens galaxies, separated by 2.0\", both deflect the light\n", + "of a single background source:\n", + "\n", + " - The two lens galaxies' light are `Sersic` profiles, centred at (0.0\", -1.0\") and (0.0\", 1.0\").\n", + " - The two lens galaxies' mass distributions are `Isothermal` profiles with comparable Einstein radii (1.0\" and\n", + " 0.8\") \u2014 this comparability is what makes them co-dominant.\n", + " - The source galaxy's light is an `ExponentialCoreSph`.\n", + "\n", + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"x2_lens_galaxies\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name\n", + "\n", + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/lens_x2.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When we plot the data, the difference from every lens we have modeled so far is obvious: there are two bright\n", + "lens galaxies, and the lensed source's arcs wrap around the *pair as a whole* rather than around either galaxy\n", + "individually. The mass distribution the source's light responds to is the sum of both galaxies' mass." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "We define a 3.0\" circular mask. For a multi-galaxy lens, sizing the mask needs more care than at galaxy scale:\n", + "the Einstein radius that matters is that of the *combined* mass distribution, not either galaxy's individually.\n", + "A mask sized by eye from one galaxy's light would clip the arcs, which extend beyond both galaxies' centres." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask_radius = 3.0\n", + "\n", + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=mask_radius,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Over Sampling__\n", + "\n", + "We use the adaptive over sampling scheme introduced in earlier chapters, which evaluates the steep central\n", + "regions of the lens galaxies' light at high resolution. The one multi-galaxy specific point is that the adaptive\n", + "grid is centred on **every** deflector, not just one \u2014 each galaxy has its own steep central light profile\n", + "needing accurate evaluation, and `centre_list` takes as many centres as we give it.\n", + "\n", + "The two centres below are the observed centres of the two lens galaxies, which for this simulated dataset we\n", + "know exactly. For real data you would measure them from the image itself (the `autolens_workspace` provides a\n", + "GUI which writes them from mouse clicks)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_centres = [(0.0, -1.0), (0.0, 1.0)]\n", + "\n", + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=dataset.grid,\n", + " sub_size_list=[4, 2, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=lens_centres,\n", + ")\n", + "\n", + "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model__\n", + "\n", + "We now compose the lens model, and here the multi-galaxy regime shows its teeth. Every co-dominant deflector\n", + "needs its own free light and mass model, so the model has one `Galaxy` entry per deflector.\n", + "\n", + "We build each lens galaxy in a loop over the observed centres and store them as `lens_0`, `lens_1`, etc. This\n", + "list-based composition scales to any number of deflectors, and it is the same API the `autolens_workspace`'s\n", + "`multi_galaxy` package (and the group-scale examples above it) use \u2014 so nothing needs re-learning later.\n", + "\n", + "For each lens galaxy's light we use a Multi Gaussian Expansion (MGE) made of linear light profiles, built by the\n", + "utility function `al.model_util.mge_model_from` (this hides the longer `Basis` composition API we stepped through\n", + "in the linear profiles tutorial of chapter 2). For each lens galaxy's mass we use an `Isothermal` (SIE) profile,\n", + "initially with every parameter free. The source is also an MGE.\n", + "\n", + "The MGE choice matters more here than anywhere we have used it before. Each lens galaxy's 20 Gaussians add just\n", + "**4 non-linear parameters** (a shared centre and shared elliptical components \u2014 every `sigma` is fixed and every\n", + "`intensity` is solved for by the linear inversion). A single ordinary `Sersic` bulge would add 7 non-linear\n", + "parameters per galaxy, and would fit the blended, overlapping light of a close pair far less well. Because the\n", + "multi-galaxy regime multiplies whatever light model we choose by the number of deflectors, a light model that is\n", + "both cheap and flexible is close to essential \u2014 this is exactly where the MGE shines.\n", + "\n", + "Each Isothermal mass model adds 5 free parameters: its centre (2), elliptical components (2) and Einstein\n", + "radius (1).\n", + "\n", + "(The `autolens_workspace` multi-galaxy examples additionally include a single external shear for the whole\n", + "system, held in its own model entry at the system centre rather than attached to any one deflector \u2014 the shear\n", + "describes the tidal field of structure *outside* the system, so it belongs to no individual galaxy. We omit it\n", + "here to keep the parameter accounting simple.)" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_dict = {}\n", + "\n", + "for i, centre in enumerate(lens_centres):\n", + "\n", + " bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=20,\n", + " centre_prior_is_uniform=True,\n", + " centre=(centre[0], centre[1]),\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + " )\n", + "\n", + " mass = af.Model(al.mp.Isothermal)\n", + "\n", + " lens_dict[f\"lens_{i}\"] = af.Model(\n", + " al.Galaxy,\n", + " redshift=0.5,\n", + " bulge=bulge,\n", + " mass=mass,\n", + " )\n", + "\n", + "bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=20,\n", + " gaussian_per_basis=1,\n", + " centre_prior_is_uniform=False,\n", + ")\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0, bulge=bulge)\n", + "\n", + "model = af.Collection(galaxies=af.Collection(**lens_dict, source=source))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The model's `info` shows `lens_0` and `lens_1` each carrying their own free mass model \u2014 the signature of the\n", + "multi-galaxy regime \u2014 and its `prior_count` gives the total number of free parameters." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)\n", + "\n", + "print(f\"Free parameters (free mass centres): {model.prior_count}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The count is **22**: each deflector contributes 4 (MGE light) + 5 (SIE mass) = 9 free parameters, and the MGE\n", + "source contributes the final 4. Compare this to the equivalent single-galaxy model, which has just 13 \u2014 one\n", + "co-dominant deflector added 9 parameters, and every further deflector will add 9 more. Model complexity compounds\n", + "fast in this regime, and with plain Sersic light profiles instead of MGEs it would compound faster still (12 per\n", + "deflector rather than 9).\n", + "\n", + "__Fixing the Mass Centres__\n", + "\n", + "Before fitting, we make one change that is standard practice for multi-galaxy lenses: we **fix each galaxy's\n", + "mass centre to its observed light centre**, removing 2 free parameters per deflector.\n", + "\n", + "Why? With a single lens galaxy, the data usually constrains the mass centre well: the arcs pin down where the\n", + "deflection field is centred, and there is only one plausible culprit. With multiple deflectors this breaks down.\n", + "The source's light responds to the *summed* deflection field, so a small shift of one galaxy's mass centre can be\n", + "compensated by shifts of the other's centre, ellipticity or Einstein radius \u2014 the free centres become extremely\n", + "degenerate with one another. The search wanders these degeneracies, converging slowly and often settling in\n", + "unphysical corners of parameter space where one galaxy's mass has drifted far from any light.\n", + "\n", + "Fixing each mass centre to the galaxy's light centre resolves this at minimal cost. Light traces stellar mass,\n", + "and the stellar body sits at the bottom of the galaxy's potential well, so the light centre is an excellent\n", + "estimate of the mass centre. (Measuring a genuine offset between mass and light \u2014 which interacting pairs can\n", + "show at the kiloparsec level \u2014 is real science, but it is a *follow-up* fit performed after a robust model with\n", + "fixed centres has been found.)\n", + "\n", + "Assigning a tuple to the mass model's `centre` fixes it, so it is no longer a free parameter with a prior." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_dict = {}\n", + "\n", + "for i, centre in enumerate(lens_centres):\n", + "\n", + " bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=20,\n", + " centre_prior_is_uniform=True,\n", + " centre=(centre[0], centre[1]),\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + " )\n", + "\n", + " mass = af.Model(al.mp.Isothermal)\n", + " mass.centre = (centre[0], centre[1])\n", + "\n", + " lens_dict[f\"lens_{i}\"] = af.Model(\n", + " al.Galaxy,\n", + " redshift=0.5,\n", + " bulge=bulge,\n", + " mass=mass,\n", + " )\n", + "\n", + "bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=20,\n", + " gaussian_per_basis=1,\n", + " centre_prior_is_uniform=False,\n", + ")\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0, bulge=bulge)\n", + "\n", + "model = af.Collection(galaxies=af.Collection(**lens_dict, source=source))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Printing the model's `info` again, each mass `centre` is now listed as a fixed value with no prior, and the\n", + "free parameter count has dropped from 22 to **18** \u2014 each SIE now contributes 3 free parameters instead of 5." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)\n", + "\n", + "print(f\"Free parameters (fixed mass centres): {model.prior_count}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model Fit__\n", + "\n", + "We fit the model with the nested sampling algorithm `Nautilus`, as in previous chapters. Because this model has\n", + "more free parameters than the single-galaxy fits of chapter 2 (which used 100 live points), we raise `n_live`\n", + "to 200 \u2014 a multi-galaxy parameter space is more multi-modal, and too few live points risks converging on a local\n", + "maximum where the two galaxies' Einstein radii have been mis-apportioned." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\", \"chapter_4\"),\n", + " name=\"tutorial_2_multi_galaxy\",\n", + " unique_tag=dataset_name,\n", + " n_live=200,\n", + " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", + ")\n", + "\n", + "analysis = al.AnalysisImaging(dataset=dataset)\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_2_multi_galaxy\"\n", + " \" folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result = search.fit(model=model, analysis=analysis)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Result__\n", + "\n", + "The result's `info` shows the inferred model, with each deflector's parameters listed under its own `lens_0` /\n", + "`lens_1` entry." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The tracer subplot is worth a close look for a multi-galaxy lens: the critical curve is that of the *combined*\n", + "mass distribution, so it wraps around the pair as a whole rather than encircling either galaxy individually \u2014\n", + "just as the arcs in the data do." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_tracer(tracer=result.max_log_likelihood_tracer, grid=result.grids.lp)\n", + "\n", + "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mass Degeneracies__\n", + "\n", + "The corner plot of the posterior is where the multi-galaxy regime reveals its most important lesson. Below we\n", + "plot the probability density function of every parameter; when you run this tutorial for real (rather than\n", + "skimming the generated output), focus on the panel showing `lens_0`'s Einstein radius against `lens_1`'s." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.corner_anesthetic(samples=result.samples)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You will see a strongly elongated, tilted degeneracy: the two Einstein radii are *anti-correlated*. When one\n", + "galaxy's Einstein radius goes up, the other's goes down, tracing out a narrow ridge in parameter space.\n", + "\n", + "The physical reason is the summed deflection field. The arcs constrain the **total** mass enclosed by the\n", + "combined Einstein ring extremely well \u2014 move along the ridge and the sum of the two galaxies' deflections barely\n", + "changes, so the fit to the data barely changes either. What the data constrains much more weakly is the **split**\n", + "of that mass between the two galaxies. The closer the pair (relative to the combined Einstein radius), the less\n", + "information the arcs carry about which galaxy the mass belongs to, and the longer the ridge grows.\n", + "\n", + "This has a real scientific cost. For a single-galaxy lens, the Einstein mass is one of the cleanest measurements\n", + "in astrophysics. For a multi-galaxy lens, the total is still clean, but per-galaxy quantities \u2014 each galaxy's\n", + "mass, its mass-to-light ratio, its dark matter fraction \u2014 inherit the full width of the degeneracy, and their\n", + "error bars can be many times larger than the total's. Any interpretation built on the split (e.g. comparing the\n", + "two galaxies' dark matter content) must honestly propagate this, which is why we fit these systems with a\n", + "sampler that maps the full posterior rather than an optimizer that returns a single best-fit point: the best-fit\n", + "point sits somewhere on the ridge and tells you nothing about the ridge's length.\n", + "\n", + "__No Shared Halo__\n", + "\n", + "It is worth being explicit about an assumption we did *not* make. We modeled two galaxies, each with its own\n", + "untruncated `Isothermal` mass profile, and simply added their deflections. We did not assume the pair inhabits a\n", + "single, large, shared dark matter halo \u2014 and because there is no host halo in the model, we also made no\n", + "assumptions about tidal stripping or the truncation of each galaxy's individual halo.\n", + "\n", + "At this scale, that is the right call: a pair (or triple) of comparable galaxies has no dominant halo for its\n", + "members to be stripped by, and the data cannot demand one. But climb the mass ladder and it changes. At the\n", + "group scale, a dominant group-sized halo enters the model as an explicit choice, and the member galaxies orbiting\n", + "within it are tidally truncated by its potential \u2014 assumptions that reshape the entire mass model, as we will\n", + "see later in this chapter.\n", + "\n", + "__Three Lens Galaxies__\n", + "\n", + "What happens when a third co-dominant deflector joins the system? Let us load a triple-galaxy lens, simulated\n", + "with three `Isothermal` mass profiles of comparable Einstein radii (0.9\", 0.8\" and 0.7\") arranged in a triangle,\n", + "all lensing a single source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"x3_lens_galaxies\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name\n", + "\n", + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/lens_x3.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We compose the three-deflector model with exactly the same loop as before \u2014 this is the payoff of the list-based\n", + "API, which needs no changes as the deflector count grows. Each mass centre is again fixed to its galaxy's\n", + "observed light centre; with three deflectors the centre degeneracies are even more severe than for the pair, so\n", + "the standard trick matters even more." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_centres = [(0.9, 0.0), (-0.6, -0.9), (-0.6, 0.9)]\n", + "\n", + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=mask_radius,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)\n", + "\n", + "lens_dict = {}\n", + "\n", + "for i, centre in enumerate(lens_centres):\n", + "\n", + " bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=20,\n", + " centre_prior_is_uniform=True,\n", + " centre=(centre[0], centre[1]),\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + " )\n", + "\n", + " mass = af.Model(al.mp.Isothermal)\n", + " mass.centre = (centre[0], centre[1])\n", + "\n", + " lens_dict[f\"lens_{i}\"] = af.Model(\n", + " al.Galaxy,\n", + " redshift=0.5,\n", + " bulge=bulge,\n", + " mass=mass,\n", + " )\n", + "\n", + "bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=20,\n", + " gaussian_per_basis=1,\n", + " centre_prior_is_uniform=False,\n", + ")\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0, bulge=bulge)\n", + "\n", + "model = af.Collection(galaxies=af.Collection(**lens_dict, source=source))\n", + "\n", + "print(model.info)\n", + "\n", + "print(f\"Free parameters (three deflectors, fixed mass centres): {model.prior_count}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The count is now **25**: even with fixed mass centres and cheap MGE light models, every deflector adds 7 free\n", + "parameters (4 light + 3 mass), and with free centres it would be 9. The trend is linear and unforgiving:\n", + "\n", + " - 1 deflector: 13 free parameters.\n", + " - 2 deflectors: 18 (this tutorial's fit).\n", + " - 3 deflectors: 25.\n", + " - 10 deflectors: ~74. A group-scale lens can easily have this many members.\n", + "\n", + "And the parameter count is only half the problem. As the deflectors multiply, so do the degeneracies between\n", + "their masses \u2014 every pair of galaxies gets its own version of the Einstein radius ridge we saw above, and the\n", + "posterior becomes progressively harder for any sampler to map.\n", + "\n", + "You are welcome to fit this three-galaxy model yourself \u2014 the search above works unchanged, though expect it to\n", + "take noticeably longer than the pair. Clearly, freeing every galaxy's mass cannot scale to the tens or hundreds\n", + "of member galaxies in groups and clusters. Something has to give.\n", + "\n", + "__Wrap Up__\n", + "\n", + "In this tutorial, we modeled strong lenses with multiple co-dominant deflectors. Let's summarise what we've\n", + "learnt:\n", + "\n", + "- **Co-dominant deflectors**: When two or more galaxies of comparable mass both contribute significantly to the\n", + " lensing, every one of them needs its own free light and mass model \u2014 unlike the nuisance neighbours of the\n", + " previous tutorial, none can be masked away or simplified.\n", + "\n", + "- **Physical origins**: Multi-galaxy lenses arise from merging galaxies at one redshift (single-plane, deflections\n", + " add) or chance line-of-sight alignments at different redshifts (multi-plane ray tracing, covered later in this\n", + " chapter).\n", + "\n", + "- **Parameter growth**: Each deflector added 9 free parameters (4 for its MGE light, 5 for its SIE mass) \u2014 and the\n", + " MGE is what kept it that cheap, since its Gaussian intensities are solved for by linear algebra rather than sampled.\n", + "\n", + "- **Fixed mass centres**: Fixing each galaxy's mass centre to its observed light centre (removing 2 parameters per\n", + " deflector) is the standard trick for taming the severe centre degeneracies of multi-deflector models.\n", + "\n", + "- **Mass degeneracies**: The data constrains the total mass well but its split between the galaxies poorly,\n", + " producing an anti-correlated ridge between the deflectors' Einstein radii \u2014 and inflating the uncertainties on\n", + " any per-galaxy science.\n", + "\n", + "- **No shared halo**: At this scale we do not assume the galaxies inhabit one large dark matter halo, so no tidal\n", + " truncation assumptions enter the model \u2014 that framing arrives at group scale.\n", + "\n", + "The three-galaxy model made the trajectory clear: freeing every deflector cannot scale. In the next tutorial, we\n", + "introduce the tool that stops this growth in its tracks \u2014 scaling relations, which tie the masses of many\n", + "galaxies to their observed luminosities so that an entire population of deflectors costs almost no extra free\n", + "parameters." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.ipynb b/notebooks/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.ipynb new file mode 100644 index 0000000..c2d3e4a --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.ipynb @@ -0,0 +1,865 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 3: Scaling Relations\n", + "=============================\n", + "\n", + "The previous tutorial ended on a warning: every deflector we add to a lens model brings its own free parameters.\n", + "Two galaxies were manageable, but each one cost us a mass profile's worth of dimensions, and the arithmetic only\n", + "gets worse. A group-scale lens may have ten member galaxies, a cluster hundreds. If every member keeps its own\n", + "free mass, the parameter space explodes \u2014 a 100-galaxy cluster with 5 free mass parameters per galaxy is a\n", + "500-dimensional model, which no non-linear search can sample and no dataset can constrain anyway.\n", + "\n", + "This tutorial introduces the tool that stops the explosion: the **scaling relation**. Instead of giving every\n", + "galaxy its own mass freedom, we set each galaxy's mass from something we can *measure* \u2014 its luminosity \u2014 via a\n", + "relation shared by the whole population. The relation itself has only a few global parameters (in this tutorial,\n", + "zero new ones!), so the dimensionality of the model stops growing with the number of galaxies. Ten members or a\n", + "hundred, the parameter space stays the same size.\n", + "\n", + "This pairs beautifully with a tool we already have. In chapter 2, we saw that a Multi-Gaussian Expansion (MGE)\n", + "fits a galaxy's light using linear algebra, so its intensities add no non-linear parameters. Put the two\n", + "together and the recipe for a many-galaxy lens model emerges:\n", + "\n", + "- **Light:** MGE for every galaxy \u2014 flexible light models at (almost) no non-linear cost.\n", + "- **Mass:** a scaling relation tying every galaxy's mass to its measured luminosity \u2014 a whole population of\n", + " mass profiles at no non-linear cost.\n", + "\n", + "The entire many-galaxy system then costs barely more than a single-lens fit. This tutorial builds that model on\n", + "the two-lens dataset from the previous tutorial, verifies the parameter counting explicitly, and is honest about\n", + "what a scaling relation assumes and when it can bite you.\n", + "\n", + "__Overview__\n", + "\n", + "In this tutorial, we will:\n", + "\n", + "- Explain the physics of why luminosity predicts lensing mass (mass-follows-light and the Faber-Jackson relation).\n", + "\n", + "- Compose a lens model where one galaxy \"anchors\" the relation and the other galaxy's Einstein radius is tied to\n", + " it via the ratio of their measured luminosities.\n", + "\n", + "- Verify by explicit parameter counting that tied galaxies add zero free parameters, however many there are.\n", + "\n", + "- Fit the model and inspect how a tied parameter appears in the results.\n", + "\n", + "- Discuss the limitations: real galaxies scatter around any relation, and a member that deviates can bias the\n", + " whole lens model.\n", + "\n", + "__Contents__\n", + "\n", + "- **Mass Follows Light:** The physics linking a galaxy's luminosity to its lensing mass.\n", + "- **Initial Setup:** Load the two-lens-galaxy imaging dataset from the previous tutorial.\n", + "- **Mask:** Define the 2D mask applied to the dataset for the model-fit.\n", + "- **Measured Luminosities:** The luminosities the relation needs, and where they come from.\n", + "- **The Anchor:** Identify the brightest galaxy, which anchors the relation.\n", + "- **Over Sampling:** Adaptive over-sampling centred on both lens galaxies.\n", + "- **Light Via MGE:** MGE light models for every galaxy, at a cost of two parameters each.\n", + "- **Scaling Relation:** Tie the fainter galaxy's Einstein radius to the anchor's.\n", + "- **Model:** Compose the model with the scaling tier as its own collection.\n", + "- **Parameter Counts:** Compare against the previous tutorial's per-galaxy model.\n", + "- **Scaling To Many Galaxies:** Show the model does not grow when 12 more members are tied.\n", + "- **Model Fit:** Fit the scaling-relation model with Nautilus.\n", + "- **Results:** How a tied parameter appears in the results, and how close the truth is to the relation.\n", + "- **Limitations:** Scatter, deviant members, and adding per-galaxy freedom selectively.\n", + "- **Lens Environments:** The same machinery models the environment of single-lens systems.\n", + "- **Wrap Up:** Summary of the script and next steps.\n", + "\n", + "__Mass Follows Light__\n", + "\n", + "Why should a galaxy's luminosity tell us anything about its lensing mass?\n", + "\n", + "The starting point is that light traces stars and stars trace mass. A more luminous elliptical galaxy contains\n", + "more stars, sits in a deeper potential well, and its stars therefore move faster. This is quantified by the\n", + "**Faber-Jackson relation**, an empirical scaling law for elliptical galaxies which states that luminosity grows\n", + "steeply with the velocity dispersion \\sigma of the stars:\n", + "\n", + " L ~ \\sigma^4\n", + "\n", + "Lensing connects \\sigma to the deflection of light. For the isothermal mass profiles we have used throughout\n", + "these tutorials, the Einstein radius is set directly by the velocity dispersion:\n", + "\n", + " \\theta_E = 4 \\pi (\\sigma / c)^2 (D_ls / D_s)\n", + "\n", + "where the D's are distances between observer, lens and source. So \\theta_E ~ \\sigma^2, and combining the two\n", + "scalings gives us a relation between the two things we care about \u2014 the Einstein radius we want to know and the\n", + "luminosity we can measure:\n", + "\n", + " \\theta_E ~ \\sigma^2 ~ (L^{1/4})^2 = L^{1/2}\n", + "\n", + "If one galaxy in a lens system has a measured luminosity L_anchor and Einstein radius \\theta_E_anchor, every\n", + "other galaxy's Einstein radius follows from its own luminosity:\n", + "\n", + " \\theta_E_i = \\theta_E_anchor * (L_i / L_anchor)^{0.5}\n", + "\n", + "This is the scaling relation we will build into the model below. The exponent 0.5 is the Faber-Jackson value;\n", + "more sophisticated versions exist (the \"fundamental plane\" adds a dependence on galaxy size and surface\n", + "brightness, tightening the relation), but the Faber-Jackson form captures the essential physics and is the\n", + "standard workhorse for lens modeling. It is the same idea that cluster lensing studies have used for decades:\n", + "tie the hundreds of cluster member galaxies to their luminosities via a scaling relation, so the model stays\n", + "tractable.\n", + "\n", + "The crucial property is that only *ratios* of luminosity enter the relation. The absolute calibration, the\n", + "units, even the waveband cancel out (so long as all galaxies are measured consistently) \u2014 which is what makes\n", + "the relation so easy to apply in practice." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import numpy as np\n", + "from pathlib import Path\n", + "import autofit as af\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Initial Setup__\n", + "\n", + "We use the same two-lens-galaxy imaging dataset as the previous tutorial, where:\n", + "\n", + " - Both lens galaxies' light are `Sersic` profiles.\n", + " - Both lens galaxies' mass distributions are `Isothermal` profiles.\n", + " - The source galaxy's light is an `ExponentialCoreSph`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"x2_lens_galaxies\"\n", + "dataset_path = Path(\"dataset\") / \"imaging\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/lens_x2.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "We define a 3.0\" circular mask, which contains both lens galaxies (centred at (0.0\", -1.0\") and (0.0\", 1.0\"))\n", + "and the lensed source emission." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask_radius = 3.0\n", + "\n", + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=mask_radius,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Measured Luminosities__\n", + "\n", + "The scaling relation needs two inputs for every galaxy it touches: its centre and its luminosity. Neither is a\n", + "free parameter of the model \u2014 both are **measurements**, made from the imaging data before the mass model is\n", + "ever fitted.\n", + "\n", + "In a real analysis these come from a light-only fit performed first: for example, fitting an MGE to every\n", + "galaxy and integrating it to a total flux (the `autolens_workspace`'s scaling-relation SLaM pipeline does exactly\n", + "this). Because only luminosity *ratios* enter the relation, a magnitude catalogue from any survey works just as\n", + "well, converting via `L_i / L_anchor = 10 ** (0.4 * (m_anchor - m_i))`.\n", + "\n", + "In this tutorial we take a shortcut available only with simulated data: the simulator saved the true galaxies to\n", + "a `tracer.json` file, so we load it and integrate each lens galaxy's true light profile directly. This keeps the\n", + "tutorial self-contained, but keep in mind that on real data this step is a fit to the data, with its own\n", + "(small) uncertainties." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer_true = al.from_json(file_path=dataset_path / \"tracer.json\")\n", + "\n", + "lens_galaxies_true = [\n", + " galaxy for galaxy in tracer_true.galaxies if galaxy.redshift == 0.5\n", + "]\n", + "\n", + "centres = [\n", + " tuple(float(value) for value in galaxy.bulge.centre)\n", + " for galaxy in lens_galaxies_true\n", + "]\n", + "\n", + "luminosities = [\n", + " galaxy.bulge.luminosity_within_circle_from(radius=mask_radius)\n", + " for galaxy in lens_galaxies_true\n", + "]\n", + "\n", + "print(f\"Lens galaxy centres: {centres}\")\n", + "print(f\"Lens galaxy luminosities: {luminosities}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__The Anchor__\n", + "\n", + "The relation needs an \"anchor\": one galaxy whose Einstein radius is a free parameter of the model, off which\n", + "every other galaxy's Einstein radius hangs. We use the brightest galaxy, identified by `argmax` over the\n", + "measured luminosities \u2014 a measurement, not an assumption about which galaxy happens to be listed first.\n", + "\n", + "Anchoring on a galaxy the model is already fitting is the key trick: the anchor's `einstein_radius` is not a new\n", + "parameter, so the relation itself will add **zero** free parameters to the model." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "anchor_index = int(np.argmax(luminosities))\n", + "\n", + "luminosity_anchor = luminosities[anchor_index]\n", + "centre_anchor = centres[anchor_index]\n", + "\n", + "tied_indexes = [i for i in range(len(centres)) if i != anchor_index]\n", + "\n", + "print(f\"Anchor galaxy index: {anchor_index}, L_anchor = {luminosity_anchor:.4f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Over Sampling__\n", + "\n", + "As in the previous tutorial, we apply adaptive over-sampling centred on both lens galaxies, so the steeply\n", + "varying central light of each is computed accurately." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=dataset.grid,\n", + " sub_size_list=[4, 2, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=centres,\n", + ")\n", + "\n", + "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Light Via MGE__\n", + "\n", + "Every galaxy's light is modelled with a Multi-Gaussian Expansion, composed via the `al.model_util.mge_model_from`\n", + "convenience function (this wraps the `Basis` of linear `Gaussian` profiles we built by hand in chapter 2 into a\n", + "single call \u2014 the composition it returns is the same).\n", + "\n", + "Because the galaxy centres were measured above, we fix each MGE's centre to them (passing the measured centre as\n", + "both the `centre` and `centre_fixed` inputs, which fixes it rather than giving it a prior). The Gaussian\n", + "`sigma` values are fixed by the basis and the intensities are solved for by linear algebra, so each galaxy's\n", + "light costs just **two** non-linear parameters: its elliptical components. This is the \"light adds (almost) no\n", + "parameters\" half of the recipe from the introduction.\n", + "\n", + "The anchor galaxy also gets its mass: a free `Isothermal` profile with its centre fixed at the measured light\n", + "centre. Its `einstein_radius` is an ordinary free parameter \u2014 but it is also what the scaling relation below\n", + "hangs off." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "bulge_anchor = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=10,\n", + " centre=centre_anchor,\n", + " centre_fixed=centre_anchor,\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + ")\n", + "\n", + "mass_anchor = af.Model(al.mp.Isothermal)\n", + "mass_anchor.centre = centre_anchor\n", + "\n", + "lens_anchor = af.Model(\n", + " al.Galaxy,\n", + " redshift=0.5,\n", + " bulge=bulge_anchor,\n", + " mass=mass_anchor,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The source galaxy is also an MGE, with a free centre (we have no pre-measured position for the unlensed source)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "source_bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=10,\n", + " gaussian_per_basis=1,\n", + " centre_prior_is_uniform=False,\n", + ")\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0, bulge=source_bulge)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Scaling Relation__\n", + "\n", + "Now the relation itself. The fainter galaxy's mass is a spherical isothermal profile whose `einstein_radius` is\n", + "not given a prior \u2014 instead, it is *assigned an expression*: the anchor's `einstein_radius` multiplied by the\n", + "luminosity ratio raised to the Faber-Jackson exponent of 0.5.\n", + "\n", + "This single line is the whole trick. Because `mass_anchor.einstein_radius` is the model's own free parameter,\n", + "multiplying it by a fixed number produces a **derived quantity**, not a new parameter. Whenever the non-linear\n", + "search proposes an Einstein radius for the anchor, every tied galaxy's Einstein radius follows automatically.\n", + "\n", + "The tied galaxy keeps its MGE bulge (its light is inside the mask and must be fitted), so it costs two light\n", + "parameters and **zero** mass parameters.\n", + "\n", + "An honest aside before we continue: this dataset's two galaxies are comparably bright, and for a genuine system\n", + "of two co-dominant deflectors the previous tutorial's model \u2014 full mass freedom for both \u2014 is the right choice.\n", + "A galaxy contributing half the lensing deserves its own parameters. We tie one here so you can learn the\n", + "machinery on a familiar dataset; the regime where the relation genuinely earns its keep is a *population* of\n", + "fainter members, which we build towards below and which dominates the next tutorial." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "scaling_exponent = 0.5\n", + "\n", + "scaling_galaxies_list = []\n", + "\n", + "for i in tied_indexes:\n", + " bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=10,\n", + " centre=centres[i],\n", + " centre_fixed=centres[i],\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + " )\n", + "\n", + " mass = af.Model(al.mp.IsothermalSph)\n", + " mass.centre = centres[i]\n", + " mass.einstein_radius = (\n", + " mass_anchor.einstein_radius\n", + " * (luminosities[i] / luminosity_anchor) ** scaling_exponent\n", + " )\n", + "\n", + " scaling_galaxies_list.append(\n", + " af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass)\n", + " )\n", + "\n", + "scaling_galaxies = af.Collection(scaling_galaxies_list)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model__\n", + "\n", + "The model has two top-level collections: `galaxies` (the anchor and the source) and `scaling_galaxies` (the tied\n", + "population). This is not just presentational \u2014 `scaling_galaxies` is the collection name **PyAutoLens** expects a\n", + "scaling population in: the analysis appends it to the tracer's galaxy list when fitting, and results loaded back\n", + "via the aggregator restore it. It also keeps `model.info` readable, with the tied population visibly separate\n", + "from the freely modelled galaxies.\n", + "\n", + "Inspecting `model.info` below, note that the tied galaxies have no `einstein_radius` prior of their own \u2014 it is\n", + "listed as a function of the anchor's." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "model = af.Collection(\n", + " galaxies=af.Collection(lens=lens_anchor, source=source),\n", + " scaling_galaxies=scaling_galaxies,\n", + ")\n", + "\n", + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Parameter Counts__\n", + "\n", + "Let's verify the claims above by counting, rather than believing. First we recompose the previous tutorial's\n", + "style of model on this dataset \u2014 every galaxy with its own free elliptical `Isothermal` mass (centres fixed at\n", + "the measured light centres, as above, so the comparison is like-for-like)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "per_galaxy_lens_dict = {}\n", + "\n", + "for i in range(len(centres)):\n", + " bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=10,\n", + " centre=centres[i],\n", + " centre_fixed=centres[i],\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + " )\n", + "\n", + " mass = af.Model(al.mp.Isothermal)\n", + " mass.centre = centres[i]\n", + "\n", + " per_galaxy_lens_dict[f\"lens_{i}\"] = af.Model(\n", + " al.Galaxy, redshift=0.5, bulge=bulge, mass=mass\n", + " )\n", + "\n", + "model_per_galaxy = af.Collection(\n", + " galaxies=af.Collection(**per_galaxy_lens_dict, source=source),\n", + ")\n", + "\n", + "print(f\"Free parameters, per-galaxy model (previous tutorial): {model_per_galaxy.prior_count}\")\n", + "print(f\"Free parameters, scaling-relation model: {model.prior_count}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The scaling-relation model is smaller. To isolate exactly what the relation saves, we compose one more variant:\n", + "identical to the scaling-relation model in every way, except each tied galaxy's `einstein_radius` is freed with\n", + "a uniform prior instead of tied. The difference in parameter count must equal the number of tied galaxies \u2014\n", + "one saved `einstein_radius` each." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "freed_galaxies_list = []\n", + "\n", + "for i in tied_indexes:\n", + " bulge = al.model_util.mge_model_from(\n", + " mask_radius=mask_radius,\n", + " total_gaussians=10,\n", + " centre=centres[i],\n", + " centre_fixed=centres[i],\n", + " sigma_min=dataset.pixel_scales[0] / 10.0,\n", + " )\n", + "\n", + " mass = af.Model(al.mp.IsothermalSph)\n", + " mass.centre = centres[i]\n", + " mass.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=2.0)\n", + "\n", + " freed_galaxies_list.append(\n", + " af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass)\n", + " )\n", + "\n", + "model_freed = af.Collection(\n", + " galaxies=af.Collection(lens=lens_anchor, source=source),\n", + " scaling_galaxies=af.Collection(freed_galaxies_list),\n", + ")\n", + "\n", + "print(f\"\\nGalaxies tied by the relation: {len(scaling_galaxies_list)}\")\n", + "print(f\"Free parameters, tier tied: {model.prior_count}\")\n", + "print(f\"Free parameters, tier freed: {model_freed.prior_count}\")\n", + "\n", + "assert model_freed.prior_count - model.prior_count == len(scaling_galaxies_list)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Scaling To Many Galaxies__\n", + "\n", + "Saving one parameter looks modest \u2014 the point is what happens as the population grows. Below we imagine this\n", + "lens had twelve additional member galaxies (we invent centres on a ring and plausible luminosities; this model\n", + "is composed for counting only, **not** fitted \u2014 our dataset contains no such galaxies!).\n", + "\n", + "Each imagined member is mass-only (faint distant perturbers whose light is negligible or outside the mask need\n", + "no light model \u2014 the mass-only counterpart of the intermediate options tutorial 1 discussed), with its Einstein\n", + "radius tied to the anchor by the same relation. We count the parameters of this 14-galaxy lens model, and of its nightmare per-galaxy twin." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "total_members = 12\n", + "\n", + "member_centres = [\n", + " (2.5 * float(np.sin(angle)), 2.5 * float(np.cos(angle)))\n", + " for angle in np.linspace(0.0, 2.0 * np.pi, total_members, endpoint=False)\n", + "]\n", + "\n", + "member_luminosities = [\n", + " float(luminosity) for luminosity in np.linspace(0.05, 0.5, total_members) * luminosity_anchor\n", + "]\n", + "\n", + "members_tied_list = []\n", + "members_freed_list = []\n", + "\n", + "for centre, luminosity in zip(member_centres, member_luminosities):\n", + " mass_tied = af.Model(al.mp.IsothermalSph)\n", + " mass_tied.centre = centre\n", + " mass_tied.einstein_radius = (\n", + " mass_anchor.einstein_radius\n", + " * (luminosity / luminosity_anchor) ** scaling_exponent\n", + " )\n", + "\n", + " members_tied_list.append(af.Model(al.Galaxy, redshift=0.5, mass=mass_tied))\n", + "\n", + " mass_freed = af.Model(al.mp.IsothermalSph)\n", + " mass_freed.centre = centre\n", + " mass_freed.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=2.0)\n", + "\n", + " members_freed_list.append(af.Model(al.Galaxy, redshift=0.5, mass=mass_freed))\n", + "\n", + "model_many_tied = af.Collection(\n", + " galaxies=af.Collection(lens=lens_anchor, source=source),\n", + " scaling_galaxies=af.Collection(scaling_galaxies_list + members_tied_list),\n", + ")\n", + "\n", + "model_many_freed = af.Collection(\n", + " galaxies=af.Collection(lens=lens_anchor, source=source),\n", + " scaling_galaxies=af.Collection(scaling_galaxies_list + members_freed_list),\n", + ")\n", + "\n", + "print(f\"Free parameters, 2-galaxy scaling model: {model.prior_count}\")\n", + "print(f\"Free parameters, 14-galaxy scaling model: {model_many_tied.prior_count}\")\n", + "print(f\"Free parameters, 14-galaxy per-galaxy-mass model: {model_many_freed.prior_count}\")\n", + "\n", + "assert model_many_tied.prior_count == model.prior_count" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Twelve more galaxies, zero more parameters. The tied model's dimensionality is *independent of the number of\n", + "galaxies* \u2014 this is the property that makes group- and cluster-scale lens modeling possible at all, and it is\n", + "why the assertion above is worth having in the script rather than in prose." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model Fit__\n", + "\n", + "We now fit the scaling-relation model to the data, using the familiar `Nautilus` search and `AnalysisImaging`\n", + "object. The parameter space is barely larger than a single-lens fit, which is the whole point." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\") / \"chapter_4\",\n", + " name=\"tutorial_3_scaling_relation\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", + ")\n", + "\n", + "analysis = al.AnalysisImaging(dataset=dataset)\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_3_scaling_relation\"\n", + " \" folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result = search.fit(model=model, analysis=analysis)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results__\n", + "\n", + "In `result.info`, the tied galaxy has no `einstein_radius` entry of its own \u2014 it is reported as a derived\n", + "function of the anchor's, which is what a tied parameter looks like in the output. Its posterior uncertainty\n", + "comes entirely from the anchor's." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)\n", + "\n", + "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The fit is good \u2014 but we should ask *why* it is good, because the answer is the honest heart of this tutorial.\n", + "\n", + "The tied model can only fit well if the true galaxies actually lie close to the relation we imposed. We can\n", + "check, because this is simulated data and we know the truth. Below we compare the fainter galaxy's true Einstein\n", + "radius against the value the relation predicts from the anchor's true Einstein radius and the luminosity ratio." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "einstein_radii_true = [\n", + " float(galaxy.mass.einstein_radius) for galaxy in lens_galaxies_true\n", + "]\n", + "\n", + "for i in tied_indexes:\n", + " einstein_radius_predicted = einstein_radii_true[anchor_index] * (\n", + " (luminosities[i] / luminosity_anchor) ** scaling_exponent\n", + " )\n", + "\n", + " print(f\"Tied galaxy {i}: true einstein_radius = {einstein_radii_true[i]:.3f}\")\n", + " print(f\"Tied galaxy {i}: relation predicts = {einstein_radius_predicted:.3f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The prediction lands within a few percent of the truth: this simulated pair happens to sit almost exactly on the\n", + "Faber-Jackson relation. That is why the tied fit succeeds. Real galaxies are not always so obliging.\n", + "\n", + "__Limitations__\n", + "\n", + "A scaling relation is an assumption, and it is worth being clear-eyed about what we assumed:\n", + "\n", + "**Scatter.** Faber-Jackson is a statistical relation with real intrinsic scatter \u2014 at fixed luminosity,\n", + "galaxies show a spread in velocity dispersion, and therefore in Einstein radius. A tied model has no freedom to\n", + "absorb this scatter: it asserts every member sits exactly on the relation. For a population of faint members\n", + "whose individual lensing effects are small, the errors average down and this is fine. For any single galaxy\n", + "that matters a lot to the fit, it is a risk.\n", + "\n", + "**Deviant members.** Worse than scatter is a member that is systematically off the relation \u2014 a tidally\n", + "stripped satellite that has lost dark matter but kept its stars, a galaxy with an unusual mass-to-light ratio,\n", + "or a misclassified foreground/background interloper assigned a luminosity at the wrong distance. Because the\n", + "tied model *cannot* adjust that one galaxy, the non-linear search compensates the only way it can: by biasing\n", + "the parameters it does control \u2014 the anchor's mass, the source \u2014 dragging the whole lens model away from the\n", + "truth. One bad member can poison an otherwise excellent fit, and nothing in the residuals will neatly point at\n", + "the culprit.\n", + "\n", + "**The anchor itself.** Anchoring on the brightest galaxy assumes the anchor sits on the relation too. If it is\n", + "the anomalous one, every tied galaxy inherits its anomaly.\n", + "\n", + "The defence against all three is **selective freedom**. The tiers can be mixed within one model: keep the long\n", + "tail of faint members tied, but promote any galaxy that is bright, close to the lensed images, or visibly\n", + "suspicious to its own free mass profile \u2014 exactly as the extra galaxies were treated earlier in this chapter. A\n", + "middle option bounds a member's free Einstein radius using its luminosity (free, but only within a factor of the\n", + "relation's prediction). And the relation itself can be loosened: freeing `scaling_exponent` as a fit parameter\n", + "costs a single dimension *for the entire population* and is a useful systematics test. The `autolens_workspace`'s\n", + "`imaging/features/scaling_relation` example shows all of these tiers working together.\n", + "\n", + "__Lens Environments__\n", + "\n", + "Everything above was framed around lenses with multiple deflectors, but the same machinery solves a problem we\n", + "met at the start of this chapter: the **environment** of an ordinary single-galaxy lens.\n", + "\n", + "The first tutorial of this chapter modelled a nearby extra galaxy individually, which was fine for one.\n", + "But a deep image of a real lens can reveal dozens of galaxies near the line of sight, each contributing a small\n", + "perturbation to the deflection field. Modeling them individually is hopeless; ignoring them entirely can bias\n", + "precision measurements. With a scaling relation they cost nothing: measure each galaxy's centre and luminosity\n", + "from the image, tie the lot to the main lens galaxy (which is the natural anchor \u2014 it is usually the brightest\n", + "galaxy in the field), and the model gains dozens of physically motivated perturbers with zero added parameters.\n", + "\n", + "__Wrap Up__\n", + "\n", + "In this tutorial we learned:\n", + "\n", + "1. Luminosity predicts lensing mass, via mass-follows-light arguments and the Faber-Jackson relation\n", + " (`L ~ \\sigma^4`, `\\theta_E ~ \\sigma^2`, hence `\\theta_E ~ L^{0.5}`).\n", + "\n", + "2. A scaling relation replaces per-galaxy mass freedom with a shared relation anchored on a parameter the model\n", + " already fits, so tied galaxies add **zero** free parameters \u2014 verified by explicit parameter counting.\n", + "\n", + "3. Combined with MGE light profiles, a many-galaxy lens model stays low-dimensional: dimensionality is\n", + " independent of the number of galaxies.\n", + "\n", + "4. The relation is an assumption: scatter and deviant members can bias the whole model, and the remedy is\n", + " selective freedom \u2014 tie the faint majority, free the important few.\n", + "\n", + "5. The same machinery models the environments of single-lens systems, at no parameter cost.\n", + "\n", + "So far, we tied galaxy to galaxy. But the biggest lenses in the Universe \u2014 galaxy groups and clusters \u2014 contain\n", + "something the galaxies themselves cannot account for: a massive dark matter halo enveloping the whole system,\n", + "holding most of its mass. In the next tutorial we step up to group-scale lenses, where the scaling relation\n", + "becomes essential (there are simply too many member galaxies to free) and a dark matter halo joins the model as\n", + "a new component. There, we will also meet a different way to normalise the relation \u2014 a shared free\n", + "normalization at a fixed reference luminosity, rather than an anchor galaxy \u2014 and truncated mass profiles, which\n", + "describe members whose outer dark matter has been tidally stripped by the very halo we are adding." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_4_scaling_up_lensing/tutorial_4_group_scale.ipynb b/notebooks/chapter_4_scaling_up_lensing/tutorial_4_group_scale.ipynb new file mode 100644 index 0000000..140ae3f --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/tutorial_4_group_scale.ipynb @@ -0,0 +1,923 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 4: Group Scale\n", + "=======================\n", + "\n", + "In the previous tutorials we took our first steps beyond the single lens galaxy: we included extra galaxies near\n", + "the lens in the model, we modeled systems where two or more galaxies of comparable mass share the lensing, and we\n", + "introduced scaling relations, which tie the mass of a galaxy to its light so that adding more galaxies to a model\n", + "does not mean adding more free parameters.\n", + "\n", + "In this tutorial we put all of those tools together and take the first big step up the \"regime ladder\" of strong\n", + "lensing: the **group scale**.\n", + "\n", + "In the standard cosmological model (Lambda Cold Dark Matter, or LCDM), galaxies do not live in isolation. They\n", + "form and evolve inside \"dark matter halos\", and these halos are themselves nested: small halos hosting single\n", + "galaxies merge and fall into larger halos hosting many. A **galaxy group** is the next rung above a single\n", + "galaxy in this hierarchy: a dark matter halo of roughly 10^13 to 10^14 solar masses hosting anywhere from a\n", + "handful to a few dozen galaxies. (Our own Milky Way lives in such a structure, the Local Group.) Above groups\n", + "sit **galaxy clusters**, halos of 10^14 solar masses and beyond hosting hundreds or thousands of galaxies, which\n", + "are the subject of the next tutorial.\n", + "\n", + "When a group acts as a strong lens, several of its galaxies contribute significantly to the deflection of the\n", + "source's light. The brightest galaxy of the group -- called the brightest group galaxy (BGG), or the brightest\n", + "cluster galaxy (BCG) at cluster scale -- typically sits near the centre of the group's dark matter halo and\n", + "dominates the lensing. The other group members perturb the lensed image around it. The group's shared dark\n", + "matter halo may also enter the mass model as its own component, often centred on the BGG where one exists --\n", + "though as we will see, whether it is needed is an explicit modelling choice, not an automatic ingredient.\n", + "\n", + "__Groups vs Multi-Galaxy Lenses__\n", + "\n", + "You may now be wondering how a \"group-scale lens\" differs from the multi-galaxy lenses of the previous tutorials,\n", + "and the honest answer is that the line between them is **blurry**. Every group is a multi-galaxy lens: it has\n", + "multiple galaxies whose mass matters for the lensing. But not every multi-galaxy lens is a group: two galaxies of\n", + "comparable mass that happen to sit close on the sky (or are mid-merger) share the lensing without being embedded\n", + "in a massive shared halo with a member population.\n", + "\n", + "The physically meaningful distinction is the **environment**: a group has a shared dark matter halo, a dominant\n", + "central galaxy, and members orbiting inside the host's potential. That environment leaves signatures in the mass\n", + "model, and the most important one is **tidal truncation**, which we introduce below.\n", + "\n", + "__Tidal Truncation and the dPIE Profile__\n", + "\n", + "A galaxy orbiting inside a group does not keep its full dark matter halo. The tidal field of the host group\n", + "strips the outer, loosely bound parts of the member's halo, so the member's mass distribution is cut off -- or\n", + "\"truncated\" -- at a finite radius. The isothermal profiles we have used so far are a poor description of this:\n", + "their density falls as 1/r^2 forever, so their total mass diverges with radius.\n", + "\n", + "The standard truncated profile of group- and cluster-scale lensing is the **dPIE** (dual Pseudo-Isothermal\n", + "Elliptical) profile, available in **PyAutoLens** as `al.mp.dPIEMassSph`. It behaves isothermally at intermediate\n", + "radii but its density falls off much more steeply beyond a truncation radius, giving it a finite total mass --\n", + "exactly the behaviour we expect for a tidally stripped group member.\n", + "\n", + "__Contents__\n", + "\n", + "- **Initial Setup:** Load the simulated group-scale dataset (auto-simulating it if absent) and plot it.\n", + "- **Mask:** Define the 2D mask, which is larger than at galaxy scale because the group spans more sky.\n", + "- **Galaxy Centres:** Load the centres of the BGG and member galaxies from .json files.\n", + "- **The dPIE Profile:** Introduce the truncated dPIE mass profile and its parameters.\n", + "- **Fitting a Group:** Fit the data with a tracer containing the BGG, members and source.\n", + "- **A Group Halo?:** Add a group-scale dark matter halo to the tracer and see how it changes the fit.\n", + "- **Model Fit:** Compose and fit a group-scale lens model with individually modeled members.\n", + "- **Scaling Relation Members:** Tie the members' masses to their light so they cost almost no parameters.\n", + "- **The Group Scale Sweet Spot:** Why groups are the sweet spot of the familiar lens modeling toolkit.\n", + "- **Wrap Up:** Summary of the script and next steps." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "from pathlib import Path\n", + "\n", + "import autofit as af\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Initial Setup__\n", + "\n", + "We begin by loading the group-scale strong lens dataset `simple`, which was simulated with:\n", + "\n", + " - A main lens galaxy (the BGG) at (0.0\", 0.0\"), with a `SersicSph` light profile and an `IsothermalSph` mass\n", + " profile with a 4.0\" Einstein radius.\n", + " - Two member galaxies at (3.5\", 2.5\") and (-4.4\", -5.0\"), with `SersicSph` light profiles and tidally\n", + " truncated `dPIEMassSph` mass profiles.\n", + " - A single source galaxy with a `SersicCore` light profile." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\") / \"group\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/group.py\"],\n", + " check=True,\n", + " )\n", + "\n", + "dataset = al.Imaging.from_fits(\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When we plot the dataset, the difference from a galaxy-scale lens is immediately clear: the lensed arcs form a\n", + "much larger ring (the BGG's Einstein radius is 4.0\", compared to the ~1.0\" typical of a single galaxy), and two\n", + "smaller galaxies are visible away from the centre. Their light -- and, more importantly for the lensing, their\n", + "mass -- must be included in our analysis." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask__\n", + "\n", + "We define a 7.5\" circular mask, much larger than the 2.5\"-3.0\" masks of previous chapters. It must enclose the\n", + "full lensed arc system and the member galaxies, because all of them contribute light and mass to the region we\n", + "fit." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mask_radius = 7.5\n", + "\n", + "mask = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " radius=mask_radius,\n", + ")\n", + "\n", + "dataset = dataset.apply_mask(mask=mask)\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Galaxy Centres__\n", + "\n", + "Following the convention of the previous tutorials, the centres of the galaxies are loaded from .json files in\n", + "the dataset folder. For a group we distinguish two categories:\n", + "\n", + " - `main_lens_centres`: the main lens galaxies -- here just the BGG at (0.0\", 0.0\") -- which are modeled with\n", + " full freedom.\n", + "\n", + " - `member_centres`: the group members, which are modeled with restrictive assumptions (fixed centres, and\n", + " later a scaling relation) to keep the model dimensionality low.\n", + "\n", + "For real data these centres come from the observed light of each galaxy (e.g. via a click-based GUI or a\n", + "photometry catalogue), exactly as for the extra galaxies of the earlier tutorials." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "main_lens_centres = al.from_json(file_path=dataset_path / \"main_lens_centres.json\")\n", + "\n", + "member_centres = al.from_json(file_path=dataset_path / \"extra_galaxies_centres.json\")\n", + "\n", + "print(f\"Main lens centres: {main_lens_centres}\")\n", + "print(f\"Member centres: {member_centres}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__The dPIE Profile__\n", + "\n", + "Before fitting anything, lets get to know the dPIE profile, since it is the workhorse mass profile of the group\n", + "and cluster regimes.\n", + "\n", + "The `dPIEMassSph` profile is parameterized in the convention of Lenstool (a long-established cluster lensing\n", + "code), which differs from the profiles we have met so far:\n", + "\n", + " - `sigma`: the fiducial velocity dispersion of the galaxy in km/s. This sets the overall lensing strength,\n", + " playing the role the `einstein_radius` played for the isothermal profile (for an isothermal sphere the two\n", + " are directly related: a bigger velocity dispersion means a bigger Einstein radius).\n", + "\n", + " - `r_core`: the core radius in arcseconds, within which the density profile flattens. It is common to fix\n", + " `r_core = 0.0` (a \"vanishing core\", where the profile has an exact analytic form), which we do throughout\n", + " this tutorial.\n", + "\n", + " - `r_cut`: the truncation radius in arcseconds -- the key new parameter. Outside `r_cut` the density falls off\n", + " steeply (as 1/r^4 rather than the isothermal 1/r^2), encoding the tidal stripping of the member's outer dark\n", + " matter halo and giving the profile a finite total mass.\n", + "\n", + " - `redshift_object`, `redshift_source`, `H0`, `Om0`: because `sigma` is a physical velocity, converting it to\n", + " deflection angles requires the lensing geometry and a cosmology. These are fixed inputs describing the lens\n", + " system, not parameters we fit.\n", + "\n", + "Lets make two dPIE profiles with identical `sigma` but different truncation radii -- one truncated at 8.0\" and\n", + "one truncated so far out (1000.0\") that it is effectively untruncated -- and compare their convergence profiles." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dpie_truncated = al.mp.dPIEMassSph(\n", + " centre=(0.0, 0.0),\n", + " sigma=200.0,\n", + " r_core=0.0,\n", + " r_cut=8.0,\n", + " redshift_object=0.5,\n", + " redshift_source=1.0,\n", + ")\n", + "\n", + "dpie_untruncated = al.mp.dPIEMassSph(\n", + " centre=(0.0, 0.0),\n", + " sigma=200.0,\n", + " r_core=0.0,\n", + " r_cut=1000.0,\n", + " redshift_object=0.5,\n", + " redshift_source=1.0,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We evaluate the convergence of both profiles on a 1D line of radial coordinates, using a `Grid2DIrregular` so we\n", + "can choose the radii ourselves." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "radii = np.linspace(0.01, 20.0, 200)\n", + "\n", + "radial_grid = al.Grid2DIrregular([(0.0, radius) for radius in radii])\n", + "\n", + "convergence_truncated = dpie_truncated.convergence_2d_from(grid=radial_grid)\n", + "convergence_untruncated = dpie_untruncated.convergence_2d_from(grid=radial_grid)\n", + "\n", + "plt.semilogy(radii, np.asarray(convergence_truncated), label=\"dPIE (r_cut = 8.0)\")\n", + "plt.semilogy(radii, np.asarray(convergence_untruncated), label=\"dPIE (untruncated)\")\n", + "plt.axvline(x=8.0, color=\"k\", linestyle=\"--\", label=\"r_cut\")\n", + "plt.xlabel(\"Radius (arcseconds)\")\n", + "plt.ylabel(\"Convergence\")\n", + "plt.legend()\n", + "plt.show()\n", + "plt.close()" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Inside `r_cut` the two profiles are nearly identical -- both behave isothermally. Beyond `r_cut`, the truncated\n", + "profile's convergence plummets while the untruncated one keeps its shallow isothermal decline. The truncated\n", + "member therefore contributes far less mass at large radii, which is exactly what tidal stripping does.\n", + "\n", + "This is why the dPIE, and not the isothermal profile, is the standard choice for group and cluster members: the\n", + "extra galaxies of a galaxy-scale lens have no host environment stripping them, so untruncated isothermal\n", + "profiles are fine there, but a member orbiting inside a shared group halo is physically truncated whether or\n", + "not the model includes that halo as an explicit component.\n", + "\n", + "For scale: a member with `sigma = 200.0` km/s at these redshifts has an Einstein radius of ~0.7\", roughly a\n", + "fifth of the BGG's -- a perturber, not a co-dominant lens.\n", + "\n", + "__Fitting a Group__\n", + "\n", + "Now lets fit the data. Following the fitting tutorials of earlier chapters, we first build a tracer from\n", + "galaxies whose light and mass profiles match the true values used to simulate the data, and fit it with the\n", + "`FitImaging` object.\n", + "\n", + "The tracer contains four galaxies: the BGG, the two members and the source. The fit handles all of them\n", + "simultaneously, summing the deflection field of every mass profile to ray-trace the source's light." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "bgg = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.SersicSph(\n", + " centre=(0.0, 0.0), intensity=0.7, effective_radius=2.0, sersic_index=4.0\n", + " ),\n", + " mass=al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=4.0),\n", + ")\n", + "\n", + "member_0 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.SersicSph(\n", + " centre=(3.5, 2.5), intensity=0.9, effective_radius=0.8, sersic_index=3.0\n", + " ),\n", + " mass=al.mp.dPIEMassSph(\n", + " centre=(3.5, 2.5),\n", + " sigma=200.0,\n", + " r_core=0.0,\n", + " r_cut=8.0,\n", + " redshift_object=0.5,\n", + " redshift_source=1.0,\n", + " ),\n", + ")\n", + "\n", + "member_1 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.SersicSph(\n", + " centre=(-4.4, -5.0), intensity=1.866, effective_radius=0.8, sersic_index=3.0\n", + " ),\n", + " mass=al.mp.dPIEMassSph(\n", + " centre=(-4.4, -5.0),\n", + " sigma=240.0,\n", + " r_core=0.0,\n", + " r_cut=13.3,\n", + " redshift_object=0.5,\n", + " redshift_source=1.0,\n", + " ),\n", + ")\n", + "\n", + "source = al.Galaxy(\n", + " redshift=1.0,\n", + " bulge=al.lp.SersicCore(\n", + " centre=(0.0, 0.1),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0),\n", + " intensity=3.0,\n", + " effective_radius=0.4,\n", + " sersic_index=1.0,\n", + " ),\n", + ")\n", + "\n", + "tracer = al.Tracer(galaxies=[bgg, member_0, member_1, source])\n", + "\n", + "fit = al.FitImaging(dataset=dataset, tracer=tracer)\n", + "\n", + "aplt.subplot_fit_imaging(fit=fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Because this is the true model, the residuals are consistent with noise and the log likelihood is as high as it\n", + "can be for this dataset. Note that nothing about the `FitImaging` object changed: the same fitting machinery we\n", + "learnt at galaxy scale simply receives a tracer with more galaxies in it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(f\"Log likelihood of the true model: {fit.log_likelihood}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__A Group Halo?__\n", + "\n", + "You may have noticed that the model above contains no group-scale dark matter halo, despite this tutorial\n", + "opening with the statement that a group is defined by one. Two things are going on here.\n", + "\n", + "First, the BGG's `IsothermalSph` has a 4.0\" Einstein radius -- far larger than the ~1.0\" of an isolated galaxy\n", + "of its luminosity. A mass profile centred on the BGG cannot tell \"stars and dark matter belonging to the BGG\"\n", + "apart from \"group dark matter centred on the BGG\": a single isothermal profile at the group centre soaks up\n", + "both. So the group's halo is partly hidden inside the BGG's mass profile already.\n", + "\n", + "Second, whether to add a *separate* halo component to the model is an **explicit modelling choice**, and this\n", + "dataset was deliberately simulated without one. Lets see what happens if we add one anyway. We add a fifth\n", + "galaxy carrying only mass -- an `IsothermalSph` centred on the BGG, the conventional choice for a group halo\n", + "where a BGG exists -- and refit." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "group_halo = al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=1.5),\n", + ")\n", + "\n", + "tracer_with_halo = al.Tracer(galaxies=[bgg, group_halo, member_0, member_1, source])\n", + "\n", + "fit_with_halo = al.FitImaging(dataset=dataset, tracer=tracer_with_halo)\n", + "\n", + "aplt.subplot_fit_imaging(fit=fit_with_halo)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The fit is now visibly worse -- the extra 1.5\" Einstein radius of mass at the centre over-deflects the source's\n", + "light, displacing the model arcs from the observed ones -- and the log likelihood drops accordingly." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(f\"Log likelihood with a group halo added: {fit_with_halo.log_likelihood}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "On real data the same experiment runs in reverse: fit the group with and without an explicit halo component and\n", + "let the data decide (via the residuals and the Bayesian evidence introduced in earlier chapters). Some groups\n", + "are well described by their galaxies alone, with the central galaxy's profile absorbing the halo; others --\n", + "typically those with the largest Einstein radii or image configurations no galaxy-mass model can reproduce --\n", + "require a dominant halo component, often centred on the BGG.\n", + "\n", + "__Model Fit__\n", + "\n", + "We now perform a proper model-fit with a non-linear search, as a scientist would for a group whose true\n", + "parameters are unknown. We compose the model as follows:\n", + "\n", + " - The BGG's light is a linear `SersicSph` (its centre free) and its mass an `IsothermalSph` [7 parameters].\n", + "\n", + " - Each member's light is a linear `SersicSph` with its centre fixed to the observed centre, and its mass a\n", + " `dPIEMassSph` with a free `sigma`, its centre fixed, `r_core = 0.0` and `r_cut` fixed to a fiducial 10.0\"\n", + " [3 parameters per member, 6 total].\n", + "\n", + " - The source's light is a linear `SersicCore` [6 parameters].\n", + "\n", + "Fixing the member centres to their observed light and fixing `r_cut` follows the reasoning of the extra\n", + "galaxies tutorial: the data rarely constrains a perturber's truncation radius, so we spend our parameter budget\n", + "on the quantity that matters most -- each member's overall mass, via `sigma`. The redshifts and cosmology of\n", + "the dPIE are pinned to their known values, since they are properties of the lens system, not parameters to\n", + "sample." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "# BGG:\n", + "\n", + "bulge = af.Model(al.lp_linear.SersicSph)\n", + "\n", + "mass = af.Model(al.mp.IsothermalSph)\n", + "\n", + "bgg_model = af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass)\n", + "\n", + "# Member Galaxies:\n", + "\n", + "member_list = []\n", + "\n", + "for centre in member_centres:\n", + "\n", + " bulge = af.Model(al.lp_linear.SersicSph)\n", + " bulge.centre = (centre[0], centre[1])\n", + "\n", + " mass = af.Model(al.mp.dPIEMassSph)\n", + " mass.centre = (centre[0], centre[1])\n", + " mass.sigma = af.UniformPrior(lower_limit=0.0, upper_limit=300.0)\n", + " mass.r_core = 0.0 # vanishing core -- fixed; the dPIE is analytic at r_core = 0\n", + " mass.r_cut = 10.0 # truncation fixed at a fiducial radius\n", + " mass.redshift_object = 0.5\n", + " mass.redshift_source = 1.0\n", + " mass.H0 = 67.66 # pinned: model constants, not parameters to sample\n", + " mass.Om0 = 0.30966\n", + "\n", + " member_list.append(af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass))\n", + "\n", + "# Source:\n", + "\n", + "source_model = af.Model(al.Galaxy, redshift=1.0, bulge=af.Model(al.lp_linear.SersicCore))\n", + "\n", + "# Overall Lens Model:\n", + "\n", + "model = af.Collection(\n", + " galaxies=af.Collection(lens=bgg_model, source=source_model),\n", + " extra_galaxies=af.Collection(member_list),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Printing the model's `info` confirms its composition: the BGG and source under `galaxies`, and the two members\n", + "under `extra_galaxies` with only their `sigma`, `effective_radius` and `sersic_index` free.\n", + "\n", + "The total number of free parameters is N=19 -- take note of this number, as it is central to the discussion at\n", + "the end of this tutorial." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)\n", + "\n", + "print(f\"Total free parameters: {model.prior_count}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before fitting, we set up the adaptive over-sampling scheme at the centre of every galaxy in the group (not\n", + "just the BGG), following the standard workspace approach. As in previous tutorials, the details are not\n", + "important yet -- just note that at group scale every galaxy centre needs it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=dataset.grid,\n", + " sub_size_list=[4, 2, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=list(main_lens_centres) + list(member_centres),\n", + ")\n", + "\n", + "dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We fit the model with the `Nautilus` non-linear search, using the same `AnalysisImaging` object as every\n", + "imaging fit so far -- another sign that the group scale does not require new machinery." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\") / \"chapter_4\",\n", + " name=\"tutorial_4_group_scale\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", + ")\n", + "\n", + "analysis = al.AnalysisImaging(dataset=dataset)\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_4_group_scale\"\n", + " \" folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result = search.fit(model=model, analysis=analysis)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The result's `info` and maximum likelihood fit show the model recovers the group: the BGG's Einstein radius,\n", + "the members' velocity dispersions and the source's properties are all inferred from the data." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)\n", + "\n", + "aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Scaling Relation Members__\n", + "\n", + "The model above gave every member its own free `sigma`. With two members that costs two parameters -- harmless.\n", + "But groups can host dozens of members, and one free mass parameter per member quickly bloats the model, slows\n", + "the search, and asks the data to constrain masses it barely feels individually.\n", + "\n", + "The previous tutorial's solution applies directly here: tie the members' masses to their light through a\n", + "scaling relation. More luminous galaxies are more massive, so we write each member's `sigma` and `r_cut` as\n", + "power laws of its luminosity, normalized by a single shared free parameter:\n", + "\n", + " - `sigma = sigma_ref * (L / L_ref) ** 0.25`\n", + " - `r_cut = r_cut_ref * (L / L_ref) ** 0.7`\n", + "\n", + "These exponents follow the convention of modern cluster lensing analyses, and `L_ref` is a fixed reference\n", + "luminosity. The 0.25 is the previous tutorial's Faber-Jackson relation recast for velocity dispersion:\n", + "`L ~ sigma^4` inverts to `sigma ~ L^0.25`, and since `theta_E ~ sigma^2` this reproduces the `theta_E ~ L^0.5`\n", + "scaling we anchored there. Note also the change of normalization: rather than hanging the relation off an\n", + "anchor galaxy the model already fits, we normalise it with a shared free parameter at a fixed reference\n", + "luminosity. Now the *entire member population's mass* costs one free parameter (`sigma_ref`), no matter how\n", + "many members the group has, and each member's observed luminosity acts as a physically motivated prior on its\n", + "mass.\n", + "\n", + "For real data the luminosities come from your photometry catalogue (only luminosity ratios enter the model, so\n", + "any consistent units work). Here we compute them from the true member light profiles, using an aperture\n", + "luminosity within 3.0\"." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "luminosity_list = [\n", + " member_0.bulge.luminosity_within_circle_from(radius=3.0),\n", + " member_1.bulge.luminosity_within_circle_from(radius=3.0),\n", + "]\n", + "\n", + "print(f\"Member luminosities: {luminosity_list}\")\n", + "\n", + "reference_luminosity = luminosity_list[0]\n", + "\n", + "sigma_ref = af.UniformPrior(lower_limit=0.0, upper_limit=300.0)\n", + "r_cut_ref = 8.0 # fixed, like r_cut in the previous model\n", + "\n", + "member_list = []\n", + "\n", + "for centre, luminosity in zip(member_centres, luminosity_list):\n", + "\n", + " luminosity_ratio = float(luminosity) / float(reference_luminosity)\n", + "\n", + " bulge = af.Model(al.lp_linear.SersicSph)\n", + " bulge.centre = (centre[0], centre[1])\n", + "\n", + " mass = af.Model(al.mp.dPIEMassSph)\n", + " mass.centre = (centre[0], centre[1])\n", + " mass.sigma = sigma_ref * luminosity_ratio**0.25\n", + " mass.r_core = 0.0\n", + " mass.r_cut = r_cut_ref * luminosity_ratio**0.7\n", + " mass.redshift_object = 0.5\n", + " mass.redshift_source = 1.0\n", + " mass.H0 = 67.66\n", + " mass.Om0 = 0.30966\n", + "\n", + " member_list.append(af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass))\n", + "\n", + "model = af.Collection(\n", + " galaxies=af.Collection(lens=bgg_model, source=source_model),\n", + " scaling_galaxies=af.Collection(member_list),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that the tied members now live in the `scaling_galaxies` collection \u2014 the home the previous tutorial\n", + "introduced for a scaling-relation population \u2014 whereas the individually-modelled members above sat in\n", + "`extra_galaxies`, like tutorial 1's interlopers. Both collections are included in the ray-tracing; the split\n", + "keeps `model.info` readable and tells the results machinery which galaxies are a tied population.\n", + "\n", + "The model's `info` shows that both members' `sigma` values now trace back to the single shared `sigma_ref`\n", + "prior, and the parameter count has dropped to N=18.\n", + "\n", + "A drop of one parameter looks unremarkable -- until you scale it up. A group with 20 members modeled\n", + "individually would need 20 free mass parameters; with the scaling relation it still needs exactly one. (In\n", + "practice the members' light is handled the same way, using the Multi Gaussian Expansion from earlier in the\n", + "lectures, whose fixed-centre members add almost no free parameters either -- so entire member populations can\n", + "be added to a group model almost for free.)\n", + "\n", + "This dataset was in fact simulated with members that obey this exact scaling relation, so the model remains the\n", + "true model." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(model.info)\n", + "\n", + "print(f\"Total free parameters: {model.prior_count}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We fit this model with an identical search set up." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\") / \"chapter_4\",\n", + " name=\"tutorial_4_group_scale_scaling_relation\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", + ")\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/\"\n", + " \"tutorial_4_group_scale_scaling_relation folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result_scaling = search.fit(model=model, analysis=analysis)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")\n", + "\n", + "print(result_scaling.info)\n", + "\n", + "aplt.subplot_fit_imaging(fit=result_scaling.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__The Group Scale Sweet Spot__\n", + "\n", + "Step back and consider what this tutorial did *not* require: no new fitting machinery, no new analysis object,\n", + "no new data format. The `FitImaging`, `AnalysisImaging`, linear light profiles and `Nautilus` search of the\n", + "galaxy-scale chapters handled the group unchanged -- we just put more galaxies in the tracer, reached for the\n", + "truncated dPIE profile, and used scaling relations to keep the member population cheap. Even the pixelized\n", + "source reconstructions of the previous chapter work at group scale exactly as they do at galaxy scale, and for\n", + "real groups with complex arcs they are the recommended source model.\n", + "\n", + "This is why the group scale is the **sweet spot** of strong lens modeling. The full toolkit of galaxy-scale\n", + "modeling still applies, and the model dimensionality stays low: our individually-modeled-member fit had N=19\n", + "free parameters and the scaling-relation fit N=18. Real group models are of this order -- a few tens of free\n", + "parameters at most (N < 30 or so) -- comfortably within reach of the non-linear searches we already know, with\n", + "run times of minutes to hours.\n", + "\n", + "Push the system any richer, however, and this toolkit breaks down. A cluster hosts hundreds of members and\n", + "lenses many sources at once; its image is too large and complex to fit pixel-by-pixel with a tracer's light\n", + "model, and its model would need hundreds of parameters. Cluster-scale modeling therefore switches to a\n", + "different tool set -- fitting the *positions* of the multiple images of many point-like sources, with member\n", + "catalogues driving the mass model -- which is the subject of the next tutorial.\n", + "\n", + "__Wrap Up__\n", + "\n", + "In this tutorial, we took the tools of the previous tutorials to the group scale. Lets recap what we learnt:\n", + "\n", + "- **Groups in LCDM**: A galaxy group is a dark matter halo of roughly 10^13 to 10^14 solar masses hosting a\n", + " handful to a few dozen galaxies, with the brightest group galaxy (BGG) typically near its centre. The group's\n", + " halo may enter the mass model as its own component, often centred on the BGG -- but whether it is needed is\n", + " an explicit modelling choice, tested by fitting with and without it.\n", + "\n", + "- **Groups vs multi-galaxy lenses**: The line is blurry. All groups are multi-galaxy lenses, but not all\n", + " multi-galaxy lenses are groups -- the group's defining feature is the shared halo environment its members\n", + " orbit within.\n", + "\n", + "- **Tidal truncation and the dPIE**: Members orbiting in the group's potential have their outer dark matter\n", + " tidally stripped, so their profiles are truncated. The `dPIEMassSph` profile encodes this via its `r_cut`\n", + " truncation radius, with `sigma` setting the lensing strength and `r_core` the (usually vanishing) core.\n", + "\n", + "- **Fitting and modeling groups**: The galaxy-scale machinery works unchanged -- a tracer with more galaxies,\n", + " members with fixed centres and free `sigma`, and a `Nautilus` search over N=19 parameters.\n", + "\n", + "- **Scaling relations**: Tying member `sigma` and `r_cut` to luminosity via power laws with one shared free\n", + " normalization means the whole member population costs a single parameter, however many members there are.\n", + "\n", + "- **The sweet spot**: Groups are the largest systems the familiar toolkit -- including pixelized source\n", + " reconstruction -- still handles, with model dimensionality staying below a few tens of parameters.\n", + "\n", + "In the next tutorial we climb the final rung of the ladder to galaxy clusters, where tens to hundreds of member\n", + "galaxies lens many sources simultaneously. There the toolkit changes: we will model the positions of multiply\n", + "imaged point sources rather than every pixel of the data, and drive the mass model from member catalogues via\n", + "a csv interface built on the same scaling relations we used here." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.ipynb b/notebooks/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.ipynb new file mode 100644 index 0000000..03818cf --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.ipynb @@ -0,0 +1,1073 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 5: Cluster Scale\n", + "=========================\n", + "\n", + "Throughout this chapter we have been climbing a ladder of scale: from a single lens galaxy with an extra\n", + "galaxy nearby, to multi-galaxy lenses, to galaxy groups whose members share a common dark matter halo.\n", + "\n", + "This tutorial climbs the final rung: **galaxy clusters**, the most massive gravitationally-bound structures in\n", + "the Universe. A cluster-scale strong lens contains:\n", + "\n", + "- **30+ to hundreds of lens galaxies**: a brightest cluster galaxy (BCG) and bright satellites, plus a large\n", + " population of lower-mass member galaxies, all embedded in...\n", + "\n", + "- **A cluster-scale dark matter halo of more than 10^14 solar masses** (the most massive exceed 10^15 solar\n", + " masses) \u2014 hundreds of times more massive than the halo of a single lens galaxy.\n", + "\n", + "- **Many multiply-imaged background sources** \u2014 often tens of them \u2014 and, crucially, these sources typically\n", + " all sit at **different redshifts**. Light from each source therefore traverses a different path through the\n", + " Universe, and modeling them together requires **multi-plane ray tracing**, which we explain below.\n", + "\n", + "The science a cluster lens delivers is different from the galaxy-scale lensing of earlier chapters:\n", + "\n", + "- **Extreme magnification**: near a cluster's critical curves, magnifications of tens to hundreds are common\n", + " (compared to the factors of a few of galaxy-scale lensing). Clusters act as natural telescopes, letting us\n", + " study intrinsically faint galaxies in the early Universe that would otherwise be invisible \u2014 many of the\n", + " most distant galaxies known were discovered behind lensing clusters.\n", + "\n", + "- **Mapping the cluster's mass**: every multiply-imaged source constrains the cluster's total mass\n", + " distribution along a different line of sight. With many sources, the mass model pins down the shape of the\n", + " cluster's dark matter halo and the substructure within it \u2014 the granularity of dark matter on sub-cluster\n", + " scales \u2014 providing some of the sharpest tests of the cold dark matter model.\n", + "\n", + "The modeling toolkit also changes at this scale. Fitting the extended arcs of every source, with a light\n", + "profile or pixelization each, is computationally prohibitive when there are tens of sources and hundreds of\n", + "member galaxies. The standard approach instead fits only the **positions** of each source's multiple images \u2014\n", + "point-source modeling \u2014 and drives the mass model from spreadsheet-editable **CSV catalogues**, because\n", + "writing Python model code for hundreds of galaxies by hand is not sensible either.\n", + "\n", + "__Contents__\n", + "\n", + "- **Multi-Plane Ray Tracing:** What it means for sources at different redshifts to be lensed together.\n", + "- **Dataset:** Load the simulated cluster's CCD imaging and inspect the true multi-plane tracer.\n", + "- **Point Source Modeling:** Why cluster modeling fits multiple-image positions rather than extended arcs.\n", + "- **Point Datasets:** The per-source multiple-image positions, loaded from a single hand-editable CSV.\n", + "- **The CSV Interface:** The named-galaxy model CSVs and the scaling-member catalogue CSV.\n", + "- **Point Solver:** The solver which finds the image-plane multiple images of a point source.\n", + "- **Model:** Compose the four-tier cluster model (main lenses, scaling members, host halo, sources).\n", + "- **Analysis + Factor Graph:** One analysis per source dataset, combined into a single global fit.\n", + "- **Search:** Configure the Nautilus non-linear search.\n", + "- **Model Fit:** Run the fit.\n", + "- **Results:** A thorough tour of results access for a cluster fit \u2014 tracer, per-member profiles,\n", + " solved source-plane positions, magnifications and image-plane validation.\n", + "- **Customization:** Where to go for the full point-source customization options.\n", + "- **Wrap Up:** Summary and the hand-off to weak lensing.\n", + "\n", + "__Multi-Plane Ray Tracing__\n", + "\n", + "Every lens system in the tutorials so far had two planes: an image plane (the lens galaxies, all at one\n", + "redshift) and a source plane (the lensed galaxy). The lens equation was applied once, mapping image-plane\n", + "coordinates to source-plane coordinates via the deflection angles:\n", + "\n", + "$\\beta = \\theta - \\alpha(\\theta)$\n", + "\n", + "A cluster breaks this picture, because its sources sit at different redshifts. Consider our simulated cluster,\n", + "which has lens galaxies at $z = 0.5$ and two sources, at $z = 1.0$ and $z = 2.0$:\n", + "\n", + "- Light from the $z = 1.0$ source is deflected by the $z = 0.5$ cluster and travels to us: the familiar\n", + " two-plane system.\n", + "\n", + "- Light from the $z = 2.0$ source first passes the $z = 1.0$ plane, then the $z = 0.5$ cluster plane. If any\n", + " galaxy at $z = 1.0$ had mass, it would deflect this light too \u2014 the deflections of successive planes\n", + " accumulate, each applied at the position the ray has reached by that plane.\n", + "\n", + "This is **multi-plane ray tracing**: the lens equation is applied recursively, plane by plane in redshift\n", + "order. There is a second, subtler effect: the strength of a deflection depends on the distances between the\n", + "observer, the deflector and the source. The same cluster bends light from a $z = 2.0$ source through a larger\n", + "effective angle than light from a $z = 1.0$ source, because the geometry of the light paths differs. The\n", + "recursive lens equation accounts for this by scaling each plane's deflections by ratios of cosmological\n", + "(angular diameter) distances.\n", + "\n", + "The maths is more bookkeeping than new physics, and the good news is that the `Tracer` handles all of it\n", + "automatically: give it galaxies at three or more redshifts and it groups them into planes, orders them, scales\n", + "their deflections and applies the recursion. Every source's redshift must simply be *correct* \u2014 hardcoding the\n", + "wrong redshift silently produces the wrong multi-plane geometry, which is why the dataset CSVs below carry a\n", + "redshift for every source." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "import importlib.util\n", + "import sys\n", + "\n", + "if importlib.util.find_spec(\"jax\") is None:\n", + " print(\n", + " \"Skipping this tutorial: it requires the `jax` package (used to \"\n", + " \"accelerate the multi-plane point-source solves), which is not \"\n", + " \"installed (install with `pip install autolens[optional]`).\"\n", + " )\n", + " sys.exit(0)\n", + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import numpy as np\n", + "from pathlib import Path\n", + "\n", + "import autofit as af\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset__\n", + "\n", + "We use a simulated cluster whose scale is kept deliberately small so it runs quickly, but which contains every\n", + "ingredient of the cluster regime: 2 main lens galaxies (a BCG and a satellite), 10 lower-mass member galaxies\n", + "on a luminosity-mass scaling relation, a 10^15.3 solar-mass dark matter halo, and 2 multiply-imaged sources at\n", + "different redshifts (z = 1.0 and z = 2.0). A real cluster simply has more of everything \u2014 and we will see that\n", + "scaling this model up to hundreds of members does not add a single free parameter." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\") / \"cluster\" / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Auto-Simulation__\n", + "\n", + "If the dataset does not already exist on your system, it will be created by running the corresponding\n", + "simulator script. This ensures that all example scripts can be run without manually simulating data first." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "if al.util.dataset.should_simulate(str(dataset_path)):\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.run(\n", + " [sys.executable, \"scripts/simulator/cluster.py\"],\n", + " check=True,\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We first load and plot the cluster's CCD imaging. Note the pixel scale of 0.2\" and the sheer size of the\n", + "field: 100\" x 100\", compared to the ~6\" fields of the galaxy-scale tutorials. The Einstein radius of a\n", + "10^15 solar-mass cluster is ~20-30\", so its multiple images and arcs are spread across a region hundreds of\n", + "times larger in area than a galaxy-scale lens.\n", + "\n", + "The imaging is loaded for visualization only \u2014 as we discuss next, the model is not fitted to these pixels." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "data = al.Array2D.from_fits(file_path=dataset_path / \"data.fits\", pixel_scales=0.2)\n", + "\n", + "aplt.plot_array(array=data, title=\"Cluster CCD Imaging\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The simulator saved the true `Tracer` used to make this dataset, which we can load to inspect the multi-plane\n", + "structure described above. The tracer has three planes: the lens plane at z = 0.5 (holding all 13 lens-plane\n", + "galaxies \u2014 2 main galaxies, 10 scaling members and the dark matter halo) and one source plane per source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer_true = al.from_json(file_path=dataset_path / \"tracer.json\")\n", + "\n", + "print(f\"Number of planes: {len(tracer_true.planes)}\")\n", + "print(f\"Plane redshifts: {[float(plane.redshift) for plane in tracer_true.planes]}\")\n", + "print(f\"Galaxies in lens plane: {len(tracer_true.planes[0])}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Point Source Modeling__\n", + "\n", + "In every previous tutorial we fitted the data pixel-by-pixel: a model image of the lens and source light was\n", + "computed, convolved with the PSF and compared to every image pixel. For a cluster this becomes computationally\n", + "prohibitive:\n", + "\n", + "- Every one of the tens of sources needs its own light model (or pixelization), each adding parameters (or an\n", + " expensive linear inversion) to the fit.\n", + "\n", + "- The image is enormous \u2014 our modest simulated cluster already spans 500 x 500 pixels \u2014 and the deflection\n", + " angles of hundreds of member galaxies would need evaluating at every one of them, for every likelihood\n", + " evaluation.\n", + "\n", + "The standard approach in cluster lensing therefore simplifies the data dramatically: each multiply-imaged\n", + "source is reduced to the **positions of its multiple images** \u2014 the (y,x) coordinates of the brightest pixel\n", + "of each image, measured from the imaging. The model then only has to answer a much cheaper question: does this\n", + "mass model ray-trace all of a source's observed image positions back to the same source-plane location?\n", + "\n", + "This is **point-source modeling**. We are deliberately discarding the information in the arcs' extended light\n", + "in exchange for a likelihood that is fast enough to evaluate with hundreds of galaxies and tens of sources.\n", + "(For lensed quasars and supernovae \u2014 genuine point sources \u2014 nothing is even discarded.) Extended-source\n", + "cluster modeling, where individual arcs are fitted at pixel level with the pixelizations of the previous\n", + "chapter, does exist in **PyAutoLens** as a specialised follow-up analysis \u2014 see the `autolens_workspace`\n", + "cluster examples \u2014 but position fitting is the workhorse that published cluster mass models are built on.\n", + "\n", + "__Point Datasets__\n", + "\n", + "The positions of each source's multiple images are stored in a single CSV file, `point_datasets.csv`, with\n", + "one row per observed multiple image:\n", + "\n", + " - `name`: the source identifier (e.g. `point_0`). All rows sharing a `name` belong to the same source.\n", + " - `y`, `x`: the image-plane position of the multiple image, in arc-seconds.\n", + " - `positions_noise`: the positional uncertainty in arc-seconds \u2014 how precisely the image's centroid is\n", + " measured (for space-telescope imaging ~0.005\", set by PSF-fitting precision, not the pixel scale).\n", + " - `redshift`: the source redshift, which every row of a group must share. This column is what makes the\n", + " multi-plane geometry reproducible from the file alone.\n", + "\n", + "This CSV is the recommended input format for cluster datasets because it is hand-editable: measuring your own\n", + "cluster's multiple images means filling in a spreadsheet, not writing Python.\n", + "\n", + "`al.list_from_csv` loads the file into a list of `PointDataset` objects, one per source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_list = al.list_from_csv(file_path=dataset_path / \"point_datasets.csv\")\n", + "\n", + "for dataset in dataset_list:\n", + " print(\"Point Dataset Info:\")\n", + " print(dataset.info)\n", + " print(f\"Redshift: {dataset.redshift}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting each dataset shows just how sparse point-source data is: a handful of (y,x) coordinates per source.\n", + "That a full cluster mass model can be constrained by so few numbers is because each coordinate is measured to\n", + "milli-arcsecond precision, and because every source at its own redshift probes the mass distribution\n", + "differently." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for dataset in dataset_list:\n", + " aplt.plot_grid(\n", + " grid=al.Grid2DIrregular(np.atleast_2d(dataset.positions)),\n", + " title=dataset.name,\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__The CSV Interface__\n", + "\n", + "With hundreds of member galaxies, composing the lens model in Python \u2014 one `af.Model(al.Galaxy)` per galaxy,\n", + "as in every previous tutorial \u2014 stops being sensible. The cluster workflow therefore defines the model in CSV\n", + "files as well, which the simulator wrote alongside the data:\n", + "\n", + " - `mass.csv`: every individually-modelled mass profile \u2014 the two main lens galaxies' `dPIEMassSph` profiles\n", + " and the host halo's `NFWMCRLudlowSph`.\n", + " - `light.csv`: the light profiles (not used in point-source modeling, since light does not lens).\n", + " - `point.csv`: the source galaxies' `Point` components.\n", + "\n", + "Each row of these CSVs carries:\n", + "\n", + " - `galaxy`: a galaxy name (e.g. `lens_0`, `host_halo`, `source_0`). Rows sharing a name compose into one\n", + " galaxy.\n", + " - `attr_name`: the attribute the profile is bound to on the galaxy (e.g. `mass`, `dark`, `point_0`).\n", + " - `profile_class`: the profile class name (e.g. `dPIEMassSph`), looked up in `al.mp` / `al.lp` / `al.ps`.\n", + " - The profile's constructor parameters as columns (tuples like `centre` split into `y` / `x`); parameters a\n", + " row's class does not use are left blank.\n", + " - `redshift`: the galaxy's redshift.\n", + "\n", + "The lower-mass member population lives in a fourth, simpler catalogue, `scaling_galaxies.csv`, with one row\n", + "per member and columns `y, x, luminosity`. No profile class, no mass parameters \u2014 because, as the previous\n", + "tutorial showed, the members' masses are not free parameters but derive from their luminosities via a scaling\n", + "relation. The catalogue is exactly what an observer measures: where each member is and how bright it is.\n", + "\n", + "This is the key scaling property of the CSV interface: modeling a richer cluster means appending rows to\n", + "`scaling_galaxies.csv` (and, for another individually-modelled galaxy or halo, to `mass.csv`) \u2014 the Python\n", + "below does not change, and the number of free parameters does not grow with the member count.\n", + "\n", + "We load the mass and point families (`al.galaxy_models_from_csv`) and the member catalogue\n", + "(`al.galaxy_table_from_csv`):" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mass_table = al.galaxy_models_from_csv(\n", + " file_path=dataset_path / \"mass.csv\", family=\"mass\"\n", + ")\n", + "point_table = al.galaxy_models_from_csv(\n", + " file_path=dataset_path / \"point.csv\", family=\"point\"\n", + ")\n", + "\n", + "scaling_galaxies_table = al.galaxy_table_from_csv(\n", + " file_path=dataset_path / \"scaling_galaxies.csv\"\n", + ")\n", + "scaling_galaxies_centres = scaling_galaxies_table.centres\n", + "scaling_galaxies_luminosity_list = scaling_galaxies_table.luminosities\n", + "\n", + "print(f\"Scaling galaxies in catalogue: {len(scaling_galaxies_luminosity_list)}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In a real analysis the member centres and luminosities come from light fits to the imaging data (or an\n", + "external photometric catalogue), and the main-galaxy centres in `mass.csv` are pinned to the observed light\n", + "centres. Fixing the centres to observed values is essential at cluster scale: a handful of multiple-image\n", + "positions cannot constrain hundreds of free centre coordinates, but the imaging measures them for free.\n", + "\n", + "__Point Solver__\n", + "\n", + "Point-source modeling needs one new piece of machinery: given a mass model and a source-plane (y,x)\n", + "coordinate, where in the image plane do its multiple images appear? Answering this means solving the lens\n", + "equation *backwards*, which has no analytic solution.\n", + "\n", + "The `PointSolver` solves it numerically: it tiles the image plane with triangles, ray-traces each triangle's\n", + "corners to the source plane, keeps the triangles that land on the source-plane coordinate, and subdivides\n", + "them repeatedly until the image positions are located to a precision of `pixel_scale_precision`.\n", + "\n", + "The solver's grid must span the full 100\" cluster field \u2014 the multiple images sit tens of arc-seconds from\n", + "the cluster centre. The `magnification_threshold` discards heavily demagnified images (e.g. the faint central\n", + "image of a typical lens configuration, which real observations do not detect)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = al.Grid2D.uniform(\n", + " shape_native=(100, 100),\n", + " pixel_scales=1.0, # The pixel-scale converts pixel units to arc-seconds.\n", + ")\n", + "\n", + "solver = al.PointSolver.for_grid(\n", + " grid=grid, pixel_scale_precision=0.001, magnification_threshold=0.1\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model__\n", + "\n", + "We now compose the cluster model, which has four tiers of component \u2014 the same four tiers every published\n", + "cluster mass model is built from:\n", + "\n", + " - **Main lens galaxies (2)**: the BCG and satellite, modelled individually with `dPIEMassSph` profiles. As\n", + " the previous tutorial on group-scale lenses discussed, the dPIE is the standard truncated profile of this\n", + " regime: members orbiting in the cluster's shared potential have their outer dark matter tidally stripped,\n", + " so their mass genuinely ends at a finite truncation radius `r_cut`. Each main galaxy has free `sigma` (its\n", + " fiducial velocity dispersion, setting the lens strength) and free `r_cut`, with its centre fixed to the\n", + " observed light centre [4 free parameters].\n", + "\n", + " - **Scaling-tier members (10)**: `dPIEMassSph` profiles with centres fixed from the catalogue and `sigma` /\n", + " `r_cut` derived from each member's luminosity via the scaling relation introduced earlier in this chapter:\n", + " sigma = sigma_ref * (L / L_ref)^0.25 and r_cut = r_cut_ref * (L / L_ref)^0.7. Only the normalization\n", + " `sigma_ref` \u2014 the velocity dispersion of a member at the reference luminosity \u2014 is free. Whether the\n", + " catalogue holds 10 members or 300, the tier costs [1 free parameter].\n", + "\n", + " - **Host dark matter halo (1)**: a standalone galaxy carrying the cluster's `NFWMCRLudlowSph` halo, centred\n", + " on the BCG, with a free total mass `mass_at_200` [1 free parameter].\n", + "\n", + " - **Source galaxies (2)**: parameter-free `PointSolved` components. Rather than sampling each source's\n", + " (y,x) source-plane centre as free parameters, the fit solves for the centre analytically \u2014 the\n", + " precision-weighted mean of the back-traced image positions. With tens of sources this matters: it removes\n", + " two parameters per source from the search [0 free parameters].\n", + "\n", + "**Total: N = 6 free parameters** \u2014 for a model containing 13 galaxies and 2 sources. This is the punchline of\n", + "the cluster workflow: catalogues and scaling relations decouple the model's physical richness from the\n", + "dimensionality of the non-linear search.\n", + "\n", + "`al.galaxy_af_models_from_csv_tables` converts the CSV tables into a dictionary of `af.Model(Galaxy)` objects\n", + "keyed by galaxy name, with every CSV value fixed as a default. We then selectively promote parameters to free\n", + "priors \u2014 exactly the `af.Model` customization API used throughout the earlier chapters, just applied to\n", + "models built from files instead of Python." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "redshift_lens = 0.5\n", + "source_redshifts = [dataset.redshift for dataset in dataset_list]\n", + "\n", + "galaxy_models = al.galaxy_af_models_from_csv_tables(mass_table, point_table)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The main lens galaxies get free dPIE `sigma` / `r_cut`; their centres and redshifts stay fixed at the CSV\n", + "values, and `r_core` stays fixed at the CSV's 0.0 (the vanishing-core convention discussed in the previous\n", + "tutorial). The cosmology constants `H0` / `Om0` are pinned explicitly: they are model *constants*, not\n", + "parameters to sample \u2014 left unset they would inherit priors from the configuration files and float." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for name in (\"lens_0\", \"lens_1\"):\n", + " galaxy_models[name].mass.sigma = af.UniformPrior(\n", + " lower_limit=50.0, upper_limit=600.0\n", + " )\n", + " galaxy_models[name].mass.r_cut = af.UniformPrior(lower_limit=2.0, upper_limit=40.0)\n", + " galaxy_models[name].mass.H0 = 67.66\n", + " galaxy_models[name].mass.Om0 = 0.30966" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The host halo gets a free `mass_at_200`, spanning the full range of cluster masses with a log-uniform prior\n", + "(mass scales spanning orders of magnitude are sampled in log space, as we saw for intensities in earlier\n", + "chapters)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "galaxy_models[\"host_halo\"].dark.mass_at_200 = af.LogUniformPrior(\n", + " lower_limit=10**14.5, upper_limit=10**16.0\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Each source's `point_i` component is swapped for the parameter-free `al.ps.PointSolved`. The name pairing\n", + "(`point_0`, `point_1`) is what links each model component to the `PointDataset` with the same name \u2014 in a\n", + "multi-source cluster this pairing is what guarantees every source's positions are fitted by the correct\n", + "component." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for i, dataset in enumerate(dataset_list):\n", + " setattr(galaxy_models[f\"source_{i}\"], f\"point_{i}\", af.Model(al.ps.PointSolved))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The scaling tier is composed in a loop over the catalogue, exactly as in the previous tutorial: the single\n", + "shared prior `scaling_sigma_ref` is defined once outside the loop, and each member's `sigma` / `r_cut` derive\n", + "from it via that member's luminosity ratio. The truth value used by the simulator is sigma_ref = 85 km/s." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "scaling_sigma_ref = af.UniformPrior(lower_limit=0.0, upper_limit=200.0)\n", + "scaling_sigma_exponent = 0.25 # alpha (Faber-Jackson)\n", + "scaling_gamma = 0.2 # mass-to-light tilt, fixed\n", + "scaling_rcut_exponent = 1.0 + scaling_gamma - 2.0 * scaling_sigma_exponent # 0.7\n", + "\n", + "reference_luminosity = 1.0\n", + "scaling_r_core_fixed = 0.0\n", + "scaling_r_cut_ref_fixed = 5.0\n", + "\n", + "scaling_galaxies_list = []\n", + "for centre, luminosity in zip(\n", + " scaling_galaxies_centres, scaling_galaxies_luminosity_list\n", + "):\n", + " luminosity_ratio = luminosity / reference_luminosity\n", + "\n", + " mass = af.Model(al.mp.dPIEMassSph)\n", + " mass.centre = tuple(centre)\n", + " mass.sigma = scaling_sigma_ref * luminosity_ratio**scaling_sigma_exponent\n", + " mass.r_core = scaling_r_core_fixed\n", + " mass.r_cut = scaling_r_cut_ref_fixed * luminosity_ratio**scaling_rcut_exponent\n", + " mass.redshift_object = redshift_lens\n", + " mass.redshift_source = max(source_redshifts)\n", + " mass.H0 = 67.66\n", + " mass.Om0 = 0.30966\n", + "\n", + " scaling_galaxies_list.append(af.Model(al.Galaxy, redshift=redshift_lens, mass=mass))\n", + "\n", + "scaling_galaxies = af.Collection(scaling_galaxies_list)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The named galaxies and the scaling tier combine into the overall model. Printing `model.info` confirms the\n", + "parameter count: for all its 15 galaxies, the search explores just 6 dimensions." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "model = af.Collection(\n", + " galaxies=af.Collection(**galaxy_models),\n", + " scaling_galaxies=scaling_galaxies,\n", + ")\n", + "\n", + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Analysis + Factor Graph__\n", + "\n", + "Each source's dataset gets its own `AnalysisPoint`, which defines the log likelihood for that source: how\n", + "well do the model's predictions match this source's observed image positions?\n", + "\n", + "`fit_positions_cls=al.FitPositionsSourceSolved` selects the **solved source-plane** likelihood \u2014 the default\n", + "cluster configuration. Rather than forward-solving the lens equation for the model's image positions on every\n", + "likelihood evaluation (accurate but expensive), it ray-traces the *observed* positions back to the source\n", + "plane and measures how tightly they converge around the analytically-solved source centre, with a weighting\n", + "that maps source-plane scatter back to image-plane precision. No lens-equation solve per evaluation makes\n", + "cluster-scale inference fast; the `PointSolver` is reserved for validating the final model below.\n", + "\n", + "The analyses are then combined through a **factor graph**, the same multi-dataset machinery used for\n", + "multi-wavelength imaging: each analysis becomes a factor sharing the one global model, and the total log\n", + "likelihood is the sum over the per-source likelihoods. This is what lets every source \u2014 each at its own\n", + "redshift, each with its own multi-plane geometry \u2014 constrain the same cluster mass model simultaneously." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "analysis_list = [\n", + " al.AnalysisPoint(\n", + " dataset=dataset,\n", + " solver=solver,\n", + " fit_positions_cls=al.FitPositionsSourceSolved,\n", + " use_jax=True,\n", + " )\n", + " for dataset in dataset_list\n", + "]\n", + "\n", + "analysis_factor_list = [\n", + " af.AnalysisFactor(prior_model=model, analysis=analysis)\n", + " for analysis in analysis_list\n", + "]\n", + "\n", + "factor_graph = af.FactorGraphModel(*analysis_factor_list, use_jax=True)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Search__\n", + "\n", + "We fit the model with Nautilus. The parameter space is only 6-dimensional, so a modest number of live points\n", + "suffices; a richer cluster with several individually-modelled galaxies would warrant more." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\", \"chapter_4\"),\n", + " name=\"tutorial_5_cluster_scale\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " n_batch=50,\n", + " iterations_per_quick_update=10000,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model Fit__\n", + "\n", + "We pass the factor graph's global model and the factor graph itself (as the analysis) to the search. Because\n", + "the likelihood involves no pixel-level computation, a full cluster point-source fit takes minutes \u2014 this is\n", + "the payoff of point-source modeling, given that a pixel-level fit of the same system would take days." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(\n", + " \"\"\"\n", + " The non-linear search has begun running.\n", + "\n", + " This Jupyter notebook cell will progress once the search has completed - this could take a few minutes!\n", + "\n", + " On-the-fly updates every iterations_per_quick_update are printed to the notebook.\n", + " \"\"\"\n", + ")\n", + "\n", + "result_list = search.fit(model=factor_graph.global_prior_model, analysis=factor_graph)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results__\n", + "\n", + "A cluster fit produces the most complex results object we have met: many galaxies across multiple planes,\n", + "tiers whose parameters are derived rather than sampled, and one result per source dataset. This section tours\n", + "how to pull out each quantity you will actually need \u2014 it is worth working through slowly, because navigating\n", + "results is half the practical skill of cluster modeling.\n", + "\n", + "A factor-graph fit returns one `Result` per dataset (in the same order as `dataset_list`). All results share\n", + "the same global model \u2014 and therefore the same samples and maximum likelihood model \u2014 but each carries its own\n", + "dataset and its own maximum likelihood `FitPointDataset`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(f\"Number of results (one per source dataset): {len(result_list)}\")\n", + "\n", + "result = result_list[0]" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: The Maximum Likelihood Instance__\n", + "\n", + "The `max_log_likelihood_instance` is the model instance with the highest likelihood found. Its structure\n", + "mirrors the model composition above: named galaxies under `.galaxies`, the scaling tier under\n", + "`.scaling_galaxies`. We can read off the best-fit parameters of every tier." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "instance = result.max_log_likelihood_instance\n", + "\n", + "print(\"Max Log Likelihood Model:\")\n", + "print(f\"BCG sigma: {instance.galaxies.lens_0.mass.sigma:.1f} km/s\")\n", + "print(f\"BCG r_cut: {instance.galaxies.lens_0.mass.r_cut:.2f} arcsec\")\n", + "print(f\"Satellite sigma: {instance.galaxies.lens_1.mass.sigma:.1f} km/s\")\n", + "print(f\"Halo mass_at_200: {instance.galaxies.host_halo.dark.mass_at_200:.2e} solar masses\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: Per-Member Profiles__\n", + "\n", + "The scaling tier's single free parameter was `sigma_ref`, but the instance contains the fully-realised\n", + "`dPIEMassSph` of every member \u2014 the scaling relation has been applied, so each member carries the `sigma` and\n", + "`r_cut` its luminosity implies. This is how you would tabulate the inferred mass of every member galaxy in a\n", + "real cluster analysis." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for i, member in enumerate(instance.scaling_galaxies):\n", + " print(\n", + " f\"Member {i}: centre={member.mass.centre}, \"\n", + " f\"sigma={member.mass.sigma:.1f} km/s, r_cut={member.mass.r_cut:.2f} arcsec\"\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: The Tracer__\n", + "\n", + "`max_log_likelihood_tracer` assembles every galaxy of the instance \u2014 named tiers and scaling members alike \u2014\n", + "into the best-fit multi-plane `Tracer`. This is the object to use for any lensing calculation with the\n", + "best-fit mass model: convergence maps, deflection fields, critical curves and caustics, all computed exactly\n", + "as in the earlier chapters, but now for the full cluster." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer = result.max_log_likelihood_tracer\n", + "\n", + "print(f\"Tracer planes: {[float(plane.redshift) for plane in tracer.planes]}\")\n", + "print(f\"Galaxies in lens plane: {len(tracer.planes[0])}\")\n", + "\n", + "aplt.subplot_tracer(tracer=tracer, grid=grid)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: Solved Source-Plane Positions__\n", + "\n", + "Each result's `max_log_likelihood_fit` is the `FitPointDataset` of that source. Because we fitted with the\n", + "solved source-plane likelihood, the fit's `positions` object carries the analytically-solved source-plane\n", + "centre \u2014 the model's inferred true position of the source, before lensing. Its `residual_map` lists how far\n", + "each back-traced image lands from that centre (in the source plane): for a good model these residuals are\n", + "tiny, since all images of a source originate from the same point." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for result in result_list:\n", + " fit = result.max_log_likelihood_fit\n", + "\n", + " print(f\"Source '{result.max_log_likelihood_fit.dataset.name}':\")\n", + " print(f\" Solved source-plane centre: {np.asarray(fit.positions.source_plane_coordinate)}\")\n", + " print(f\" Source-plane residuals: {np.asarray(fit.positions.residual_map)}\")\n", + " print(f\" Log likelihood: {float(fit.positions.log_likelihood):.2f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can perform the same back-tracing manually with the tracer, which makes explicit what the fit just did:\n", + "ray-trace each source's observed image positions through the multi-plane lens equation to that source's own\n", + "plane. `plane_index_via_redshift_from` maps a source redshift to its plane, and `traced_grid_2d_list_from`\n", + "returns the positions traced to every plane \u2014 the multi-plane generalisation of the two-plane ray tracing of\n", + "chapter 1." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for dataset in dataset_list:\n", + " plane_index = tracer.plane_index_via_redshift_from(redshift=dataset.redshift)\n", + " traced_grids = tracer.traced_grid_2d_list_from(grid=dataset.positions)\n", + "\n", + " print(f\"Source '{dataset.name}' (z={dataset.redshift}, plane {plane_index}):\")\n", + " print(f\" Back-traced source-plane positions: {traced_grids[plane_index].in_list}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: Magnifications__\n", + "\n", + "The magnification of each multiple image tells us how much the cluster brightens the source at that position \u2014\n", + "the \"natural telescope\" number. It is computed from the local distortion of the multi-plane deflection field\n", + "(its Hessian) at each observed position, via the `LensCalc` object. `plane_j` selects the source's plane, so\n", + "each source's magnifications are evaluated through its own multi-plane chain.\n", + "\n", + "At cluster scale, images near the critical curves can reach magnifications of tens to hundreds \u2014 this is\n", + "precisely the quantity you would quote when using the cluster to study an intrinsically faint background\n", + "galaxy, since the source's true luminosity is the observed luminosity divided by the magnification." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for dataset in dataset_list:\n", + " plane_index = tracer.plane_index_via_redshift_from(redshift=dataset.redshift)\n", + "\n", + " lens_calc = al.LensCalc.from_tracer(\n", + " tracer=tracer, use_multi_plane=True, plane_j=plane_index\n", + " )\n", + " magnifications = lens_calc.magnification_2d_via_hessian_from(grid=dataset.positions)\n", + "\n", + " print(f\"Source '{dataset.name}' image magnifications: {np.abs(np.asarray(magnifications))}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: Image-Plane Validation__\n", + "\n", + "The solved source-plane likelihood never solves the lens equation forwards, so it cannot tell us whether the\n", + "best-fit model predicts the *right number* of multiple images \u2014 a model that would produce an extra image, or\n", + "lose an observed one, looks the same to it. The standard final check is therefore to forward-solve the\n", + "best-fit model with the `PointSolver` and compare the predicted image positions to the observed ones,\n", + "per source." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for result, dataset in zip(result_list, dataset_list):\n", + " fit = result.max_log_likelihood_fit\n", + "\n", + " source_plane_centre = tuple(np.asarray(fit.positions.source_plane_coordinate))\n", + "\n", + " model_positions = solver.solve(\n", + " tracer=tracer,\n", + " source_plane_coordinate=source_plane_centre,\n", + " plane_redshift=dataset.redshift,\n", + " )\n", + "\n", + " print(f\"Source '{dataset.name}':\")\n", + " print(f\" Observed positions ({len(dataset.positions)}): {dataset.positions.in_list}\")\n", + " print(f\" Model positions ({len(model_positions)}): {model_positions.in_list}\")\n", + "\n", + " aplt.plot_grid(\n", + " grid=model_positions,\n", + " title=f\"Model Multiple Images: {dataset.name}\",\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Results: Samples and Errors__\n", + "\n", + "Everything above used the maximum likelihood model, but the `Samples` object gives the full posterior \u2014 the\n", + "same API as chapter 2, unchanged by the cluster's complexity. Of particular scientific interest here are the\n", + "inferred halo mass and the scaling-relation normalization `sigma_ref` (whose truth value in the simulator was\n", + "85 km/s): the marginalised errors on these are what a cluster paper reports." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "samples = result_list[0].samples\n", + "\n", + "median_pdf = samples.median_pdf()\n", + "\n", + "print(\"Median PDF Model:\")\n", + "print(f\"Halo mass_at_200: {median_pdf.galaxies.host_halo.dark.mass_at_200:.2e}\")\n", + "print(f\"BCG sigma: {median_pdf.galaxies.lens_0.mass.sigma:.1f} km/s\")\n", + "\n", + "upper_3_sigma = samples.values_at_upper_sigma(sigma=3.0)\n", + "lower_3_sigma = samples.values_at_lower_sigma(sigma=3.0)\n", + "\n", + "print(\"BCG sigma 3-sigma interval:\")\n", + "print(f\" {lower_3_sigma.galaxies.lens_0.mass.sigma:.1f}\"\n", + " f\" - {upper_3_sigma.galaxies.lens_0.mass.sigma:.1f} km/s\")\n", + "\n", + "aplt.corner_anesthetic(samples=samples)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Customization__\n", + "\n", + "This tutorial used the default point-source setup throughout: `Point` / `PointSolved` source components, the\n", + "solved source-plane likelihood, and default `PointSolver` settings. Point-source modeling has a rich set of\n", + "options beyond these \u2014 image-plane likelihoods and their pairing schemes, free source centres, fitting the\n", + "fluxes of the multiple images, fitting time delays (the observable behind lensed-quasar cosmology) \u2014 and a\n", + "dedicated workspace guide covers them extensively:\n", + "\n", + " - `autolens_workspace/scripts/point_source`: the galaxy-scale point-source examples (lensed quasars and\n", + " supernovae), including the `fit.py` guide to every fit variant and the `features` folder (fluxes, time\n", + " delays, multiple sources).\n", + " - `autolens_workspace/scripts/guides/point_source_pairing.py`: the full matrix of position-fitting schemes\n", + " and when to use each.\n", + " - `autolens_workspace/scripts/cluster`: the cluster-scale workflow this tutorial is built on, including a\n", + " fit to real Hubble Space Telescope data of the cluster Abell 2744 with 188 catalogue members and 7\n", + " sources, the CSV-schema guide (`csv_api.py`) and a step-by-step walkthrough of the point-source\n", + " likelihood (`likelihood_function.py`).\n", + "\n", + "__Wrap Up__\n", + "\n", + "This tutorial reached the top of the strong-lensing mass ladder. Lets recap what we learnt:\n", + "\n", + "- **Clusters**: 30+ to hundreds of lens galaxies inside a dark matter halo of more than 10^14 solar masses,\n", + " lensing many background sources at once. Their science is extreme magnification \u2014 using the cluster as a\n", + " natural telescope onto the faint early Universe \u2014 and mapping the cluster's dark matter distribution.\n", + "\n", + "- **Multi-plane ray tracing**: cluster sources sit at different redshifts, so the lens equation is applied\n", + " recursively through the planes, with each plane's deflections scaled by cosmological distance ratios. The\n", + " `Tracer` handles this automatically, provided every source's redshift is set correctly.\n", + "\n", + "- **Point-source modeling**: fitting the extended light of every source is computationally prohibitive at\n", + " this scale, so the standard workflow fits the positions of each source's multiple images, reducing the\n", + " likelihood to source-plane geometry that evaluates in milliseconds.\n", + "\n", + "- **The CSV interface**: the data (`point_datasets.csv`), the individually-modelled galaxies (`mass.csv` /\n", + " `point.csv`) and the member catalogue (`scaling_galaxies.csv`) are all spreadsheet-editable files. Scaling\n", + " the model to a richer cluster is a row-append, not a code change.\n", + "\n", + "- **Six parameters, fifteen galaxies**: fixed observed centres, a scaling relation for the member population\n", + " and solved source centres decouple the model's physical richness from the dimensionality of the search.\n", + "\n", + "- **Results access**: per-member realised profiles, the multi-plane tracer, solved source-plane centres,\n", + " per-image magnifications and forward-solved image positions \u2014 the complete toolkit for interpreting a\n", + " cluster fit.\n", + "\n", + "Strong lensing \u2014 multiple images, arcs, Einstein rings \u2014 has carried us from single galaxies to the most\n", + "massive structures in the Universe. But a cluster's gravity does not stop deflecting light at the radius\n", + "where multiple images form. Far beyond it, every background galaxy is still subtly sheared \u2014 distorted by a\n", + "percent or less \u2014 and by measuring those distortions statistically across thousands of galaxies, the\n", + "cluster's mass can be mapped out to its edges. That is **weak lensing**, a different regime with different\n", + "data and different statistics, and it is where the next tutorial takes us." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.ipynb b/notebooks/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.ipynb new file mode 100644 index 0000000..6bbf611 --- /dev/null +++ b/notebooks/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.ipynb @@ -0,0 +1,679 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Tutorial 6: Weak Lensing\n", + "========================\n", + "\n", + "Every tutorial in this series so far \u2014 indeed, every fit performed in all four chapters of **HowToLens** \u2014 has been\n", + "a *strong* lensing analysis. Strong lensing occurs when a background galaxy lies so close (in projection) to a\n", + "foreground mass that its light is bent into multiple images, arcs or a complete Einstein ring. These dramatic\n", + "features are what we simulated, fitted and modeled, from the single galaxy-scale lenses of chapters 1 and 2 up to\n", + "the group-scale and cluster-scale systems earlier in this chapter.\n", + "\n", + "Strong lensing is spectacular, but it is also rare and spatially confined. Multiple images only form inside (or very\n", + "near) the Einstein radius, so a strong-lensing analysis constrains the mass distribution only within that region \u2014\n", + "a few arc-seconds for a galaxy, tens of arc-seconds for a cluster core. Outside it, the arcs and multiple images\n", + "simply do not exist, and everything we have built so far has nothing to fit.\n", + "\n", + "Yet the gravity of the foreground mass does not stop at the Einstein radius. *Every* background galaxy, however far\n", + "from the lens centre, has its shape slightly distorted \u2014 sheared \u2014 by the foreground mass it sits behind. This is\n", + "**weak lensing**: a percent-level stretching of each background galaxy's observed ellipticity, tangentially aligned\n", + "around the foreground mass.\n", + "\n", + "The catch is that we cannot see this distortion in any single galaxy. Galaxies are not intrinsically round: a\n", + "typical galaxy has a random unlensed ellipticity of ~0.25 per component, whereas the weak-lensing shear it receives\n", + "is ~0.01-0.1. The signal is buried an order of magnitude below the intrinsic \"shape noise\" of any one object.\n", + "\n", + "The solution is statistical. The intrinsic ellipticities of different galaxies are randomly oriented and average to\n", + "zero, whereas the lensing shear is coherent \u2014 every galaxy near a given point on the sky is sheared the same way.\n", + "Average the measured shapes of many background galaxies and the random part cancels while the lensing part remains.\n", + "This is the fundamental contrast with everything the series has done so far: strong lensing extracts an exquisite\n", + "signal from a single special system, weak lensing extracts a faint signal from the ensemble of many ordinary ones.\n", + "\n", + "This changes what the data even *is*. There is no image to fit, no PSF to convolve, no mask, no pixelized source\n", + "reconstruction. A weak-lensing dataset is a **shear catalogue**: a table of background-galaxy sky positions, each\n", + "with a measured shear estimate (two ellipticity components) and a per-galaxy uncertainty. Fitting one means\n", + "comparing a mass model's predicted shear field, evaluated at those positions, against the measured values.\n", + "\n", + "In this tutorial we simulate a weak-lensing shear catalogue around a cluster-scale lens (entirely in memory \u2014 no\n", + "files are written), visualize its shear field, fit it with a mass model whose parameters we know, and then infer\n", + "those parameters with a non-linear search. You will recognise every step: the `Tracer`, `Fit`, `Analysis` and\n", + "search workflow of the whole series, applied to a completely different kind of data.\n", + "\n", + "This tutorial closes both this chapter and the four core chapters of **HowToLens**.\n", + "\n", + "__Contents__\n", + "\n", + "- **Shear Catalogues:** What a weak-lensing dataset is and where the catalogue comes from in a real analysis.\n", + "- **Mass Scales:** Why weak lensing is a cluster-scale tool, tying to the mass scales of the previous tutorials.\n", + "- **Ray Tracing:** Build a `Tracer` for a cluster-scale lens, the mass whose shear field we will measure.\n", + "- **Source Galaxy Positions:** Draw background galaxies in an annulus outside the strong-lensing core.\n", + "- **Simulate:** Create a `WeakDataset` in memory with `SimulatorShearYX`, adding realistic shape noise.\n", + "- **Visualize:** Plot the shear field as a whisker map and the dataset's 2x2 summary mosaic.\n", + "- **Mass Map:** Invert the shear catalogue into a model-independent convergence map (Kaiser-Squires).\n", + "- **Fitting:** Fit the catalogue with the true mass model via `FitWeak` and inspect its goodness-of-fit.\n", + "- **Model Fit:** Infer the mass model from the shear catalogue alone with `AnalysisWeak` and a non-linear search.\n", + "- **Result:** The inferred model, the fit mosaic and the tangential shear profile.\n", + "- **Joint Strong and Weak Lensing:** How PyAutoLens expects weak lensing to be used \u2014 combined with strong lensing.\n", + "- **Wrap Up:** The end of the chapter, and of the core HowToLens lectures." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import numpy as np\n", + "from pathlib import Path\n", + "\n", + "import autofit as af\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Shear Catalogues__\n", + "\n", + "Before we simulate anything, lets be clear about what a real weak-lensing dataset looks like, because it is unlike\n", + "any data this series has fitted before.\n", + "\n", + "A **shear catalogue** is a table with one row per background galaxy, containing:\n", + "\n", + " - The galaxy's sky position (for us, (y, x) arc-second offsets from a chosen centre).\n", + " - A measured shear estimate (gamma_1, gamma_2) \u2014 in practice the galaxy's measured ellipticity components, which\n", + " estimate the lensing shear because the intrinsic shape averages to zero over many galaxies.\n", + " - A per-galaxy uncertainty, combining the intrinsic shape dispersion (~0.25 per component) with the measurement\n", + " error, added in quadrature.\n", + "\n", + "Producing this catalogue from raw telescope images is a substantial data-reduction task in its own right: galaxies\n", + "must be detected, stars separated from galaxies, the PSF measured and its blurring of galaxy shapes corrected\n", + "(the PSF makes every galaxy look rounder, diluting the shear signal), and calibration biases characterised.\n", + "Dedicated shape-measurement pipelines exist for exactly this job, and entire survey collaborations are built\n", + "around getting it right.\n", + "\n", + "**All of that is outside PyAutoLens's scope.** PyAutoLens assumes the shear catalogue has already been produced\n", + "upstream by such a pipeline, and starts where the catalogue ends: fitting mass models to the measured shears. This\n", + "division of labour is why the weak-lensing API is so lean \u2014 no PSF, no mask, no image \u2014 the hard observational\n", + "systematics were dealt with before the data reached us.\n", + "\n", + "__Mass Scales__\n", + "\n", + "This chapter has climbed a ladder of mass scales, and weak lensing is its top rung. The strength of the statistical\n", + "shear signal around a lens scales with its mass, so whether weak lensing is even measurable depends on where on the\n", + "ladder we stand:\n", + "\n", + " - **A single galaxy** (Einstein radius ~1-2\", the lenses of chapters 1-3): the shear imprinted on background\n", + " galaxies beyond the Einstein radius is far too weak to detect around any one system \u2014 there are too few\n", + " background galaxies carrying too little shear. (Surveys recover it only by *stacking* thousands of lens\n", + " galaxies, a technique called galaxy-galaxy lensing that averages over the lens population.)\n", + "\n", + " - **A group** (Einstein radius ~4-10\", the scale of the group tutorial earlier in this chapter): the signal is\n", + " marginal \u2014 detectable for well-observed individual groups, but noisy.\n", + "\n", + " - **A cluster** (Einstein radius ~10-30\"+, the previous tutorial's scale): the signal is strong enough to measure\n", + " around a *single* system. Hundreds to thousands of background galaxies within a few arc-minutes each carry a\n", + " |gamma| ~ 0.01-0.2 shear, and their average traces the cluster's mass out to radii of arc-minutes \u2014 ten or more\n", + " times further out than the strong-lensing core.\n", + "\n", + "This reach is the scientific point. The dark matter halo of a cluster extends to Mpc scales, far beyond where arcs\n", + "form. Weak lensing is the only lensing probe of that outer halo, which is why it is the tool of choice for cluster\n", + "mass measurements, and it is why this tutorial simulates a cluster-scale lens.\n", + "\n", + "__Ray Tracing__\n", + "\n", + "We begin exactly as the series always has: with a `Tracer`. The lens is a cluster-scale `Isothermal` mass profile\n", + "with an Einstein radius of 25.0\" \u2014 the same order as the previous tutorial's cluster, and a mass for which the\n", + "weak shear signal is genuinely measurable. The source galaxy carries no light profile: weak lensing measures the\n", + "lens's shear field at the background galaxies' positions, so their appearance is irrelevant \u2014 they are pure probes,\n", + "included only to give the `Tracer` its source-plane redshift.\n", + "\n", + "Note also what the lens galaxy lacks: no light profile. A shear catalogue contains no image of the lens, so there\n", + "is no lens light to model \u2014 another simplification compared to every imaging fit in this series." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=25.0,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0),\n", + " ),\n", + ")\n", + "\n", + "source_galaxy = al.Galaxy(redshift=1.0)\n", + "\n", + "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Source Galaxy Positions__\n", + "\n", + "Real weak-lensing measurements avoid the strong-lensing core: inside it the shear is no longer weak (the linear\n", + "shear approximation breaks down), and cluster-member galaxies contaminate the background sample. We therefore draw\n", + "our background galaxies in an **annulus**, from an inner radius of 50\" (twice the Einstein radius, safely into the\n", + "weak regime) to an outer radius of 200\" (over 3 arc-minutes), distributed uniformly in area.\n", + "\n", + "At 50\" from this lens the shear is |gamma| ~ 0.25, and by 200\" it has fallen to |gamma| ~ 0.06 \u2014 every galaxy is a\n", + "weak probe whose individual distortion is at or below the shape noise. With 1500 galaxies (a deep-survey source\n", + "density of ~45 per square arc-minute) the *ensemble* nonetheless carries a decisive signal, exactly as in a real\n", + "cluster analysis.\n", + "\n", + "The positions are stored as an `al.Grid2DIrregular` of (y, x) coordinates \u2014 the same irregular-grid structure the\n", + "point-source tutorials used, because a catalogue of sky positions is not a uniform pixel grid." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "rng = np.random.default_rng(1)\n", + "\n", + "n_galaxies = 1500\n", + "radius_inner = 50.0 # arc-seconds \u2014 inside this we are in the strong-lensing core.\n", + "radius_outer = 200.0 # arc-seconds \u2014 the edge of our simulated weak-lensing field.\n", + "\n", + "radii = np.sqrt(rng.uniform(radius_inner**2.0, radius_outer**2.0, n_galaxies))\n", + "phi = rng.uniform(0.0, 2.0 * np.pi, n_galaxies)\n", + "\n", + "positions = al.Grid2DIrregular(\n", + " values=np.stack([radii * np.sin(phi), radii * np.cos(phi)], axis=1)\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Simulate__\n", + "\n", + "The `SimulatorShearYX` object simulates a weak-lensing shear catalogue from a tracer: its `via_tracer_from` method\n", + "evaluates the tracer's shear field at every galaxy position (by differentiating the deflection-angle field) and\n", + "adds Gaussian shape noise to each measurement.\n", + "\n", + "A `noise_sigma` of 0.25 per shear component is the standard intrinsic shape dispersion of real galaxies \u2014 this is\n", + "the dominant noise source in weak lensing, and it is what makes the per-galaxy signal invisible: the noise on each\n", + "measurement is several times larger than the shear it contains.\n", + "\n", + "Unlike every previous tutorial, we do not write this dataset to disk and reload it. The whole catalogue is a few\n", + "thousand numbers, so we keep it in memory and use it directly \u2014 which also mirrors how you might resimulate\n", + "catalogues on the fly when testing survey sensitivities. (This repo's `scripts/simulator/weak_lensing.py` \u2014 the\n", + "script chapter 1's data tutorial auto-ran to give you your first glimpse of a shear catalogue \u2014 shows the\n", + "disk-based version, with the same JSON output pattern as the imaging simulators.)" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_name = \"weak_lensing\"\n", + "\n", + "simulator = al.SimulatorShearYX(noise_sigma=0.25, seed=1)\n", + "\n", + "dataset = simulator.via_tracer_from(\n", + " tracer=tracer,\n", + " grid=positions,\n", + " name=dataset_name,\n", + ")\n", + "\n", + "print(dataset.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "Weak-lensing data has its own visual language. The shear field is drawn as a **whisker map**: at each galaxy\n", + "position, a line segment whose length is the shear magnitude |gamma| and whose orientation is the shear's position\n", + "angle. The segments are deliberately *headless* (no arrowheads), because shear is a spin-2 quantity \u2014 rotating a\n", + "shear by 180 degrees maps it back onto itself, so a whisker has an orientation but no direction, and an arrowhead\n", + "would suggest information the data does not contain.\n", + "\n", + "Around a massive lens, the whiskers align *tangentially* \u2014 each traces a small piece of a circle around the lens\n", + "centre, the statistical echo of the rings and arcs of strong lensing. Squint at the plot below and you can see it,\n", + "though the shape noise makes it far from obvious galaxy-by-galaxy: this is what a signal an order of magnitude\n", + "below the per-object noise looks like." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_shear_yx_2d(shear_yx=dataset.shear_yx)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`aplt.subplot_weak_dataset` summarises the full dataset as a 2x2 mosaic: the whisker map, the per-galaxy noise-map,\n", + "the shear magnitude |gamma| and the position angle phi at every galaxy." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_weak_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mass Map__\n", + "\n", + "A remarkable property of weak lensing is that the shear catalogue can be inverted directly into a map of the\n", + "convergence `kappa` \u2014 the dimensionless projected mass density we have used throughout the series \u2014 *without\n", + "assuming any mass model at all*. In Fourier space, shear and convergence are related algebraically, so two FFTs\n", + "turn the catalogue into a \"dark matter map\". This is the Kaiser-Squires technique, and it produced some of the most\n", + "famous images in cosmology, such as the mass map of the Bullet Cluster showing dark matter offset from the\n", + "colliding gas.\n", + "\n", + "`aplt.plot_convergence_map` bins the catalogue onto a regular grid, applies a small Gaussian smoothing (per-cell\n", + "shears are shape-noise dominated) and plots the reconstruction. For our simulated cluster it peaks at the lens\n", + "centre at (0.0\", 0.0\"). Two caveats: the map's mean is unconstrained (the mass-sheet degeneracy, which chapter 3\n", + "met in its strong-lensing form) and FFT periodicity produces edge artefacts \u2014 for quantitative masses we fit a\n", + "mass model, which is what the rest of this tutorial does." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_convergence_map(\n", + " shear_yx=dataset.shear_yx,\n", + " shape_native=(30, 30),\n", + " smoothing_sigma_pixels=1.0,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Fitting__\n", + "\n", + "We now fit the catalogue, following the same pattern as every fit in this series: build a model `Tracer`, pass it\n", + "with the dataset to a `Fit` object, and inspect residuals and the log likelihood. For weak lensing the fit object\n", + "is `FitWeak`, and because we simulated the data ourselves we can hand it the *true* tracer and see what a perfect\n", + "model looks like.\n", + "\n", + "`FitWeak` evaluates the model tracer's shear field at the dataset's galaxy positions and compares it with the\n", + "measured shears, assuming each component is independently Gaussian-distributed around the model with the\n", + "per-galaxy noise. Each galaxy contributes **two** independent data points (gamma_1 and gamma_2), so the number of\n", + "degrees of freedom is 2 * n_galaxies, and for a good fit whose residuals are pure shape noise the chi-squared\n", + "should be close to that number.\n", + "\n", + "Compare this to the imaging fits of chapter 1: there, the fit convolved a model image with the PSF and compared\n", + "tens of thousands of pixels; here there is no convolution, no mask and only a few thousand numbers. A weak-lensing\n", + "likelihood is orders of magnitude cheaper \u2014 which is precisely why it is so inexpensive to add to a strong-lensing\n", + "analysis, as discussed at the end of this tutorial." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "fit = al.FitWeak(dataset=dataset, tracer=tracer)\n", + "\n", + "print()\n", + "print(\"Fit Summary (true model)\")\n", + "print(\"------------------------\")\n", + "print(f\"n_galaxies : {dataset.n_galaxies}\")\n", + "print(f\"degrees_of_freedom: {2 * dataset.n_galaxies}\")\n", + "print(f\"chi_squared : {fit.chi_squared:.3f}\")\n", + "print(f\"log_likelihood : {fit.log_likelihood:.3f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As always, an incorrect mass model produces a worse fit. Halving the Einstein radius halves the predicted shear\n", + "everywhere, leaving coherent tangential residuals across the field, and the log likelihood drops accordingly." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer_wrong = al.Tracer(\n", + " galaxies=[\n", + " al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=12.5,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0),\n", + " ),\n", + " ),\n", + " source_galaxy,\n", + " ]\n", + ")\n", + "\n", + "fit_wrong = al.FitWeak(dataset=dataset, tracer=tracer_wrong)\n", + "\n", + "print()\n", + "print(f\"log_likelihood (true model) : {fit.log_likelihood:.3f}\")\n", + "print(f\"log_likelihood (wrong model): {fit_wrong.log_likelihood:.3f}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`aplt.subplot_fit_weak` visualises a weak-lensing fit as a 2x2 mosaic: the observed shear field, the model shear\n", + "field, the two overlaid on one axes (data in black, model in red), and the per-galaxy chi-squared map. For the true\n", + "model the overlay's residuals are short and randomly oriented \u2014 the shape-noise floor \u2014 and the chi-squared map is\n", + "spatially uniform." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_fit_weak(fit=fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model Fit__\n", + "\n", + "In a real analysis we do not know the mass model \u2014 we infer it. The workflow is the one you have used since\n", + "chapter 2: compose a model with `af.Model` and `af.Collection`, choose a non-linear search, create an `Analysis`\n", + "object and fit. For weak lensing the analysis object is `AnalysisWeak`, whose `log_likelihood_function` builds a\n", + "`Tracer` from each sampled parameter set, evaluates its shear field at the catalogue positions and returns the\n", + "Gaussian log likelihood \u2014 the same `FitWeak` machinery we just used by hand.\n", + "\n", + "The model is the lens's `Isothermal` mass [5 parameters]; the source galaxy again carries no components. Note the\n", + "tiny parameter space \u2014 no lens light, no source light, no shear-as-nuisance-parameter (the shear field *is* the\n", + "data here) \u2014 compared to the N=20+ models of the imaging tutorials.\n", + "\n", + "One adjustment is needed, familiar from the group and cluster tutorials: the default priors are tuned for\n", + "galaxy-scale lenses (Einstein radii of 0\"-8\", centres within a fraction of an arc-second). Our cluster's 25\"\n", + "Einstein radius lies entirely outside that prior, so we widen the priors to cluster scales \u2014 always match your\n", + "priors to the mass scale of the system you are fitting." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mass = af.Model(al.mp.Isothermal)\n", + "mass.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=60.0)\n", + "mass.centre.centre_0 = af.GaussianPrior(mean=0.0, sigma=20.0)\n", + "mass.centre.centre_1 = af.GaussianPrior(mean=0.0, sigma=20.0)\n", + "\n", + "lens = af.Model(al.Galaxy, redshift=0.5, mass=mass)\n", + "\n", + "source = af.Model(al.Galaxy, redshift=1.0)\n", + "\n", + "model = af.Collection(galaxies=af.Collection(lens=lens, source=source))\n", + "\n", + "print(model.info)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We fit the model with the nested sampling algorithm Nautilus, as throughout the series. A weak-lensing likelihood\n", + "evaluation takes milliseconds (one shear-field evaluation at 1500 positions plus a chi-squared sum), so this\n", + "5-parameter fit completes in minutes on an ordinary CPU \u2014 a refreshing contrast to the imaging fits of the earlier\n", + "chapters." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "search = af.Nautilus(\n", + " path_prefix=Path(\"howtolens\", \"chapter_4\"),\n", + " name=\"tutorial_6_weak_lensing\",\n", + " unique_tag=dataset_name,\n", + " n_live=100,\n", + " iterations_per_quick_update=5000,\n", + ")\n", + "\n", + "analysis = al.AnalysisWeak(dataset=dataset)\n", + "\n", + "print(\n", + " \"The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_6_weak_lensing\"\n", + " \" folder for live output of the results, images and model.\"\n", + " \" This Jupyter notebook cell with progress once search has completed - this could take some time!\"\n", + ")\n", + "\n", + "result = search.fit(model=model, analysis=analysis)\n", + "\n", + "print(\"The search has finished run - you may now continue the notebook.\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Result__\n", + "\n", + "The result's `info` attribute confirms the search recovers the input mass model \u2014 Einstein radius, ellipticity and\n", + "centre \u2014 from nothing but the statistically-averaged shapes of background galaxies. No arc, no ring, no multiple\n", + "images: the ensemble alone constrains the cluster's mass." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "print(result.info)\n", + "\n", + "aplt.subplot_fit_weak(fit=result.max_log_likelihood_fit)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Cluster weak-lensing measurements are conventionally presented as the **tangential shear profile**: the mean\n", + "tangential shear gamma_t of background galaxies in radial bins about the lens centre, which traces the projected\n", + "mass profile. `aplt.plot_shear_profile` bins our catalogue and, given the fit, overlays the model's profile as a\n", + "line.\n", + "\n", + "The plot also shows the *cross* component gamma_x \u2014 the shear rotated by 45 degrees. Gravitational lensing produces\n", + "no cross component at leading order, so cross points scattering around zero is the standard \"B-mode\" null test of a\n", + "weak-lensing measurement: a systematic contaminating the shapes would show up there." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_shear_profile(\n", + " result.max_log_likelihood_fit,\n", + " centre=(0.0, 0.0),\n", + " bins=8,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Joint Strong and Weak Lensing__\n", + "\n", + "We fitted this catalogue on its own, and weak-lensing-only fits of real catalogues are a fully supported PyAutoLens\n", + "workflow. But the way PyAutoLens really expects weak lensing to be used is **jointly with strong lensing**, because\n", + "the two probes are perfectly complementary:\n", + "\n", + " - **Strong lensing** (everything before this tutorial) pins the mass in the inner region with exquisite precision \u2014\n", + " the positions of multiple images and arcs nail the Einstein mass and the inner density profile \u2014 but says\n", + " nothing beyond the Einstein radius.\n", + "\n", + " - **Weak lensing** (this tutorial) constrains the mass profile out to arc-minute radii \u2014 the outskirts of the dark\n", + " matter halo \u2014 but its centre and inner profile are only weakly pinned by the field's geometry.\n", + "\n", + "Fit both datasets simultaneously with a single mass model \u2014 one analysis for the imaging or point-source data, one\n", + "`AnalysisWeak` for the shear catalogue, combined exactly as the multi-dataset fits earlier in this chapter combined\n", + "their analyses \u2014 and the model is anchored at the centre by strong lensing while its outer profile is measured by\n", + "weak lensing. This hybrid is the state of the art for cluster mass measurements, used across systems like the\n", + "Frontier Fields clusters, and PyAutoLens's shared `Tracer` infrastructure makes it natural: the same mass profiles\n", + "that ray-traced arcs in the previous tutorial produce the shear field fitted here.\n", + "\n", + "The workspace's `autolens_workspace/*/weak` package has the complete treatment:\n", + "\n", + " - `weak/start_here.py`: fits a real shear catalogue of the merging cluster Abell 2744 (\"Pandora's Cluster\") \u2014\n", + " download, quality cuts, mass map, NFW halo fit.\n", + " - `weak/simulator.py`, `weak/fit.py`, `weak/modeling.py`: the disk-based versions of this tutorial's workflow.\n", + " - `weak/likelihood_function.py`: a step-by-step walkthrough of the weak-lensing likelihood.\n", + " - `weak/features/strong_lensing`: the joint strong-plus-weak fits described above.\n", + "\n", + "__Wrap Up__\n", + "\n", + "This tutorial completed the chapter's journey up the mass ladder, and with it the four core chapters of\n", + "**HowToLens**. Lets take stock of how far we have come.\n", + "\n", + "Chapter 1 built lensing from first principles: grids, light and mass profiles, ray tracing, the lens equation, and\n", + "fitting imaging data with a `Tracer`. Chapter 2 turned fitting into inference \u2014 non-linear searches, priors,\n", + "Bayesian model comparison \u2014 so we could *infer* lens models rather than guess them. Chapter 3 replaced analytic\n", + "sources with pixelized reconstructions, letting the data itself reveal the source's morphology.\n", + "\n", + "This chapter then scaled those tools up through every regime nature offers, and in this tutorial:\n", + "\n", + "- **Weak Lensing:** Outside the strong-lensing region every background galaxy is still weakly sheared; the signal\n", + " is invisible per galaxy beneath intrinsic shape noise but emerges statistically over an ensemble.\n", + "\n", + "- **Shear Catalogues:** The weak-lensing dataset is a catalogue of galaxy positions, measured shear estimates and\n", + " uncertainties, produced upstream by a dedicated shape-measurement pipeline \u2014 PyAutoLens fits mass models to it\n", + " and leaves the shape measurement to those pipelines.\n", + "\n", + "- **Mass Scales:** The statistical signal is hopeless around a single galaxy, marginal around a group and\n", + " measurable around a cluster \u2014 and it probes the dark matter halo to radii far beyond the reach of arcs.\n", + "\n", + "- **The Same Workflow:** `SimulatorShearYX`, `FitWeak` and `AnalysisWeak` slot into the identical\n", + " simulate-fit-model pattern the series has used since chapter 1, and a non-linear search recovered the cluster's\n", + " mass from galaxy shapes alone.\n", + "\n", + "- **Joint Fits:** Strong lensing pins the inner mass, weak lensing measures the outskirts; fitting both with one\n", + " model is the expected PyAutoLens workflow for cluster and group science.\n", + "\n", + "And that is the series: from a single lens galaxy, to lenses with extra galaxies, to multi-galaxy lenses, to\n", + "scaling relations that keep many-galaxy models tractable, to group-scale and cluster-scale systems, and finally\n", + "beyond the Einstein radius altogether into the weak-lensing regime. You have every conceptual tool the modern\n", + "lensing literature uses, and you have used each one on data.\n", + "\n", + "Where next? The `autolens_workspace` is the destination for real science. Its `imaging`, `point_source`, `group`,\n", + "`cluster` and `weak` packages hold the production-ready versions of everything taught here \u2014 including the\n", + "`start_here.py` scripts that are each topic's canonical, always-current reference \u2014 plus the features this series\n", + "could only gesture at: multi-wavelength fits, interferometry, automated SLaM pipelines and more. Take your own\n", + "data there, and good luck with your science!" + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/chapter_optional/tutorial_searches.ipynb b/notebooks/chapter_optional/tutorial_searches.ipynb index 42cda46..5f258a3 100644 --- a/notebooks/chapter_optional/tutorial_searches.ipynb +++ b/notebooks/chapter_optional/tutorial_searches.ipynb @@ -68,7 +68,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", @@ -228,7 +228,7 @@ " unique_tag=dataset_name,\n", " n_live=400,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "analysis = al.AnalysisImaging(dataset=dataset)\n", @@ -294,7 +294,7 @@ " unique_tag=dataset_name,\n", " n_live=75,\n", " n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2.\n", - " iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations\n", + " iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations\n", ")\n", "\n", "print(\n", @@ -357,7 +357,7 @@ "error estimates (perhaps this is our final lens model fit before we publish the results in a paper), these extra\n", "iterations are acceptable. \n", "\n", - "However, we often don't care about the errors. For example, in the previous tutorial when chaining searches, the only\n", + "However, we often don't care about the errors. For example, in chapter 2's search chaining tutorial, the only\n", "result we used from the fit performed in the first search was the maximum log likelihood model, omitting the errors\n", "entirely! Its seems wasteful to use a nested sampling algorithm like Nautilus to map out the entirity of parameter\n", "space when we don't use this information!\n", @@ -471,7 +471,7 @@ "__Wrap Up__\n", "\n", "This tutorial showed how the same lens model can be fitted using different non-linear searches (e.g. `Nautilus`,\n", - "`Emcee`, `PySwarms`). Each search explores parameter space in a different way, and the best choice depends on the\n", + "`Emcee`, `Zeus`). Each search explores parameter space in a different way, and the best choice depends on the\n", "dimensionality and complexity of the model you are fitting.\n", "\n", "For the vast majority of lens models in **PyAutoLens**, the default nested sampling search `Nautilus` is recommended,\n", diff --git a/notebooks/simulator/cluster.ipynb b/notebooks/simulator/cluster.ipynb new file mode 100644 index 0000000..64c501c --- /dev/null +++ b/notebooks/simulator/cluster.ipynb @@ -0,0 +1,795 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulator: Cluster\n", + "==================\n", + "\n", + "This script simulates a strong lens on the 'cluster' scale: 2 main lens galaxies (a brightest cluster\n", + "galaxy and a satellite), 10 lower-mass cluster member galaxies on a luminosity-mass scaling relation,\n", + "a cluster-scale dark matter halo not tied to any individual galaxy, and 2 multiply-imaged background\n", + "source galaxies at *different* redshifts (z = 1.0 and z = 2.0) \u2014 a genuine multi-plane lens.\n", + "\n", + "It is used to illustrate cluster-scale lens modeling in the HowToLens lecture series.\n", + "\n", + "Cluster modeling uses the *point source* API: rather than fitting the extended arc light of each\n", + "lensed source, only the image-plane positions of its multiple images are fitted. This script therefore\n", + "simulates point-source datasets (one per source) alongside CCD imaging, which in real datasets is used\n", + "to measure those positions and to visually confirm the lens configuration.\n", + "\n", + "__Contents__\n", + "\n", + "- **Dataset Paths:** The `dataset_path` describes where the simulated cluster dataset is output to.\n", + "- **Redshifts:** The lens redshift and the two distinct source redshifts of the multi-plane system.\n", + "- **Galaxy Centres:** The centres of the main lens galaxies, scaling members, halo and sources.\n", + "- **Grids:** The imaging grid (with adaptive over sampling) and a coarse visualization grid.\n", + "- **Main Lens Galaxies:** The 2 individually-modelled cluster members, each a `SersicSph` light and `dPIEMassSph` mass.\n", + "- **Scaling Member Galaxies:** 10 lower-mass members whose masses derive from a luminosity-mass scaling relation.\n", + "- **Host Dark Matter Halo:** A standalone `NFWMCRLudlowSph` halo with `mass_at_200 = 10^15.3`.\n", + "- **Source Galaxies:** The 2 multi-plane background sources, each a `SersicCore` light + a `Point` component.\n", + "- **Ray Tracing:** Combine all galaxies into a single multi-plane `Tracer`.\n", + "- **Point Solver:** Solve the lens equation for the image-plane multiple-image positions of each source.\n", + "- **Point Datasets:** Collect the per-source positions (with noise) into `PointDataset` objects.\n", + "- **Combined CSV:** Write all point datasets to a single hand-editable CSV.\n", + "- **Scaling Galaxies CSV:** Write the scaling-member centres and luminosities to `scaling_galaxies.csv`.\n", + "- **Model CSVs:** Write the truth model to `mass.csv` + `light.csv` + `point.csv` (the named-galaxy CSV API).\n", + "- **Tracer json:** Save the true `Tracer` in the dataset folder as a .json file.\n", + "- **Imaging:** Simulate CCD imaging of the cluster and output it to .fits files.\n", + "- **Visualize:** Output subplots of the point datasets, tracer and imaging to the dataset path.\n", + "\n", + "__Model__\n", + "\n", + "This script simulates a 'cluster-scale' strong lens where:\n", + "\n", + " - The 2 main lens galaxies have `SersicSph` light profiles and `dPIEMassSph` mass profiles.\n", + " - The 10 scaling-tier member galaxies have `dPIEMassSph` masses set by a luminosity-mass scaling relation.\n", + " - The cluster's dark matter halo is a standalone `NFWMCRLudlowSph` profile.\n", + " - The 2 source galaxies have `SersicCore` light profiles and `Point` components at z = 1.0 and z = 2.0.\n", + "\n", + "__Start Here Notebook__\n", + "\n", + "If any code in this script is unclear, refer to the `autolens_workspace/*/cluster/simulator.ipynb` notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "import importlib.util\n", + "import sys\n", + "\n", + "if importlib.util.find_spec(\"jax\") is None:\n", + " print(\n", + " \"Skipping this simulator: it requires the `jax` package, which is not \"\n", + " \"installed (install with `pip install autolens[optional]`).\"\n", + " )\n", + " sys.exit(0)\n", + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "import jax\n", + "import jax.numpy as jnp\n", + "import numpy as np\n", + "from pathlib import Path\n", + "\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Paths__\n", + "\n", + "The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a\n", + "descriptive name. The cluster dataset is output to `howtolens/dataset/cluster/simple`, which the\n", + "tutorial on cluster-scale lensing loads via the auto-simulation guard." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_type = \"cluster\"\n", + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\", dataset_type, dataset_name)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Redshifts__\n", + "\n", + "All lens galaxies and the host dark matter halo sit at the lens redshift z = 0.5. The two sources sit\n", + "at *different* redshifts (z = 1.0 and z = 2.0), making this a genuine multi-plane lens: light from the\n", + "z = 2.0 source passes through the z = 1.0 plane on its way to us, so the `Tracer` must ray-trace\n", + "through every plane in redshift order." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "redshift_lens = 0.5\n", + "source_redshifts = [1.0, 2.0]" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Galaxy Centres__\n", + "\n", + "The host halo is anchored at the cluster centre (the origin); the two main galaxies are the central\n", + "brightest cluster galaxy (BCG) and a satellite offset to the upper-right. The 10 scaling-member\n", + "centres sit at radii of 5-15\" from the centre \u2014 inside the strongly-lensed region of the host halo but\n", + "clear of the cores of the two main galaxies. The source centres are chosen so both sources land in the\n", + "strongly-lensed region and are multiply imaged." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "main_lens_centres = [\n", + " (0.0, 0.0), # BCG at cluster centre\n", + " (10.0, 8.0), # satellite member\n", + "]\n", + "\n", + "scaling_galaxies_centres = [\n", + " (5.5, -6.5),\n", + " (-7.5, 3.0),\n", + " (12.0, -5.0),\n", + " (-4.0, -9.0),\n", + " (3.0, 13.0),\n", + " (-14.0, 4.0),\n", + " (15.0, 9.0),\n", + " (-9.0, -12.0),\n", + " (8.5, 5.5),\n", + " (-6.5, 11.0),\n", + "]\n", + "\n", + "scaling_galaxies_luminosities = [\n", + " 0.40,\n", + " 0.32,\n", + " 0.25,\n", + " 0.20,\n", + " 0.16,\n", + " 0.13,\n", + " 0.10,\n", + " 0.08,\n", + " 0.06,\n", + " 0.05,\n", + "]\n", + "\n", + "host_halo_centre = (0.0, 0.0)\n", + "\n", + "source_centres = [\n", + " (0.3, 0.5),\n", + " (-0.8, 1.2),\n", + "]" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Grids__\n", + "\n", + "The typical Einstein radius of a 10^15 solar-mass halo is ~20-30\" and the member galaxies span ~30\"\n", + "across, so the field must be large (100\" x 100\") to capture the multiple images and arcs \u2014 far bigger\n", + "than the galaxy-scale fields of earlier simulators.\n", + "\n", + "Two grids are used: a high-resolution `imaging_grid` for rendering the CCD image, with adaptive over\n", + "sampling around the centre of every cluster member, and a coarse `viz_grid` used only for the\n", + "visualization plots at the end of the script (which do not need full resolution)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "imaging_grid = al.Grid2D.uniform(\n", + " shape_native=(500, 500),\n", + " pixel_scales=0.2,\n", + ")\n", + "\n", + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=imaging_grid,\n", + " sub_size_list=[32, 8, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=main_lens_centres + scaling_galaxies_centres,\n", + ")\n", + "\n", + "imaging_grid = imaging_grid.apply_over_sampling(over_sample_size=over_sample_size)\n", + "\n", + "viz_grid = al.Grid2D.uniform(shape_native=(200, 200), pixel_scales=0.5)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Main Lens Galaxies__\n", + "\n", + "The 2 individually-modelled cluster members. Each has a `SersicSph` light profile (used for the CCD\n", + "imaging \u2014 the light plays no role in point-source modeling) and a `dPIEMassSph` mass profile, the\n", + "standard truncated mass profile for cluster members. Its parameters are Lenstool's native ones:\n", + "\n", + " - `sigma` (km/s): the fiducial velocity dispersion, setting the lens strength.\n", + " - `r_core` (arcsec): the core radius, fixed to 0 here (the standard vanishing-core convention).\n", + " - `r_cut` (arcsec): the truncation radius, beyond which the density falls off rapidly \u2014 the tidal\n", + " stripping physics of galaxies orbiting inside a cluster's shared potential." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "main_lens_dpie_params = [\n", + " # (r_core, r_cut, sigma) per galaxy \u2014 arcsec, arcsec, km/s\n", + " (0.0, 20.0, 330.0), # BCG \u2014 strongest\n", + " (0.0, 12.0, 210.0), # satellite\n", + "]\n", + "\n", + "main_lens_sersic_params = [\n", + " # (intensity, effective_radius, sersic_index)\n", + " (1.5, 3.0, 4.0), # BCG \u2014 bright and extended\n", + " (0.8, 1.5, 3.5), # satellite\n", + "]\n", + "\n", + "main_lens_galaxies = []\n", + "for centre, (r_core, r_cut, sigma), (intensity, effective_radius, sersic_index) in zip(\n", + " main_lens_centres, main_lens_dpie_params, main_lens_sersic_params\n", + "):\n", + " bulge = al.lp.SersicSph(\n", + " centre=centre,\n", + " intensity=intensity,\n", + " effective_radius=effective_radius,\n", + " sersic_index=sersic_index,\n", + " )\n", + " mass = al.mp.dPIEMassSph(\n", + " centre=centre,\n", + " sigma=sigma,\n", + " r_core=r_core,\n", + " r_cut=r_cut,\n", + " redshift_object=redshift_lens,\n", + " redshift_source=max(source_redshifts),\n", + " )\n", + " main_lens_galaxies.append(al.Galaxy(redshift=redshift_lens, bulge=bulge, mass=mass))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Scaling Member Galaxies__\n", + "\n", + "The 10 lower-mass cluster members are modelled collectively via a luminosity-mass scaling relation \u2014\n", + "the convention used by essentially every published cluster strong-lensing analysis:\n", + "\n", + " sigma_i = sigma_ref * (L_i / L_ref) ** 0.25\n", + " r_cut_i = r_cut_ref * (L_i / L_ref) ** 0.7\n", + " r_core_i = 0\n", + "\n", + "where `L_ref` is a fixed reference luminosity and `sigma_ref` is the fiducial velocity dispersion of a\n", + "galaxy at that reference luminosity. The truth value simulated here is `sigma_ref = 85.0` km/s \u2014 the\n", + "value the tutorial's model-fit seeks to recover with a single free parameter for the whole tier." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "scaling_sigma_ref_truth = 85.0\n", + "scaling_sigma_exponent = 0.25 # alpha (Faber-Jackson)\n", + "scaling_gamma = 0.2 # mass-to-light tilt, fixed\n", + "scaling_rcut_exponent = 1.0 + scaling_gamma - 2.0 * scaling_sigma_exponent # 0.7\n", + "reference_luminosity = 1.0\n", + "scaling_r_core = 0.0 # vanishing core \u2014 fixed, never scaled\n", + "scaling_r_cut_ref = 5.0\n", + "\n", + "scaling_galaxies = []\n", + "for centre, luminosity in zip(scaling_galaxies_centres, scaling_galaxies_luminosities):\n", + " bulge = al.lp.SersicSph(\n", + " centre=centre,\n", + " intensity=luminosity,\n", + " effective_radius=0.8,\n", + " sersic_index=3.0,\n", + " )\n", + " luminosity_ratio = luminosity / reference_luminosity\n", + " mass = al.mp.dPIEMassSph(\n", + " centre=centre,\n", + " sigma=scaling_sigma_ref_truth * luminosity_ratio**scaling_sigma_exponent,\n", + " r_core=scaling_r_core,\n", + " r_cut=scaling_r_cut_ref * luminosity_ratio**scaling_rcut_exponent,\n", + " redshift_object=redshift_lens,\n", + " redshift_source=max(source_redshifts),\n", + " )\n", + " scaling_galaxies.append(al.Galaxy(redshift=redshift_lens, bulge=bulge, mass=mass))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Host Dark Matter Halo__\n", + "\n", + "A standalone galaxy holding the cluster's dark matter halo. It has no light profile \u2014 it sits in the\n", + "tracer solely to contribute mass. `NFWMCRLudlowSph` is parameterised by the physical halo mass within\n", + "r_200 (`mass_at_200 = 10^15.3`, ~2 x 10^15 solar masses) and the redshifts; its concentration follows\n", + "from a concentration-mass relation. The `redshift_source` is anchored to the *furthest* source\n", + "(z = 2.0), the multi-plane convention used throughout (deflections are normalized to the final plane)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "host_halo = al.mp.NFWMCRLudlowSph(\n", + " centre=host_halo_centre,\n", + " mass_at_200=10**15.3,\n", + " redshift_object=redshift_lens,\n", + " redshift_source=max(source_redshifts),\n", + ")\n", + "\n", + "host_halo_galaxy = al.Galaxy(redshift=redshift_lens, dark=host_halo)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Source Galaxies__\n", + "\n", + "The 2 background sources at *different* redshifts. Each carries a `SersicCore` light profile (so the\n", + "lensed arcs appear in the CCD imaging) and a `Point` component whose multiple-image positions we solve\n", + "for below \u2014 those positions are the data the tutorial's model-fit uses.\n", + "\n", + "The attribute name of each `Point` (`point_0`, `point_1`) is important: it is the name that pairs each\n", + "model component to its `PointDataset` during modeling." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "source_galaxies = []\n", + "for i, (centre, src_z) in enumerate(zip(source_centres, source_redshifts)):\n", + " bulge = al.lp.SersicCore(\n", + " centre=centre,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0 + 30.0 * i),\n", + " intensity=2.0,\n", + " effective_radius=0.3,\n", + " sersic_index=1.0,\n", + " )\n", + " point = al.ps.Point(centre=centre)\n", + " source_galaxies.append(\n", + " al.Galaxy(redshift=src_z, bulge=bulge, **{f\"point_{i}\": point})\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "Combine the main lens galaxies, scaling members, host halo and sources into a single tracer. With\n", + "sources at distinct redshifts, the tracer automatically performs multi-plane ray tracing." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer = al.Tracer(\n", + " galaxies=main_lens_galaxies\n", + " + scaling_galaxies\n", + " + [host_halo_galaxy]\n", + " + source_galaxies\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Point Solver__\n", + "\n", + "The `PointSolver` solves the lens equation for the image-plane positions of each point source, by\n", + "ray-tracing triangles from the image plane to the source plane and iteratively refining those that\n", + "contain the source centre. At cluster scale (13 lens galaxies, multi-plane ray tracing) this solve is\n", + "expensive, so we accelerate it with JAX: `use_jax=True` plus a `jax.jit` wrapper around the solve call\n", + "compiles the triangle-refinement kernel once and reuses it for both sources.\n", + "\n", + "The solver's grid must span the full cluster field, because the multiple images sit at radii of\n", + "~20-30\" from the cluster centre \u2014 far outside the arc-second scale grids of galaxy-scale lensing.\n", + "\n", + "`plane_redshift` is per-source and must be passed: it tells the solver which source plane to solve\n", + "for. Without it, the solver defaults to the tracer's final plane, which would silently place the\n", + "z = 1.0 source's images as if it sat at z = 2.0." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "from autolens.jax import register_tracer_classes\n", + "\n", + "register_tracer_classes(tracer)\n", + "\n", + "solver = al.PointSolver.for_grid(\n", + " grid=al.Grid2D.uniform(shape_native=(400, 400), pixel_scales=0.25),\n", + " pixel_scale_precision=0.001,\n", + " magnification_threshold=0.1,\n", + " use_jax=True,\n", + ")\n", + "\n", + "\n", + "def jitted_solve_for(plane_redshift):\n", + " @jax.jit\n", + " def jitted_solve(tracer, source_plane_coordinate):\n", + " return solver.solve(\n", + " tracer=tracer,\n", + " source_plane_coordinate=source_plane_coordinate,\n", + " plane_redshift=plane_redshift,\n", + " ).array\n", + "\n", + " return jitted_solve\n", + "\n", + "\n", + "positions_list = []\n", + "for i, (src_centre, src_z) in enumerate(zip(source_centres, source_redshifts)):\n", + " coord = jnp.asarray(src_centre)\n", + " jitted_solve = jitted_solve_for(float(src_z))\n", + " raw = np.asarray(jitted_solve(tracer, coord))\n", + " finite = ~(np.isinf(raw).any(axis=1) | np.isnan(raw).any(axis=1))\n", + " positions_list.append(al.Grid2DIrregular(raw[finite]))\n", + "\n", + " print(f\"point_{i} (z={src_z}): {int(finite.sum())} multiple images solved\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Point Datasets__\n", + "\n", + "One `PointDataset` per source. The `name` (`point_0`, `point_1`) pairs each dataset with the matching\n", + "`Point` component in the lens model during modeling, and the `redshift` records each source's\n", + "redshift so the multi-plane geometry can be rebuilt when the dataset is loaded.\n", + "\n", + "The position uncertainty is 0.005\" (5 mas), the centroid precision achievable by PSF fitting on\n", + "space-telescope imaging \u2014 not the imaging pixel scale." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "position_noise = 0.005\n", + "\n", + "dataset_list = []\n", + "for i, positions in enumerate(positions_list):\n", + " dataset = al.PointDataset(\n", + " name=f\"point_{i}\",\n", + " positions=positions,\n", + " positions_noise_map=position_noise,\n", + " redshift=source_redshifts[i],\n", + " )\n", + " dataset_list.append(dataset)\n", + "\n", + "for i, dataset in enumerate(dataset_list):\n", + " al.output_to_json(\n", + " obj=dataset,\n", + " file_path=dataset_path / f\"point_dataset_{i}.json\",\n", + " )" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Combined CSV__\n", + "\n", + "For cluster workflows with many sources, a single CSV with one row per observed multiple image \u2014\n", + "grouped by source `name`, with `y`, `x`, `positions_noise` and `redshift` columns \u2014 is far easier to\n", + "edit in a spreadsheet than many per-source files. `al.output_to_csv` writes every dataset into one\n", + "file, which the tutorial loads back with `al.list_from_csv`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_csv(\n", + " datasets=dataset_list,\n", + " file_path=dataset_path / \"point_datasets.csv\",\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Scaling Galaxies CSV__\n", + "\n", + "The scaling-tier members are written to `scaling_galaxies.csv` \u2014 one row per member carrying its\n", + "centre and luminosity (`y, x, luminosity`). This is the catalogue the tutorial's scaling relation\n", + "consumes: scaling a real cluster up to hundreds of members is a CSV-level edit, with the model's\n", + "free-parameter count unchanged." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.galaxy_table_to_csv(\n", + " centres=scaling_galaxies_centres,\n", + " luminosities=scaling_galaxies_luminosities,\n", + " file_path=dataset_path / \"scaling_galaxies.csv\",\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Model CSVs__\n", + "\n", + "Write the truth model out as three family-level CSVs \u2014 `mass.csv`, `light.csv`, `point.csv` \u2014 keyed\n", + "by galaxy name. The tutorial loads these with `al.galaxy_models_from_csv` and composes them into\n", + "`af.Model` galaxies ready for the non-linear search. See `autolens_workspace/*/cluster/csv_api.py`\n", + "for the full schema walkthrough." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "mass_profiles = {\n", + " **{f\"lens_{i}\": {\"mass\": g.mass} for i, g in enumerate(main_lens_galaxies)},\n", + " \"host_halo\": {\"dark\": host_halo_galaxy.dark},\n", + "}\n", + "\n", + "light_profiles = {\n", + " **{f\"lens_{i}\": {\"bulge\": g.bulge} for i, g in enumerate(main_lens_galaxies)},\n", + " **{f\"source_{i}\": {\"bulge\": g.bulge} for i, g in enumerate(source_galaxies)},\n", + "}\n", + "\n", + "point_profiles = {\n", + " f\"source_{i}\": {f\"point_{i}\": getattr(g, f\"point_{i}\")}\n", + " for i, g in enumerate(source_galaxies)\n", + "}\n", + "\n", + "redshifts_by_galaxy = {\n", + " **{f\"lens_{i}\": redshift_lens for i in range(len(main_lens_galaxies))},\n", + " \"host_halo\": redshift_lens,\n", + " **{f\"source_{i}\": z for i, z in enumerate(source_redshifts)},\n", + "}\n", + "\n", + "al.galaxy_models_to_csv(\n", + " profiles_by_galaxy=mass_profiles,\n", + " file_path=dataset_path / \"mass.csv\",\n", + " family=\"mass\",\n", + " redshifts=redshifts_by_galaxy,\n", + ")\n", + "\n", + "al.galaxy_models_to_csv(\n", + " profiles_by_galaxy=light_profiles,\n", + " file_path=dataset_path / \"light.csv\",\n", + " family=\"light\",\n", + " redshifts=redshifts_by_galaxy,\n", + ")\n", + "\n", + "al.galaxy_models_to_csv(\n", + " profiles_by_galaxy=point_profiles,\n", + " file_path=dataset_path / \"point.csv\",\n", + " family=\"point\",\n", + " redshifts=redshifts_by_galaxy,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Tracer json__\n", + "\n", + "Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass\n", + "profiles and galaxies are safely stored and available to check how the dataset was simulated in the\n", + "future.\n", + "\n", + "This can be loaded via the method `tracer = al.from_json()`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=tracer,\n", + " file_path=Path(dataset_path, \"tracer.json\"),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Imaging__\n", + "\n", + "Strong lens clusters come with imaging data, used to *measure* the point positions and to visually\n", + "confirm the lens configuration. Although modeling is point-source only, we simulate CCD imaging so\n", + "the dataset looks like a realistic cluster observation." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "psf = al.Convolver.from_gaussian(\n", + " shape_native=(11, 11), sigma=0.2, pixel_scales=imaging_grid.pixel_scales\n", + ")\n", + "\n", + "simulator = al.SimulatorImaging(\n", + " exposure_time=300.0,\n", + " psf=psf,\n", + " background_sky_level=0.1,\n", + " add_poisson_noise_to_data=True,\n", + ")\n", + "\n", + "dataset = simulator.via_tracer_from(tracer=tracer, grid=imaging_grid)\n", + "\n", + "aplt.fits_imaging(\n", + " dataset=dataset,\n", + " data_path=dataset_path / \"data.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " overwrite=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "Output .png plots of the per-source point datasets, the tracer, and the simulated imaging to the\n", + "dataset path." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "for pd in dataset_list:\n", + " aplt.subplot_point_dataset(\n", + " dataset=pd, output_path=dataset_path, output_format=\"png\"\n", + " )\n", + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)\n", + "aplt.plot_array(array=dataset.data, title=\"Data\")\n", + "\n", + "aplt.subplot_tracer(\n", + " tracer=tracer, grid=viz_grid, output_path=dataset_path, output_format=\"png\"\n", + ")\n", + "aplt.subplot_galaxies_images(\n", + " tracer=tracer, grid=viz_grid, output_path=dataset_path, output_format=\"png\"\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The dataset can be viewed in the folder `howtolens/dataset/cluster/simple`." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/simulator/group.ipynb b/notebooks/simulator/group.ipynb new file mode 100644 index 0000000..0197a34 --- /dev/null +++ b/notebooks/simulator/group.ipynb @@ -0,0 +1,576 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulator: Group Scale Lens\n", + "===========================\n", + "\n", + "This script simulates `Imaging` of a 'group-scale' strong lens, which is used in chapter 4 of the **HowToLens**\n", + "lectures to illustrate lens modeling at the group scale.\n", + "\n", + "The group consists of a single main lens galaxy -- the brightest group galaxy (BGG) -- and two smaller member\n", + "galaxies nearby, whose mass contributes significantly to the ray-tracing and is therefore included in the strong\n", + "lens model. A single source galaxy is lensed by the combined mass of all three galaxies.\n", + "\n", + "__Model__\n", + "\n", + "This script simulates `Imaging` of a 'group-scale' strong lens where:\n", + "\n", + " - The main lens galaxy's light is a `SersicSph` and its total mass distribution is an `IsothermalSph`.\n", + " - The two member galaxies have `SersicSph` light profiles and tidally truncated `dPIEMassSph` mass profiles.\n", + " - The source galaxy's light is a `SersicCore`.\n", + "\n", + "The member galaxies are simulated to follow the light-mass scaling relation used in the group-scale modeling\n", + "tutorial: `sigma = sigma_ref * (L / L_ref) ** 0.25` and `r_cut = r_cut_ref * (L / L_ref) ** 0.7`, where `L` is\n", + "each member's luminosity. The second member is 2.07 times more luminous than the first, so its velocity\n", + "dispersion is 1.2 times higher (240 km/s vs 200 km/s) and its truncation radius 1.67 times larger.\n", + "\n", + "__Contents__\n", + "\n", + "- **Dataset Paths:** The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a.\n", + "- **Grid:** Define the 2D grid of (y,x) coordinates the simulated image is evaluated on.\n", + "- **Galaxy Centres:** Define the centres of the main lens galaxy and member galaxies.\n", + "- **Over Sampling:** Set up the adaptive over-sampling scheme, centred on every galaxy in the group.\n", + "- **PSF / Simulator:** The Point Spread Function and simulator defining the observational properties of the data.\n", + "- **Main Lens Galaxy:** The brightest group galaxy (BGG) at the origin (0.0\", 0.0\").\n", + "- **Member Galaxies:** The two smaller group members, with tidally truncated `dPIEMassSph` mass profiles.\n", + "- **Source Galaxy:** The source galaxy whose lensed image we simulate.\n", + "- **Ray Tracing:** Use all galaxies to set up a tracer, which generates the image that is simulated.\n", + "- **Output:** Output the simulated dataset to the dataset path as .fits files.\n", + "- **Visualize:** Output a subplot of the simulated dataset and the tracer's quantities to the dataset path.\n", + "- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file.\n", + "- **Centre JSON Files:** Save the centres of the main lens galaxy and member galaxies as .json files.\n", + "- **Positions:** Solve for the multiple-image positions of the lensed source and save them as a .json file.\n", + "\n", + "__Start Here Notebook__\n", + "\n", + "If any code in this script is unclear, refer to the `autolens_workspace/*/group/simulator.ipynb` notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Paths__\n", + "\n", + "The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name.\n", + "\n", + "The dataset is output to `dataset/group/simple`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_type = \"group\"\n", + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\", dataset_type, dataset_name)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Grid__\n", + "\n", + "Define the 2D grid of (y,x) coordinates the simulated image is evaluated on.\n", + "\n", + "A group-scale lens spans a wider area of sky than a galaxy-scale lens, because it contains multiple lens\n", + "galaxies which are separated by many arc-seconds. The grid is therefore larger (250 x 250 pixels, or 25.0\")\n", + "than those used in previous chapters." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = al.Grid2D.uniform(\n", + " shape_native=(250, 250),\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Galaxy Centres__\n", + "\n", + "Define the centres of the main lens galaxy and member galaxies. These are used for over-sampling and are also\n", + "output to .json files so that the modeling tutorial can load them." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "main_lens_centres = [(0.0, 0.0)]\n", + "member_centres = [(3.5, 2.5), (-4.4, -5.0)]" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Over Sampling__\n", + "\n", + "Over sampling evaluates the light profiles on a higher resolution grid in their bright central regions, to\n", + "ensure the calculation is accurate.\n", + "\n", + "The adaptive over-sampling scheme below is applied at the centre of every galaxy in the group, not just the\n", + "main lens galaxy." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=grid,\n", + " sub_size_list=[32, 8, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=main_lens_centres + member_centres,\n", + ")\n", + "\n", + "grid = grid.apply_over_sampling(over_sample_size=over_sample_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__PSF / Simulator__\n", + "\n", + "Simulate a simple Gaussian PSF for the image, and create the simulator for the imaging data, which defines the\n", + "exposure time, background sky, noise levels and psf." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "psf = al.Convolver.from_gaussian(\n", + " shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales\n", + ")\n", + "\n", + "simulator = al.SimulatorImaging(\n", + " exposure_time=300.0,\n", + " psf=psf,\n", + " background_sky_level=0.1,\n", + " add_poisson_noise_to_data=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Main Lens Galaxy__\n", + "\n", + "The main lens galaxy is the brightest group galaxy (BGG), located at the origin (0.0\", 0.0\"). It has a spherical\n", + "Sersic light profile and an isothermal mass profile with a large Einstein radius of 4.0\", reflecting that it\n", + "dominates the mass of the group." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.SersicSph(\n", + " centre=(0.0, 0.0), intensity=0.7, effective_radius=2.0, sersic_index=4.0\n", + " ),\n", + " mass=al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=4.0),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Member Galaxies__\n", + "\n", + "The two member galaxies are smaller galaxies orbiting within the group. They have spherical Sersic light\n", + "profiles and tidally truncated `dPIEMassSph` mass profiles (vanishing core `r_core = 0.0` and a finite\n", + "truncation radius `r_cut`), reflecting that their outer dark matter has been stripped by the tides of the\n", + "group's potential.\n", + "\n", + "The members follow the light-mass scaling relation used in the modeling tutorial: the second member is 2.07\n", + "times more luminous than the first (`intensity` of 1.866 vs 0.9, with identical `effective_radius` and\n", + "`sersic_index`), so its `sigma` is a factor 2.07 ** 0.25 = 1.2 higher (240 km/s vs 200 km/s) and its `r_cut`\n", + "a factor 2.07 ** 0.7 = 1.67 larger (13.3\" vs 8.0\").\n", + "\n", + "The `dPIEMassSph` profile is parameterized in Lenstool's native convention -- `sigma` (fiducial velocity\n", + "dispersion, km/s), `r_core` and `r_cut` (arcsec) -- and converts these to a lensing strength internally using\n", + "the object/source redshifts and cosmology (`H0` / `Om0`, which default to Planck-like values)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "member_galaxy_0 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.SersicSph(\n", + " centre=(3.5, 2.5), intensity=0.9, effective_radius=0.8, sersic_index=3.0\n", + " ),\n", + " mass=al.mp.dPIEMassSph(\n", + " centre=(3.5, 2.5),\n", + " sigma=200.0,\n", + " r_core=0.0,\n", + " r_cut=8.0,\n", + " redshift_object=0.5,\n", + " redshift_source=1.0,\n", + " ),\n", + ")\n", + "\n", + "member_galaxy_1 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.SersicSph(\n", + " centre=(-4.4, -5.0), intensity=1.866, effective_radius=0.8, sersic_index=3.0\n", + " ),\n", + " mass=al.mp.dPIEMassSph(\n", + " centre=(-4.4, -5.0),\n", + " sigma=240.0,\n", + " r_core=0.0,\n", + " r_cut=13.3,\n", + " redshift_object=0.5,\n", + " redshift_source=1.0,\n", + " ),\n", + ")\n", + "\n", + "member_galaxies = [member_galaxy_0, member_galaxy_1]" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Source Galaxy__\n", + "\n", + "The source galaxy whose lensed image we simulate. It uses a cored Sersic profile, whose light changes gradually\n", + "in its central regions and can therefore be evaluated accurately without adaptive over-sampling." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "source_galaxy = al.Galaxy(\n", + " redshift=1.0,\n", + " bulge=al.lp.SersicCore(\n", + " centre=(0.0, 0.1),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0),\n", + " intensity=3.0,\n", + " effective_radius=0.4,\n", + " sersic_index=1.0,\n", + " ),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "Use all galaxies to set up a tracer, which will generate the image for the simulated `Imaging` dataset.\n", + "\n", + "The tracer combines the main lens galaxy, the member galaxies and the source galaxy." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer = al.Tracer(galaxies=[lens_galaxy] + member_galaxies + [source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lets look at the tracer`s image, this is the image we'll be simulating." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=tracer.image_2d_from(grid=grid), title=\"Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pass the simulator a tracer, which creates the image which is simulated as an imaging dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = simulator.via_tracer_from(tracer=tracer, grid=grid)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plot the simulated `Imaging` dataset before outputting it to fits." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output__\n", + "\n", + "Output the simulated dataset to the dataset path as .fits files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.fits_imaging(\n", + " dataset=dataset,\n", + " data_path=dataset_path / \"data.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " overwrite=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset path as .png files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)\n", + "aplt.plot_array(array=dataset.data, title=\"Data\")\n", + "\n", + "aplt.subplot_tracer(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")\n", + "aplt.subplot_galaxies_images(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Tracer json__\n", + "\n", + "Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and\n", + "galaxies are safely stored and available to check how the dataset was simulated in the future.\n", + "\n", + "This can be loaded via the method `tracer = al.from_json()`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=tracer,\n", + " file_path=Path(dataset_path, \"tracer.json\"),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Centre JSON Files__\n", + "\n", + "Save the centres of the main lens galaxy and member galaxies as .json files. These are loaded by the group-scale\n", + "modeling tutorial to set up the lens model (e.g. fixing the centres of the member galaxies)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=al.Grid2DIrregular(main_lens_centres),\n", + " file_path=Path(dataset_path, \"main_lens_centres.json\"),\n", + ")\n", + "\n", + "al.output_to_json(\n", + " obj=al.Grid2DIrregular(member_centres),\n", + " file_path=Path(dataset_path, \"extra_galaxies_centres.json\"),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Positions__\n", + "\n", + "Solve for the multiple-image positions of the lensed source galaxy, which can be used as input to group-scale\n", + "modeling to help the non-linear search converge (and are the primary observable of cluster-scale modeling,\n", + "introduced in the next chapter 4 tutorial)." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "solver = al.PointSolver.for_grid(\n", + " grid=al.Grid2D.uniform(shape_native=(500, 500), pixel_scales=0.1),\n", + " pixel_scale_precision=0.001,\n", + " magnification_threshold=0.01,\n", + ")\n", + "\n", + "positions = solver.solve(\n", + " tracer=tracer, source_plane_coordinate=source_galaxy.bulge.centre\n", + ")\n", + "\n", + "al.output_to_json(\n", + " obj=positions,\n", + " file_path=dataset_path / \"positions.json\",\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The dataset can be viewed in the folder `dataset/group/simple`." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/simulator/interferometer.ipynb b/notebooks/simulator/interferometer.ipynb new file mode 100644 index 0000000..b5513b0 --- /dev/null +++ b/notebooks/simulator/interferometer.ipynb @@ -0,0 +1,404 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulator: Interferometer\n", + "=========================\n", + "\n", + "This script simulates `Interferometer` data of a 'galaxy-scale' strong lens, as would be observed by a radio or\n", + "sub-mm interferometer like ALMA or the JVLA.\n", + "\n", + "Unlike CCD imaging, an interferometer does not observe an image of the lens. It measures \"visibilities\", which are\n", + "the Fourier transform of the sky brightness sampled at a set of points in the \"uv-plane\", where each point\n", + "corresponds to the separation of a pair of antennas in the array.\n", + "\n", + "It is used to illustrate interferometer data in the HowToLens lecture series. HowToLens does not cover\n", + "interferometry beyond this glimpse; the `autolens_workspace/scripts/interferometer` package is the dedicated\n", + "resource for uv-plane lens modeling.\n", + "\n", + "__Contents__\n", + "\n", + "- **Dataset Paths:** The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a.\n", + "- **Simulate:** Simulate the image using a (y,x) real-space grid and a synthetic set of uv-plane baselines.\n", + "- **Ray Tracing:** Setup the lens galaxy's mass and source galaxy light for this simulated lens.\n", + "- **Output:** Output the simulated dataset to the dataset path as .fits files.\n", + "- **Visualize:** Output a subplot of the simulated dataset's dirty images and the tracer's quantities.\n", + "- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass.\n", + "\n", + "__Model__\n", + "\n", + "This script simulates `Interferometer` data of a 'galaxy-scale' strong lens where:\n", + "\n", + " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", + " - The source galaxy's light is a `SersicCore`.\n", + "\n", + "__Start Here Notebook__\n", + "\n", + "If any code in this script is unclear, refer to the `autolens_workspace/*/interferometer/simulator.ipynb` notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "import numpy as np\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Paths__\n", + "\n", + "The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_type = \"interferometer\"\n", + "dataset_name = \"simple\"\n", + "dataset_path = Path(\"dataset\", dataset_type, dataset_name)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Simulate__\n", + "\n", + "For interferometer data, the strong lens's image is evaluated in real space on a (y,x) grid and then Fourier\n", + "transformed to the uv-plane, where it is compared to the observed visibilities.\n", + "\n", + "Interferometers do not observe galaxies in a way where over sampling is necessary, therefore the grid uses no\n", + "over sampling." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = al.Grid2D.uniform(\n", + " shape_native=(100, 100),\n", + " pixel_scales=0.1,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To perform the Fourier transform we need the wavelengths of the baselines, which describe where in the uv-plane\n", + "each visibility samples the sky's Fourier transform.\n", + "\n", + "For real data these are determined by the layout of the interferometer's antennas. The `autolens_workspace`\n", + "bundles baselines of real instruments (e.g. the Square Mile Array (SMA) and ALMA). For this tutorial dataset we\n", + "instead draw a small synthetic set of baselines from a Gaussian distribution in the uv-plane, with a scale\n", + "comparable to the SMA's (a few hundred visibilities), keeping the simulation fast and self-contained." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "rng = np.random.default_rng(1)\n", + "\n", + "total_visibilities = 200\n", + "\n", + "uv_wavelengths = rng.normal(loc=0.0, scale=1.0e5, size=(total_visibilities, 2))" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To simulate the interferometer dataset we first create a simulator, which defines the exposure time, noise levels\n", + "and Fourier transform method used in the simulation.\n", + "\n", + "We use the `TransformerDFT`, an exact Discrete Fourier Transform which is fast for datasets with a low number of\n", + "visibilities like this one." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "simulator = al.SimulatorInterferometer(\n", + " uv_wavelengths=uv_wavelengths,\n", + " exposure_time=300.0,\n", + " noise_sigma=1000.0,\n", + " transformer_class=al.TransformerDFT,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "Setup the lens galaxy's mass and source galaxy light for this simulated lens.\n", + "\n", + "The lens galaxy has no light profile, because at the radio and sub-mm wavelengths an interferometer observes the\n", + "foreground lens galaxy typically emits negligibly." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=1.6,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", + " ),\n", + " shear=al.mp.ExternalShear(gamma_1=0.05, gamma_2=0.05),\n", + ")\n", + "\n", + "source_galaxy = al.Galaxy(\n", + " redshift=1.0,\n", + " bulge=al.lp.SersicCore(\n", + " centre=(0.0, 0.0),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0),\n", + " intensity=10.0,\n", + " effective_radius=1.0,\n", + " sersic_index=2.5,\n", + " ),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Use these galaxies to setup a tracer, which will generate the image for the simulated interferometer dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lets look at the tracer`s image, this is the image we'll be simulating." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=tracer.image_2d_from(grid=grid), title=\"Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pass the simulator a tracer, which creates the ray-traced image plotted above and simulates it as an\n", + "interferometer dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = simulator.via_tracer_from(tracer=tracer, grid=grid)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plot the simulated interferometer dataset's dirty images before outputting it to fits." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_interferometer_dirty_images(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output__\n", + "\n", + "Output the simulated dataset to the dataset path as .fits files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.fits_interferometer(\n", + " dataset=dataset,\n", + " data_path=dataset_path / \"data.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " uv_wavelengths_path=dataset_path / \"uv_wavelengths.fits\",\n", + " overwrite=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "Output a subplot of the simulated dataset's dirty images and the tracer's quantities to the dataset path as\n", + ".png files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_interferometer_dirty_images(\n", + " dataset=dataset, output_path=dataset_path, output_format=\"png\"\n", + ")\n", + "\n", + "aplt.subplot_tracer(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")\n", + "aplt.subplot_galaxies_images(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Tracer json__\n", + "\n", + "Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and galaxies\n", + "are safely stored and available to check how the dataset was simulated in the future.\n", + "\n", + "This can be loaded via the method `tracer = al.from_json()`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=tracer,\n", + " file_path=Path(dataset_path, \"tracer.json\"),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The dataset can be viewed in the folder `dataset/interferometer/simple`." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/simulator/lens_extra_galaxy.ipynb b/notebooks/simulator/lens_extra_galaxy.ipynb new file mode 100644 index 0000000..f03457b --- /dev/null +++ b/notebooks/simulator/lens_extra_galaxy.ipynb @@ -0,0 +1,514 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulator: Lens With Extra Galaxy\n", + "=================================\n", + "\n", + "This script simulates `Imaging` of a 'galaxy-scale' strong lens which is identical to the `lens_sersic` dataset\n", + "simulated for chapter 2 (lens light + mass + lensed source), but with one extra galaxy located a few arc-seconds\n", + "from the lens galaxy.\n", + "\n", + "The extra galaxy has its own light profile, which blends into the outskirts of the image, and its own mass profile,\n", + "which contributes to the lensing of the source. Chapter 4's first tutorial uses this dataset to illustrate the two\n", + "approaches to dealing with such interloper galaxies: removing their light via noise scaling, or including them in\n", + "the lens model explicitly.\n", + "\n", + "It is used to illustrate scaling up lens modeling in the HowToLens lecture series.\n", + "\n", + "__Contents__\n", + "\n", + "- **Model:** Compose the lens model fitted to the data.\n", + "- **Dataset Paths:** The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a.\n", + "- **Simulate:** Simulate the image using a (y,x) grid with the adaptive over sampling scheme.\n", + "- **Ray Tracing:** Setup the lens galaxy's light, mass and source galaxy light for this simulated lens.\n", + "- **Extra Galaxy:** Include one extra galaxy, whose light and mass must be masked or modeled in the tutorial.\n", + "- **Output:** Output the simulated dataset to the dataset path as .fits files.\n", + "- **Mask Extra Galaxy:** Build and save `mask_extra_galaxies.fits` so the tutorial can load it directly.\n", + "- **Visualize:** Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset.\n", + "- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass.\n", + "- **Extra Galaxy Centre:** Output the centre of the extra galaxy to a .json file for use in the lens model.\n", + "\n", + "__Model__\n", + "\n", + "This script simulates `Imaging` of a 'galaxy-scale' strong lens where:\n", + "\n", + " - The lens galaxy's light profile is an `Sersic`.\n", + " - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`.\n", + " - The source galaxy's light is an `Sersic`.\n", + " - There is one extra galaxy whose light is near the lens and whose mass perturbs the lensed source's emission.\n", + "\n", + "__Start Here Notebook__\n", + "\n", + "If any code in this script is unclear, refer to the `autolens_workspace/*/imaging/simulator.ipynb` notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Paths__\n", + "\n", + "The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_type = \"imaging\"\n", + "dataset_name = \"lens_extra_galaxy\"\n", + "dataset_path = Path(\"dataset\", dataset_type, dataset_name)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Simulate__\n", + "\n", + "Simulate the image using a (y,x) grid with the adaptive over sampling scheme.\n", + "\n", + "This simulated lens has an extra galaxy offset from the main lens galaxy centre of (0.0\", 0.0\"). The adaptive over\n", + "sampling grid has both centres input to account for this." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "extra_galaxy_centre = (1.5, 2.5)\n", + "\n", + "grid = al.Grid2D.uniform(\n", + " shape_native=(100, 100),\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=grid,\n", + " sub_size_list=[32, 8, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=[(0.0, 0.0), extra_galaxy_centre],\n", + ")\n", + "\n", + "grid = grid.apply_over_sampling(over_sample_size=over_sample_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulate a simple Gaussian PSF for the image." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "psf = al.Convolver.from_gaussian(\n", + " shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create the simulator for the imaging data, which defines the exposure time, background sky, noise levels and psf." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "simulator = al.SimulatorImaging(\n", + " exposure_time=300.0,\n", + " psf=psf,\n", + " background_sky_level=0.1,\n", + " add_poisson_noise_to_data=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "Setup the lens galaxy's light, mass and source galaxy light for this simulated lens.\n", + "\n", + "These are identical to the `lens_sersic` dataset simulated for chapter 2, so that the only difference between the\n", + "two datasets is the extra galaxy included below." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.Sersic(\n", + " centre=(0.0, 0.0),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", + " intensity=1.0,\n", + " effective_radius=0.8,\n", + " sersic_index=4.0,\n", + " ),\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=1.6,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0),\n", + " ),\n", + " shear=al.mp.ExternalShear(gamma_1=0.05, gamma_2=0.05),\n", + ")\n", + "\n", + "source_galaxy = al.Galaxy(\n", + " redshift=1.0,\n", + " bulge=al.lp.SersicCore(\n", + " centre=(0.0, 0.0),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0),\n", + " intensity=4.0,\n", + " effective_radius=0.1,\n", + " sersic_index=1.0,\n", + " ),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Extra Galaxy__\n", + "\n", + "Include one extra galaxy, which is offset a few arc-seconds from the lens galaxy.\n", + "\n", + "Its light is an `ExponentialSph` profile, which blends into the image near the lensed source's emission, and its\n", + "mass is an `IsothermalSph` profile, which perturbs the ray-tracing of the source's light.\n", + "\n", + "Note that its redshift is the same as the main lens galaxy, which is not necessarily the case in real observations.\n", + "If it were at a different redshift, the tools for masking or modeling the extra galaxy are equipped to handle this,\n", + "with mass modeling leading to multi-plane ray-tracing being performed." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "extra_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " light=al.lp.ExponentialSph(\n", + " centre=extra_galaxy_centre, intensity=2.0, effective_radius=0.5\n", + " ),\n", + " mass=al.mp.IsothermalSph(centre=extra_galaxy_centre, einstein_radius=0.15),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Use these galaxies to setup a tracer, which will generate the image for the simulated `Imaging` dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer = al.Tracer(galaxies=[lens_galaxy, extra_galaxy, source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lets look at the tracer`s image, this is the image we'll be simulating." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.plot_array(array=tracer.image_2d_from(grid=grid), title=\"Image\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pass the simulator a tracer, which creates the image which is simulated as an imaging dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = simulator.via_tracer_from(tracer=tracer, grid=grid)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plot the simulated `Imaging` dataset before outputting it to fits." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_imaging_dataset(dataset=dataset)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output__\n", + "\n", + "Output the simulated dataset to the dataset path as .fits files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.fits_imaging(\n", + " dataset=dataset,\n", + " data_path=dataset_path / \"data.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " overwrite=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Mask Extra Galaxy__\n", + "\n", + "Build and output a `mask_extra_galaxies.fits` covering the extra galaxy's region, so that the chapter 4 tutorial\n", + "which uses this dataset can load the mask directly without a separate data-preparation step.\n", + "\n", + "The circle is sized to ~3x the galaxy's `effective_radius`, which comfortably covers the light extent of the\n", + "`ExponentialSph` profile used above. The geometry is derived from the same centre + radius defined for the extra\n", + "galaxy in this script, so it stays in sync with any future tweak to those values.\n", + "\n", + "`Mask2D.circular` honours the `PYAUTO_SMALL_DATASETS=1` env var (caps to 16x16 at 0.6\"/px), so the mask\n", + "automatically shrinks alongside the small-dataset image and never raises an out-of-bounds error." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "extra_galaxies_mask = np.zeros(dataset.shape_native, dtype=bool)\n", + "\n", + "circle = al.Mask2D.circular(\n", + " shape_native=dataset.shape_native,\n", + " pixel_scales=dataset.pixel_scales,\n", + " centre=extra_galaxy_centre,\n", + " radius=3.0 * 0.5,\n", + " invert=True, # True inside the circle (i.e. masked region)\n", + ")\n", + "extra_galaxies_mask = np.logical_or(extra_galaxies_mask, circle.native)\n", + "\n", + "mask_extra_galaxies = al.Mask2D(\n", + " mask=extra_galaxies_mask,\n", + " pixel_scales=dataset.pixel_scales,\n", + ")\n", + "\n", + "aplt.fits_array(\n", + " array=mask_extra_galaxies,\n", + " file_path=dataset_path / \"mask_extra_galaxies.fits\",\n", + " overwrite=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset path as .png files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)\n", + "aplt.plot_array(array=dataset.data, title=\"Data\")\n", + "\n", + "aplt.subplot_tracer(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")\n", + "aplt.subplot_galaxies_images(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Tracer json__\n", + "\n", + "Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and galaxies\n", + "are safely stored and available to check how the dataset was simulated in the future.\n", + "\n", + "This can be loaded via the method `tracer = al.from_json()`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=tracer,\n", + " file_path=Path(dataset_path, \"tracer.json\"),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Extra Galaxy Centre__\n", + "\n", + "Output the centre of the extra galaxy to a .json file, so that it can be used to set up the lens model in the\n", + "chapter 4 tutorial." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=al.Grid2DIrregular(values=[extra_galaxy_centre]),\n", + " file_path=dataset_path / \"extra_galaxies_centres.json\",\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The dataset can be viewed in the folder `autolens_workspace/dataset/imaging/lens_extra_galaxy`." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/simulator/lens_sersic.ipynb b/notebooks/simulator/lens_sersic.ipynb index 927643a..95295e1 100644 --- a/notebooks/simulator/lens_sersic.ipynb +++ b/notebooks/simulator/lens_sersic.ipynb @@ -80,7 +80,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", diff --git a/notebooks/simulator/lens_x2.ipynb b/notebooks/simulator/lens_x2.ipynb index 759824f..99271d7 100644 --- a/notebooks/simulator/lens_x2.ipynb +++ b/notebooks/simulator/lens_x2.ipynb @@ -36,9 +36,9 @@ "\n", " - The lens galaxy's light is two `Sersic`'s.\n", " - The lens galaxy's mass distribution is two `Isothermal`'s.\n", - " - The source galaxy's light is an `Sersic`.\n", + " - The source galaxy's light is an `ExponentialCoreSph`.\n", "\n", - "This dataset is used in chapter 3 of the **HowToLens** lectures.\n", + "This dataset is used in chapter 4 of the **HowToLens** lectures.\n", "\n", "__Start Here Notebook__\n", "\n", diff --git a/notebooks/simulator/lens_x3.ipynb b/notebooks/simulator/lens_x3.ipynb new file mode 100644 index 0000000..5a031b3 --- /dev/null +++ b/notebooks/simulator/lens_x3.ipynb @@ -0,0 +1,408 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulator: Lens x3\n", + "==================\n", + "\n", + "This script simulates `Imaging` of a 'galaxy-scale' lens where there are three lens galaxies, each with their own\n", + "light and mass profiles, which all contribute significantly to the lensing of a single background source.\n", + "\n", + "Strong lenses with this complex mass distribution are more challenging to model than those with one or two lens\n", + "galaxies, because every additional deflector adds its own free light and mass parameters to the model.\n", + "\n", + "A system with three co-dominant lens galaxies and no shared dark-matter halo is exactly what the\n", + "`autolens_workspace`'s `multi_galaxy` package is dedicated to \u2014 its `start_here` and `modeling` examples are\n", + "directly applicable to this lens. (Systems where a dominant group-scale halo joins the galaxies belong one rung\n", + "up the regime ladder, in the `group` package.)\n", + "\n", + "This dataset is modeled in HowToLens chapter 4 and is used to illustrate how the parameter count of a lens model\n", + "grows with every co-dominant deflector, motivating the scaling relations introduced later in that chapter.\n", + "\n", + "__Contents__\n", + "\n", + "- **Model:** Compose the lens model fitted to the data.\n", + "- **Dataset Paths:** The `dataset_type` describes the type of data being simulated (in this case, `Imaging` data) and.\n", + "- **Simulate:** Simulate the image using a (y,x) grid with the adaptive over sampling scheme.\n", + "- **Ray Tracing:** Setup the lens galaxies' light, mass and source galaxy light for this simulated lens.\n", + "- **Output:** Output the simulated dataset to the dataset path as .fits files.\n", + "- **Visualize:** Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset.\n", + "- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass.\n", + "\n", + "__Model__\n", + "\n", + "This script simulates `Imaging` of a 'galaxy-scale' strong lens where:\n", + "\n", + " - The lens galaxies' light are three `Sersic`'s.\n", + " - The lens galaxies' mass distributions are three `Isothermal`'s.\n", + " - The source galaxy's light is an `ExponentialCoreSph`.\n", + "\n", + "This dataset is used in chapter 4 of the **HowToLens** lectures.\n", + "\n", + "__Start Here Notebook__\n", + "\n", + "If any code in this script is unclear, refer to the `autolens_workspace/*/imaging/simulator.ipynb` notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from pathlib import Path\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Paths__\n", + "\n", + "The `dataset_type` describes the type of data being simulated (in this case, `Imaging` data) and `dataset_name`\n", + "gives it a descriptive name." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_type = \"imaging\"\n", + "dataset_name = \"x3_lens_galaxies\"\n", + "dataset_path = Path(\"dataset\", dataset_type, dataset_name)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Simulate__\n", + "\n", + "Simulate the image using a (y,x) grid with the adaptive over sampling scheme.\n", + "\n", + "This simulated lens has three galaxies whose centres are offset from (0.0\", 0.0\"), forming a triangle around the\n", + "origin. The adaptive over sampling grid has all three centres input to account for this." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "grid = al.Grid2D.uniform(\n", + " shape_native=(100, 100),\n", + " pixel_scales=0.1,\n", + ")\n", + "\n", + "over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from(\n", + " grid=grid,\n", + " sub_size_list=[32, 8, 2],\n", + " radial_list=[0.3, 0.6],\n", + " centre_list=[(0.9, 0.0), (-0.6, -0.9), (-0.6, 0.9)],\n", + ")\n", + "\n", + "grid = grid.apply_over_sampling(over_sample_size=over_sample_size)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulate a simple Gaussian PSF for the image." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "psf = al.Convolver.from_gaussian(\n", + " shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create the simulator for the imaging data, which defines the exposure time, background sky, noise levels and psf." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "simulator = al.SimulatorImaging(\n", + " exposure_time=300.0,\n", + " psf=psf,\n", + " background_sky_level=0.1,\n", + " add_poisson_noise_to_data=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "Setup the lens galaxies' light, mass and source galaxy light for this simulated lens.\n", + "\n", + "The image plane is made of three separate lens galaxies, whose Einstein radii (0.9\", 0.8\" and 0.7\") are\n", + "deliberately comparable \u2014 no single galaxy dominates the deflection of the source's light, which is what makes\n", + "every one of them a co-dominant deflector that must be modeled individually.\n", + "\n", + "All three galaxies are at the same redshift, so ray tracing is single-plane and their deflection fields simply add." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy_0 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.Sersic(\n", + " centre=(0.9, 0.0),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=30.0),\n", + " intensity=0.1,\n", + " effective_radius=0.7,\n", + " sersic_index=3.0,\n", + " ),\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.9, 0.0),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.85, angle=30.0),\n", + " einstein_radius=0.9,\n", + " ),\n", + ")\n", + "\n", + "lens_galaxy_1 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.Sersic(\n", + " centre=(-0.6, -0.9),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.7, angle=120.0),\n", + " intensity=0.1,\n", + " effective_radius=0.6,\n", + " sersic_index=2.5,\n", + " ),\n", + " mass=al.mp.Isothermal(\n", + " centre=(-0.6, -0.9),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=120.0),\n", + " einstein_radius=0.8,\n", + " ),\n", + ")\n", + "\n", + "lens_galaxy_2 = al.Galaxy(\n", + " redshift=0.5,\n", + " bulge=al.lp.Sersic(\n", + " centre=(-0.6, 0.9),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=60.0),\n", + " intensity=0.1,\n", + " effective_radius=0.5,\n", + " sersic_index=3.5,\n", + " ),\n", + " mass=al.mp.Isothermal(\n", + " centre=(-0.6, 0.9),\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=60.0),\n", + " einstein_radius=0.7,\n", + " ),\n", + ")\n", + "\n", + "source_galaxy = al.Galaxy(\n", + " redshift=1.0,\n", + " bulge=al.lp.ExponentialCoreSph(\n", + " centre=(0.05, 0.15), intensity=0.2, effective_radius=0.5, radius_break=0.025\n", + " ),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Use these galaxies to setup a tracer, which will generate the image for the simulated `Imaging` dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "tracer = al.Tracer(\n", + " galaxies=[lens_galaxy_0, lens_galaxy_1, lens_galaxy_2, source_galaxy]\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Pass the simulator a tracer, which creates the image which is simulated as an imaging dataset." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset = simulator.via_tracer_from(tracer=tracer, grid=grid)" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output__\n", + "\n", + "Output the simulated dataset to the dataset path as .fits files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.fits_imaging(\n", + " dataset=dataset,\n", + " data_path=dataset_path / \"data.fits\",\n", + " psf_path=dataset_path / \"psf.fits\",\n", + " noise_map_path=dataset_path / \"noise_map.fits\",\n", + " overwrite=True,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset path as .png files." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "aplt.subplot_imaging_dataset(dataset=dataset)\n", + "aplt.plot_array(array=dataset.data, title=\"Data\")\n", + "\n", + "aplt.subplot_tracer(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")\n", + "aplt.subplot_galaxies_images(\n", + " tracer=tracer, grid=grid, output_path=dataset_path, output_format=\"png\"\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Tracer json__\n", + "\n", + "Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and galaxies\n", + "are safely stored and available to check how the dataset was simulated in the future.\n", + "\n", + "This can be loaded via the method `tracer = al.from_json()`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "al.output_to_json(\n", + " obj=tracer,\n", + " file_path=Path(dataset_path, \"tracer.json\"),\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The dataset can be viewed in the folder `autolens_workspace/dataset/imaging/x3_lens_galaxies`." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/notebooks/simulator/no_lens_light.ipynb b/notebooks/simulator/no_lens_light.ipynb index 28451a8..30de813 100644 --- a/notebooks/simulator/no_lens_light.ipynb +++ b/notebooks/simulator/no_lens_light.ipynb @@ -81,7 +81,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", diff --git a/notebooks/simulator/no_lens_light__mass_sis.ipynb b/notebooks/simulator/no_lens_light__mass_sis.ipynb index 2caa1d7..ea30376 100644 --- a/notebooks/simulator/no_lens_light__mass_sis.ipynb +++ b/notebooks/simulator/no_lens_light__mass_sis.ipynb @@ -79,7 +79,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", diff --git a/notebooks/simulator/source_complex.ipynb b/notebooks/simulator/source_complex.ipynb index 1e05f91..43d5984 100644 --- a/notebooks/simulator/source_complex.ipynb +++ b/notebooks/simulator/source_complex.ipynb @@ -83,7 +83,7 @@ "\n", "from autolens import jax_wrapper # Sets JAX environment before other imports\n", "\n", - "# from autolens import setup_notebook; setup_notebook()\n", + "from autolens import setup_notebook; setup_notebook()\n", "\n", "from pathlib import Path\n", "import autolens as al\n", diff --git a/notebooks/simulator/weak_lensing.ipynb b/notebooks/simulator/weak_lensing.ipynb new file mode 100644 index 0000000..638a310 --- /dev/null +++ b/notebooks/simulator/weak_lensing.ipynb @@ -0,0 +1,315 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Simulator: Weak Lensing\n", + "=======================\n", + "\n", + "This script simulates a weak gravitational lensing shear catalogue. Unlike the imaging simulators (which produce\n", + "a 2D image of the lensed source) the weak-lensing simulator produces a *catalogue* of (gamma_2, gamma_1) shear\n", + "measurements at the (y, x) positions of a population of background source galaxies.\n", + "\n", + "The lens is a **cluster-scale** mass (Einstein radius 25\", the regime where weak lensing is actually used) and the\n", + "background galaxies are placed in an **annulus outside the strong-lensing core** \u2014 this is where real weak-lensing\n", + "measurements are made, so the simulated shears are genuinely weak (|gamma| ~ 0.06-0.25) rather than the order-unity\n", + "shears found among galaxies projected near the Einstein radius.\n", + "\n", + "The shear computation itself comes from `Tracer.shear_yx_2d_via_hessian_from`, which differentiates the\n", + "deflection-angle field. On top of that the simulator adds Gaussian shape noise per galaxy (the dominant noise\n", + "source in real weak-lensing data \u2014 each galaxy has a random unlensed ellipticity around 0.2-0.4 per component).\n", + "\n", + "It is used to illustrate weak lensing data in the HowToLens lecture series.\n", + "\n", + "__Contents__\n", + "\n", + "- **Dataset Paths:** The `dataset_type` and `dataset_name` define the on-disk output folder.\n", + "- **Ray Tracing:** Build a Tracer from a cluster-scale Isothermal lens galaxy.\n", + "- **Source Positions:** Draw background source galaxies in an annulus outside the strong-lensing core.\n", + "- **Simulator:** Construct a `SimulatorShearYX` with the desired shape-noise level and random seed.\n", + "- **Output:** Save the simulated `WeakDataset` and the `Tracer` to JSON.\n", + "- **Visualize:** Plot the shear field and the dataset subplot mosaic via `aplt`.\n", + "\n", + "__Start Here Notebook__\n", + "\n", + "If any code in this script is unclear, refer to the `autolens_workspace/*/weak/simulator.ipynb` notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Google Colab Setup__\n", + "\n", + "This cell sets up the environment when the notebook is run on Google Colab: it installs the\n", + "required PyAuto packages, clones the workspace (configuration files and example datasets) and\n", + "points the configuration at it. If you are running the notebook elsewhere (e.g. locally via\n", + "your own installation) it does nothing, and you can run it safely.\n", + "\n", + "Colab tip: model-fits run much faster on a GPU \u2014 enable one via \"Runtime\" -> \"Change runtime\n", + "type\" -> \"Hardware accelerator\" before running the notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " import google.colab\n", + "except ImportError:\n", + " from autolens import setup_colab as _setup_colab\n", + "else:\n", + " import importlib\n", + " import subprocess\n", + " import sys\n", + "\n", + " subprocess.check_call(\n", + " [sys.executable, \"-m\", \"pip\", \"install\", \"autonerves\", \"--no-deps\"]\n", + " )\n", + " _setup_colab = importlib.import_module(\"autonerves.setup_colab\")\n", + "\n", + "_setup_colab.setup(\"howtolens\")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "\n", + "from autolens import jax_wrapper # Sets JAX environment before other imports\n", + "\n", + "from autolens import setup_notebook; setup_notebook()\n", + "\n", + "from pathlib import Path\n", + "\n", + "import numpy as np\n", + "\n", + "import autolens as al\n", + "import autolens.plot as aplt" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Dataset Paths__\n", + "\n", + "The `dataset_type` describes the type of data being simulated (in this case, a weak-lensing shear catalogue) and\n", + "`dataset_name` gives it a descriptive name. They define the folder the dataset is output to on your hard-disk:\n", + "\n", + " - The shear catalogue will be output to `/dataset/weak_lensing/simple/dataset.json`.\n", + " - The tracer used to simulate the dataset will be output alongside as `tracer.json`." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_type = \"weak_lensing\"\n", + "dataset_name = \"simple\"\n", + "\n", + "dataset_path = Path(\"dataset\") / dataset_type / dataset_name" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Ray Tracing__\n", + "\n", + "We define the lens galaxy's mass distribution as a **cluster-scale** `Isothermal` profile with an Einstein radius\n", + "of 25\" (no external shear, no source light \u2014 weak-lensing measurements are sensitive to the shear field induced\n", + "by the lens mass alone). A 25\" Einstein radius corresponds to a very massive cluster (velocity dispersion of order\n", + "1200-1400 km/s) \u2014 the mass scale on which weak lensing is the tool of choice, because the shear signal extends to\n", + "the many-arc-minute radii where strong lensing has no features.\n", + "\n", + "Because the source-galaxy positions are an irregular catalogue rather than a 2D pixel grid, this simulator\n", + "does not need PSF convolution, over-sampling, or background-sky modelling \u2014 those are all imaging-specific\n", + "concerns." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "lens_galaxy = al.Galaxy(\n", + " redshift=0.5,\n", + " mass=al.mp.Isothermal(\n", + " centre=(0.0, 0.0),\n", + " einstein_radius=25.0,\n", + " ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0),\n", + " ),\n", + ")\n", + "\n", + "source_galaxy = al.Galaxy(redshift=1.0)\n", + "\n", + "tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy])" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Source Positions__\n", + "\n", + "Real weak-lensing measurements avoid the strong-lensing core, where the linear-shear approximation breaks down\n", + "and cluster-member galaxies contaminate the sample. We therefore draw the background galaxies in an **annulus**\n", + "between an inner radius of 50\" (~2 Einstein radii, safely into the weak regime) and an outer radius of 200\"\n", + "(~3.3 arc-minutes), uniformly in area.\n", + "\n", + "This is the physically genuine weak-lensing geometry: at 50\" the shear is |gamma| ~ 0.25 and by 200\" it has\n", + "fallen to |gamma| ~ 0.06, so every galaxy is a *weak* probe whose individual shear is well below the 0.25 shape\n", + "noise. The signal lives in the ensemble of 1500 galaxies (a deep-survey source density of ~45 / arc-minute^2),\n", + "exactly as in a real cluster weak-lensing analysis.\n", + "\n", + "We build the positions as an `al.Grid2DIrregular` of (y, x) coordinates and pass them to the simulator's\n", + "`via_tracer_from` method." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "rng = np.random.default_rng(1)\n", + "\n", + "n_galaxies = 1500\n", + "radius_inner = 50.0 # arc-seconds \u2014 inside this we are in the strong-lensing core.\n", + "radius_outer = 200.0 # arc-seconds \u2014 the edge of the simulated weak-lensing field.\n", + "\n", + "radii = np.sqrt(rng.uniform(radius_inner**2.0, radius_outer**2.0, n_galaxies))\n", + "phi = rng.uniform(0.0, 2.0 * np.pi, n_galaxies)\n", + "\n", + "positions = al.Grid2DIrregular(\n", + " values=np.stack([radii * np.sin(phi), radii * np.cos(phi)], axis=1)\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Simulator__\n", + "\n", + "`SimulatorShearYX` takes a shape-noise level and an optional random seed. A `noise_sigma` of 0.25 is a typical\n", + "per-component shape-noise value for a weak-lensing survey; reduce it to 0.0 to inspect the noise-free shear field.\n", + "\n", + "`via_tracer_from` evaluates the tracer's shear at the supplied (y, x) positions and adds the shape noise." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "simulator = al.SimulatorShearYX(noise_sigma=0.25, seed=1)\n", + "\n", + "dataset = simulator.via_tracer_from(\n", + " tracer=tracer,\n", + " grid=positions,\n", + " name=dataset_name,\n", + ")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Output__\n", + "\n", + "Save the simulated `WeakDataset` and the `Tracer` to the dataset folder as JSON, ensuring the inputs to the\n", + "simulation are reproducible and inspectable later." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "dataset_path.mkdir(parents=True, exist_ok=True)\n", + "\n", + "al.output_to_json(obj=dataset, file_path=dataset_path / \"dataset.json\")\n", + "al.output_to_json(obj=tracer, file_path=dataset_path / \"tracer.json\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "__Visualize__\n", + "\n", + "The shear field is visualised with `matplotlib.quiver` rendered as *headless line segments*\n", + "(`headwidth=0, headlength=0, headaxislength=0`) \u2014 the standard weak-lensing convention, because shear is a\n", + "spin-2 quantity and a 180-degree rotation maps it back to itself, so an arrowhead would suggest a\n", + "directionality the data does not have.\n", + "\n", + "`aplt.subplot_weak_dataset` produces a 2x2 mosaic combining the shear field, the per-galaxy noise map, the\n", + "shear magnitude `|gamma|`, and the position angle `phi`. `aplt.plot_shear_yx_2d` writes a single-panel\n", + "quiver of the shear field alone \u2014 useful for high-resolution figures where the mosaic is too dense." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "source": [ + "aplt.subplot_weak_dataset(\n", + " dataset=dataset,\n", + " output_path=dataset_path,\n", + " output_format=\"png\",\n", + ")\n", + "\n", + "aplt.plot_shear_yx_2d(\n", + " shear_yx=dataset.shear_yx,\n", + " output_path=dataset_path,\n", + " output_format=\"png\",\n", + ")\n", + "\n", + "print(dataset.info)\n", + "print(f\"Wrote dataset to {dataset_path}\")" + ], + "outputs": [], + "execution_count": null + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The dataset can be viewed in the folder `dataset/weak_lensing/simple`." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.1" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} \ No newline at end of file diff --git a/scripts/README.md b/scripts/README.md index c103a00..9717845 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -4,9 +4,11 @@ a strong lens. # Folders - `chapter_1_introduction`: An introduction to strong gravitational lensing and **PyAutolens**. -- `chapter_2_lens_modeling`: How to model strong lenses, including a primer on Bayesian non-linear analysis. -- `chapter_3_search_chaining`: How to fit complex lens models using non-linear search chaining. -- `chapter_4_pixelizations`: How to perform pixelized reconstructions of the source-galaxy. +- `chapter_2_lens_modeling`: How to model strong lenses, including a primer on Bayesian non-linear analysis + and non-linear search chaining. +- `chapter_3_pixelizations`: How to perform pixelized reconstructions of the source-galaxy. +- `chapter_4_scaling_up_lensing`: How to scale lens modeling up to extra galaxies, multi-galaxy lenses, + scaling relations, groups, clusters and weak lensing. - `chapter_optional`: Optional tutorials. # Full Explanation @@ -62,15 +64,15 @@ and would like to adapt it to your own code, check out the Python auto-code form # HOW TO TACKLE HowToLens -The **HowToLens** lecture series current sits at 5 chapters, and each will take a day or so to go through +The **HowToLens** lecture series currently sits at 4 chapters, and each will take a day or so to go through properly. You probably want to be modeling lenses faster than that! Furthermore, the concepts in the -later chapters are pretty challenging, and familiarity and lens modeling is desirable before you +later chapters are pretty challenging, and familiarity with lens modeling is desirable before you tackle them. Therefore, we recommend that you complete chapters 1 & 2 and then apply what you've learnt to the modeling of simulated and real strong lens data, using the scripts found in the 'autolens_workspace'. Once you're happy with the results and confident with your use of **PyAutoLens**, you can then begin to cover the advanced functionality -covered in chapters 3, 4 & 5. +covered in chapters 3 & 4. # OVERVIEW OF CHAPTER 1 (Beginner) @@ -97,49 +99,40 @@ fit CCD imaging data of a strong gravitational lens with a lens model. At the en 2. How to fit a lens model to strong lens CCD imaging via a non-linear search. 3. The trade-off between realism and complexity when choosing a lens model. 4. Why an incorrect lens model may be inferred and how to prevent this from happening. -5. The challenges that are involved in inferred a robust lens model in a computationally reasonable run-time. +5. The challenges that are involved in inferring a robust lens model in a computationally reasonable run-time. +6. How to chain non-linear searches together to build automated lens modeling pipelines, including the + Source, Light and Mass (SLaM) pipelines. **Once completed, you'll be ready to model your own strong gravitational lenses with PyAutoLens!** # OVERVIEW OF CHAPTER 3 (Intermediate) -**Automated Modeling with non-linear search chaining** +**Using an inversion to perform a pixelized source reconstruction** -In chapter 3, we'll learn how to chain multiple non-linear searches together to build automated lens modeling pipelines -which can: - -1. Break-down the fitting of a model using multiple non-linear searches and prior passing. -2. Fit CCD imaging of a strong lens where the lens light and source light are fitted separately. -3. Use a custom pipeline to fit a strong lens with multiple lens galaxies or source galaxies. -4. Know how to use advanced pipelines called the Source, Light and Mass (SLaM) pipelines. - -# OVERVIEW OF CHAPTER 4 (Intermediate) - -**Using an inverison to perform a pixelized source reconstructions** - -In chapter 4, we'll learn how to reconstruct the lensed source galaxy using a pixel-grid, ensuring that we can fit an +In chapter 3, we'll learn how to reconstruct the lensed source galaxy using a pixel-grid, ensuring that we can fit an accurate lens model to sources with complex and irregular morphologies. You'll learn how to: 1. Pixelize a source-plane into a set of source-plane pixels defined by mappings to image pixels. 2. Perform a linear inversion on this source-plane pixelization to reconstruct the source's light. 3. Apply a smoothness prior on the source reconstruction, called regularization. 4. Apply smoothing within a Bayesian framework to objectively quantify the source reconstruction's complexity. -5. Define a border in the source-plane to prevent pixels tracing outside the source reconstruction. -6. Use alternative pixelizations, for example a Voronoi mesh whose pixels adapt to the lens's mass model. -7. Use these features to fit a lens model via non-linear searches. - -# OVERVIEW OF CHAPTER 5 (Advanced) - -**Hyper-Mode** - -In hyper-mode, we introduced advanced functionality that adapts various parts of the lens modeling procedure to the -data that we are fitting. - -NOTE: Hyper-mode is conceptually quite challenging, and I advise that you make sure you are very familiar with -PyAutoLens before covering chapter 5! - -1. Adapt an inversions's `Pixelization` to the morphology of the reconstructed source galaxy. -2. Adapt the `Regularization` scheme applied to this source to its surface brightness profile. -3. Use hyper-galaxies to scale the image's noise-map during fitting, to prevent over-fitting regions of the image. -4. include aspects of the data reduction in the model fitting, for example the background sky subtraction. -5. Use these features in PyAutoLens's search chaining framework. +5. Write down the linear algebra and Bayesian evidence equations that underpin the whole framework. +6. Define a border in the source-plane to prevent pixels tracing outside the source reconstruction. +7. Use alternative pixelizations, for example a Voronoi mesh whose pixels adapt to the lens's mass model + or to the source's own brightness, alongside adaptive regularization schemes. +8. Use these features to fit a lens model via non-linear searches. + +# OVERVIEW OF CHAPTER 4 (Advanced) + +**Scaling Up Lensing** + +In chapter 4, we'll scale lens modeling up beyond a single lens galaxy, learning how to: + +1. Handle extra galaxies near a lens, by scaling their light out of the fit or modeling them explicitly. +2. Model systems with two or more co-dominant lens galaxies and understand the degeneracies this creates. +3. Use scaling relations to tie galaxy masses to their luminosities, so model complexity stops growing + with every galaxy. +4. Model group-scale lenses, using truncated dPIE mass profiles and an optional group dark matter halo. +5. Model cluster-scale lenses, using point-source positions, a CSV interface for member catalogues and + multi-plane ray tracing. +6. Fit weak-lensing shear catalogues, which probe dark matter far beyond the strong lensing region. diff --git a/scripts/chapter_1_introduction/README.md b/scripts/chapter_1_introduction/README.md index f4be8f1..f3514ba 100644 --- a/scripts/chapter_1_introduction/README.md +++ b/scripts/chapter_1_introduction/README.md @@ -12,7 +12,7 @@ In chapter 1, we introduce you to strong gravitational lensing and the core **Py - [Tutorial 3: More Ray Tracing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb) — Advanced strong lens ray-tracing. -- [Tutorial 4: Point Sources](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb) — How lensing calculations when the source galaxy is a point-source (e.g. a quasar). +- [Tutorial 4: Point Sources](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb) — How lensing calculations change when the source galaxy is a point source (e.g. a quasar). - [Tutorial 5: Lensing Formalism](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb) — The algebraic lensing formalism used to describe strong lensing. diff --git a/scripts/chapter_1_introduction/tutorial_0_visualization.py b/scripts/chapter_1_introduction/tutorial_0_visualization.py index c37fee3..e9b8ed1 100644 --- a/scripts/chapter_1_introduction/tutorial_0_visualization.py +++ b/scripts/chapter_1_introduction/tutorial_0_visualization.py @@ -7,8 +7,9 @@ __Contents__ -- **Directories:** **PyAutoLens assumes** the working directory is `autolens_workspace` on your hard-disk. +- **Directories:** **HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk. - **Dataset:** Load and plot the strong lens dataset. +- **Dataset Auto-Simulation:** Create the dataset via its simulator script if it is not on your hard-disk. - **Subplots:** In addition to plotting individual figures, **PyAutoLens** can plot `subplots` which show multiple. - **Plot Customization:** Does the figure display correctly on your computer screen? - **Overlays:** Overlays such as critical curves and image positions are added using the `lines=` and `positions=`. @@ -21,11 +22,16 @@ # from autolens import setup_notebook; setup_notebook() """ -If the printed working directory does not match the workspace path on your computer, you can manually set it -as follows (the example below shows the path I would use on my laptop. The code is commented out so you do not -use this path in this tutorial! +__Directories__ + +**HowToLens** assumes the working directory is the `HowToLens` repository root on your hard-disk, so that relative +paths to `dataset/` and `output/` resolve correctly. + +If your working directory does not match this path on your computer, you can manually set it as follows (the +example below shows the path I would use on my laptop. The code is commented out so you do not use this path in +this tutorial! """ -# workspace_path = "/Users/Jammy/Code/PyAuto/autolens_workspace" +# workspace_path = "/Users/Jammy/Code/PyAuto/HowToLens" # #%cd $workspace_path # print(f"Working Directory has been set to `{workspace_path}`") @@ -33,10 +39,10 @@ __Dataset__ The `dataset_path` specifies where the dataset is located, which is the -directory `autolens_workspace/dataset/imaging/simple__no_lens_light`. +directory `dataset/imaging/simple__no_lens_light` of the HowToLens repository. -There are many example simulated images of strong lenses in this directory that will be used throughout the -**HowToLens** lectures. +The simulated images of strong lenses used throughout the **HowToLens** lectures are written to the `dataset` +directory at runtime by the simulator scripts in `scripts/simulator/`. """ from pathlib import Path @@ -92,7 +98,7 @@ If not, the default matplotlib settings can be customized via the config files in: - autolens_workspace/config/visualize/ + config/visualize/ Key config entries: diff --git a/scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py b/scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py index 56cada6..c35ea02 100644 --- a/scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py +++ b/scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py @@ -57,6 +57,10 @@ - **One Dimension Projection:** We often want to calculative 1D quantities of a light profile, for example to plot how its light. - **Galaxies:** Galaxies are collections of light profiles that represent a galaxy's luminous emission. - **Units:** By assuming a redshift for a galaxy we can convert its quantities from arcseconds to kiloparsecs. +- **Wrap Up:** Summary of the script and next steps. +- **Advanced Topics:** Optional reference material for expert users, not needed on a first read. +- **Other Unit Conversion:** Other unit conversions a galaxy's redshift enables (e.g. AB magnitudes). +- **Over Sampling:** The adaptive over-sampling scheme used to evaluate light profiles accurately. """ @@ -93,7 +97,7 @@ shape_native=( 101, 101, - ), # The dimensions of the grid, which here is 100 x 100 pixels. + ), # The dimensions of the grid, which here is 101 x 101 pixels. pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds. ) @@ -150,7 +154,7 @@ We can shift the grid to a new center, (y_c, x_c), by subtracting this center from each coordinate. """ -centre = (0.3, 0.5) # Shifting the grid to be centered at y=1.0", x=2.0". +centre = (0.3, 0.5) # Shifting the grid to be centered at y=0.3", x=0.5". grid_shifted = grid grid_shifted[:, 0] = grid_shifted[:, 0] - centre[0] # Shift in y-direction. diff --git a/scripts/chapter_1_introduction/tutorial_2_ray_tracing.py b/scripts/chapter_1_introduction/tutorial_2_ray_tracing.py index 5fed74a..c914104 100644 --- a/scripts/chapter_1_introduction/tutorial_2_ray_tracing.py +++ b/scripts/chapter_1_introduction/tutorial_2_ray_tracing.py @@ -65,6 +65,7 @@ - **Galaxies:** A `Galaxy` can contain both light and mass profiles, forming realistic lens and source galaxies. - **Tracer:** The `Tracer` object automates ray-tracing for a system of galaxies at different redshifts. - **Mappings:** Every image-plane coordinate maps to a source-plane coordinate via the lens equation. +- **Wrap Up:** Summary of the script and next steps. """ diff --git a/scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py b/scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py index 7b28149..79e01a5 100644 --- a/scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py +++ b/scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py @@ -1,5 +1,5 @@ """ -Tutorial 5: More Ray Tracing +Tutorial 3: More Ray Tracing ============================ We'll now reinforce the ideas that we learnt about ray-tracing in the previous tutorial and introduce the following @@ -27,7 +27,7 @@ - **Initial Setup:** To begin, lets setup the grid we'll ray-trace using. - **Concise Code:** Lets set up the tracer used in the previous tutorial. -- **Critical Curves:** To end, we can finally explain what the black lines that have appeared on many of the plots. +- **Critical Curves:** We can finally explain what the black lines that have appeared on many of the plots. - **Caustics:** In the previous tutorial, we plotted the critical curves of the mass profile on the image-plane. - **Units:** Lets plot the lensing quantities again. - **More Complexity:** We now make a lens with some attributes we didn`t in the last tutorial. @@ -89,8 +89,8 @@ """ __Critical Curves__ -To end, we can finally explain what the black lines that have appeared on many of the plots throughout this chapter -actually are. +We can finally explain what the black lines that have appeared on many of the plots throughout this chapter +actually are. These lines are called the 'critical curves', and they define line of infinite magnification due to a mass profile. They therefore mark where in the image-plane a mass profile perfectly `focuses` light rays such that if a source is @@ -104,7 +104,7 @@ when its inner mass slope is less steep than a steep power-law). To make it appear below, we therefore change the mass profile of our lens galaxy to a `PowerLawSph` with a slope of 1.8. -In the next tutorial, we'll introduce 'caustics', which are where the critical curves map too in the source-plane. +In the next section, we'll introduce 'caustics', which are where the critical curves map to in the source-plane. """ mass_profile = al.mp.PowerLawSph(centre=(0.0, 0.0), einstein_radius=1.6, slope=1.8) @@ -121,12 +121,12 @@ """ __Caustics__ -In the previous tutorial, we plotted the critical curves of the mass profile on the image-plane. We will now plot the +In the previous section, we plotted the critical curves of the mass profile on the image-plane. We will now plot the 'caustics', which correspond to each critical curve ray-traced to the source-plane. This is computed by using the lens galaxy mass profile's to calculate the deflection angles at the critical curves and ray-trace them to the source-plane. -As discussed in the previous tutorial, critical curves mark regions of infinite magnification. Thus, if a source +As discussed in the previous section, critical curves mark regions of infinite magnification. Thus, if a source appears near a caustic in the source plane it will appear significantly brighter than its true luminosity. We again have to use a mass profile with a slope below 2.0 to ensure a radial critical curve and therefore radial @@ -172,9 +172,9 @@ aplt.plot_array(array=tracer.image_2d_list_from(grid=grid)[1], title="Plane 1 Image") """ -Caustics also mark the regions in the source-plane where the multiplicity of the strong lens changes. That is, -if a source crosses a caustic, it goes from 2 images to 1 image. Try and show this yourself by changing the (y,x) -centre of the source-plane galaxy's light profile! +Caustics also mark the regions in the source-plane where the multiplicity of the strong lens changes. Each time +a source crosses a caustic, the number of images it produces changes by two. Try and show this yourself by changing +the (y,x) centre of the source-plane galaxy's light profile! """ source = al.Galaxy( redshift=1.0, @@ -203,7 +203,7 @@ aplt.subplot_galaxies_images(tracer=tracer, grid=grid) """ -If you're too familiar with Cosmology, it will be unclear how exactly we converted the distance units from +If you're not too familiar with Cosmology, it will be unclear how exactly we converted the distance units from arcseconds to kiloparsecs. You'll need to read up on your Cosmology lecture to understand this properly. You can create a `Cosmology` object, which provides many methods for calculation different cosmological quantities, @@ -325,9 +325,9 @@ Now lets pass our 4 galaxies to a `Tracer`, which means the following will occur: - - Using the galaxy redshift`s, and image-plane and source-plane will be created each with two galaxies galaxies. + - Using the galaxy redshifts, an image-plane and source-plane will be created, each with two galaxies. -We've also pass the tracer below a Planck15 cosmology, where the cosomology of the Universe describes exactly how +We also pass the tracer below a Planck15 cosmology, where the cosmology of the Universe describes exactly how ray-tracing is performed. """ tracer = al.Tracer( @@ -350,7 +350,7 @@ 5) The image of the source galaxies is computed by summing both of their images and ray-tracing their light back to the image-plane. -This process is pretty much the same as we have single in previous tutorials when there is one galaxy per plane. We +This process is pretty much the same as we have seen in previous tutorials when there is one galaxy per plane. We are simply summing the images and deflection angles of the galaxies before using them to perform ray-tracing. """ aplt.plot_array(array=tracer.image_2d_from(grid=grid), title="Image") @@ -369,7 +369,7 @@ """ __Wrap Up__ -Tutorial 6 completed! Try the following: +Tutorial 3 completed! Try the following: 1) If you change the lens and source galaxy redshifts, does the tracer's image change? diff --git a/scripts/chapter_1_introduction/tutorial_4_point_sources.py b/scripts/chapter_1_introduction/tutorial_4_point_sources.py index 83b0ff2..30467bb 100644 --- a/scripts/chapter_1_introduction/tutorial_4_point_sources.py +++ b/scripts/chapter_1_introduction/tutorial_4_point_sources.py @@ -2,15 +2,65 @@ Tutorial 4: Point Sources ========================= -This tutorial is not wrriten yet, but will explain how point source lensing works. +In the previous tutorials, the background sources we lensed were galaxies: extended objects whose light spreads over +many thousands of light years. When lensed, their light is warped into the arcs and Einstein rings we produced with +the `Tracer`, spread across many pixels of the image. -This tutorial is not necesary for using PyAutoLens or doing strong lens analysis, so don't worry that it is not -written yet! +However, not every source is a galaxy. Some of the most scientifically valuable strong lenses have a background +source that is physically tiny: a **quasar** (the bright accretion disk around a supermassive black hole, light-days +across) or a **supernova** (an exploding star, even smaller). These are called **point sources**. -Tutorial 8 summary is written and you should check that out instead! +When a point source is strongly lensed, we do not see arcs. Instead, we see the same source appear multiple times, +as two or four (or more) distinct, unresolved points of light in the image. Modeling these systems is fundamentally +different from modeling extended sources — different enough that it deserves its own tutorial before we go any +further. + +In this tutorial, we will see why. The short version: for an extended source, we ray-trace light *forwards* from the +image-plane to the source-plane, which is computationally simple. For a point source, we must go *backwards* — we +know where the source is, and we must find every image-plane position its light appears at. That means solving the +lens equation, an inverse problem with no analytic solution, which requires a dedicated numerical tool called the +`PointSolver`. + +Only the lens equation from tutorial 2 is needed to follow this tutorial. The full lensing formalism — convergence, +potentials, and how deflection angles arise from a mass distribution — is covered in the next tutorial. + +Here is an overview of what we'll cover: + +- **Initial Setup**: Create a strong lens with a compact extended source, showing how a small source produces + distinct multiple images rather than arcs. + +- **Point Sources**: What a point source is physically, and why quasars and supernovae appear as multiple + unresolved images. + +- **Point Source Tracer**: Represent a point source in PyAutoLens using the `Point` profile and build a `Tracer`. + +- **The Lens Equation**: Why finding a point source's multiple images means solving the lens equation — an inverse + problem with no analytic solution. + +- **Point Solver**: The `PointSolver`, which solves the lens equation numerically by ray-tracing triangles. + +- **Multiple Images and Critical Curves**: Plot the solved image positions over the lensed image and the tracer's + critical curves. + +- **Magnifications**: Compute the magnification at each multiple image, and why observed fluxes are often + unreliable for modeling. + +- **Time Delays**: The arrival-time differences between multiple images, and why they matter for cosmology. + +- **Extended Versus Point Computations**: A recap of why the two regimes require fundamentally different + calculations and fitting strategies. __Contents__ +- **Initial Setup:** Create a strong lens with a compact extended source, producing distinct multiple images. +- **Point Sources:** What a point source is physically and why it appears as multiple unresolved images. +- **Point Source Tracer:** Represent a point source with the `Point` profile and build a `Tracer`. +- **The Lens Equation:** Finding multiple images means solving the lens equation, an inverse problem. +- **Point Solver:** The `PointSolver` solves the lens equation numerically via triangle ray-tracing. +- **Multiple Images and Critical Curves:** Plot the solved positions over the image and critical curves. +- **Magnifications:** The magnification of each multiple image, and why fluxes are often unreliable. +- **Time Delays:** The relative arrival times of the multiple images and their use in cosmology. +- **Extended Versus Point Computations:** Why the two regimes require fundamentally different calculations. - **Wrap Up:** Summary of the script and next steps. """ @@ -19,9 +69,323 @@ # from autolens import setup_notebook; setup_notebook() +import numpy as np + import autolens as al import autolens.plot as aplt """ +__Initial Setup__ + +Let's begin with a strong lens system like those of the previous tutorials: an elliptical isothermal mass profile +for the lens galaxy, and an extended light profile for the source galaxy. + +There is one deliberate difference: the source's `effective_radius` is just 0.1", making it far more compact than +the sources we lensed before. +""" +grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.05, +) + +lens_galaxy = al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=1.6, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0), + ), +) + +source_galaxy_extended = al.Galaxy( + redshift=1.0, + light=al.lp.ExponentialCore( + centre=(0.07, 0.07), intensity=0.1, effective_radius=0.1 + ), +) + +tracer_extended = al.Tracer(galaxies=[lens_galaxy, source_galaxy_extended]) + +""" +When we plot the image of this system, something new happens: instead of a sweeping arc or Einstein ring, the +source appears as distinct blobs of light — the same source, imaged multiple times at different locations in +the image-plane. + +This is a general rule of strong lensing: the more compact the source, the more its lensed image breaks up into +separate multiple images. An extended galaxy blurs its multiple images together into arcs; a compact source keeps +them distinct. +""" +aplt.plot_array( + array=tracer_extended.image_2d_from(grid=grid), + title="Compact Source Multiple Images", +) + +""" +__Point Sources__ + +Now imagine shrinking the source further — not to 0.01", but to the physical size of a quasar accretion disk or a +supernova photosphere. These objects are light-days to light-weeks across, which at cosmological distances +corresponds to micro-arcseconds on the sky. That is millions of times smaller than the resolution of any telescope. + +Strictly speaking, such a source still has a finite size, but the telescope cannot resolve it. Each multiple image +appears as a single unresolved point of light, with all of its flux effectively contained within one pixel (spread +only by the telescope's blurring, which we will meet in a later tutorial). + +This is what we call a **point source**. Lensed quasars and lensed supernovae are the classic examples, and they +are prized objects: the arrival-time differences between their multiple images can be used to measure the expansion +rate of the Universe (the Hubble constant), and their fluxes are sensitive probes of small-scale dark matter +structure. + +For a point source, the extended-source picture of lensing breaks down: + +- There is no arc or ring of surface brightness to fit, just a handful of image positions. + +- The observable quantities are the (y,x) **positions** of the multiple images, and optionally their **fluxes** + and **time delays**. + +- The concept of evaluating a light profile's surface brightness on a ray-traced grid — the engine of the previous + two tutorials — no longer applies, because the source has no extent to evaluate. + +__Point Source Tracer__ + +PyAutoLens represents a point source with the `Point` profile, found in the `al.ps` module (`ps` for point source). +Unlike a light profile it has no `intensity`, `effective_radius` or elliptical components — it is fully described +by its (y,x) centre in the source-plane. + +We give the point source the same centre, (0.07", 0.07"), as the compact extended source above, so its multiple +images will coincide with the bright blobs in the image we just plotted. +""" +point_source = al.ps.Point(centre=(0.07, 0.07)) + +source_galaxy = al.Galaxy(redshift=1.0, point_0=point_source) + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) + +""" +Note that we attached the point source to its galaxy with the name `point_0`. This name is a label that PyAutoLens +uses when fitting real data to pair each point source in the model with the dataset containing its observed image +positions. With one source the name is a formality, but group- and cluster-scale lenses can contain many point +sources, and the names keep each one matched to its own data. + +__The Lens Equation__ + +In tutorial 2, we met the **lens equation**, which relates a coordinate in the image-plane to the coordinate in +the source-plane its light ray comes from: + +$\beta = \theta - \alpha(\theta)$ + +where $\theta$ is the image-plane (y,x) coordinate, $\alpha(\theta)$ is the deflection angle at that coordinate, +and $\beta$ is the source-plane (y,x) coordinate. + +For the extended sources of tutorials 2 and 3, we always used this equation in the *forward* direction: take every +image-plane coordinate $\theta$ on our grid, compute its deflection angles, and subtract to get $\beta$. This is +easy — it is just an evaluation, one subtraction per coordinate. + +For a point source, the question is reversed. We *know* the source's position $\beta$ (the centre of the `Point` +profile), and we want to find every image-plane position $\theta$ that satisfies the lens equation for that +$\beta$. Each solution is one of the multiple images. + +This is called **solving the lens equation**, and it is much harder than evaluating it: + +- The equation is non-linear, because the deflection angles $\alpha(\theta)$ vary with position in a way set by the + mass profile. For all but the simplest mass profiles, no analytic solution exists. + +- There are multiple solutions — that is the whole point! A single $\beta$ maps to two, four or more distinct + $\theta$ positions, and we must find all of them. + +We therefore need a numerical root-finding method that searches the image-plane for every position whose ray-traced +source-plane coordinate lands on the source. (Where the deflection angles themselves come from is the subject of +the next tutorial — for now, we just need the equation.) + +__Point Solver__ + +PyAutoLens solves the lens equation with the `PointSolver`, which uses a triangle-based scheme: + +1. Overlay the image-plane with a grid of triangles. + +2. Ray-trace the corners of every triangle to the source-plane using the lens equation, giving a set of + ray-traced source-plane triangles. + +3. Keep every triangle that contains the source's (y,x) centre — somewhere inside that image-plane triangle is a + multiple image. + +4. Subdivide the retained triangles into smaller triangles and repeat, homing in on each multiple image with + progressively finer precision. + +The `pixel_scale_precision` input sets the triangle size at which the solver stops refining, and therefore the +precision of the image positions. Smaller values are more precise but cost more computation; 0.001" is a good +balance for most lens modeling. + +Mass models also predict a faint "central image" near the centre of the lens, which is usually so heavily +demagnified that real observations never detect it. The `magnification_threshold` input discards solved images +whose magnification is below the threshold, removing this central image to match what the data would contain. +""" +solver_grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.2, +) + +solver = al.PointSolver.for_grid( + grid=solver_grid, pixel_scale_precision=0.001, magnification_threshold=0.1 +) + +""" +We now solve the lens equation, passing the solver our tracer (which provides the mass model's deflection angles) +and the source-plane centre of the point source. + +The result is the set of image-plane (y,x) coordinates of the multiple images. +""" +positions = solver.solve(tracer=tracer, source_plane_coordinate=(0.07, 0.07)) + +print("Multiple Image Positions (y,x) in arc-seconds:") +print(positions) + +""" +__Multiple Images and Critical Curves__ + +For our elliptical mass profile, the solver finds four multiple images (a fifth, central image existed but was +removed by the `magnification_threshold`). The number of images depends on the mass model: + +- Spherical mass profiles produce three images (including the demagnified central image). + +- Elliptical mass profiles produce five images (again including the central image). + +- More complex systems, with multiple lens galaxies, can produce even more. + +To see that the solver got the right answer, we plot the solved positions on top of the compact extended source's +image from the start of the tutorial. The positions land on the centres of the bright multiple images. + +We also overlay the tracer's tangential critical curve (introduced in tutorial 3 — the line of infinite +magnification in the image-plane). The multiple images of a strongly lensed point source straddle the critical +curve, and images close to it are the most highly magnified. +""" +tangential_critical_curve_list = al.LensCalc.from_tracer( + tracer=tracer +).tangential_critical_curve_list_from(grid=grid) + +aplt.plot_array( + array=tracer_extended.image_2d_from(grid=grid), + positions=positions, + lines=tangential_critical_curve_list, + title="Multiple Images and Critical Curve", +) + +""" +__Magnifications__ + +Lensing does not just relocate a point source's light — it magnifies it. Each multiple image has its own +magnification, set by how strongly the mass model focuses light rays at that image-plane position, and it can make +an image tens of times brighter than the unlensed source. + +For an extended source, magnification stretches the source over more pixels. For a point source, the image stays +unresolved — so the magnification instead multiplies its observed **flux**. If the source's intrinsic flux is $F$, +the flux of a multiple image with magnification $\mu$ is $|\mu| * F$. + +We compute the magnification at each solved image position below (the sign of $\mu$ encodes the image's parity — +whether it is mirror-flipped — which is why we take its absolute value for fluxes). +""" +magnifications = al.LensCalc.from_tracer( + tracer=tracer +).magnification_2d_via_hessian_from(grid=positions) + +print("Magnification of each Multiple Image:") +print(magnifications) + +flux = 1.0 +fluxes = [flux * np.abs(magnification) for magnification in magnifications] + +print("Flux of each Multiple Image (for a source of flux 1.0):") +print(fluxes) + +""" +In principle, comparing the observed flux ratios of the images to the model's magnification ratios is a powerful +constraint on the mass model. In practice, point-source fluxes are treated with caution, because effects the smooth +mass model does not include can change them substantially: + +- **Microlensing**: individual stars in the lens galaxy lens the point source again on micro-arcsecond scales, + magnifying or demagnifying each image by unpredictable amounts that change over time. + +- **Dark matter substructure**: small invisible clumps of dark matter near an image perturb its magnification + (this is a systematic for mass modeling, but also exactly why lensed quasars are used to *search* for + substructure). + +- Dust in the lens galaxy and the source's own intrinsic variability further distort the observed flux ratios. + +Image positions are essentially immune to all of these, which is why point-source lens modeling is built on +positions first, with fluxes included only when the science demands it and the systematics are under control. + +__Time Delays__ + +There is a third observable unique to point sources. The light of each multiple image travels a different path +through the Universe, and each path takes a different time — partly because the geometric path lengths differ, and +partly because light is slowed as it climbs through the lens galaxy's gravitational field (the Shapiro delay). + +For an extended, steady source this is unobservable. But quasars flicker and supernovae explode: when the source +varies, the variation appears in each multiple image at a different time. By monitoring the images, these +**time delays** can be measured — typically days to months apart. + +Time delays are a cosmological gold mine: their absolute scale depends on the distances between us, the lens and +the source, so a lens model plus measured delays yields the Hubble constant. This is time-delay cosmography, one +of the headline applications of point-source lensing. + +The tracer computes the arrival time at each image position from its mass model and the galaxies' redshifts. Only +the *differences* between images are observable — a delay of the same length along every path is invisible — so it +is the relative values below that matter. +""" +time_delays = tracer.time_delays_from(grid=positions) + +print("Time Delay of each Multiple Image (days):") +print(time_delays) + +print("Time Delays relative to first image (days):") +print(time_delays - time_delays[0]) + +""" +__Extended Versus Point Computations__ + +We can now state precisely why point-source lensing works so differently from everything in tutorials 2 and 3, on +both the computation and the data. + +**Extended sources are forward-modeled.** To predict the data, we ray-trace every image-plane pixel to the +source-plane via the lens equation (an evaluation, not a solve) and compute the source light profile's surface +brightness at each ray-traced coordinate. This is computationally cheap, requires no equation solving, and a fit +compares the predicted and observed images pixel-by-pixel. + +**Point sources require solving an inverse problem.** The source has no surface brightness to evaluate — the model +must predict the discrete positions of the multiple images, which means numerically solving the lens equation with +the `PointSolver` every time the mass model changes. A fit then compares the predicted image positions to the +observed ones (and optionally the predicted fluxes and time delays to their measurements) — a handful of numbers, +rather than thousands of pixels. + +This trade shapes the whole analysis. The point-source dataset is tiny — a few positions with uncertainties — but +each model evaluation involves root finding over the image plane rather than simple forward ray-tracing. Whole +questions that never arose for extended sources, such as how to pair predicted images with observed ones when the +model predicts too many or too few, become central to the likelihood (the workspace covers these in detail). + __Wrap Up__ + +In this tutorial, we met point sources and the tools PyAutoLens uses to model them. Let's summarise what we've +learnt: + +- **Point Sources**: Quasars and supernovae are so much smaller than a telescope's resolution that each of their + multiple images appears as a single unresolved point of light — no arcs, no rings. + +- **The Lens Equation**: Point-source modeling means solving $\beta = \theta - \alpha(\theta)$ for the image + positions $\theta$ given the source position $\beta$ — a non-linear inverse problem with no analytic solution, + in contrast to the cheap forward evaluation used for extended sources. + +- **Point Solver**: The `PointSolver` solves the lens equation numerically, ray-tracing progressively finer + triangles until every multiple image is located to sub-pixel precision, and filtering out the demagnified + central image. + +- **Observables**: A point-source dataset consists of image positions, and optionally fluxes and time delays. + Positions are the bedrock; fluxes are often compromised by microlensing and substructure; time delays enable + measurement of the Hubble constant. + +When you are ready to model real lensed quasars and supernovae — composing mass models, fitting observed positions +with a non-linear search, and including fluxes and time delays — the `autolens_workspace/scripts/point_source` +package is the place to go, starting with its `start_here` example. + +In the next tutorial, we return to the lensing formalism itself: where deflection angles come from, and the +quantities (convergence, potential, magnification) that describe a mass distribution's lensing power. """ diff --git a/scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py b/scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py index a2c4991..7ff511f 100644 --- a/scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py +++ b/scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py @@ -2,16 +2,37 @@ Tutorial 5: Lensing Formalism ============================= -This tutorial is not wrriten yet, but will explain what all the different lens quantities are and give a more -formal description of them. +This tutorial is the equations lecture of **HowToLens**. -This tutorial is not necesary for using PyAutoLens or doing strong lens analysis, so don't worry that it is not -written yet! +Every other tutorial in this series deliberately defers the formal mathematics of gravitational lensing, so that you +could first learn lensing hands-on: in the earlier tutorials we created grids, evaluated light and mass profiles, +ray-traced coordinates from the image-plane to the source-plane and looked at quantities like the convergence, +deflection angles and critical curves visually, without ever writing down their formal definitions. -Tutorial 8 summary is written and you should check that out instead! +This tutorial gathers all of that algebra in one place. Each quantity you have already computed and plotted — +deflection angles, convergence, potential, magnification, critical curves, the Einstein radius — is now given its +formal mathematical definition, explained in plain words, and tied back to the exact **PyAutoLens** method that +computes it. The goal is that after this tutorial, every symbol in a lensing paper maps to a line of code you know. + +None of this mathematics is required to use **PyAutoLens** or to analyse strong lenses — the library computes +everything for you. But understanding where each quantity comes from will make you a better lens modeler, and this +is the single place in **HowToLens** where the formalism lives, so you can return here whenever you need it. + +We will build one simple strong lens system — an elliptical isothermal (SIE) lens galaxy and a Sersic source — and +reuse it throughout, computing every quantity from the same tracer so you can see how they all fit together. __Contents__ +- **Initial Setup:** Build the grid, SIE lens, Sersic source and `Tracer` reused throughout the tutorial. +- **Cosmological Distances:** Angular diameter distances D_l, D_s, D_ls and how arcseconds convert to kiloparsecs. +- **The Lens Equation:** beta = theta - alpha(theta), the mapping from image-plane to source-plane, and multiple images. +- **Convergence:** Surface mass density Sigma, the critical surface density Sigma_cr and kappa = Sigma / Sigma_cr. +- **Deflection Angles:** The integral relating the convergence field to the deflection angles. +- **The Lensing Potential:** The scalar potential psi, with alpha = grad psi and kappa = (1/2) laplacian psi. +- **Shear and Magnification:** The Hessian, the Jacobian matrix A, shear gamma and magnification mu = 1 / det(A). +- **Critical Curves and Caustics:** Where det(A) = 0 in the image-plane, and its map to the source-plane. +- **Einstein Radius:** Its definition, the SIS and point-mass closed forms, and why it is the robust mass measurement. +- **Time Delays:** The Fermat potential and the time-delay surface, in brief. - **Wrap Up:** Summary of the script and next steps. """ @@ -20,9 +41,531 @@ # from autolens import setup_notebook; setup_notebook() +import matplotlib.pyplot as plt +import numpy as np import autolens as al +import autoarray as aa import autolens.plot as aplt +""" +__Initial Setup__ + +We first build the strong lens system used throughout this tutorial, mirroring the objects introduced in tutorials +2 and 3: a 2D grid of (y,x) arcsecond coordinates, a lens galaxy with an elliptical isothermal (`Isothermal`) mass +profile — the singular isothermal ellipsoid, or SIE, the workhorse mass model of galaxy-scale lensing — and a +source galaxy with a Sersic light profile. +""" +grid = al.Grid2D.uniform(shape_native=(100, 100), pixel_scales=0.05) + +mass_profile = al.mp.Isothermal( + centre=(0.0, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0), + einstein_radius=1.6, +) + +lens_galaxy = al.Galaxy(redshift=0.5, mass=mass_profile) + +source_galaxy = al.Galaxy( + redshift=1.0, + bulge=al.lp.SersicCore( + centre=(0.0, 0.1), + ell_comps=(0.1, 0.0), + intensity=0.3, + effective_radius=0.3, + sersic_index=1.0, + ), +) + +""" +We combine these into a `Tracer` with a Planck 2015 cosmology. The redshifts of the two galaxies (0.5 and 1.0) and +the cosmology together fix every distance that appears in the equations below. +""" +cosmology = al.cosmo.Planck15() + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy], cosmology=cosmology) + +""" +Here is the lensed image of this system, which by now should look familiar. Every equation in this tutorial is +describing some aspect of how this image forms. +""" +image = tracer.image_2d_from(grid=grid) +aplt.plot_array(array=image, title="Lensed Image of the Source Galaxy") + +""" +We also create a `LensCalc` object from the tracer. As we saw in tutorial 3, this is **PyAutoLens**'s calculator for +derived lensing quantities — everything it computes is derived from the tracer's deflection angles, which is a theme +we will see repeatedly below. +""" +lens_calc = al.LensCalc.from_tracer(tracer=tracer) + +""" +__Cosmological Distances__ + +Gravitational lensing is a geometric effect, so distances are everywhere in its equations. The distances used are +**angular diameter distances**, defined so that an object of physical size l at angular diameter distance D subtends +an angle (in radians): + + theta = l / D + +Three distances appear in every lensing formula: + +- D_l — the angular diameter distance from the observer to the lens (deflector) galaxy. +- D_s — the angular diameter distance from the observer to the source galaxy. +- D_ls — the angular diameter distance from the lens to the source. + +In an expanding Universe angular diameter distances do not add linearly, so D_ls != D_s - D_l. Each is computed by +integrating the cosmological expansion history between the two redshifts, which is why the tracer requires both +galaxy redshifts and a cosmology. + +The `Cosmology` object computes each of these directly (values below are in kiloparsecs): +""" +D_l = cosmology.angular_diameter_distance_to_earth_in_kpc_from(redshift=0.5) +D_s = cosmology.angular_diameter_distance_to_earth_in_kpc_from(redshift=1.0) +D_ls = cosmology.angular_diameter_distance_between_redshifts_in_kpc_from( + redshift_0=0.5, redshift_1=1.0 +) + +print(f"D_l (kpc) = {D_l:.3e}") +print(f"D_s (kpc) = {D_s:.3e}") +print(f"D_ls (kpc) = {D_ls:.3e}") + +""" +The angular diameter distance is also what converts **PyAutoLens**'s internal angular units of arcseconds into +physical distances. An angle of 1.0" at redshift z corresponds to a physical size: + + l = D(z) * (1.0" in radians) + +The `Cosmology` object wraps this up as a `kpc_per_arcsec` conversion factor, which differs between the image-plane +(lens redshift) and source-plane (source redshift) because the two planes are at different distances. +""" +kpc_per_arcsec_lens = cosmology.kpc_per_arcsec_from(redshift=0.5) +kpc_per_arcsec_source = cosmology.kpc_per_arcsec_from(redshift=1.0) + +print(f"kpc per arcsec at the lens (z=0.5) = {kpc_per_arcsec_lens:.4f}") +print(f"kpc per arcsec at the source (z=1.0) = {kpc_per_arcsec_source:.4f}") + +""" +So our lens's Einstein radius of 1.6" corresponds to a physical scale of roughly 1.6 * 6.3 ~ 10 kpc at the lens +redshift — a sensible size for the inner regions of a massive elliptical galaxy. + +__The Lens Equation__ + +The fundamental equation of gravitational lensing is the **lens equation**, which we met in tutorial 2: + + beta = theta - alpha(theta) + +where: + +- theta is the observed (image-plane) angular position of a light ray, in arcseconds. +- alpha(theta) is the (scaled) deflection angle — how much the ray is bent by the lens's gravity at position theta. +- beta is the true (source-plane) position — where the source would appear if there were no lens. + +In plain words: to find where a light ray we observe at theta really came from, subtract the deflection the lens +applied to it. This is exactly the ray-tracing calculation we performed in tutorial 2, and it is worth knowing that +the deflection angle here is the "scaled" deflection: the physical bending angle of the ray multiplied by the +distance ratio D_ls / D_s, which is the convention that makes the lens equation take the simple form above. + +The lens equation is trivial to evaluate one way (given theta, compute beta), but it cannot be inverted +analytically: for a strong lens, *multiple* image-plane positions theta can satisfy the equation for the *same* +source position beta. This is why strong lenses produce multiple images of a single source — and why lens modeling +works "forwards", ray-tracing image-plane grids to the source-plane rather than the other way around. + +In code, the lens equation is one line — the same `grid_2d_via_deflection_grid_from` call from tutorial 2: +""" +deflections = tracer.deflections_yx_2d_from(grid=grid) + +source_plane_grid = grid.grid_2d_via_deflection_grid_from(deflection_grid=deflections) + +aplt.plot_grid(grid=grid, title="Image-Plane Grid (theta)") +aplt.plot_grid(grid=source_plane_grid, title="Source-Plane Grid (beta)") + +""" +__Convergence__ + +The lens galaxy's mass enters lensing via its **surface mass density** Sigma(theta): the galaxy's 3D mass density +integrated along the line of sight, giving a projected 2D mass per unit area (e.g. in solar masses per kpc^2). +Lensing only cares about this projection — two very different 3D distributions with the same projected Sigma +deflect light identically. + +Whether a lens is "strong" is set by comparing Sigma to the **critical surface density**: + + Sigma_cr = (c^2 / (4 pi G)) * (D_s / (D_l * D_ls)) + +This is a purely cosmological quantity — it depends only on the constants c and G and the three distances from the +previous section. It has a beautiful interpretation: it is the surface density a lens needs, given this observer / +lens / source geometry, to be capable of producing multiple images. +""" +sigma_cr_kpc = cosmology.critical_surface_density_between_redshifts_solar_mass_per_kpc2_from( + redshift_0=0.5, redshift_1=1.0 +) +sigma_cr_arcsec = cosmology.critical_surface_density_between_redshifts_from( + redshift_0=0.5, redshift_1=1.0 +) + +print(f"Sigma_cr (solar masses per kpc^2) = {sigma_cr_kpc:.3e}") +print(f"Sigma_cr (solar masses per arcsec^2) = {sigma_cr_arcsec:.3e}") + +""" +The **convergence** kappa is the surface mass density in units of the critical density: + + kappa(theta) = Sigma(theta) / Sigma_cr + +This is the dimensionless quantity **PyAutoLens** has been plotting since tutorial 2 whenever we called +`convergence_2d_from`: + +- kappa >= 1 — the lens is super-critical at that point; multiple imaging is possible. +- kappa < 1 — the lens is sub-critical there. + +Because kappa is dimensionless, lens models can be composed and fitted without knowing the galaxy redshifts at all — +the redshifts and Sigma_cr are only needed at the end, to convert the inferred kappa back into physical masses. +""" +convergence = tracer.convergence_2d_from(grid=grid) + +aplt.plot_array(array=convergence, title="Convergence (kappa)", use_log10=True) + +""" +__Deflection Angles__ + +The convergence field determines the deflection angles everywhere, via a 2D integral over the whole image-plane: + + alpha(theta) = (1 / pi) * integral d^2 theta' kappa(theta') * (theta - theta') / |theta - theta'|^2 + +In plain words: every patch of mass in the lens plane pulls on every light ray. The deflection at position theta is +the sum (integral) of the pulls from all mass elements kappa(theta'), each pointing from the mass element towards +the ray and falling off as 1/distance — the 2D (projected) analogue of Newtonian gravity's inverse-square law. + +Two things follow from this integral being over *all* theta': + +- The deflection at a point depends on the mass distribution *everywhere*, not just the mass at that point. Even + mass well outside the region where images form contributes deflections (this is why external shear from + neighbouring galaxies matters, as we saw in tutorial 3). + +- For simple analytic profiles (isothermal, power-law, Sersic-like) this integral has closed-form solutions, which + is exactly what a **PyAutoLens** mass profile is: an analytic kappa(theta) paired with its analytic alpha(theta). + +The `deflections_yx_2d_from` method we have used since tutorial 2 evaluates this integral's closed-form solution. +The deflections form a 2D vector field, so we plot its y and x components separately: +""" +deflections = tracer.deflections_yx_2d_from(grid=grid) + +deflections_y = aa.Array2D(values=deflections.slim[:, 0], mask=grid.mask) +aplt.plot_array(array=deflections_y, title="Deflection Angles (y component)") + +deflections_x = aa.Array2D(values=deflections.slim[:, 1], mask=grid.mask) +aplt.plot_array(array=deflections_x, title="Deflection Angles (x component)") + +""" +__The Lensing Potential__ + +The deflection field is not arbitrary — it is the gradient of a scalar field, the **lensing potential** psi(theta): + + alpha(theta) = grad psi(theta) + +The potential is itself an integral over the convergence: + + psi(theta) = (1 / pi) * integral d^2 theta' kappa(theta') * ln|theta - theta'| + +and taking the divergence of the gradient (the Laplacian) recovers the convergence with a factor of a half: + + kappa(theta) = (1/2) * laplacian psi(theta) + +This is the 2D Poisson equation of lensing — the projected analogue of Newtonian gravity's del^2 Phi = 4 pi G rho. +The potential is the single most economical description of a lens: one scalar field from which the deflections +(first derivatives), and the convergence, shear and magnification (second derivatives) all follow. + +The `potential_2d_from` method returns psi, which we have plotted before without defining it: +""" +potential = tracer.potential_2d_from(grid=grid) + +aplt.plot_array(array=potential, title="Lensing Potential (psi)") + +""" +We can verify alpha = grad psi numerically. Mass profiles have a `deflections_2d_via_potential_2d_from` method which +computes the deflections by numerically differentiating the potential, rather than using the profile's closed-form +deflection formula. Across the grid the two agree closely — we print the median absolute difference, since the +finite-difference derivative is inaccurate right at the isothermal profile's central cusp, where the potential is +not smooth (the closed-form deflections have no such problem): +""" +deflections_analytic = mass_profile.deflections_yx_2d_from(grid=grid) +deflections_via_potential = mass_profile.deflections_2d_via_potential_2d_from(grid=grid) + +difference = np.median( + np.abs(np.asarray(deflections_analytic) - np.asarray(deflections_via_potential)) +) +print(f"Median |alpha_analytic - grad psi| = {difference:.3e} arcsec") + +""" +__Shear and Magnification__ + +How a small image is distorted by lensing is governed by how the deflection angles *change* across it — the second +derivatives of the potential. These form the 2x2 **Hessian** matrix, which `LensCalc` computes by finite +differences of the deflection field (as we saw in the workspace guides, this works for any mass distribution): + + H_yy = d(alpha_y) / d(theta_y) H_xy = d(alpha_x) / d(theta_y) + H_yx = d(alpha_y) / d(theta_x) H_xx = d(alpha_x) / d(theta_x) +""" +hessian_yy, hessian_xy, hessian_yx, hessian_xx = lens_calc.hessian_from(grid=grid) + +print(f"Hessian components at pixel 0: H_yy = {hessian_yy[0]:.4f}, H_xx = {hessian_xx[0]:.4f}") + +""" +Differentiating the lens equation beta = theta - alpha(theta) gives the **Jacobian matrix** A, which maps a small +displacement in the source-plane to the corresponding displacement in the image-plane: + + A = d(beta) / d(theta) = I - H = | 1 - H_yy -H_xy | + | -H_yx 1 - H_xx | + +The Jacobian decomposes into two physically distinct distortions: + +- The **convergence** kappa = (1/2) * (H_yy + H_xx) — the isotropic part, which magnifies an image uniformly + without changing its shape. (Note this is the same kappa as before: the trace of the Hessian recovers the + Poisson equation kappa = (1/2) laplacian psi.) + +- The **shear** gamma — the anisotropic part, which stretches an image along one axis and squeezes it along the + perpendicular axis. It has two components and a magnitude: + + gamma_1 = (1/2) * (H_xx - H_yy) + gamma_2 = H_xy + |gamma| = sqrt(gamma_1^2 + gamma_2^2) + +The shear is why lensed images near the lens are stretched into tangential arcs — the tidal field of the lens +elongates them around it. +""" +shear = lens_calc.shear_yx_2d_via_hessian_from(grid=grid) + +print(f"Shear magnitude at pixel 0: |gamma| = {shear.magnitudes[0]:.4f}") + +""" +The **magnification** mu is the inverse of the Jacobian's determinant: + + mu = 1 / det(A) = 1 / [ (1 - kappa)^2 - |gamma|^2 ] + +Lensing conserves surface brightness, so a lensed image that covers more sky than the unlensed source appears +brighter in total by exactly the factor |mu|: + +- |mu| > 1 — the image is magnified (larger and brighter than the unlensed source). +- |mu| < 1 — the image is demagnified. +- mu < 0 — the image has negative parity: it is a mirror image of the source. + +The determinant factorises into two eigenvalues, giving the **tangential** and **radial** magnifications: + + lambda_t = 1 - kappa - |gamma| (tangential eigenvalue) + lambda_r = 1 - kappa + |gamma| (radial eigenvalue) + + mu = 1 / (lambda_t * lambda_r) + +An image is stretched by 1/lambda_t in the tangential direction (around the lens) and 1/lambda_r in the radial +direction (towards/away from the lens). Giant tangential arcs form where lambda_t is close to zero. +""" +magnification = lens_calc.magnification_2d_from(grid=grid) + +aplt.plot_array(array=magnification, title="Magnification (mu)") + +tangential_eigen_values = lens_calc.tangential_eigen_value_from(grid=grid) +radial_eigen_values = lens_calc.radial_eigen_value_from(grid=grid) + +print(f"Tangential eigenvalue at pixel 0: {tangential_eigen_values[0]:.4f}") +print(f"Radial eigenvalue at pixel 0: {radial_eigen_values[0]:.4f}") + +""" +__Critical Curves and Caustics__ + +Where either eigenvalue passes through zero, det(A) = 0 and the magnification formally diverges to infinity. The +closed curves in the image-plane where this happens are the **critical curves** — the white and yellow lines that +have appeared on plots since tutorial 2, and which we explored visually in tutorial 3: + +- The **tangential critical curve** (lambda_t = 0) — roughly traces the Einstein ring; sources near its source-plane + counterpart form giant tangential arcs. +- The **radial critical curve** (lambda_r = 0) — an inner curve associated with radially stretched central images. + +Ray-tracing each critical curve through the lens equation maps it to the source-plane, where it is called a +**caustic**: + + caustic = critical_curve - alpha(critical_curve) + +Caustics divide the source-plane into regions of different image multiplicity: each time a source crosses a +caustic, the number of images it produces changes by two. For our SIE lens, a source inside the tangential caustic +produces four images (plus a faint central image); between the tangential and radial caustics, two; and outside +both caustics, just one — the source is no longer multiply imaged at all. + +`LensCalc` computes both by locating the zero-contours of the eigenvalue fields on the grid: +""" +tangential_critical_curve_list = lens_calc.tangential_critical_curve_list_from(grid=grid) +radial_critical_curve_list = lens_calc.radial_critical_curve_list_from(grid=grid) + +tangential_caustic_list = lens_calc.tangential_caustic_list_from(grid=grid) +radial_caustic_list = lens_calc.radial_caustic_list_from(grid=grid) + +print(f"Number of tangential critical curves: {len(tangential_critical_curve_list)}") +print(f"Number of radial critical curves: {len(radial_critical_curve_list)}") + +""" +Lets plot the critical curves (image-plane) and caustics (source-plane) of our SIE side by side. Note how the +elliptical lens produces a tangential caustic with four cusps — the origin of the four-image "quad" configurations +seen in many real lenses. +""" +plt.figure(figsize=(10, 5)) + +plt.subplot(1, 2, 1) +for curve in tangential_critical_curve_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="black", label="Tangential") +for curve in radial_critical_curve_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="orange", label="Radial") +plt.gca().set_aspect("equal") +plt.title("Critical Curves (Image-Plane)") +plt.xlabel("x (arcsec)") +plt.ylabel("y (arcsec)") +plt.legend() + +plt.subplot(1, 2, 2) +for curve in tangential_caustic_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="black", label="Tangential") +for curve in radial_caustic_list: + curve = np.asarray(curve) + plt.plot(curve[:, 1], curve[:, 0], color="orange", label="Radial") +plt.gca().set_aspect("equal") +plt.title("Caustics (Source-Plane)") +plt.xlabel("x (arcsec)") +plt.ylabel("y (arcsec)") +plt.legend() + +plt.show() +plt.close() + +""" +__Einstein Radius__ + +The **Einstein radius** theta_E is the characteristic angular scale of a strong lens. For a circular lens with the +source perfectly aligned behind it, it is the radius of the Einstein ring the source forms. More generally (and +this is the definition **PyAutoLens** uses), it is the radius of the circle enclosing the same area as the +tangential critical curve: + + theta_E = sqrt(A_crit / pi) + +sometimes called the "effective" Einstein radius, since an elliptical lens's critical curve is not a circle. + +Two closed-form results are worth memorising. For a **point mass** M: + + theta_E = sqrt( (4 G M / c^2) * (D_ls / (D_l * D_s)) ) + +and for a **singular isothermal sphere** (SIS) with velocity dispersion sigma_v: + + theta_E = 4 pi * (sigma_v / c)^2 * (D_ls / D_s) + +The Einstein radius is also the radius within which the *mean* convergence equals exactly one — so measuring +theta_E directly measures the projected mass enclosed within it: + + M(< theta_E) = pi * theta_E^2 * Sigma_cr + +This is why the Einstein radius is celebrated as one of the most robust mass measurements in all of astrophysics: +the data pin down theta_E via the image separations almost independently of the assumed mass profile, so the +enclosed Einstein mass is trusted even when the profile's slope is not. (The main caveat, the mass-sheet +degeneracy, is discussed in the lens modeling chapters.) + +`LensCalc` computes the Einstein radius from the area of the tangential critical curve, and the enclosed +"Einstein mass" in angular units (pi * theta_E^2), which Sigma_cr converts to solar masses: +""" +einstein_radius = lens_calc.einstein_radius_from(grid=grid) + +print(f"Einstein radius (arcsec) = {einstein_radius:.4f}") +print(f"Einstein radius (kpc) = {einstein_radius * kpc_per_arcsec_lens:.4f}") + +einstein_mass_angular = lens_calc.einstein_mass_angular_from(grid=grid) +einstein_mass_solar = einstein_mass_angular * sigma_cr_arcsec + +print(f"Einstein mass (angular, arcsec^2) = {einstein_mass_angular:.4f}") +print(f"Einstein mass (solar masses) = {einstein_mass_solar:.4e}") + +""" +Reassuringly, the Einstein radius computed from the critical curve area (~1.6") matches the `einstein_radius=1.6` +parameter we gave the `Isothermal` profile — for isothermal profiles the model parameter *is* the effective +Einstein radius, which is exactly why **PyAutoLens** parameterizes its mass profiles this way: the non-linear +search then varies the quantity the data constrain most directly. + +As a second check, an `IsothermalSph` (the SIS) has a perfectly circular critical curve, so its recovered Einstein +radius equals its input parameter even more precisely: +""" +sis = al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=1.6) + +sis_einstein_radius = al.LensCalc.from_mass_obj(mass_obj=sis).einstein_radius_from( + grid=grid +) + +print(f"SIS input Einstein radius = 1.6, recovered = {sis_einstein_radius:.4f}") + +""" +__Time Delays__ + +The final piece of the formalism is time. Light rays forming different images of the same source travel different +paths and through different depths of the lens's gravitational potential, so they arrive at different times. Both +effects are captured by the **Fermat potential** (or time-delay surface): + + phi(theta) = (1/2) * |theta - beta|^2 - psi(theta) + +The first term is the **geometric delay** — the extra path length of a bent ray. The second is the **gravitational +(Shapiro) delay** — light slowing as it climbs through the lens's potential. Fermat's principle states that images +form at the stationary points (minima, maxima and saddle points) of this surface, which is a wonderfully compact +restatement of the lens equation: grad phi = 0 is exactly beta = theta - alpha(theta). + +The observable **time delay** between two images A and B is the difference in their Fermat potentials, scaled by +the cosmological distances: + + Delta t_AB = (1 + z_l) / c * (D_l * D_s / D_ls) * [ phi(theta_A) - phi(theta_B) ] + +The distance combination (1 + z_l) * D_l * D_s / D_ls is called the **time-delay distance**, and because it is +inversely proportional to the Hubble constant, measuring time delays between the images of a variable source (a +quasar or supernova) turns a strong lens into a cosmological probe. + +We only touch on this here — time delays belong to the modeling of lensed point sources, introduced in tutorial 4 +and covered in depth by the `autolens_workspace` point-source material. For now, we simply plot the Fermat +potential of our lens system: +""" +fermat_potential = lens_calc.fermat_potential_from(grid=grid) + +aplt.plot_array(array=fermat_potential, title="Fermat Potential (phi)") + """ __Wrap Up__ + +This was the mathematics lecture of **HowToLens** — every formal definition deferred by the other tutorials, in one +place. Lets summarise the chain of quantities, because it has a beautiful logical structure where everything flows +from the mass distribution and a handful of distances: + +- **Distances**: the angular diameter distances D_l, D_s and D_ls set the geometry, convert arcseconds to + kiloparsecs and define the critical surface density Sigma_cr. + +- **Convergence**: kappa = Sigma / Sigma_cr is the dimensionless projected mass; kappa >= 1 marks super-critical + regions capable of multiple imaging. + +- **Deflections and potential**: kappa determines the deflection field alpha (a 2D gravity integral) and the + lensing potential psi, tied together by alpha = grad psi and kappa = (1/2) laplacian psi. + +- **The lens equation**: beta = theta - alpha(theta) maps the image-plane to the source-plane; its + non-invertibility is why multiple images form. + +- **Distortion**: the Jacobian A = I - H decomposes into convergence (isotropic) and shear gamma (anisotropic); + magnification is mu = 1 / det(A) = 1 / (lambda_t * lambda_r). + +- **Critical curves and caustics**: where det(A) = 0, magnification diverges; caustics are their source-plane + images and set the image multiplicity. + +- **Einstein radius**: the area-equivalent radius of the tangential critical curve, whose enclosed mass + M(< theta_E) = pi * theta_E^2 * Sigma_cr is the most robust measurement strong lensing delivers. + +- **Time delays**: the Fermat potential phi = (1/2)|theta - beta|^2 - psi locates images at its stationary points + and its differences, scaled by the time-delay distance, give observable delays. + +You do not need to memorise any of this to continue — **PyAutoLens** computes every one of these quantities via the +methods used above — but you now know what each method is computing and can return to this tutorial whenever a +symbol needs unpacking. + +In the next tutorial, we turn from theory to observation: how telescope optics, exposure times and noise turn the +pristine images of a tracer into the CCD imaging data we actually observe, and how to simulate such data ourselves. + +Finally, a signpost for much later: chapter 3's tutorial 5 on the Bayesian formalism is this tutorial's twin — the +equivalent equations lecture for pixelized source reconstruction, deriving the linear inversion and Bayesian +evidence that underpin chapter 3 just as this tutorial derived the lensing quantities underpinning chapter 1. """ diff --git a/scripts/chapter_1_introduction/tutorial_6_data.py b/scripts/chapter_1_introduction/tutorial_6_data.py index 68f2f99..ec6ea19 100644 --- a/scripts/chapter_1_introduction/tutorial_6_data.py +++ b/scripts/chapter_1_introduction/tutorial_6_data.py @@ -13,8 +13,8 @@ **Exposure Time:** The time the detector collects light, affecting the clarity of the image. Longer exposure times gather more light, improving the signal-to-noise ratio and creating a clearer image. -**Background Sky:** Light from the sky itself, such as distant stars or zodiacal light, adds noise to the image. -adds additional noise to the image. +**Background Sky:** Light from the sky itself, such as distant stars or zodiacal light, adds additional noise to +the image. In this tutorial, we'll simulate a strong lens image by applying these real-world effects to the light and mass profiles and images we created earlier. @@ -25,6 +25,7 @@ - **Poisson Noise:** We'll add Poisson noise to the image, simulating the randomness in the photon-to-electron conversion process on the CCD. - **Background Sky:** We'll add a background sky to the image, simulating the light from the sky that adds noise to the image. - **Simulator:** We'll use the `SimulatorImaging` object to simulate imaging data that includes all these effects. +- **Other Data Types:** We'll finish with a brief look at interferometer and weak lensing data, two other types of data used to study strong lenses. __Contents__ @@ -34,6 +35,8 @@ - **Background Sky:** The final effect we will consider when simulating imaging data is the background sky. - **Simulator:** The `SimulatorImaging` object lets us create simulated imaging data while including the effects of. - **Output:** We will now save these simulated data to `.fits` files, the standard format used by astronomers for. +- **Interferometer Data:** Radio / sub-mm interferometers like ALMA observe visibilities in the uv-plane rather than images. +- **Weak Lensing Data:** Weak lensing datasets are catalogues of weakly sheared background galaxy ellipticities. - **Wrap Up:** Summary of the script and next steps. """ @@ -58,7 +61,7 @@ shape_native=( 101, 101, - ), # The dimensions of the grid, which here is 100 x 100 pixels. + ), # The dimensions of the grid, which here is 101 x 101 pixels. pixel_scales=0.1, # The conversion factor between pixel units and arc-seconds. ) @@ -128,7 +131,7 @@ in its tail, which are much smaller than the central peak yet critical for many scientific analyses. The tail values may significantly affect the spread and detail captured in the data. """ -aplt.plot_array(array=psf.kernel, title="PSF 2D Kernel") +aplt.plot_array(array=psf.kernel, title="PSF 2D Kernel (Log10)", use_log10=True) """ Next, we'll manually perform a 2D convolution of the PSF with the image of the galaxy. This convolution simulates the @@ -162,7 +165,7 @@ """ aplt.plot_array(array=image, title="Tracer Image Before PSF") -aplt.plot_array(array=blurred_image, title="") +aplt.plot_array(array=blurred_image, title="Tracer Image After PSF") """ @@ -189,8 +192,8 @@ We also need to consider the units of our image data. Let’s assume that the tracer image is measured in units of electrons per second ($e^- s^{-1}$), which is standard for CCD imaging data. To simulate the number of electrons -actually detected in each pixel, we multiply the image by the observation’s exposure time. This conversion changes t -he units to the total number of electrons collected per pixel over the entire exposure time. +actually detected in each pixel, we multiply the image by the observation’s exposure time. This conversion changes +the units to the total number of electrons collected per pixel over the entire exposure time. Once the image is converted, we add Poisson noise, simulating the randomness in the photon-to-electron conversion process. After adding the noise, we convert the image back to units of electrons per second for analysis, as @@ -235,7 +238,7 @@ In addition to light from the strong lens, the telescope also picks up light from the sky. This background sky light is primarily due to two sources: zodiacal light, which is light scattered by interplanetary dust in the solar system, -and the unresolved emission from distant stars and tracer. +and the unresolved emission from distant stars and galaxies. For our simulation, we'll assume that the background sky has a uniform brightness across the image, measured at 0.1 electrons per second per pixel. The background sky is added to the image before applying the PSF convolution @@ -356,9 +359,8 @@ We will now save these simulated data to `.fits` files, the standard format used by astronomers for storing images. Most imaging data from telescopes like the Hubble Space Telescope (HST) are stored in this format. -The `dataset_path` specifies where the data will be saved, in this case, in the directory -`autolens_workspace/dataset/imaging/howtolens/`, which contains many example images distributed with -the `autolens_workspace`. +The `dataset_path` specifies where the data will be saved, in this case the `dataset/imaging/howtolens/` +directory of the HowToLens repository (paths are relative to the repository root, from which scripts are run). The files are named `data.fits`, `noise_map.fits`, and `psf.fits`, and will be used in the next tutorial. """ @@ -373,6 +375,93 @@ overwrite=True, ) +""" +__Interferometer Data__ + +CCD imaging is not the only type of data used to study strong lenses. Radio and sub-mm interferometers, like the +Atacama Large Millimeter Array (ALMA) and the Jansky Very Large Array (JVLA), observe strong lenses at wavelengths +where a CCD cannot. + +An interferometer does not observe an image of the lens. Each pair of antennas in the array measures a "visibility", +a Fourier component of the sky brightness, at a point in what is called the "uv-plane" set by the separation of the +two antennas. The dataset is therefore a set of complex visibilities in Fourier space, not a 2D image, and its +noise properties are very different from those of CCD data — there is no PSF convolution, Poisson noise or +background sky; instead each visibility has Gaussian noise. + +One could Fourier transform the visibilities into an image (called a "dirty image") and fit that, but the transform +correlates the noise between pixels, making the fit statistically incorrect. **PyAutoLens** therefore fits lens +models directly in visibility space: the tracer's image is evaluated in real space on a grid (defined by a +real-space mask), Fourier transformed to the uv-plane and compared with the observed visibilities there. + +Below, we load a simulated interferometer dataset (creating it first via the `scripts/simulator/interferometer.py` +script if it does not exist on your hard-disk, using the same auto-simulation idiom as later tutorials) and plot +its dirty images — the closest an interferometer dataset comes to the CCD images we simulated above. + +HowToLens will not cover interferometry any further than this. The lecture series teaches lensing using CCD +imaging, and everything you learn transfers to visibility-space fitting. If you need to model interferometer data, +go to the `autolens_workspace/scripts/interferometer` package, which is the dedicated resource for uv-plane lens +modeling. +""" +dataset_path = Path("dataset") / "interferometer" / "simple" + +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/interferometer.py"], + check=True, + ) + +real_space_mask = al.Mask2D.circular( + shape_native=(100, 100), + pixel_scales=0.1, + radius=3.0, +) + +dataset = al.Interferometer.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + uv_wavelengths_path=dataset_path / "uv_wavelengths.fits", + real_space_mask=real_space_mask, + transformer_class=al.TransformerDFT, +) + +aplt.subplot_interferometer_dirty_images(dataset=dataset) + +""" +__Weak Lensing Data__ + +There is one more type of lensing data to glimpse before we move on: weak lensing. + +A weak lensing dataset looks nothing like the images above. Instead of the spectacular arcs and rings of strong +lensing, it is a *catalogue* of many background galaxies, each with a measured ellipticity — a shear — at its +(y, x) position on the sky. Each galaxy is only weakly sheared by the foreground mass, a distortion far too small +to see in any single galaxy, and the plot below shows this: a field of short line segments tracing the subtle, +coherent stretching of the background galaxy population, rather than any arc-like feature. + +We load a simulated weak lensing shear catalogue (again auto-simulating it, via `scripts/simulator/weak_lensing.py`, +if it is not on your hard-disk) and plot it, just so you have seen what this data looks like. + +That is deliberately all we will say for now — this glimpse is foreshadowing. Weak lensing gets a full treatment in +the final tutorial of chapter 4, which describes what these shear measurements are, why they are made far from the +lens centre and how they are fitted; until then, we defer all further description to that tutorial. +""" +dataset_path = Path("dataset") / "weak_lensing" / "simple" + +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/weak_lensing.py"], + check=True, + ) + +dataset = al.from_json(file_path=dataset_path / "dataset.json") + +aplt.subplot_weak_dataset(dataset=dataset) + """ __Wrap Up__ @@ -381,7 +470,7 @@ Let's summarise what we've covered: -- **Optics Blurring**: The optics of a telescope blur the light from tracer, reducing the clarity and sharpness of +- **Optics Blurring**: The optics of a telescope blur the light of the tracer, reducing the clarity and sharpness of the images. - **Poisson Noise**: The process of converting photons to electrons on a CCD introduces Poisson noise, which is random diff --git a/scripts/chapter_1_introduction/tutorial_7_fitting.py b/scripts/chapter_1_introduction/tutorial_7_fitting.py index 406c272..58e0239 100644 --- a/scripts/chapter_1_introduction/tutorial_7_fitting.py +++ b/scripts/chapter_1_introduction/tutorial_7_fitting.py @@ -2,7 +2,7 @@ Tutorial 7: Fitting =================== -In previous tutorials, we used light profiles to create simulated images of tracer and visualized how these images +In previous tutorials, we used light profiles to create simulated images of a tracer and visualized how these images would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope. However, this simulation process is the reverse of what astronomers typically do when analyzing real data. Usually, @@ -33,7 +33,9 @@ __Contents__ -- **Dataset & Mask:** Standard set up of the dataset and mask that is fitted. +- **Dataset:** Load the imaging dataset that we previously simulated, consisting of the image, noise map, and PSF. +- **Dataset Auto-Simulation:** Create the dataset by running the tutorial 6 script if it is not on your hard-disk. +- **Mask:** Apply a mask to the data, excluding regions with low signal-to-noise ratios from the analysis. - **Masked Grid:** In tutorials 1 and 2, we emphasized that the `Grid2D` object is crucial for evaluating a lens's. - **Fitting:** Fit the lens model to the dataset and inspect the results. - **Incorrect Fit:** In the previous section, we successfully created and fitted a lens model to the image data. @@ -42,6 +44,8 @@ """ +from autolens import jax_wrapper # Sets JAX environment before other imports + # from autolens import setup_notebook; setup_notebook() import numpy as np @@ -55,11 +59,11 @@ We begin by loading the imaging dataset that we will use for fitting in this tutorial. This dataset is identical to the one we simulated in the previous tutorial, representing how a lens would appear if captured by a CCD camera. -In the previous tutorial, we saved this dataset as .fits files in the `autolens_workspace/dataset/imaging/howtolens` -folder. The `.fits` format is commonly used in astronomy for storing image data along with metadata, making it a -standard for CCD imaging. +In the previous tutorial, we saved this dataset as .fits files in the `dataset/imaging/howtolens` folder of the +HowToLens repository. The `.fits` format is commonly used in astronomy for storing image data along with metadata, +making it a standard for CCD imaging. -The `dataset_path` below specifies where these files are located: `autolens_workspace/dataset/imaging/howtolens/`. +The `dataset_path` below specifies where these files are located: `dataset/imaging/howtolens/`. """ dataset_path = Path("dataset") / "imaging" / "howtolens" @@ -112,7 +116,7 @@ """ __Mask__ -The signal-to-noise map of the image highlights areas where the signal (light from the lens and source tracer) +The signal-to-noise map of the image highlights areas where the signal (light from the lens and source galaxies) is detected above the background noise. Values above 3.0 indicate regions where the light is detected with a signal-to-noise ratio of at least 3, while values below 3.0 are dominated by noise, where the light is not clearly distinguishable. @@ -427,7 +431,7 @@ """ Another quantity that contributes to our final assessment of the goodness-of-fit is the `noise_normalization`. -The `noise_normalization` is computed as the logarithm of the sum of squared noise values in our data: +The `noise_normalization` is computed by summing, over every pixel, the logarithm of 2 pi times the squared noise value: \[ \text{{noise\_normalization}} = \sum \log(2 \pi \text{{noise\_map}}^2) @@ -501,7 +505,7 @@ These metrics are standard ways to quantify the quality of a model fit. They are applicable not only to 1D data but also to more complex data structures like 2D images, 3D data cubes, or any other multidimensional datasets. -__Incorrect Fit___ +__Incorrect Fit__ In the previous section, we successfully created and fitted a lens model to the image data, resulting in an excellent fit. The residual map and chi-squared map showed no significant discrepancies, indicating that the @@ -586,7 +590,7 @@ aplt.subplot_fit_imaging(fit=fit_very_bad) """ -It is now evident that this model provides a terrible fit to the data. The tracer do not resemble a plausible +It is now evident that this model provides a terrible fit to the data. The tracer does not resemble a plausible representation of our simulated strong lens dataset, which we already anticipated given that we generated the data ourselves! As expected, the log likelihood has dropped dramatically with this poorly fitting model. @@ -606,8 +610,8 @@ In practice, however, we don't know the "true" model. For example, we might have an image of a strong lens observed with the Hubble Space Telescope, but the values for parameters like its `einstein_radius` and others are -unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of -Chapter 2 of *HowToGalaxy*. +unknown. The process of determining the best-fit model is called model fitting, and it is the main topic of +Chapter 2 of **HowToLens**. To conclude this section, let's perform a basic, hands-on model fit to develop some intuition about how we can find the best-fit model. We'll start by loading a simple dataset that was simulated without any lens light, using diff --git a/scripts/chapter_1_introduction/tutorial_8_summary.py b/scripts/chapter_1_introduction/tutorial_8_summary.py index 4d50143..ec1cd23 100644 --- a/scripts/chapter_1_introduction/tutorial_8_summary.py +++ b/scripts/chapter_1_introduction/tutorial_8_summary.py @@ -1,5 +1,5 @@ """ -Tutorial 9: Summary +Tutorial 8: Summary =================== In this chapter, we have learnt that: diff --git a/scripts/chapter_2_lens_modeling/README.md b/scripts/chapter_2_lens_modeling/README.md index 8d51818..16f952f 100644 --- a/scripts/chapter_2_lens_modeling/README.md +++ b/scripts/chapter_2_lens_modeling/README.md @@ -1,4 +1,5 @@ -In chapter 2, we'll take you through how to model strong lenses using a non-linear search. +In chapter 2, we'll take you through how to model strong lenses using a non-linear search, ending with how +searches are chained together into automated lens modeling pipelines. **Colab** links to every tutorial are included. @@ -19,3 +20,9 @@ In chapter 2, we'll take you through how to model strong lenses using a non-line - [Tutorial 7: Results](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_7_results.ipynb) — Overview of the results available after successfully fitting a lens model. - [Tutorial 8: Need for Speed](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb) — How to fit complex models whilst balancing efficiency and run-time. + +- [Tutorial 9: Search Chaining](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_9_search_chaining.ipynb) — Breaking the lens modeling procedure into a chained sequence of model-fits. + +- [Tutorial 10: Prior Passing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_10_prior_passing.ipynb) — How the results of earlier searches are passed to later searches. + +- [Tutorial 11: SLaM](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_2_lens_modeling/tutorial_11_slam.ipynb) — Template pipelines for fitting lens models in standardized ways. diff --git a/scripts/chapter_3_search_chaining/tutorial_2_prior_passing.py b/scripts/chapter_2_lens_modeling/tutorial_10_prior_passing.py similarity index 86% rename from scripts/chapter_3_search_chaining/tutorial_2_prior_passing.py rename to scripts/chapter_2_lens_modeling/tutorial_10_prior_passing.py index 0fa346d..1090356 100644 --- a/scripts/chapter_3_search_chaining/tutorial_2_prior_passing.py +++ b/scripts/chapter_2_lens_modeling/tutorial_10_prior_passing.py @@ -1,6 +1,6 @@ """ -Tutorial 2: Prior Passing -========================= +Tutorial 10: Prior Passing +========================== In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two non-linear searches. This used an initial search to fit a simple lens model, whose results were used to tune and @@ -12,13 +12,15 @@ __Contents__ -- **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where. +- **Initial Setup:** Load the same strong lensing data as the previous tutorial. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Model:** Compose the lens model fitted to the data. - **Search:** Configure the non-linear search used to fit the model. -- **Prior Passing:** We are now going to use the prior passing API to pass these results, in a way which does not. +- **Result (Search 1):** The results of search 1, which are used for prior passing. +- **Prior Passing:** Use the prior passing API to pass the results of search 1 without writing out values manually. - **Result:** Overview of the results of the model-fit. - **Wrap Up:** Summary of the script and next steps. -- **Detailed Explanation Of Prior Passing:** To end, I provide a detailed overview of how prior passing works and illustrate tools that can be. +- **Detailed Explanation Of Prior Passing:** A detailed overview of how prior passing works and tools to customize it. - **EXAMPLE:** Lets go through an example using a real parameter. """ @@ -119,11 +121,11 @@ run it. """ search_1 = af.Nautilus( - path_prefix=Path("howtolens", "chapter_3"), - name="tutorial_1_search_chaining_1", + path_prefix=Path("howtolens", "chapter_2"), + name="tutorial_9_search_chaining_1", unique_tag=dataset_name, n_live=100, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis_1 = al.AnalysisImaging(dataset=dataset) @@ -178,8 +180,8 @@ We can use a special prior passing method to do this, called `take_attributes`. This scans the `Exponential` passed to the `take_attributes` method for all parameters which have the same name as the `Sersic` model, -and if their names are the same it passes their prior as a `model` (like we did above). Thus, it will locate all 6 -parameters in common between the two profiles (centre_, ell_comps, intensity, effective_radius) and pass those, +and if their names are the same it passes their prior as a `model` (like we did above). Thus, it will locate the +parameters in common between the two profiles (`centre`, `ell_comps`, `effective_radius`) and pass those, leaving the `sersic_index`'s priors as the default values. The `take_attributes` method is used in many examples of prior passing, when we pass a simpler parameterization of a @@ -210,17 +212,17 @@ that were passed. """ search_2 = af.Nautilus( - path_prefix=Path("howtolens", "chapter_3"), - name="tutorial_2_search_chaining_2", + path_prefix=Path("howtolens", "chapter_2"), + name="tutorial_10_prior_passing_2", unique_tag=dataset_name, n_live=100, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis_2 = al.AnalysisImaging(dataset=dataset) print( - "The non-linear search has begun running - checkout the workspace/output/5_chaining_searches" + "The non-linear search has begun running - checkout the output/howtolens/chapter_2" " folder for live output of the results, images and lens model." " This Jupyter notebook cell with progress once search has completed - this could take some time!" ) @@ -244,18 +246,19 @@ """ __Wrap Up__ -We will expand on the prior passing API in the following tutorials. The main thing to note is that we can pass -entire profiles or galaxies using prior passing, if their model does not change (which for the bulge, mass and -source_bulge above, was not true). The API to pass a whole profile or galaxy is as follows: - +This tutorial has covered the core of the prior passing API; the chaining examples in the `autolens_workspace` +(e.g. `autolens_workspace/scripts/guides/modeling/chaining.py`) expand on it further. The main thing to note is that +we can pass entire profiles or galaxies using prior passing, if their model does not change (which for the bulge, +mass and source_bulge above, was not true). The API to pass a whole profile or galaxy is as follows: + bulge = result_1.model.galaxies.lens.bulge lens = result_1.model.galaxies.lens source = result_1.model.galaxies.source - + We can also pass priors using an `instance` instead of a `model`. When an `instance` is used, the maximum likelihood -parameter values are passed as fixed values that are therefore not fitted for nby the non-linear search (reducing its -dimensionality). We will use this in the next tutorial to fit the lens light, fix it to the best-fit model in a second -search, and then go on to fit it as a model in the final search. +parameter values are passed as fixed values that are therefore not fitted for by the non-linear search (reducing its +dimensionality). Chained fits often use this to fix the lens light to the best-fit model of an early search, reducing +the dimensionality of the searches that follow — the workspace's chaining examples show this in action. Lets now think about how priors are passed. Checkout the `model.info` file of the second search of this tutorial. The parameters do not use the default priors we saw in search 1 (which are typically broad UniformPriors). Instead, @@ -265,7 +268,7 @@ - The sigma values are specified in the `width_modifier` field of the profile's entry in the `priors.yaml' config file (we will discuss why this is used in a moment). -Like the manual `GaussianPrior`'s that were used in tutorial 1, the prior passing API sets up the prior on each +Like the manual `GaussianPrior`'s that were used in the previous tutorial, the prior passing API sets up the prior on each parameter with a `GaussianPrior` centred on the high likelihood regions of parameter space! __Detailed Explanation Of Prior Passing__ diff --git a/scripts/chapter_2_lens_modeling/tutorial_11_slam.py b/scripts/chapter_2_lens_modeling/tutorial_11_slam.py new file mode 100755 index 0000000..c8c3a1f --- /dev/null +++ b/scripts/chapter_2_lens_modeling/tutorial_11_slam.py @@ -0,0 +1,67 @@ +""" +Tutorial 11: SLaM +================= + +In the previous two tutorials, we learnt how search chaining breaks a lens model-fit into a sequence of simpler +non-linear searches, and how prior passing carries the results of each search into the next. Together, they give us +the flexibility to juggle the dimensionality, priors and settings of each search — the three drivers of run-time we +met in tutorial 8 — whilst still fitting a complex and realistic lens model at the end. + +You could now write your own chained sequences of searches, tailored to whatever lens model is of particular interest +to your scientific study. However, for most lens models there are standardized approaches one can take to fitting +them. For example, an effective approach is to first fit a model for the lens's light, then a model for its mass and +the source. It would be wasteful for all **PyAutoLens** users to have to write their own chained pipelines to perform +the same tasks. + +For this reason, the `autolens_workspace` comes with standardized chained pipelines, which fit common lens models in +ways we have tested are efficient and robust. This closing tutorial does not run any code; it is a pointer to where +that material lives, so you know where to go once you want to scale your lens modeling up. + +__Contents__ + +- **Search Chaining In The Workspace:** Where the workspace documents the chaining API introduced in tutorials 9 and 10. +- **SLaM (Source, Light and Mass):** The standardized pipelines that chain searches for you. +- **Wrap Up:** Choosing between single searches, your own chained fits and the SLaM pipelines. + +__Search Chaining In The Workspace__ + +The chaining and prior passing API we introduced in tutorials 9 and 10 is documented in full in the workspace guide +`autolens_workspace/scripts/guides/modeling/chaining.py`. It covers the concise model composition API used in chained +fits, prior passing and the tools for customizing it, and how chaining is used in practice for automated lens +modeling. + +__SLaM (Source, Light and Mass)__ + +The workspace's standardized pipelines are called the **SLaM** (Source, Light and Mass) pipelines. They are composed +of a specific sequence of chained searches: + + - `Source`: A pipeline that focuses on producing a robust model for the source's light, using simpler models for the + lens's light and mass. + + - `Light`: A pipeline that fits a complex lens light model (e.g. one with many components), using the initialized + source model to cleanly deblend the lens and source light. + + - `Mass`: A pipeline that fits a complex lens mass model, benefitting from the good models for the lens's light and + source. + +For fitting very complex lens models, for example ones which decompose the mass into its stellar and dark components, +the **SLaM** pipelines have been carefully crafted to do this in a reliable and automated way that is still efficient. +They also make fitting many different models to a single dataset efficient, as they reuse the results of earlier +searches (e.g. in the Source pipeline) to fit different models in the `Light` and `Mass` pipelines for the lens's +light and mass. + +The canonical introduction is `autolens_workspace/scripts/guides/modeling/slam_start_here.py`, which walks through the +pipeline structure end-to-end. Runnable SLaM examples then live as `slam.py` scripts inside the feature folders of +each topic, for example `autolens_workspace/scripts/imaging/features/pixelization/slam.py` and +`autolens_workspace/scripts/imaging/features/multi_gaussian_expansion/slam.py`. + +__Wrap Up__ + +Whether you should use individual searches, your own chained fits, the SLaM pipelines or write your own model-fitting +script depends on the scope of your scientific analysis. I would advise you begin by trying to adapt the scripts in +the `autolens_workspace` to fit your data, and also try using the SLaM pipelines once you are a confident +**PyAutoLens** user. + +This tutorial concludes the lens modeling chapter. In the next chapter, we introduce pixelizations, which reconstruct +the source galaxy on a pixel-grid rather than with light profiles. +""" diff --git a/scripts/chapter_2_lens_modeling/tutorial_9_summary b/scripts/chapter_2_lens_modeling/tutorial_12_summary similarity index 78% rename from scripts/chapter_2_lens_modeling/tutorial_9_summary rename to scripts/chapter_2_lens_modeling/tutorial_12_summary index be22534..7ba51a1 100755 --- a/scripts/chapter_2_lens_modeling/tutorial_9_summary +++ b/scripts/chapter_2_lens_modeling/tutorial_12_summary @@ -8,11 +8,13 @@ In this chapter, you learnt: 4) How one must carefully balance complexity and realism when fitting a model to a data-set. 5) The importance of factoring in run-speed when modeling a lens, and tricks to speed up the analysis. 6) About masking data and using positions in the analysis. +7) How to chain searches together into automated lens modeling pipelines, passing priors from one search + to the next, and how the SLaM pipelines standardize this. At this point, you are ready to begin modeling lenses. The 'modeling' folders in the autolens_workspace contains a number of scripts that can be easily adopted to model lens using a variety of different approaches, lens models and non-linear searches. If you have your own lens data, I'd recommend you adapt these scripts to your data. If you don't have your own data, checkout the 'simulators' folder to simulate your own dataset! -Alternatively, you may wish to continue on to chapter 3 on search chaining. +Alternatively, you may wish to continue on to chapter 3 on pixelized source reconstructions. diff --git a/scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py b/scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py index 9f28674..a94c841 100644 --- a/scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py +++ b/scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py @@ -4,7 +4,7 @@ The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a strong lens using a telescope like the Hubble Space Telescope, and seek to learn about the lens galaxy, source galaxy and the -Universe from these observations. With **PyAutoLen**, we seek to learn about the lenses may and ray-tracing, asking questions like +Universe from these observations. With **PyAutoLens**, we seek to learn about the lens's mass and ray-tracing, asking questions like how big is the lens galaxy and what does the unlensed source galaxy look like? To answer these questions, we must therefore fit the dataset with a lens model, where the lens model defines the @@ -27,7 +27,7 @@ and begin learning about real galaxies in the Universe. This first tutorial introduces a number of key statistical concepts that are fundamental to understanding how -model-fitting works, both for **PyAutoLen** and in general. +model-fitting works, both for **PyAutoLens** and in general. __Overview__ @@ -40,22 +40,27 @@ model instance to the data. - Fit datasets with different non-linear searches, including a maximum likelihood estimator (MLE), - Markok Chain Monte Carlo (MCMC) and nested sampling. + Markov Chain Monte Carlo (MCMC) and nested sampling. __Contents__ - **Overview:** In this tutorial, we will use a non-linear search to fit a lens model to simulated imaging of. - **Parameter Space:** In mathematics, a function is defined by its parameters, which map inputs to outputs. +- **Non-Linear Search:** The method used to fit the model to the data, by sampling the non-linear parameter space. - **Search Types:** There are different types of non-linear searches, each of which explores parameter space in a. -- **Deeper Background:** **The descriptions of how searches work in this example are simplfied and phoenomenological and do. +- **Deeper Background:** **The descriptions of how searches work in this example are simplified and phenomenological and do. - **PyAutoFit:** Modeling uses the probabilistic programming language. - **Initial Setup:** Let's first load the `Imaging` dataset, which we will use to fit a model with a non-linear search. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Mask:** Define the 2D mask applied to the dataset for the model-fit. - **Model:** Compose the lens model fitted to the data. - **Priors:** When we examine the `.info` of our model, we notice that each parameter (like `centre`. - **Analysis:** Create the Analysis object that defines how the model is fitted to the data. - **Searches:** To perform a non-linear search, we create an instance of a `NonLinearSearch` object. +- **Maximum Likelihood Estimation (MLE):** The simplest search type, which moves "up" the likelihood gradient. +- **Markov Chain Monte Carlo (MCMC):** Walkers explore parameter space, mapping out the likelihood surface. - **Nested Sampling:** **Nested Sampling** is an advanced method for model-fitting that excels in handling complex models. +- **What is The Best Search To Use?:** Choosing between MLE, MCMC and nested sampling for lens modeling. - **Wrap Up:** Summary of the script and next steps. __Parameter Space__ @@ -78,8 +83,8 @@ This defines a parameter space in three dimensions, representing the relationships between \(x\), \(y\), \(z\), and the output \(f(x, y, z)\). -This concept of parameter space is closely related to how we approach model-fitting. For instance, in chapter 1, w -e created instances of `Galaxy` objects with +This concept of parameter space is closely related to how we approach model-fitting. For instance, in chapter 1, +we created instances of `Galaxy` objects with parameters like \( (\text{`centre_0`}, \text{`centre_1`}, \text{`ell_comps_0`}, \text{`ell_comps_1`}, \text{`intensity`}, \text{`effective_radius`}, \text{`sersic_index`}) \). These parameters were used to fit data and compute a log likelihood. @@ -144,7 +149,7 @@ __Deeper Background__ -**The descriptions of how searches work in this example are simplfied and phoenomenological and do not give a full +**The descriptions of how searches work in this example are simplified and phenomenological and do not give a full description of how they work at a deep statistical level. The goal is to provide you with an intuition for how to use them and when different searches are appropriate for different problems. Later tutorials will provide a more formal description of how these searches work.** @@ -172,11 +177,11 @@ [PyAutoFit](https://github.com/PyAutoLabs/PyAutoFit), an open-source project that allows complex model fitting techniques to be straightforwardly integrated into scientific modeling software. -**PyAutoFit** is actually a spin-off project of **PyAutoLen**. whereby we found that the statistic techniques and +**PyAutoFit** is actually a spin-off project of **PyAutoLens**, whereby we found that the statistical techniques and methods we applied to model galaxies could be used in a more general setting to many different scientific problems. Check it out if you are interested in developing your own software to perform advanced model-fitting! -We import this library separately from **PyAutoLen**. +We import this library separately from **PyAutoLens**. """ import autofit as af @@ -185,12 +190,12 @@ Let's first load the `Imaging` dataset, which we will use to fit a model with a non-linear search. -The strong lens in this image was generated using an `Isothermal` lens mass profile and `SersicCore` source light profile, +The strong lens in this image was generated using an `IsothermalSph` lens mass profile and `ExponentialCoreSph` source light profile, which we'll also use in our model fitting in this tutorial. This means the model we are going to fit is identical to the one used to simulate the data, allowing us to assess the fitting process under controlled conditions. -The dataset, as well as all subsequent datasets used in future tutorials, is stored in -the `autolens_workspace/dataset/imaging` folder. +The dataset, as well as all subsequent datasets used in future tutorials, is stored in the HowToLens +repository's `dataset/imaging` folder (simulated at runtime by the simulator scripts, as shown below). """ dataset_name = "simple__no_lens_light__mass_sis" dataset_path = Path("dataset") / "imaging" / dataset_name @@ -300,10 +305,10 @@ been chosen to be broad, and contain the breadth of plausible solutions one should expect when fitting light and mass profiles of a real galaxy. -For instance, consider the `centre` parameter of our `IsothermalSph` light profile. In theory, it could take on any value from -negative to positive infinity. However, imaging datasets are typically reduced such that the galaxy centre is close -to (0.0", 0.0"). Therefore, a `TruncatedGaussianPrior` with `mean=0.0` and `sigma=0.1` is a good description of where the -galaxy `centre` is. +For instance, consider the `centre` parameter of our `IsothermalSph` mass profile. In theory, it could take on any value from +negative to positive infinity. However, imaging datasets are typically reduced such that the galaxy centre is close +to (0.0", 0.0"). Therefore, a `GaussianPrior` with `mean=0.0` and `sigma=0.1` is a good description of where the +galaxy `centre` is. If the galaxy had a different centre in the dataset, we would change the mean of the prior to reflect this. However, in general, we advise all galaxy images are reduced such that the galaxy is at (0.0", 0.0"). @@ -524,7 +529,7 @@ one, other walkers will explore other regions of parameter space and encourage the walker in local maxima to move away from it and head towards higher likelihood regions. -In the example below, we use the `Emcee` MCMC search to fit the strong lens The search starts with walkers +In the example below, we use the `Emcee` MCMC search to fit the strong lens. The search starts with walkers initialized in a "ball" around the center of the model’s priors, similar to the MLE search that failed earlier. """ search = af.Emcee( @@ -556,7 +561,7 @@ A major advantage of MCMC is that it provides estimates of parameter uncertainties by "mapping out" the likelihood surface, unlike MLE, which only finds the maximum likelihood solution. These error estimates are accessible in -the `result.info` string and through the `result.samples` object, which is explained fully in tutorial 5. +the `result.info` string and through the `result.samples` object, which is explained fully in tutorial 7. While a good starting point wasn't necessary for this simple model, it becomes essential for efficiently mapping the likelihood surface in more complex models with many parameters. The code below shows an MCMC fit using a good starting @@ -635,7 +640,7 @@ search = af.Nautilus( n_live=100, n_batch=50, # Explained in next tutorial - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) """ @@ -695,12 +700,12 @@ When approaching a model-fitting problem, it's usually advisable to try various search methods to find the most effective one. -However, after extensive testing within **PyAutoLen**, a clear recommendation has emerged for fitting strong lens models. +However, after extensive testing within **PyAutoLens**, a clear recommendation has emerged for fitting strong lens models. The nested sampling method `Nautilus` consistently proves to be the most effective. It requires fewer iterations than MCMC or MLE methods (when no starting point is used), provides robust sampling even for complex models, includes a built-in stopping criterion, and delivers reliable error estimates. -All examples in the `autolens_workspace` use the `Nautilus` search, and future tutorials in the **HowToGalaxy** +All examples in the `autolens_workspace` use the `Nautilus` search, and future tutorials in the **HowToLens** series will also use it. We strongly recommend using `Nautilus` from now on. That said, MLE and MCMC searches can still be effective, and you're encouraged to experiment with them. If you have diff --git a/scripts/chapter_2_lens_modeling/tutorial_2_practicalities.py b/scripts/chapter_2_lens_modeling/tutorial_2_practicalities.py index c0cbdf3..84d53e4 100755 --- a/scripts/chapter_2_lens_modeling/tutorial_2_practicalities.py +++ b/scripts/chapter_2_lens_modeling/tutorial_2_practicalities.py @@ -19,6 +19,7 @@ - **PyAutoFit:** Modeling uses the probabilistic programming language. - **Initial Setup:** Lets first load the `Imaging` dataset we'll fit a model with using a non-linear search. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Mask:** Define the 2D mask applied to the dataset for the model-fit. - **Model:** Compose the lens model fitted to the data. - **Search:** Configure the non-linear search used to fit the model. @@ -27,6 +28,7 @@ - **Analysis:** Create the Analysis object that defines how the model is fitted to the data. - **VRAM Use:** When running AutoLens with JAX on a GPU, the analysis must fit within the GPU’s available VRAM. - **Run Times:** Profiling the expected run time of the model-fit. +- **Model-Fit:** Begin the model-fit by passing the model and analysis to the search. - **Result Info:** A concise readable summary of the results is given by printing its `info` attribute. - **Output Folder:** Now checkout the `autolens_workspace/output` folder. - **Unique Identifier:** In the output folder, you will note that results are in a folder which is a collection of random. @@ -47,7 +49,7 @@ The ideal number of live points depends on model complexity. More parameters generally require more live points, but the default of 200 is sufficient for most lens models. Lower values can still yield reliable results, particularly -for simpler models. For this example (7 parameters), we reduce the live points to 100 to speed up runtime without +for simpler models. For this example (6 parameters), we reduce the live points to 100 to speed up runtime without compromising accuracy. Tuning non-linear search settings (e.g., the number of live points) to match model complexity is essential. We aim @@ -55,8 +57,8 @@ is excessive. In practice, the optimal number of live points is often found through trial and error, guided by summary statistics -on how well the search is performing, which we’ll cover below. For this single Sersic model with a linear light -profile, 80 live points is sufficient to achieve reliable results. +on how well the search is performing, which we’ll cover below. For the simple model fitted here, with a linear light +profile, 100 live points is sufficient to achieve reliable results. __Iterations Per Update__ @@ -66,8 +68,8 @@ Depending on how long it takes for the model to be fitted to the data (see discussion about run times below), this can take up a large fraction of the run-time of the non-linear search. -For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=10000` to ensure these updates -so not slow down the overall speed of the model-fit. +For this fit, the fit is very fast, thus we set a high value of `iterations_per_quick_update=2500` to ensure these updates +do not slow down the overall speed of the model-fit. **If the iteration per update is too low, the model-fit may be significantly slowed down by the time it takes to output results and visualization frequently to hard-disk. If your fit is consistent displaying a log saying that it @@ -172,13 +174,18 @@ print(model.info) +""" +__Search__ + +We again use the `Nautilus` non-linear search, with settings chosen following the Search Settings discussion above. +""" search = af.Nautilus( path_prefix=Path("howtolens") / "chapter_2", name="tutorial_2_practicalities", unique_tag=dataset_name, n_live=100, n_batch=50, # GPU batching and VRAM use explained in VRAM section below. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) """ @@ -211,7 +218,7 @@ The method below prints the VRAM usage estimate for the analysis and model with the specified batch size, it takes about 20-30 seconds to run so you may want to comment it out once you are familiar with your GPU's VRAM limits. -For a MGE model with the low resolution dataset fitted in this example VRAM use is relatively low (~0.027GB) For other +For the simple model and low resolution dataset fitted in this example VRAM use is relatively low (~0.027GB). For other models (e.g. pixelized sources) and higher resolution datasets it can be much higher (> 1GB going beyond 10GB). """ analysis.print_vram_use(model=model, batch_size=search.batch_size) @@ -315,9 +322,9 @@ __Output Folder Contents__ -Now this is running you should checkout the `autolens_workspace/output` folder. This is where the results of the -search are written to hard-disk (in the `start_here` folder), where all outputs are human readable (e.g. as .json, -.csv or text files). +Now this is running you should checkout the `output` folder. This is where the results of the +search are written to hard-disk (in the `tutorial_2_practicalities` folder), where all outputs are human readable +(e.g. as .json, .csv or text files). As the fit progresses, results are written to the `output` folder on the fly using the highest likelihood model found by the non-linear search so far. This means you can inspect the results of the model-fit as it runs, without having to @@ -378,7 +385,7 @@ The following are examples of other practicalities which I will document fully in this example script in the future, but so far have no found the time: -- `config`: The files in `autogalaxy_workspace/config` which control many aspects of how PyAutoGalaxy runs, +- `config`: The files in the `config` folder which control many aspects of how **PyAutoLens** runs, including visualization, the non-linear search settings. - `config/priors`: Folder containing the default priors on all model components. diff --git a/scripts/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.py b/scripts/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.py index acf4ea7..7cb3258 100644 --- a/scripts/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.py +++ b/scripts/chapter_2_lens_modeling/tutorial_3_realism_and_complexity.py @@ -25,8 +25,10 @@ __Contents__ - **Initial Setup:** we'll use new strong lensing data, where. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Mask:** Define the 2D mask applied to the dataset for the model-fit. - **Model:** Compose the lens model fitted to the data. +- **Search + Analysis:** Create the non-linear search and analysis used to fit the model. - **Run Time:** Profiling the expected run time of the model-fit. - **Result:** Overview of the results of the model-fit. - **Global and Local Maxima:** Up to now, all our non-linear searches have successfully found lens models that provide visibly. @@ -125,7 +127,7 @@ __Search + Analysis__ We set up `Nautilus` as we did in the previous tutorial, however given the increase in model complexity we'll use -a higher `n_live` value of 150 to ensure we sample the complex parameter space efficiently. +a higher `n_live` value of 200 to ensure we sample the complex parameter space efficiently. """ search = af.Nautilus( path_prefix=Path("howtolens") / "chapter_2", @@ -133,7 +135,7 @@ unique_tag=dataset_name, n_live=200, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis = al.AnalysisImaging(dataset=dataset) @@ -198,7 +200,7 @@ unique_tag=dataset_name, n_live=75, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) print( @@ -251,13 +253,13 @@ inferring a local maxima is still very real, especially as we make our lens model more complex. Lets think about *complexity*. As we make our lens model more realistic, we also made it more complex. For this -tutorial, our non-linear parameter space went from 7 dimensions to 18. This means there was a much larger *volume* of +tutorial, our non-linear parameter space went from 6 dimensions to 20. This means there was a much larger *volume* of parameter space to search. As this volume grows, there becomes a higher chance that our non-linear search gets lost and infers a local maxima, especially if we don't set it up with enough live points! At its core, lens modeling is all about learning how to get a non-linear search to find the global maxima region of -parameter space, even when the lens model is complex. This will be the main theme throughout the rest of this chapter -and is the main subject of chapter 3. +parameter space, even when the lens model is complex. This will be the main theme throughout the rest of this chapter, +culminating in the search chaining technique introduced in tutorials 9 and 10. In the next exercise, we'll learn how to deal with failure and begin thinking about how we can ensure our non-linear search finds the global-maximum log likelihood solution. First, think about the following: @@ -268,7 +270,7 @@ 2) The non-linear search failed because parameter space was too complex. Could we make it less complex, whilst still keeping our lens model fairly realistic? - 3) The source galaxy in this example had only 7 non-linear parameters. Real source galaxies may have multiple + 3) The source galaxy in this example had only 6 non-linear parameters. Real source galaxies may have multiple components (e.g. a disk, bulge, bar, star-forming knot) and there may even be more than 1 source galaxy! Do you think there is any hope of us navigating a parameter space if the source contributes 20+ parameters by itself? """ diff --git a/scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py b/scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py index 4c4ba7d..8ad6d0f 100644 --- a/scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py +++ b/scripts/chapter_2_lens_modeling/tutorial_4_dealing_with_failure.py @@ -21,11 +21,15 @@ __Contents__ - **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Mask:** Define the 2D mask applied to the dataset for the model-fit. - **Prior Tuning:** First, we will try to assist our non-linear search by tuning our priors. - **Run Time:** Profiling the expected run time of the model-fit. - **Result:** Overview of the results of the model-fit. - **Discussion:** By tuning our priors to the specific lens model we are fitting, we increase the chances of finding. +- **Approach 2: Reducing Complexity:** Simplify the lens model, for example by assuming light-traces-mass. +- **Approach 3: Look Harder:** Brute force the global maxima with more thorough non-linear search settings. +- **Wrap Up:** Summary of the script and next steps. """ @@ -229,7 +233,7 @@ unique_tag=dataset_name, n_live=150, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis = al.AnalysisImaging(dataset=dataset) @@ -336,7 +340,7 @@ unique_tag=dataset_name, n_live=200, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) """ @@ -401,11 +405,8 @@ number of live points used by `Nautilus`. The more thoroughly the search examines the space, the more likely it is to find the global maximum lens model. However, this also means the search will take longer! -Below, we configure a more thorough Nautilus search with `n_live=200`. These settings and what they change are +Below, we configure a more thorough Nautilus search with `n_live=300`. These settings and what they change are discussed in the optional tutorial `chapter_optional/tutorial_searches.ipynb`. - -Due to the long run times of this search, the output is commented out below. Feel free to uncomment it to run the -script faster. """ lens = af.Model( al.Galaxy, @@ -424,7 +425,7 @@ name="tutorial_4_look_harder", unique_tag=dataset_name, n_live=300, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) """ __Run Time__ @@ -467,8 +468,8 @@ weeks, or even months! In conclusion, we can now fit strong lenses effectively. When the process fails, we know how to make it work. -In chapter 3 of **HowToLens**, we will introduce a technique called 'non-linear search chaining', which performs a model -fit by chaining together multiple searches back-to-back . This allows us to combine the 3 different approaches +In tutorials 9 and 10 of this chapter, we will introduce a technique called 'non-linear search chaining', which performs +a model fit by chaining together multiple searches back-to-back. This allows us to combine the 3 different approaches discussed and exploit the advantages of each, whilst not being hindered by their disadvantages. With search chaining, we can: @@ -485,6 +486,6 @@ __Wrap Up__ -To wrap up chapter 2, we have a few more tutorials, where we will discuss masking in more detail, the `Result` object -and how to make **PyAutoLens** run faster. +Before we get to search chaining, we have a few more tutorials, where we will discuss masking in more detail, the +`Result` object and how to make **PyAutoLens** run faster. """ diff --git a/scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py b/scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py index a62ccef..bd9bd56 100644 --- a/scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py +++ b/scripts/chapter_2_lens_modeling/tutorial_5_linear_profiles.py @@ -26,6 +26,7 @@ __Contents__ - **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Mask:** Define the 2D mask applied to the dataset for the model-fit. - **Linear Light Profiles:** We use a variant of a light profile discussed called a "linear light profile", which is accessed. - **Run Time:** Profiling the expected run time of the model-fit. @@ -163,7 +164,7 @@ unique_tag=dataset_name, n_live=100, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis = al.AnalysisImaging(dataset=dataset) @@ -437,7 +438,7 @@ unique_tag=dataset_name, n_live=100, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) print( @@ -536,7 +537,7 @@ lens modeling problems. If you have a desire to fit sources with even more complex morphologies we recommend you look at how to reconstruct -sources using pixelizations in the `modeling/features` section or chapter 4 of **HowToLens**. +sources using pixelizations in the `modeling/features` section or chapter 3 of **HowToLens**. __Wrap Up__ @@ -549,7 +550,7 @@ galaxies which may have multiple components with multiple centres (e.g. many star forming knots) which such a basis cannot capture. -In chapter 4 of **HowToLens** we introduce non-parametric pixelizations, which reconstruct the data in way +In chapter 3 of **HowToLens** we introduce non-parametric pixelizations, which reconstruct the data in way that does not make assumptions like a centre and can thus reconstruct even more complex, asymmetric and irregular galaxy morphologies. """ diff --git a/scripts/chapter_2_lens_modeling/tutorial_6_masking_and_positions.py b/scripts/chapter_2_lens_modeling/tutorial_6_masking_and_positions.py index f1e0929..b4dbaeb 100644 --- a/scripts/chapter_2_lens_modeling/tutorial_6_masking_and_positions.py +++ b/scripts/chapter_2_lens_modeling/tutorial_6_masking_and_positions.py @@ -9,7 +9,9 @@ __Contents__ - **Initial Setup:** we'll use the same strong lensing data as tutorials 1 & 2, where. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Mask:** Define the 2D mask applied to the dataset for the model-fit. +- **Model + Analysis:** Compose the lens model and create the analysis that fits it using the chosen mask. - **Run Time:** Profiling the expected run time of the model-fit. - **Search:** Configure the non-linear search used to fit the model. - **Discussion:** So, we can choose the mask we use in a model-fit. @@ -33,7 +35,7 @@ we'll use the same strong lensing data as tutorials 1 & 2, where: - The lens galaxy's total mass distribution is a `IsothermalSph`. - - The source galaxy's light is a `ExponentialSph`. + - The source galaxy's light is a `ExponentialCoreSph`. """ dataset_name = "simple__no_lens_light__mass_sis" dataset_path = Path("dataset") / "imaging" / dataset_name @@ -126,7 +128,7 @@ unique_tag=dataset_name, n_live=80, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis = al.AnalysisImaging(dataset=dataset) @@ -255,7 +257,7 @@ unique_tag=dataset_name, n_live=80, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) print( diff --git a/scripts/chapter_2_lens_modeling/tutorial_7_results.py b/scripts/chapter_2_lens_modeling/tutorial_7_results.py index f784a99..450eeca 100644 --- a/scripts/chapter_2_lens_modeling/tutorial_7_results.py +++ b/scripts/chapter_2_lens_modeling/tutorial_7_results.py @@ -8,6 +8,7 @@ __Contents__ - **Initial Setup:** Lets use the model-fit performed in tutorial 1 to get a `Result` object. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Tracer & Fit:** In the previous tutorials, we saw that this result contains the maximum log likelihood fit, which. - **Samples:** The result contains a lot more information about the model-fit. - **Workspace:** We are not going into any more detail on the result variable in this tutorial, or in the. @@ -78,7 +79,7 @@ unique_tag=dataset_name, n_live=80, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis = al.AnalysisImaging(dataset=dataset) diff --git a/scripts/chapter_2_lens_modeling/tutorial_8_need_for_speed.py b/scripts/chapter_2_lens_modeling/tutorial_8_need_for_speed.py index 61c4b1b..72faed8 100755 --- a/scripts/chapter_2_lens_modeling/tutorial_8_need_for_speed.py +++ b/scripts/chapter_2_lens_modeling/tutorial_8_need_for_speed.py @@ -14,53 +14,126 @@ fit performed in this chapter. Lets quickly remind ourselves of the main factors that drive the run-time, and how we might reduce it. +The overall run-time of a model-fit is, to a good approximation, a simple product of two numbers: + + - The number of log likelihood evaluations the non-linear search performs before it converges. + + - The time each individual log likelihood evaluation takes. + +Everything we discuss below reduces one (or both) of these numbers. + +__Contents__ + +- **Searching Non-linear Parameter Space:** What drives the number of likelihood evaluations a search performs. +- **Cost Per Evaluation:** What drives the time a single log likelihood evaluation takes. +- **JAX:** How **PyAutoLens** compiles the likelihood function with JAX and runs it on CPUs and GPUs. +- **Run Time Estimation:** How to estimate the overall run-time of a fit before committing to it. +- **Data Quantity:** How the amount of data fitted, and the mask applied to it, drives run-times. +- **Wrap Up:** Summary of the script and next steps. + __Searching Non-linear Parameter Space__ The time it takes for the non-linear search to sample parameter space and find the high likelihood models is driven by: - Dimensionality: A more complex parameter space (e.g. more parameters) takes longer to search. - Priors: The broader the priors on each parameter the longer the search. - - Settings: Non-linear search settings which sample parameter space more thoroughly lead to longer run-times. + - Settings: Non-linear search settings which sample parameter space more thoroughly (e.g. a higher `n_live`) lead to + longer run-times. When we use only one search to fit a lens model, we are somewhat restricted in how we can try to achieve faster run -times by changing these 3 aspects of the search. +times by changing these 3 aspects of the search. We have already seen one powerful exception: the linear light +profiles of tutorial 5 remove every `intensity` parameter from the non-linear parameter space, reducing its +dimensionality and removing some of its most difficult degeneracies at the same time. -In the next chapter, we introduce 'non-linear search chaining', which fits a lens model using multiple searches that -are performed back-to-back. A key motivation for this is that it gives us a lot more flexibility in juggling the -dimensionality, priors and settings so as to perform faster and more efficient lens modeling. +The final three tutorials of this chapter (tutorials 9, 10 and 11) introduce 'non-linear search chaining', which fits +a lens model using multiple searches that are performed back-to-back, passing the results of earlier searches to +later ones. A key motivation for this is that it gives us a lot more flexibility in juggling the dimensionality, +priors and settings so as to perform faster and more efficient lens modeling, culminating in the fully automated +SLaM pipelines of tutorial 11. -In the optional **HowToGalaxy** tutorial `chapter_optional/tutorial_searches.ipynb` we discuss other non-linear -searches supported by **HowToGalaxy** which use a different approach to sample parameter sample than `Nautilus`. For -those familiar with statistical inference, this includes maximum likelihood estimators and MCMC algorithms. +In the optional **PyAutoLens** tutorial `chapter_optional/tutorial_searches.py` we discuss other non-linear searches +supported by **PyAutoLens** which use a different approach to sample parameter space than `Nautilus`. For those +familiar with statistical inference, this includes maximum likelihood estimators and MCMC algorithms. For lens modeling, we have not found another non-linear search that provides as robust and efficient results as -Nautilus. We therefore recommend uses stick to `Nautilus`. +Nautilus. We therefore recommend users stick to `Nautilus`. -__Contents__ +__Cost Per Evaluation__ -- **Algorithmic Optimization:** Every operation **PyAutoLens** performs to fit strong lens data with a model takes time, for. -- **Data Quantity:** The final factor driving run-speed is the quantity of data that is fitted. -- **Wrap Up:** Summary of the script and next steps. +The second number is the time a single log likelihood evaluation takes, which is set by every operation +**PyAutoLens** performs to fit the strong lens data with a model, for example: -__Algorithmic Optimization__ + - Computing the deflection angles of the mass model and ray-tracing the image-plane grid to the source-plane. + - Computing the intensity values of every light profile on those grids. + - Over-sampling: sub-dividing image pixels into finer sub-pixels where light profiles change rapidly (e.g. the + centre of the lens galaxy), which multiplies the number of intensity evaluations performed. + - Convolving the image that comes from the tracer with the telescope's PSF to compare it to the data. The larger + the PSF, the more expensive this convolution becomes. + - For linear light profiles, basis functions like the MGE of tutorial 5 and the pixelizations of chapter 3, solving + the linear algebra of the inversion, which computes the `intensity` values that best fit the data. -Every operation **PyAutoLens** performs to fit strong lens data with a model takes time, for example: +More complex fitting techniques therefore cost more per evaluation: a couple of standard light profiles are cheap, +an MGE costs more (its inversion solves for many Gaussians), and pixelized source reconstructions cost the most. +As we saw in tutorial 5, this trade often still favours the more advanced technique, because the simpler parameter +space it produces reduces the number of evaluations the search needs. - - Computing the intensity values from a light profile. - - Convolving the image that comes from a tracer with the PSF to compare it to the data. +__JAX__ -One can therefore in principle make **PyAutoLens** run faster by using more efficient algorithms. However, I am -confident that for many tasks and operations we have written code that is already very fast! +How are these operations made fast? **PyAutoLens** uses JAX (https://github.com/jax-ml/jax), Google's numerical +computing library, which just-in-time (JIT) compiles the entire log likelihood function into optimized machine code. -I often get asked, given that **PyAutoLens** is written in Python (a synonymously slow programming language), is it not -really slow? **PyAutoLens** uses a library called `numba` to ensure that it runs fast, which recompiles Python -functions into C functions before **PyAutoLens** runs. This gives us C-like speed, but in Python code. If you`ve got -your own code that needs speeding up, I strongly recommend that you look up Numba: +JIT compilation means the first time the likelihood function is called, JAX traces through every operation it +performs (ray-tracing, light profile evaluation, PSF convolution, the linear algebra of an inversion) and compiles +them into a single optimized program. This compilation is a one-off overhead, typically taking seconds to a couple +of minutes depending on model complexity. Every one of the tens of thousands of likelihood evaluations that follow +then reuses the compiled program and runs dramatically faster than ordinary Python. When a search begins you will +see a log message like `JAX jit compiling vectorized (vmap) likelihood function, could take seconds or minutes...` +-- that is the compilation happening. -http://numba.pydata.org/ +JAX's second superpower is that the same compiled code runs on either a CPU or a GPU. On a GPU, likelihood +evaluations are not only faster individually but can be batched, with many evaluated simultaneously (the +`n_batch` search input and its VRAM implications were discussed in tutorial 2). GPU speed-ups over CPU +of an order of magnitude or more are common, and they grow with the size of the dataset and model. If you have +access to a GPU (even a modest laptop GPU), it is the single biggest speed-up available to you. -Therefore, **PyAutoLens** is pretty well optimized and there are no 'low hanging fruit' speed ups available by writing -the code in a different language. +You do not need to do anything to switch this on. If JAX is installed (`pip install autolens[jax]`), every +`AnalysisImaging` object we have created in this chapter defaults to `use_jax=True` and the search compiles and +batches the likelihood function automatically. You can force the plain NumPy path with +`al.AnalysisImaging(dataset=dataset, use_jax=False)` (or by setting the environment variable +`PYAUTO_DISABLE_JAX=1`), which is mainly useful for debugging, as NumPy errors and stack traces are much easier +to read than JAX ones. + +The one trade-off to keep in mind is the compilation overhead itself. For a full model-fit performing tens of +thousands of likelihood evaluations it is negligible. For a quick one-off calculation (e.g. fitting a single +tracer to data once, as we did in chapter 1) compilation can take longer than the calculation it speeds up, which +is why short interactive computations sometimes feel slower than you might expect the first time they run. + +Long-time users may remember that **PyAutoLens** previously used `numba`, which recompiles individual Python +functions into fast machine code. JAX supersedes it: rather than accelerating functions one at a time, it compiles +and optimizes the likelihood function end-to-end, and adds GPU support and batching on top. + +Therefore, **PyAutoLens** is pretty well optimized and there are no 'low hanging fruit' speed ups available by +writing the code in a different language. + +__Run Time Estimation__ + +Because run-time is the product of the two numbers above, we can estimate it before committing to a fit: + + - Time the log likelihood evaluation: perform a fit of the model to the data once (e.g. via a `FitImaging`) and + time it, remembering to discard the first call, which includes JIT compilation. + + - Estimate the number of evaluations: for `Nautilus` a reasonable rule of thumb is of order 10000 evaluations per + free parameter, with simpler parameter spaces (e.g. linear light profiles, chained searches with tight priors) + converging in fewer. + +Multiplying the two gives a ballpark overall run-time. In tutorial 2 we used exactly this arithmetic: a log +likelihood evaluation time below 0.01 seconds on CPU (below 0.001 seconds on GPU) and 50000-100000 evaluations gave +expected run-times of roughly 30 minutes on CPU and 10 minutes on GPU. If your own estimate comes out at days or +weeks, that is the moment to simplify the model, tighten the search settings, reach for search chaining or find +a GPU -- before starting the fit, not after. + +On a GPU, tutorial 2 also showed the other pre-flight check: `analysis.print_vram_use` estimates whether the fit +will fit within the GPU's memory for your chosen batch size. __Data Quantity__ @@ -75,9 +148,15 @@ There are ways that we can reduce the number of image-pixels we fit, via masking. If we mask out more of the image, we will fit fewer pixels and **PyAutoLens** will run faster. If you want the best, most perfect lens model possible, -aggressive masking and cutting the data in this way is a bad idea, as discussed in tutorial 5. +aggressive masking and cutting the data in this way is a bad idea, as discussed in tutorial 6. __Wrap Up__ -This tutorial simply wanted to get you thinking about *why* a lens model takes as long to fit as it does. +This tutorial simply wanted to get you thinking about *why* a lens model takes as long to fit as it does: the number +of likelihood evaluations the search performs, times the cost of each evaluation, with JAX's compiled likelihood +function (and a GPU, if you have one) driving the latter down as far as it will go. + +The remaining tutorials of this chapter attack the other half of the product: search chaining (tutorial 9) and prior +passing (tutorial 10) reduce the number of evaluations needed to fit complex models, and the SLaM pipelines +(tutorial 11) package these ideas into automated pipelines for fitting large samples of lenses. """ diff --git a/scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py b/scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py similarity index 85% rename from scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py rename to scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py index a85bd93..afe17cf 100644 --- a/scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py +++ b/scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py @@ -1,15 +1,19 @@ """ -Tutorial 1: Search Chaining +Tutorial 9: Search Chaining =========================== -In chapter 2, we learnt how to perform lens modeling using a non-linear search. In all of the tutorials, we fitted the -data using just one non-linear search. In this chapter, we introduce a technique called 'non-linear search chaining', -fits a lens model using a sequence of non-linear searches. The initial searches fit simpler lens models whose parameter -spaces can be more accurately and efficiently sampled. The results of this search are then passed to later searches -which fit lens models of gradually increasing complexity. +Throughout this chapter, we have fitted the data using just one non-linear search. The previous tutorial discussed +the run-time cost of doing this: with a single search, the dimensionality of the model, the priors on its parameters +and the search settings must all be juggled at once, leaving us little flexibility to trade them off against one +another. -Lets think back to tutorial 4 of chapter 2. We learnt there were three approaches one could take fitting a lens model -accurately if we found that a model fit failed. These were: +In this closing part of the chapter, we introduce a technique called 'non-linear search chaining', which fits a lens +model using a sequence of non-linear searches. The initial searches fit simpler lens models whose parameter spaces can +be more accurately and efficiently sampled. The results of this search are then passed to later searches which fit +lens models of gradually increasing complexity. + +Lets think back to tutorial 4 of this chapter. We learnt there were three approaches one could take fitting a lens +model accurately if we found that a model fit failed. These were: 1) Tuning our priors to the strong lens we're fitting. 2) Making our lens model less complex. @@ -24,7 +28,7 @@ we'll run 2 searches, and chain the lens model inferred in the first search to the priors of the second search`s lens model. -Our first search will make the same light-traces-mass assumption we made in the previous tutorial. We saw that this +Our first search will make the same light-traces-mass assumption we made in tutorial 4. We saw that this gives a reasonable lens model. However, we'll make a couple of extra simplifying assumptions, to really try and bring our lens model complexity down and get the non-linear search running fast. @@ -40,10 +44,12 @@ __Contents__ -- **Initial Setup:** we'll use the same strong lensing data as the previous tutorial, where. +- **Initial Setup:** Load the same strong lensing data fitted throughout this chapter. +- **Dataset Auto-Simulation:** Simulate the dataset via its simulator script if it is not on your hard-disk. - **Model:** Compose the lens model fitted to the data. +- **Search + Analysis:** Create the non-linear search and analysis used to fit the first model. - **Result:** Overview of the results of the model-fit. -- **Prior Passing:** Now all we need to do is look at the results of search 1 and pass the results as priors for search. +- **Prior Passing:** Use the results of search 1 to set up the priors of search 2. - **Run Time:** Profiling the expected run time of the model-fit. - **Model Fit:** Perform the model-fit using the search and analysis. - **Wrap Up:** Summary of the script and next steps. @@ -63,7 +69,7 @@ """ __Initial Setup__ -we'll use the same strong lensing data as the previous tutorial, where: +we'll use the same strong lensing data as tutorials 3, 4 and 5 of this chapter, where: - The lens galaxy's light is an `Sersic`. - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. @@ -167,11 +173,11 @@ Now lets create the search and analysis. """ search_1 = af.Nautilus( - path_prefix=Path("howtolens", "chapter_3"), - name="tutorial_1_search_chaining_1", + path_prefix=Path("howtolens", "chapter_2"), + name="tutorial_9_search_chaining_1", unique_tag=dataset_name, n_live=100, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis_1 = al.AnalysisImaging(dataset=dataset) @@ -181,7 +187,7 @@ 11 parameters. """ print( - "The non-linear search has begun running - checkout the workspace/output/5_chaining_searches" + "The non-linear search has begun running - checkout the output/howtolens/chapter_2" " folder for live output of the results, images and lens model." " This Jupyter notebook cell with progress once search has completed - this could take some time!" ) @@ -311,11 +317,11 @@ faster than we are used to! """ search_2 = af.Nautilus( - path_prefix=Path("howtolens", "chapter_3"), - name="tutorial_1_search_chaining_2", + path_prefix=Path("howtolens", "chapter_2"), + name="tutorial_9_search_chaining_2", unique_tag=dataset_name, n_live=150, - iterations_per_quick_update=25000, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=25000, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis_2 = al.AnalysisImaging(dataset=dataset) @@ -334,7 +340,7 @@ Run the search. """ print( - "The non-linear search has begun running - checkout the workspace/output/5_chaining_searches" + "The non-linear search has begun running - checkout the output/howtolens/chapter_2" " folder for live output of the results, images and lens model." " This Jupyter notebook cell with progress once search has completed - this could take some time!" ) diff --git a/scripts/chapter_3_pixelizations/README.md b/scripts/chapter_3_pixelizations/README.md new file mode 100644 index 0000000..c5d9288 --- /dev/null +++ b/scripts/chapter_3_pixelizations/README.md @@ -0,0 +1,29 @@ +In chapter 3, we use **Pixelizations** to reconstruct complex source galaxies on pixelized grids. + +**Colab** links to every tutorial are included. + +# Files + +- [Tutorial 1: Pixelizations](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_1_pixelizations.ipynb) — Creating a pixel-grid in the source-plane. + +- [Tutorial 2: Mappers](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_2_mappers.ipynb) — How a pixelization maps source-pixels to image-pixels. + +- [Tutorial 3: Inversions](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_3_inversions.ipynb) — Inverting the mappings to reconstruct the source's light. + +- [Tutorial 4: Bayesian Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_4_bayesian_regularization.ipynb) — Smoothing the source within a Bayesian framework. + +- [Tutorial 5: Bayesian Formalism](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_5_bayesian_formalism.ipynb) — The linear algebra and Bayesian evidence equations behind pixelized source reconstruction. + +- [Tutorial 6: Borders](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_6_borders.ipynb) — Preventing highly demagnified image-pixels ruining the inversion. + +- [Tutorial 7: Lens Modeling](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_7_lens_modeling.ipynb) — How to use inversions to fit a lens model. + +- [Tutorial 8: Adaptive Pixelization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.ipynb) — A Delaunay mesh which adapts to the mass model's magnification. + +- [Tutorial 9: Model Fit](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_9_model_fit.ipynb) — An example lens modeling pipeline which uses an inversion. + +- [Tutorial 10: Fit Problems](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_10_fit_problems.ipynb) — The shortcomings of our lens models and inversions. + +- [Tutorial 11: Brightness Adaption](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_11_brightness_adaption.ipynb) — Adapting the pixelization to the source's morphology. + +- [Tutorial 12: Adaptive Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_pixelizations/tutorial_12_adaptive_regularization.ipynb) — Adapting the regularization to the source's morphology. diff --git a/scripts/chapter_3_search_chaining/__init__.py b/scripts/chapter_3_pixelizations/__init__.py similarity index 100% rename from scripts/chapter_3_search_chaining/__init__.py rename to scripts/chapter_3_pixelizations/__init__.py diff --git a/scripts/chapter_4_pixelizations/introduction b/scripts/chapter_3_pixelizations/introduction similarity index 86% rename from scripts/chapter_4_pixelizations/introduction rename to scripts/chapter_3_pixelizations/introduction index d52b456..fbfe5a3 100755 --- a/scripts/chapter_4_pixelizations/introduction +++ b/scripts/chapter_3_pixelizations/introduction @@ -1,4 +1,4 @@ -So, we've learnt how to build pipelines that model strong lenses and customize them to our science case. +So, we've learnt how to model strong lenses and build pipelines that customize the process to our science case. However, we ran into a pretty huge barrier when trying to fit complex sources. It required an unwieldy number of non-linear parameters, and even when we included them, the results were rubbish. The non-linear parameter space @@ -18,7 +18,7 @@ In particular, you'll learn how we: 3) Apply a smoothness prior on our source reconstruction, called 'regularization', to ensure our solution is physical. 4) Apply this prior in a Bayesian framework to objectively quantify our source reconstruction's log likelihood. 5) Define a border in the source-plane to prevent pixels tracing outside the source reconstruction. -6) Can use alternative pixelizations that use Voronoi pixels adapted to the lens's mass model. +6) Can use alternative pixelizations that use Delaunay pixels adapted to the lens's mass model. 7) Use these features in PyAutoLens pipelines. Sound good? I agree, so lets get on with it. \ No newline at end of file diff --git a/scripts/chapter_4_pixelizations/tutorial_9_fit_problems.py b/scripts/chapter_3_pixelizations/tutorial_10_fit_problems.py similarity index 89% rename from scripts/chapter_4_pixelizations/tutorial_9_fit_problems.py rename to scripts/chapter_3_pixelizations/tutorial_10_fit_problems.py index 0ea9268..2ccff98 100644 --- a/scripts/chapter_4_pixelizations/tutorial_9_fit_problems.py +++ b/scripts/chapter_3_pixelizations/tutorial_10_fit_problems.py @@ -1,18 +1,19 @@ """ -Tutorial 9: Fit Problems -======================== +Tutorial 10: Fit Problems +========================= -To begin, make sure you have read the `introduction` file carefully, as a clear understanding of how the Bayesian -evidence works is key to understanding this chapter! +To begin, make sure you have read tutorials 4 and 5 carefully, as a clear understanding of how the Bayesian +evidence works is key to understanding the adaption tutorials that close this chapter! -In the previous chapter we investigated two pixelization's: `RectangularAdaptDensity` and `RectangularAdaptDensity`. We argued that the -latter was better than the former, because it dedicated more source-pixels to the regions of the source-plane where we -had more data, e.g, the high-magnification regions. Therefore, we could fit the data using fewer source pixels, -which improved computational efficiency and increased the Bayesian evidence. +Earlier in this chapter (tutorial 8) we investigated two pixelizations: the `RectangularAdaptDensity` mesh and the +`Delaunay` mesh with an `Overlay` image-mesh. We argued that the latter was better than the former, because it +dedicated more source-pixels to the regions of the source-plane where we had more data, e.g. the high-magnification +regions. Therefore, we could fit the data using fewer source pixels, which improved computational efficiency and +increased the Bayesian evidence. So far, we've used just one regularization scheme; `Constant`. As the name suggests, this scheme applies just one regularization coefficient when comparing source pixel fluxes to apply smoothing. Here is a recap of our discussion -about regularization from chapter 4: +about regularization from tutorial 4: -------------------------------------------- @@ -157,8 +158,8 @@ def simulate_for_source_galaxy(source_galaxy): """ __Fitting__ -we'll make one more convenience function which fits the simulated imaging data with an `Overlay` image-mesh, -`RectangularAdaptDensity` mesh and `Constant` regularization scheme pixelization. +we'll make one more convenience function which fits the simulated imaging data with a +`RectangularAdaptDensity` mesh and `Constant` regularization scheme pixelization. We'll input the `coefficient` of each fit, so that for each simulated source we regularize it at an appropriate level. There is nothing new in this function you haven't seen before. @@ -256,14 +257,15 @@ def fit_with_Rectangular_from(dataset, mask, coefficient): *Image-Mesh / Mesh*: -The problem is the same one we discussed when we compared the `RectangularAdaptDensity` and `RectangularAdaptDensity` meshes in tutorial 7. +The problem is the same one we discussed in tutorial 8, when we motivated pixelizations which adapt to where the +data constrains the source. -We are simply not dedicating enough source-pixels to the central regions of the source reconstruction, -e.g. where it`s brightest. As the source becomes more compact, the source reconstruction no longer has enough +We are simply not dedicating enough source-pixels to the central regions of the source reconstruction, +e.g. where it's brightest. As the source becomes more compact, the source reconstruction no longer has enough resolution to resolve its fine-detailed central structure, causing the fit to the image to degrade. -As we made our sources more compact we go from reconstructing them using ~100 source pixels, to ~20 source pixels -to ~ 10 source pixels. This is why we advocated not using the `RectangularAdaptDensity` mesh previously! +As we made our sources more compact we went from reconstructing them using ~100 source pixels, to ~20 source pixels +to ~10 source pixels. Adapting to the mass model magnification is not the best approach. As we simulated more compact sources the magnification (which is determined via the mass model) does not change. We therefore reconstructed each source @@ -293,7 +295,7 @@ def fit_with_Rectangular_from(dataset, mask, coefficient): ) """ -As you can see, all pixels are regularized with our input regularization_coefficient value of 3.6. +As you can see, all pixels are regularized with our single input regularization coefficient value. This is not the best approach to regularizing the source. In fact, different regions of the source prefer different levels of regularization: @@ -315,7 +317,7 @@ def fit_with_Rectangular_from(dataset, mask, coefficient): By using a single regularization coefficient, we infer an intermediate regularization coefficient that over-smooths the source's central regions whilst failing to fully correlate exterior pixels. -An adaptive regularization scheme, where the regularization coefficient varies from the outskirts to the centrel +An adaptive regularization scheme, where the regularization coefficient varies from the outskirts to the central regions, will produce solutions that further increase the Bayesian evidence. __Wrap Up__ diff --git a/scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py b/scripts/chapter_3_pixelizations/tutorial_11_brightness_adaption.py similarity index 98% rename from scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py rename to scripts/chapter_3_pixelizations/tutorial_11_brightness_adaption.py index 1ddf685..e449df1 100644 --- a/scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py +++ b/scripts/chapter_3_pixelizations/tutorial_11_brightness_adaption.py @@ -1,5 +1,5 @@ """ -Tutorial 10: Brightness Adaption +Tutorial 11: Brightness Adaption ================================ In the previous tutorial we motivated our need to adapt the pixelization to the source's morphology, such that source @@ -116,7 +116,7 @@ fit = al.FitImaging(dataset=dataset, tracer=tracer) """ -Lets have a quick look to make sure it has the same residuals we saw in tutorial 1. +Lets have a quick look to make sure it has the same residuals we saw in the previous tutorial. """ mapper = fit.inversion.cls_list_from(al.Mapper)[0] @@ -358,9 +358,9 @@ """ __Wrap Up__ -To end, lets think about the Bayesian evidence, which we saw now goes to significantly higher values than for a -magnification-based grid. At this point, it might be worth reminding yourself how the Bayesian evidence works by -going back to description in this chapters `introduction` text file. +To end, lets think about the Bayesian evidence, which we saw now goes to significantly higher values than for a +magnification-based grid. At this point, it might be worth reminding yourself how the Bayesian evidence works by +going back to tutorials 4 and 5 of this chapter. So, why do you think why adapting to the source's brightness increases the log evidence? @@ -374,5 +374,5 @@ **PyAutoLens** can now access those solutions, this means that adapt-mode will run much faster than the magnification based grid! Put simply, fewer source-pixels means lower computational overheads. YAY! -Tutorial 2 done, next up, adaptive regularization! +Tutorial 11 done, next up, adaptive regularization! """ diff --git a/scripts/chapter_4_pixelizations/tutorial_11_adaptive_regularization.py b/scripts/chapter_3_pixelizations/tutorial_12_adaptive_regularization.py similarity index 89% rename from scripts/chapter_4_pixelizations/tutorial_11_adaptive_regularization.py rename to scripts/chapter_3_pixelizations/tutorial_12_adaptive_regularization.py index 05b67cf..a133646 100644 --- a/scripts/chapter_4_pixelizations/tutorial_11_adaptive_regularization.py +++ b/scripts/chapter_3_pixelizations/tutorial_12_adaptive_regularization.py @@ -1,8 +1,8 @@ """ -Tutorial 11: Adaptive Regularization +Tutorial 12: Adaptive Regularization ==================================== -In tutorial 7, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source +In tutorial 10, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source demand different levels of regularization, motivating a regularization scheme which adapts to the reconstructed source's surface brightness. @@ -174,8 +174,8 @@ def fit_via_source_galaxy_from(dataset, source_galaxy, adapt_images=None): Combining the adaptive mesh and regularization will only further benefit lens modeling! -However, as shown below, we don't fit the source as well as the morphology based mesh did in the last chapter. -This is because although the adaptive regularization scheme improves the fit, the magnification based +However, as shown below, we don't fit the source as well as the morphology based mesh did in the previous tutorial. +This is because although the adaptive regularization scheme improves the fit, the magnification based mesh simply does not have sufficient resolution to resolve the source's cuspy central light. """ aplt.subplot_fit_imaging(fit=fit) @@ -267,20 +267,25 @@ def fit_via_source_galaxy_from(dataset, source_galaxy, adapt_images=None): __Wrap Up__ To end, lets consider what this adaptive regularization scheme means in the context of maximizing the Bayesian -evidence. In the previous tutorial, we noted that by using a brightness-based adaptive pixelization we increased -the Bayesian evidence by allowing for new solutions which fit the data user fewer source pixels; the key criteria +evidence. In the previous tutorial, we noted that by using a brightness-based adaptive pixelization we increased +the Bayesian evidence by allowing for new solutions which fit the data using fewer source pixels; the key criteria in making a source reconstruction 'more simple' and 'less complex'. -As you might of guessed, adaptive regularization increases the Bayesian log evidence by making the source +As you might have guessed, adaptive regularization increases the Bayesian log evidence by making the source reconstruction simpler: - 1) Reducing regularization in the source's brightest regions produces a `simpler` solution in that we are not + 1) Reducing regularization in the source's brightest regions produces a `simpler` solution in that we are not over-smoothing our reconstruction of its brightest regions. - - 2) Increasing regularization in the outskirts produces a simpler solution by correlating more source-pixels, + + 2) Increasing regularization in the outskirts produces a simpler solution by correlating more source-pixels, effectively reducing the number of pixels used by the reconstruction. -Together, brightness based pixelization's and regularization allow us to find the objectively `simplest` source -solution possible and therefore ensure that our Bayesian evidence has a well defined maximum value. This was not the -case for magnification based pixelization's and constant regularization schemes. +Together, brightness based pixelizations and regularization allow us to find the objectively `simplest` source +solution possible and therefore ensure that our Bayesian evidence has a well defined maximum value. This was not the +case for magnification based pixelizations and constant regularization schemes. + +That completes this chapter on pixelized source reconstruction. You can now reconstruct sources of arbitrary +complexity and adapt every part of the analysis to the data itself. In chapter 4, `Scaling Up Lensing`, we take these +tools to bigger systems: lenses with extra galaxies along the line of sight, multi-galaxy lenses, scaling relations, +group and cluster scale lenses and weak lensing. """ diff --git a/scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py b/scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py similarity index 82% rename from scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py rename to scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py index 230863a..98146f2 100644 --- a/scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py +++ b/scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py @@ -1,5 +1,5 @@ """ -Tutorial 1: pixelizations +Tutorial 1: Pixelizations ========================= In the previous chapters, we used light profiles to model the light of a strong lens's source galaxy, where the light @@ -72,10 +72,10 @@ """ By itself, a pixelization does not tell us much. It has no grid of $(y,x)$ coordinates, no image, and no information -about the lens we are fitting. +about the lens we are fitting. -This information comes when we use the pixelization to create up a `Mapper`, which we -perform below using the (traced) source-plane grid that we created above. +This information comes when we use the pixelization to create a `Mapper`, which we +do below using the (traced) source-plane grid that we created above. """ interpolator = mesh.interpolator_from( source_plane_data_grid=source_plane_grid, source_plane_mesh_grid=None @@ -84,7 +84,7 @@ mapper = al.Mapper(interpolator=interpolator) """ -This `Mapper` is a `RectangularMapper`, every `Pixelization` generates it owns mapper. +This `Mapper` is a `RectangularMapper` -- every `Pixelization` generates its own mapper. """ print(type(mapper)) @@ -94,8 +94,8 @@ aplt.plot_grid(grid=mapper.source_plane_mesh_grid, title="Source-Plane Mesh Grid") """ -However, the `Mapper` does contain lots of interesting information about our `Pixelization`, for example its -pixelization_grid tells us where the pixel centers are located. +The `Mapper` contains lots of interesting information about our `Pixelization`, for example its +`source_plane_mesh_grid` tells us where the pixel centres are located. """ print("RectangularAdaptDensity Grid2D Pixel Centre 1:") print(mapper.source_plane_mesh_grid[0]) @@ -112,7 +112,7 @@ aplt.plot_grid(grid=mapper.source_plane_mesh_grid, title="Source-Plane Mesh Grid") """ -The `Mapper` also has the (source-plane) grid that we passed when we set it up. Lets check they`re the same. +The `Mapper` also has the (source-plane) grid that we passed when we set it up. Lets check they're the same. """ print("Source Grid2D Pixel 1") print(source_plane_grid[0]) @@ -132,7 +132,7 @@ aplt.plot_grid(grid=mapper.source_plane_mesh_grid, title="Source-Plane Mesh Grid") """ -Finally, the mapper`s `mesh_grid` has lots of information about the pixelization, for example, the arc-second +Finally, the mapper's mesh grid has lots of information about the pixelization, for example, its arc-second size and dimensions. """ print(mapper.source_plane_mesh_grid.geometry.shape_native_scaled) @@ -145,10 +145,10 @@ This was a relatively gentle overview of pixelizations, but one that was hopefully easy to follow. Think about the following questions before moving on to the next tutorial: - 1) Look at how the source-grid coordinates are distributed over the rectangular pixel-grid. Are these points - distributed evenly over the rectangular grid`s pixels? Do some pixels have a lot more grid-points inside of them? + 1) Look at how the source-grid coordinates are distributed over the rectangular pixel-grid. Are these points + distributed evenly over the rectangular grid's pixels? Do some pixels have a lot more grid-points inside of them? Do some pixels have no grid-points in them? - 2) The rectangular pixelization`s edges are aligned with the most exterior coordinates of the source-grid. This is - intentional, why do you think this is? + 2) The rectangular pixelization's edges are aligned with the most exterior coordinates of the source-grid. This is + intentional -- why do you think this is? """ diff --git a/scripts/chapter_4_pixelizations/tutorial_2_mappers.py b/scripts/chapter_3_pixelizations/tutorial_2_mappers.py similarity index 89% rename from scripts/chapter_4_pixelizations/tutorial_2_mappers.py rename to scripts/chapter_3_pixelizations/tutorial_2_mappers.py index 085b7e7..a980779 100644 --- a/scripts/chapter_4_pixelizations/tutorial_2_mappers.py +++ b/scripts/chapter_3_pixelizations/tutorial_2_mappers.py @@ -2,12 +2,12 @@ Tutorial 2: Mappers =================== -In the previous tutorial, we used a pixelization to create made a `Mapper`. However, it was not clear what a `Mapper` +In the previous tutorial, we used a pixelization to create a `Mapper`. However, it was not clear what a `Mapper` does, why it was called a mapper and whether it was mapping anything at all! Therefore, in this tutorial, we'll cover mappers in more detail. -WARNING: THHIS TUTORIAL VISUALS ARE SLIGHTLY BUGGY CURRENTLY AND WILL BE FIXED IN THE FUTURE. +WARNING: THIS TUTORIAL'S VISUALS ARE SLIGHTLY BUGGY CURRENTLY AND WILL BE FIXED IN THE FUTURE. __Contents__ @@ -71,8 +71,8 @@ ) """ -Our `Tracer` will use the same lens galaxy and source galaxy that we used to Simulate the imaging data (although, -becuase we're modeling the source with a pixel-grid, we do not pass it any light profiles. +Our `Tracer` will use the same lens galaxy and source galaxy that we used to simulate the imaging data (although, +because we're modeling the source with a pixel-grid, we do not pass the source any light profiles). """ lens_galaxy = al.Galaxy( redshift=0.5, @@ -116,8 +116,8 @@ aplt.plot_grid(grid=mapper.source_plane_mesh_grid, title="Source-Plane Mesh Grid") """ -Using the `lines=`/`positions=` overlays object we are also going to highlight specific grid coordinates certain colors, such that we -can see how they map from the image-plane to source-plane and visa versa. +Using the `positions=` overlay input we are also going to highlight specific grid coordinates in certain colors, such +that we can see how they map from the image-plane to the source-plane and vice versa. We do this by specifying their integer indexes, corresponding to the index of each data point in the image and source plane grids. These indexes are used to highlight the grid coordinates in the image and source-plane grids that map @@ -149,8 +149,8 @@ aplt.plot_grid(grid=mapper.source_plane_mesh_grid, title="Source-Plane Mesh Grid") """ -We can now make these mappings appear the other way round. That is, we can input a source-pixel index (of our -rectangular grid) and highlight how all of the (sub-)image-pixels that it contains map to the image-plane. +We can now make these mappings appear the other way round. That is, we can input a source-pixel index (of our +rectangular grid) and highlight how all of the image-pixels that it contains map to the image-plane. To make the indexes appear in the image-plane, we have to convert them from their source-plane pixel indexes to image plane image-pixel indexes using the mapper. @@ -224,7 +224,7 @@ aplt.plot_array(array=dataset.data, title="Data") """ -To create the mapper, we need to set up the masked imaging's grid as the source-plane gird via the tracer. +To create the mapper, we need to trace the masked imaging's grid to the source-plane via the tracer. """ tracer = al.Tracer(galaxies=[lens_galaxy, al.Galaxy(redshift=1.0)]) @@ -280,7 +280,7 @@ radius deviates from 1.6" (the input value of the simulated lens), what do you notice about where the points map from the centre of the source-plane (where the source-galaxy is simulated, e.g. (0.0", 0.0"))? - 2) Think about how this could help us actually model lenses. We have said we're going to reconstruct our source - galaxies on the pixel-grid. So, how does knowing how each pixel maps to the image actually help us? If you`ve not got - any bright ideas, then worry not, that exactly what we're going to cover in the next tutorial. + 2) Think about how this could help us actually model lenses. We have said we're going to reconstruct our source + galaxies on the pixel-grid. So, how does knowing how each pixel maps to the image actually help us? If you've not got + any bright ideas, then worry not, that's exactly what we're going to cover in the next tutorial. """ diff --git a/scripts/chapter_4_pixelizations/tutorial_3_inversions.py b/scripts/chapter_3_pixelizations/tutorial_3_inversions.py similarity index 89% rename from scripts/chapter_4_pixelizations/tutorial_3_inversions.py rename to scripts/chapter_3_pixelizations/tutorial_3_inversions.py index e473626..eb09720 100644 --- a/scripts/chapter_4_pixelizations/tutorial_3_inversions.py +++ b/scripts/chapter_3_pixelizations/tutorial_3_inversions.py @@ -7,7 +7,7 @@ - `Pixelization`'s: which place a pixel-grid in the source-plane. - `Mappers`'s: which describe how each source-pixel maps to one or more image pixels. -However, non of this has actually helped us fit strong lens data or reconstruct the source galaxy. This is the subject +However, none of this has actually helped us fit strong lens data or reconstruct the source galaxy. This is the subject of this tutorial, where the process of reconstructing the source's light on the pixelization is called an `Inversion`. __Contents__ @@ -63,7 +63,7 @@ ) """ -Lets create an annular mask which traces the stongly lensed source's ring of light. +Lets create an annular mask which traces the strongly lensed source's ring of light. """ mask = al.Mask2D.circular_annular( shape_native=dataset.shape_native, @@ -222,9 +222,10 @@ ) """ -Pretty great, huh? If you ran the complex source pipeline in chapter 3, you'll remember that getting a model image -that looked this good simply *was not possible*. With an inversion, we can do this with ease and without having to -perform model-fitting with 20+ parameters for the source's light! +Pretty great, huh? If you fitted complex sources using light profiles in the search chaining tutorials at the end of +chapter 2 (tutorials 9-11), you'll remember that getting a model image that looked this good simply *was not possible*. +With an inversion, we can do this with ease and without having to perform model-fitting with 20+ parameters for the +source's light! We will now briefly discuss how an inversion actually works, however the explanation I give in this tutorial will be overly-simplified. To be good at lens modeling you do not need to understand the details of how an inversion works, you @@ -249,11 +250,16 @@ 1) The mappings between every source-pixel and sets of image-pixels. 2) The flux values in every observed image-pixel, which are the values we want to fit successfully. -It turns out that with these two pieces of information we can linearly solve for the set of source-pixel fluxes that -best-fit (e.g. maximize the log likelihood) our observed image. Essentially, we set up the mappings between source and -image pixels as a large matrix and solve for the source-pixel fluxes in an analogous fashion to how you would solve a +It turns out that with these two pieces of information we can linearly solve for the set of source-pixel fluxes that +best-fit (e.g. maximize the log likelihood) our observed image. Essentially, we set up the mappings between source and +image pixels as a large matrix and solve for the source-pixel fluxes in an analogous fashion to how you would solve a set of simultaneous linear equations. This process is called a `linear inversion`. +In the language of linear algebra: the mappings form a `mapping matrix` $f$, which together with the data and +noise-map defines a linear system whose solution is the vector of source-pixel fluxes $s$. Tutorial 5 of this +chapter writes this system out in full and computes it step-by-step in code -- for now, the intuition above is all +you need. + There are three more things about a linear inversion that are worth knowing: 1) When performing fits using light profiles, we discussed how a `model_image` was generated by convolving the light @@ -265,7 +271,7 @@ the mapping between every sub-pixel and source-pixel that is computed and used to perform the inversion. This prevents aliasing effects degrading the image reconstruction. By default **PyAutoLens** uses sub-gridding of degree 4x4. - 3) The inversion`s solution is regularized. But wait, that`s what we'll cover in the next tutorial! + 3) The inversion's solution is regularized. But wait, that's what we'll cover in the next tutorial! Finally, let me show you how easy it is to fit an image with an `Inversion` using a `FitImaging` object. Instead of giving the source galaxy a light profile, we simply pass it a `Pixelization` and regularization, and pass it to a @@ -319,7 +325,8 @@ __Detailed Explanation__ -If you are interested in a more detailed description of how inversions work, then checkout the file -`autolens_workspace/*/imaging/features/pixelization/likelihood_function.ipynb` which gives a visual step-by-step -guide of the process alongside equations and references to literature on the subject. +If you are interested in a more detailed description of how inversions work, tutorial 5 of this chapter collects the +full linear algebra in one place. The file +`autolens_workspace/*/imaging/features/pixelization/likelihood_function.ipynb` gives a complementary visual +step-by-step guide of the process alongside equations and references to literature on the subject. """ diff --git a/scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py b/scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py similarity index 89% rename from scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py rename to scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py index 42e6d94..2ec0d6a 100644 --- a/scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py +++ b/scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py @@ -68,11 +68,11 @@ """ __Convenience Function__ -we're going to perform a lot of fits using an `Inversion` this tutorial. This would create a lot of code, so to keep +we're going to perform a lot of fits using an `Inversion` this tutorial. This would create a lot of code, so to keep things tidy, I've setup this function which handles it all for us. -(You may notice we include an option to `use_pixelization_border, ignore this for now, as we'll be covering borders in -the next tutorial) +(Borders, which relocate demagnified image pixels in the source-plane, are used behind the scenes here -- we'll be +covering them in tutorial 6.) """ @@ -142,12 +142,15 @@ def perform_fit_with_source_galaxy(dataset, source_galaxy): So, what is happening here? Why does reducing the `coefficient` do this to our source reconstruction? First, we need to understand what regularization actually does! -When the inversion reconstructs the source, it does not *only* compute the set of source-pixel fluxes that best-fit -the image. It also regularizes this solution, whereby it goes to every pixel on the rectangular source-plane grid -and computes the different between the reconstructed flux values of every source pixel with its 4 neighboring pixels. -If the difference in flux is large the solution is penalized, reducing its log likelihood. You can think of this as +When the inversion reconstructs the source, it does not *only* compute the set of source-pixel fluxes that best-fit +the image. It also regularizes this solution, whereby it goes to every pixel on the rectangular source-plane grid +and computes the difference between the reconstructed flux values of every source pixel with its 4 neighboring pixels. +If the difference in flux is large the solution is penalized, reducing its log likelihood. You can think of this as us applying a 'smoothness prior' on the reconstructed source galaxy's light. +(In the language of linear algebra, this smoothness prior is encoded in a `regularization matrix` $H$ which is +added to the linear system the inversion solves -- tutorial 5 writes this out in full.) + This smoothing adds a 'penalty term' to the log likelihood of an inversion which is the summed difference between the reconstructed fluxes of every source-pixel pair multiplied by the `coefficient`. By setting the regularization coefficient to zero, we set this penalty term to zero, meaning that regularization is completely omitted. @@ -279,14 +282,16 @@ def perform_fit_with_source_galaxy(dataset, source_galaxy): aplt.subplot_fit_imaging(fit=fit) """ - 2) Can you think of any other ways we might increase the Bayesian evidence even further? In future tutorials we will - learn how **PyAutoLens** can adapts the source reconstructions to the properties of the image so as to maximize the + 2) Can you think of any other ways we might increase the Bayesian evidence even further? In future tutorials we will + learn how **PyAutoLens** can adapt the source reconstruction to the properties of the image so as to maximize the Bayesian evidence! - + __Detailed Description__ Below, I provide a more detailed discussion of the Bayesian evidence. It is not paramount that you understand this to -use **PyAutoLens**, but I recommend you give it a read to get an intuition for how the evidence works. +use **PyAutoLens**, but I recommend you give it a read to get an intuition for how the evidence works. The evidence +itself is a sum of chi-squared, regularization and matrix log-determinant terms -- tutorial 5 derives the exact +expression and computes every term in code. The Bayesian log evidence quantifies the following 3 aspects of a fit to strong lens imaging data: @@ -297,10 +302,10 @@ def perform_fit_with_source_galaxy(dataset, source_galaxy): However, this raises the question of what constitutes a ‘good’ solution? The Bayesian evidence defines this by assuming that the image data consists of independent Gaussian noise in every image pixel. A ‘good’ solution is one - whose chi-squared residuals are consistent with Gaussian noise, producing a reduced chi-squared near 1.0 .Solutions - which give a reduced chi squared below 1 are penalized for being overly complex and fitting the image’s noise, whereas - solutions with a reduced chi-squared above are penalized for not invoking a more complex source model when the data it - is necessary to fit the data bettter. In both circumstances, these penalties reduce the inferred Bayesian evidence! + whose chi-squared residuals are consistent with Gaussian noise, producing a reduced chi-squared near 1.0. Solutions + which give a reduced chi-squared below 1 are penalized for being overly complex and fitting the image’s noise, whereas + solutions with a reduced chi-squared above 1 are penalized for not invoking a more complex source model when the data + necessitates it. In both circumstances, these penalties reduce the inferred Bayesian evidence! 2) *The complexity of the source reconstruction:* The log evidence estimates the number of source pixels that are used to reconstruct the image, after accounting for their correlation with one another due to regularization. Solutions that @@ -308,9 +313,9 @@ def perform_fit_with_source_galaxy(dataset, source_galaxy): reconstructions are favoured. 3) *The signal-to-noise (S/N) of the image that is fitted:* The Bayesian evidence favours models which fit higher S/N - realizations of the observed data (where the S/N is determined using the image-pixel variances, e.g. the noise-map). Up - to now, all **PyAutoLens** fits assumed fixed variances, meaning that this aspect of the Bayeisan evidence has no impact - on the inferred evidence values. However, in hyper-mode we will invoke functionality that increases the variances + realizations of the observed data (where the S/N is determined using the image-pixel variances, e.g. the noise-map). Up + to now, all **PyAutoLens** fits assumed fixed variances, meaning that this aspect of the Bayesian evidence has no impact + on the inferred evidence values. However, adaptive features can invoke functionality that increases the variances of image-pixels where the lens model fits the data poorly. The premise is that whilst increasing the variances of image pixels lowers their S/N values and therefore also diff --git a/scripts/chapter_3_pixelizations/tutorial_5_bayesian_formalism.py b/scripts/chapter_3_pixelizations/tutorial_5_bayesian_formalism.py new file mode 100644 index 0000000..2a3b757 --- /dev/null +++ b/scripts/chapter_3_pixelizations/tutorial_5_bayesian_formalism.py @@ -0,0 +1,645 @@ +""" +Tutorial 5: Bayesian Formalism +============================== + +In tutorials 1 to 4, we built an intuition for how pixelized source reconstruction works: pixelizations place a +pixel-grid in the source-plane, mappers pair source-pixels with image-pixels, inversions solve for the source-pixel +fluxes that best fit the data, and regularization smooths the solution within a Bayesian framework. + +This tutorial collects the linear algebra behind all of that in one place. It is the counterpart of chapter 1's +tutorial 5 on the lensing formalism: the hands-on tutorials built intuition, and now we write down the equations. +We will construct, step-by-step and in code, every matrix and vector the inversion uses, solve for the source +reconstruction ourselves and compute the Bayesian evidence by hand, comparing our answer at the end to the +`FitImaging` object which performs this calculation internally. + +None of this is required to *use* **PyAutoLens** for lens modeling. However, if you publish results which use a +pixelization, this is the calculation your paper's likelihood function section will describe, and understanding it +removes any sense that the source reconstruction is a "black box". + +The formalism follows Warren & Dye 2003 (https://arxiv.org/abs/astro-ph/0302587), hereafter WD03, with the data +vector notation of Nightingale & Dye 2015 (https://arxiv.org/abs/1412.7436), hereafter N15. The Bayesian evidence +was derived by Suyu et al. 2006 (https://arxiv.org/abs/astro-ph/0601493) and translated to **PyAutoLens** notation +in Dye et al. 2008 (https://arxiv.org/abs/0804.4002). + +__Contents__ + +- **Initial Setup:** Load the dataset, mask it and disable over sampling so the algebra stays simple. +- **Mesh Shape:** Fix the rectangular mesh's shape and note why edge pixels are zeroed. +- **Ray Tracing:** Trace the masked image-pixel coordinates to the source-plane via the lens equation. +- **Border Relocation:** Relocate demagnified coordinates that trace far outside the source-plane border. +- **Source Pixel Centres:** Overlay the rectangular mesh over the traced coordinates. +- **Interpolation:** Pair every traced image-pixel with source pixels via bilinear interpolation. +- **Mapper:** Package the interpolation into a `Mapper` describing all image-to-source mappings. +- **Mapping Matrix:** Express the mappings as the 2D matrix $f$. +- **Blurred Mapping Matrix:** Convolve every column of $f$ with the imaging PSF. +- **Data Vector:** Compute the data vector $D$ from the blurred mapping matrix, data and noise-map. +- **Curvature Matrix:** Compute the curvature matrix $F$. +- **Unregularized Solve:** Solve $s = F^{-1} D$ and see the over-fitted mess this produces. +- **Regularization Matrix:** Compute the regularization matrix $H$ encoding the smoothness prior. +- **Source Reconstruction:** Solve the regularized system $s = [F + H]^{-1} D$. +- **Image Reconstruction:** Map the reconstruction back to the image-plane via the blurred mapping matrix. +- **Likelihood Function:** The five terms which combine into the log evidence. +- **Chi Squared:** The goodness-of-fit of the reconstructed image to the data. +- **Regularization Term:** The penalty $s^{T} H s$ applied by the smoothness prior. +- **Complexity Terms:** The log determinant terms which penalize complex source reconstructions. +- **Noise Normalization Term:** The Gaussian noise normalization. +- **Log Evidence:** Combine all five terms into the log evidence. +- **Fit:** Compare our by-hand log evidence to the `FitImaging` object's internal calculation. +- **Wrap Up:** Summary and next steps. + +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +import matplotlib.pyplot as plt +import numpy as np +from pathlib import Path + +import autolens as al +import autolens.plot as aplt + +""" +__Initial Setup__ + +we'll use the same strong lensing data as the previous tutorials, where: + + - The lens galaxy's light is omitted. + - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. + - The source galaxy's light is an `Sersic`. + +Because the lens galaxy's light is omitted, the data is already "lens subtracted". WD03 make the same assumption, +so in the equations below the lens light model $b_{j}$ is zero everywhere. For a lens with light, one simply computes +the lens galaxy's PSF-convolved image first and subtracts it from the data before the steps below (the workspace +guide referenced at the end shows this in full). +""" +dataset_name = "simple__no_lens_light" +dataset_path = Path("dataset") / "imaging" / dataset_name + +""" +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it will be created by running the corresponding +simulator script. This ensures that all example scripts can be run without manually simulating data first. +""" +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/no_lens_light.py"], + check=True, + ) + +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Mask__ + +The likelihood is only evaluated within a mask, which we define as a 3.0" circle containing all of the lensed +source's light. +""" +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=3.0, +) + +masked_dataset = dataset.apply_mask(mask=mask) + +aplt.subplot_imaging_dataset(dataset=masked_dataset) + +""" +__Over Sampling__ + +Over sampling splits every image pixel into a sub-grid of sub-pixels, each of which is traced to the source-plane +and paired with source pixels individually. It improves accuracy, but it also multiplies the number of rows in every +matrix below by the number of sub-pixels. + +To keep the algebra as easy to follow as possible, we disable over sampling by setting both sub-grid sizes to 1, so +each image pixel is represented by the single coordinate at its centre. +""" +masked_dataset = masked_dataset.apply_over_sampling( + over_sample_size_lp=1, + over_sample_size_pixelization=1, +) + +""" +__Mesh Shape__ + +The `mesh_shape` defines the number of pixels in the rectangular mesh used to reconstruct the source, set below +to 20 x 20 = 400 source pixels. + +We use the `RectangularUniform` mesh, where all rectangular source pixels have the same size, rather than the +`RectangularAdaptDensity` mesh used in the previous tutorials. The uniform mesh keeps the geometry simple, and every +equation below applies unchanged to the adaptive meshes -- only the source pixel centres move. + +By default, source pixels at the edge of the mesh are forced to solutions of zero flux by the linear algebra solver. +This prevents unphysical solutions where the mesh edge lights up to fit residuals, and does not change any of the +formalism below. +""" +mesh_pixels_yx = 20 +mesh_shape = (mesh_pixels_yx, mesh_pixels_yx) + +""" +__Tracer__ + +We use the same lens galaxy mass model as the previous tutorials (an `Isothermal` plus `ExternalShear`, the true +model of the simulated data) and a source galaxy whose `Pixelization` pairs the `RectangularUniform` mesh with +`Constant` regularization (whose role appears later, when we reach the matrix $H$). +""" +lens_galaxy = al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=1.6, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0), + ), + shear=al.mp.ExternalShear(gamma_1=0.05, gamma_2=0.05), +) + +pixelization = al.Pixelization( + mesh=al.mesh.RectangularUniform(shape=mesh_shape), + regularization=al.reg.Constant(coefficient=1.0), +) + +source_galaxy = al.Galaxy(redshift=1.0, pixelization=pixelization) + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) + +""" +__Ray Tracing__ + +Every 2D (y,x) image-plane coordinate $\theta$ is ray-traced to its source-plane coordinate $\beta$ by subtracting +the deflection angles $\alpha$ of the mass profiles, via the lens equation we met in chapter 1: + + $\beta = \theta - \alpha(\theta)$ + +The pixelization uses its own grid of coordinates, `masked_dataset.grids.pixelization`, with one coordinate at the +centre of every masked image pixel (because we disabled over sampling above). We trace this grid to the source-plane, +taking the final entry of the traced grid list (the source-plane). +""" +traced_grid_pixelization = tracer.traced_grid_2d_list_from( + grid=masked_dataset.grids.pixelization +)[-1] + +aplt.plot_grid(grid=traced_grid_pixelization, title="Traced Source-Plane Grid") + +""" +__Border Relocation__ + +Coordinates which pass close to the mass profile's centre are heavily demagnified and can trace to the far outskirts +of the source-plane. + +We relocate these coordinates to the edge of the source-plane border (defined via the border of the image-plane +mask), exactly as tutorial 6 explains in detail (see also figure 2 of https://arxiv.org/abs/1708.07377). This stops +a handful of demagnified coordinates stretching the mesh over a huge, empty region of the source-plane. +""" +from autoarray.inversion.mesh.border_relocator import BorderRelocator + +border_relocator = BorderRelocator(mask=masked_dataset.mask, sub_size=1) + +relocated_grid = border_relocator.relocated_grid_from(grid=traced_grid_pixelization) + +aplt.plot_grid(grid=relocated_grid, title="Relocated Source-Plane Grid") + +""" +__Source Pixel Centres__ + +To reconstruct the source on a `RectangularUniform` mesh, we need the centres of its rectangular source pixels. + +We compute these by overlaying a uniform rectangular grid over the relocated traced grid, sized so the mesh fully +contains the region of the source-plane the traced image-pixels cover, without wasted edge pixels beyond it. +""" +from autoarray.inversion.mesh.mesh.rectangular_adapt_density import overlay_grid_from + +mesh_grid = overlay_grid_from( + shape_native=mesh_shape, grid=al.Grid2DIrregular(relocated_grid) +) + +""" +__Interpolation__ + +We now combine the two grids to create an `Interpolator`, which describes how every traced image-pixel coordinate +maps to the rectangular mesh pixels. + +For a rectangular mesh the scheme is bilinear interpolation: every image pixel maps to the rectangular pixel it +lands in *and* its three nearest neighbours, with weights set by how close the coordinate is to each pixel centre. +Interpolation is what lets the mesh reconstruct smooth source morphologies. We can print the mappings and weights of +the first image pixel to confirm it pairs with four source pixels. +""" +interpolator = pixelization.mesh.interpolator_from( + source_plane_data_grid=relocated_grid, + source_plane_mesh_grid=mesh_grid, +) + +print(interpolator.mappings[0]) +print(interpolator.weights[0]) + +""" +__Mapper__ + +The interpolator creates a `Mapper`, the object tutorials 1 and 2 introduced. It packages up the mapping between +every image pixel and every rectangular source pixel. + +The key attribute is `pix_indexes_for_sub_slim_index`, mapping every image-pixel index (`sub_slim_index`) to the +source-pixel indexes (`pix_indexes`) it interpolates onto, alongside the number of mappings per image pixel and +their interpolation weights. +""" +mapper = al.Mapper(interpolator=interpolator) + +pix_indexes_for_sub_slim_index = mapper.pix_indexes_for_sub_slim_index + +print(pix_indexes_for_sub_slim_index[0:9]) +print(mapper.pix_sizes_for_sub_slim_index[0:9]) +print(mapper.pix_weights_for_sub_slim_index[0:9]) + +""" +__Mapping Matrix__ + +The `mapping_matrix` expresses these image-pixel to source-pixel mappings as a single 2D matrix, with +dimensions `(total_image_pixels, total_source_pixels)`. + +Each column is the "image" of one source pixel: entries are the interpolation weights for image pixels which map to +that source pixel and zero everywhere else. +""" +mapping_matrix = al.util.mapper.mapping_matrix_from( + pix_indexes_for_sub_slim_index=pix_indexes_for_sub_slim_index, + pix_size_for_sub_slim_index=mapper.pix_sizes_for_sub_slim_index, + pix_weights_for_sub_slim_index=mapper.pix_weights_for_sub_slim_index, + pixels=mapper.pixels, + total_mask_pixels=mapper.source_plane_data_grid.mask.pixels_in_mask, + slim_index_for_sub_slim_index=mapper.slim_index_for_sub_slim_index, + sub_fraction=mapper.over_sampler.sub_fraction, +) + +plt.imshow(mapping_matrix, aspect=(mapping_matrix.shape[1] / mapping_matrix.shape[0])) +plt.show() +plt.close() + +""" +Because each column is an image of zeros and interpolation weights, we can plot any column as a 2D image showing all +image pixels its source pixel maps to. For a source pixel near the mesh centre these image pixels trace the multiple +images of that patch of the source. +""" +source_pixel_index = 210 + +array_2d = al.Array2D( + values=mapping_matrix[:, source_pixel_index], mask=masked_dataset.mask +) + +aplt.plot_array(array=array_2d, title="Image of Source Pixel 210") + +""" +__Blurred Mapping Matrix__ + +The imaging data is blurred by the telescope's PSF, so the model must be too. Since each column of the mapping +matrix is an image, we simply convolve each column with the PSF via 2D convolution. + +This produces the `blurred_mapping_matrix`, with the same dimensions `(total_image_pixels, total_source_pixels)`. + +In WD03 this matrix is denoted $f_{ij}$, where $i$ runs over all $I$ source pixels and $j$ over all $J$ image +pixels. For example: + + - $f_{0, 2} = 0.3$ indicates that image-pixel $2$ maps to source-pixel $0$ with a weight of $0.3$ after PSF + convolution. + - $f_{4, 8} = 0$ indicates that image-pixel $8$ does not map to source-pixel $4$, even after PSF convolution. + +(The indexing of the code's `mapping_matrix` is transposed relative to WD03's $f$: image pixels are the first index +in the code, but the second index in the equations.) + +PSF blurring has an important consequence: whereas before, no two source pixels shared an image pixel, blurring +correlates neighbouring columns, so the images of nearby source pixels now overlap. +""" +blurred_mapping_matrix = masked_dataset.psf.convolved_mapping_matrix_from( + mapping_matrix=mapping_matrix, mask=masked_dataset.mask +) + +plt.imshow( + blurred_mapping_matrix, + aspect=(blurred_mapping_matrix.shape[1] / blurred_mapping_matrix.shape[0]), +) +plt.colorbar() +plt.show() +plt.close() + +""" +__Data Vector (D)__ + +We now pose the reconstruction as a linear inversion, converting the blurred mapping matrix, data and noise-map +into two objects: the data vector $D$ and the curvature matrix $F$. + +The data vector has dimensions `(total_source_pixels,)` and is given by (WD03 / N15): + + $\vec{D}_{i} = \sum_{j=1}^{J} f_{ij} (d_{j} - b_{j}) / \sigma_{j}^2 \, \, .$ + +Where: + + - $d_{j}$ are the image-pixel data values. + - $b_{j}$ are the lens light model values ($d_{j} - b_{j}$ is the lens-subtracted image; zero for this dataset). + - $\sigma_{j}^2$ are the statistical uncertainties of each image pixel. + +Each entry of $D$ is therefore the noise-weighted overlap between one source pixel's blurred image and the data: +it measures how much evidence the data provides for flux in that source pixel, with the PSF fully accounted for. +""" +data_vector = al.util.inversion_imaging.data_vector_via_blurred_mapping_matrix_from( + blurred_mapping_matrix=blurred_mapping_matrix, + image=np.array(masked_dataset.data), + noise_map=np.array(masked_dataset.noise_map), +) + +plt.imshow( + data_vector.reshape(data_vector.shape[0], 1), aspect=10.0 / data_vector.shape[0] +) +plt.colorbar() +plt.show() +plt.close() + +""" +__Curvature Matrix (F)__ + +The curvature matrix has dimensions `(total_source_pixels, total_source_pixels)` and is given by (WD03): + + ${F}_{ik} = \sum_{j=1}^{J} f_{ij} f_{kj} / \sigma_{j}^2 \, \, .$ + +Every entry of $F$ is the noise-weighted overlap between the blurred images of two source pixels: $F_{ik}$ sums the +product of columns $i$ and $k$ of $f$ over all image pixels. For $F_{ik}$ to be non-zero, source pixels $i$ and $k$ +must share at least one image pixel, which (interpolation aside) only happens because of PSF blurring. + +$F$ describes how degenerate pairs of source pixels are with one another: two source pixels whose blurred images +overlap heavily can trade flux between themselves whilst fitting the data almost equally well. +""" +curvature_matrix = al.util.inversion.curvature_matrix_via_mapping_matrix_from( + mapping_matrix=blurred_mapping_matrix, noise_map=masked_dataset.noise_map +) + +plt.imshow(curvature_matrix) +plt.colorbar() +plt.show() +plt.close() + +""" +__Unregularized Solve__ + +The inversion seeks the source-pixel fluxes $s$ (a vector with one entry per source pixel) that minimize the +chi-squared: + + $\chi^2 = \sum_{j=1}^{J} \bigg[ \frac{(\sum_{i=1}^{I} s_{i} f_{ij}) + b_{j} - d_{j}}{\sigma_{j}} \bigg]^2$ + +Setting the derivative of $\chi^2$ with respect to each $s_{i}$ to zero gives the linear system whose solution is +(equation 5 of WD03): + + $s = F^{-1} D$ + +We can solve this directly with NumPy. (Without regularization the curvature matrix is often singular, so the loop +below adds a tiny value to its diagonal to avoid a `LinAlgError` -- it is a numerical crutch, not part of the +formalism.) +""" +for i in range(curvature_matrix.shape[0]): + curvature_matrix[i, i] += 1e-8 + +reconstruction = np.linalg.solve(curvature_matrix, data_vector) + +print(reconstruction) + +""" +The reconstructed source-pixel fluxes are a noisy, unsmooth mess -- exactly the over-fitting we saw in tutorial 4 +when we set the regularization coefficient to zero. The linear inversion is fitting the noise in the data, because +this system of equations is ill-posed: we need a smoothness prior. + +__Regularization Matrix (H)__ + +Regularization adds a linear regularization term $G_{L}$ to the merit function we minimize (equation 11 of WD03): + + $G = \chi^2 + \lambda \, G_{L}$ + +where $\lambda$ is the `regularization_coefficient` controlling the degree of smoothing. The `Constant` scheme uses +gradient regularization (equation 14 of WD03): + + $G_{L} = \sum_{i}^{I} \sum_{n=1}^{N} [s_{i} - s_{i, n}]^2$ + +In words: for every source pixel, compare its flux with each of its $N$ neighbours $n$, and penalize solutions where +the differences are large. This is precisely the "smoothness prior" of tutorial 4, now written as an equation. + +To fold this into the linear algebra we define the regularization matrix $H$, with +dimensions `(total_source_pixels, total_source_pixels)` (equation 13 of WD03): + + $H_{ik} = \frac{1}{2} \frac{\partial^{2} G_{L}}{\partial s_{i} \partial s_{k}}$ + +$H$ has the coefficient $\lambda$ folded into it. Its non-zero off-diagonal entries mark pairs of source pixels +which are neighbours and therefore regularized with one another; most entries are zero because most source pixels +are not neighbours. +""" +regularization_matrix = al.util.regularization.constant_regularization_matrix_from( + coefficient=source_galaxy.pixelization.regularization.coefficient, + neighbors=mapper.neighbors, + neighbors_sizes=mapper.neighbors.sizes, +) + +plt.imshow(regularization_matrix) +plt.colorbar() +plt.show() +plt.close() + +""" +__Source Reconstruction (s)__ + +$H$ enters the linear system as follows (equation 12 of WD03): + + $s = [F + H]^{-1} D$ + +We add the two matrices and solve again. The diagonal jitter used above is no longer needed, because $H$ makes the +system well-posed. +""" +curvature_reg_matrix = np.add(curvature_matrix, regularization_matrix) + +reconstruction = np.linalg.solve(curvature_reg_matrix, data_vector) + +print(reconstruction) + +""" +The reconstructed fluxes are now smooth and physical: regularization has suppressed the noisy solution and the +reconstruction actually looks like a galaxy, without over-fitting the noise. + +__Image Reconstruction__ + +Using the reconstructed source-pixel fluxes, we map the source back to the image-plane via the blurred mapping +matrix (so the reconstructed image includes PSF blurring) to produce the model image of the lensed source. +""" +mapped_reconstructed_data = ( + al.util.inversion.mapped_reconstructed_data_via_mapping_matrix_from( + mapping_matrix=blurred_mapping_matrix, reconstruction=reconstruction + ) +) + +mapped_reconstructed_data = al.Array2D( + values=mapped_reconstructed_data, mask=masked_dataset.mask +) + +aplt.plot_array(array=mapped_reconstructed_data, title="Reconstructed Image") + +""" +__Likelihood Function__ + +We now quantify the goodness-of-fit of the source reconstruction, computing the quantity tutorial 4 called the +Bayesian evidence. The log evidence consists of five terms: + + $-2 \, \mathrm{ln} \, \epsilon = \chi^2 + s^{T} H s + \mathrm{ln} \, [ \mathrm{det} (F + H) ] - \mathrm{ln} \, [ \mathrm{det} (H) ] + \sum_{j=1}^{J} \mathrm{ln} \, [2 \pi (\sigma_{j})^2 ] \, .$ + +This expression was first derived by Suyu et al. 2006 (https://arxiv.org/abs/astro-ph/0601493), equation (19), and +is given in **PyAutoLens** notation by Dye et al. 2008 (https://arxiv.org/abs/0804.4002), equation (5). + +We now compute each term in turn. + +__Chi Squared__ + +The first term is the $\chi^2$ statistic from the merit function above, computed as: + + - `model_data` = the reconstructed image of the lensed source (plus the lens light model, zero here). + - `residual_map` = (`data` - `model_data`) + - `normalized_residual_map` = (`data` - `model_data`) / `noise_map` + - `chi_squared_map` = (`normalized_residual_map`) ** 2.0 + - `chi_squared` = sum(`chi_squared_map`) + +High chi-squared values indicate image pixels the reconstruction fits poorly, lowering the likelihood. +""" +model_image = mapped_reconstructed_data + +residual_map = masked_dataset.data - model_image +normalized_residual_map = residual_map / masked_dataset.noise_map +chi_squared_map = normalized_residual_map**2.0 + +chi_squared = np.sum(chi_squared_map) + +print(chi_squared) + +""" +__Regularization Term__ + +The second term, $s^{T} H s$, is the $\lambda \, G_{L}$ regularization penalty evaluated at the solution: the summed +difference in flux between all neighbouring source pixels, weighted by the regularization coefficient (which is +already folded into $H$). + +Less smooth solutions have larger values of this term and therefore lower likelihoods. +""" +regularization_term = np.matmul( + reconstruction.T, np.matmul(regularization_matrix, reconstruction) +) + +print(regularization_term) + +""" +__Complexity Terms__ + +Up to this point, nothing has justified our choice of `regularization_coefficient=1.0`. We cannot choose it using +the two terms above, because increasing the coefficient smooths the solution more, which *both* worsens the +chi-squared *and* (for a fixed solution) raises the regularization penalty. Optimizing those two terms alone would +drive the coefficient to zero and put us right back at the over-fitted mess. + +The two log determinant terms, $\mathrm{ln} \, [ \mathrm{det} (F + H) ]$ and $- \mathrm{ln} \, [ \mathrm{det} (H) ]$, +fix this. Together they measure how *complex* the source reconstruction is -- roughly, how many effective degrees +of freedom the source uses after regularization correlates its pixels -- and penalize more complex solutions. +Lowering the regularization coefficient frees the source to use more of its flexibility, increasing this complexity +penalty. + +These terms therefore counteract the chi-squared and regularization terms, so the highest evidence goes to solutions +which fit the data well with the *simplest* source reconstruction. This is the Occam's razor behaviour that +tutorial 4 demonstrated empirically. +""" +log_curvature_reg_matrix_term = np.linalg.slogdet(curvature_reg_matrix)[1] +log_regularization_matrix_term = np.linalg.slogdet(regularization_matrix)[1] + +print(log_curvature_reg_matrix_term) +print(log_regularization_matrix_term) + +""" +__Noise Normalization Term__ + +The likelihood function assumes the imaging data consists of independent Gaussian noise in every image pixel, and +the final term is the normalization of those Gaussians: the sum of the log of every noise-map value squared. + +Because the noise-map is fixed, this term is constant throughout lens modeling and has no impact on the model +we infer -- it simply normalizes the likelihood. +""" +noise_normalization = float(np.sum(np.log(2 * np.pi * masked_dataset.noise_map**2.0))) + +print(noise_normalization) + +""" +__Log Evidence__ + +We can now combine the five terms into the log evidence of the source reconstruction. +""" +log_evidence = float( + -0.5 + * ( + chi_squared + + regularization_term + + log_curvature_reg_matrix_term + - log_regularization_matrix_term + + noise_normalization + ) +) + +print(log_evidence) + +""" +__Fit__ + +Everything above is what the `FitImaging` object does internally when it fits a tracer whose source galaxy has a +pixelization. We can see this by performing the fit and comparing its `log_evidence` to ours. + +The two values are close but not identical, because the real fit improves on our simplified solve in two ways +mentioned along the way: it uses the positive-only solver (tutorial 3), which forbids the negative source-pixel +fluxes our unconstrained `np.linalg.solve` permits, and it zeroes the pixels at the edge of the mesh. Neither +changes the formalism -- the same $f$, $D$, $F$ and $H$ feed a solver with extra constraints. +""" +fit = al.FitImaging( + dataset=masked_dataset, + tracer=tracer, + settings=al.Settings(use_border_relocator=True), +) + +print(fit.log_evidence) + +aplt.subplot_fit_imaging(fit=fit) + +""" +__Wrap Up__ + +We have walked through the complete linear algebra of a pixelized source reconstruction: + + - The `mapping_matrix` and PSF-blurred mapping matrix $f$, whose columns are the blurred images of each + source pixel. + + - The data vector $D$ and curvature matrix $F$, the noise-weighted overlaps of those images with the data and with + each other. + + - The regularization matrix $H$, which encodes the smoothness prior, and the linear solve $s = [F + H]^{-1} D$ for + the source reconstruction. + + - The five terms of the Suyu et al. 2006 log evidence -- chi-squared, the regularization penalty, the two log + determinant complexity terms and the noise normalization -- and their Bayesian interpretation as an Occam's + razor which favours the simplest source reconstruction the data allows. + +During lens modeling, this whole calculation is one likelihood evaluation: the non-linear search varies the mass +model (and, later in this chapter, the pixelization and regularization parameters), and each sample triggers the +full ray-trace, solve and evidence computation above. + +Two simplifications are worth remembering: real fits use over sampling (each image pixel contributes several +sub-pixel rows to $f$) and **PyAutoLens** uses a positive-only solver for $s$ rather than the unconstrained +`np.linalg.solve` used here (see tutorial 3). The workspace +guide `autolens_workspace/*/imaging/features/pixelization/likelihood_function.ipynb` repeats this walk-through with +lens light included and additional visualization of every step. + +In the next tutorial, we return to hands-on territory and look at borders, which deal with the demagnified traced +coordinates whose relocation we performed in a single line above. +""" diff --git a/scripts/chapter_4_pixelizations/tutorial_5_borders.py b/scripts/chapter_3_pixelizations/tutorial_6_borders.py similarity index 83% rename from scripts/chapter_4_pixelizations/tutorial_5_borders.py rename to scripts/chapter_3_pixelizations/tutorial_6_borders.py index fc8904f..bd72bd5 100644 --- a/scripts/chapter_4_pixelizations/tutorial_5_borders.py +++ b/scripts/chapter_3_pixelizations/tutorial_6_borders.py @@ -1,5 +1,5 @@ """ -Tutorial 5: Borders +Tutorial 6: Borders =================== In the previous tutorials, the source-plane pixel grid perfectly mapped over the traced image-pixel $(y,x)$ coordinates @@ -7,7 +7,8 @@ increase its size so as to cover every source-plane coordinate. In this tutorial, we will consider how the size of the pixelization grid is chosen and introduce the concept of a -border. +border. The previous tutorial applied border relocation as a single step of the likelihood algebra -- here we see +why that step is needed. __Contents__ @@ -68,7 +69,7 @@ So, what is a border? In the image-plane, a border is the set of exterior pixels in a mask that are at its border. -Lets plot the image with a circular circular and tell our `aplt.subplot_imaging_dataset` to plot the border. +Lets apply a circular mask to the image and plot the dataset -- the border is the ring of pixels at the mask's edge. """ mask_circular = al.Mask2D.circular( shape_native=dataset.shape_native, @@ -83,8 +84,8 @@ aplt.subplot_imaging_dataset(dataset=dataset) """ -As you can see, for a circular mask the border *is* the edge of the mask (the ring of black dots we're used to -seeing whenever we plot a mask. +As you can see, for a circular mask the border *is* the edge of the mask (the ring of black dots we're used to +seeing whenever we plot a mask). For an annular mask, pixels on its inner edge are not a part of the border, whereas those on its outer edge are. """ @@ -133,10 +134,9 @@ def perform_fit_with_source_galaxy_mask_and_border( """ -Okay, so lets first look at the mapper without using a border and using annular mask. +Okay, so lets first look at the mapper without using a border, using the annular mask. -First, note how we set up the border, using a `al.Settings` object. This behaves analogously to the -`SettingsLens` objects we have used in previous tutorials. +Note how we control the border via the `al.Settings` object's `use_border_relocator` input. """ pixelization = al.Pixelization( mesh=al.mesh.RectangularAdaptDensity(shape=(40, 40)), @@ -191,12 +191,12 @@ def perform_fit_with_source_galaxy_mask_and_border( So, why is this happening? What is the mass profile physically doing to create these source plane coordinates at extremely large radial values? -Towards the centre of th elliptical isothermal mass profile, the density begins to rise very sharply, it becomes -extremely steep or 'cuspy'. This cuspy behaviour towards its centre can cause extremely large deflection angles to be -calculated: +Towards the centre of the elliptical isothermal mass profile, the density begins to rise very sharply, it becomes +extremely steep or 'cuspy'. This cuspy behaviour towards its centre causes extremely large deflection angles to be +calculated. -Central image pixel can therefore be subjected to 'demagnification', whereby they trace to extremely large values in -the source plane! +Central image pixels are therefore subjected to 'demagnification', whereby they trace to extremely large values in +the source plane! Physically, this is not a problem, and it is the reason we do not see a 'central image' in most strong lenses, as the light-rays which take this path through the centre of the lens are demagnified. However, if the lens galaxy had a less @@ -205,15 +205,15 @@ def perform_fit_with_source_galaxy_mask_and_border( Demagnification is a problem for the pixelization and inversion though, which reconstruct the flux of these demagnified pixels just like the other pixels in the image-pixel. There are two negative consequences: - 1) The rectangular pixel-grid that we 'lay over' the source-plane is very larger because it expands to include the - demagnified image-pixels. As a result, larger source-pixels are used to reconstruct the central regions of the - source-plane (where the source galaxy is actually located), meaning we reconstruct the source-galaxy at a lower + 1) The rectangular pixel-grid that we 'lay over' the source-plane becomes very large, because it expands to include + the demagnified image-pixels. As a result, larger source-pixels are used to reconstruct the central regions of the + source-plane (where the source galaxy is actually located), meaning we reconstruct the source-galaxy at a lower effective resolution. - - 2) The inversion reconstructs the flux of the demanigified image pixels using source-pixels which contain *only* - demagnified image pixels (these are the source pixels at the edge of the source plane). These source-pixels *should* - have had other image-pixels traced within them via image-pixels at even larger radii from the centre of the lens - galaxy. However, these image-pixels are at radii above 3.0", meaning the circular mask removed them from the inversion. + + 2) The inversion reconstructs the flux of the demagnified image pixels using source-pixels which contain *only* + demagnified image pixels (these are the source pixels at the edge of the source plane). These source-pixels *should* + have had other image-pixels traced within them via image-pixels at even larger radii from the centre of the lens + galaxy. However, these image-pixels are at radii above 2.5", meaning the circular mask removed them from the inversion. Lets quickly use a large circular mask to confirm that these pixels exist when we don't mask them. """ @@ -226,7 +226,7 @@ def perform_fit_with_source_galaxy_mask_and_border( fit = perform_fit_with_source_galaxy_mask_and_border( dataset=dataset_unmasked, source_galaxy=source_galaxy, - mask=mask_circular, + mask=mask_circular_large, settings=al.Settings(use_border_relocator=False), ) @@ -270,7 +270,7 @@ def perform_fit_with_source_galaxy_mask_and_border( and lensing configurations produce nasty edge effects where image pixels not just in the centre of mask, but anywhere in the mask, trace beyond the source-plane border. -we'll use new strong lensing data as the previous tutorial, where: +we'll use new strong lensing data, where: - The lens galaxy's light is omitted. - There are two lens galaxies whose `MassProfile`'s are `Isothermal`. @@ -298,11 +298,11 @@ def perform_fit_with_source_galaxy_mask_and_border( data_path=dataset_path / "data.fits", noise_map_path=dataset_path / "noise_map.fits", psf_path=dataset_path / "psf.fits", - pixel_scales=0.05, + pixel_scales=0.1, ) """ -We again must define a mask around this image, lets start with a 2.8" mask. we'll use larger masks to illustrate the +We again must define a mask around this image, lets start with a 2.8" mask. we'll vary the mask size to illustrate the effects of the border in a moment. """ dataset_unmasked_x2 = dataset @@ -340,7 +340,7 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( sersic_index=2.5, ), mass=al.mp.Isothermal( - centre=(1.1, 0.51), ell_comps=(0.0, 0.15), einstein_radius=1.07 + centre=(0.0, -1.0), ell_comps=(0.17647, 0.0), einstein_radius=1.0 ), ) @@ -354,7 +354,7 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( sersic_index=3.0, ), mass=al.mp.Isothermal( - centre=(-0.20, -0.35), ell_comps=(0.06, 0.1053), einstein_radius=0.71 + centre=(0.0, 1.0), ell_comps=(0.0, -0.111111), einstein_radius=0.8 ), ) @@ -376,10 +376,6 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( mapper = fit.inversion.cls_list_from(al.Mapper)[0] -border = mapper.source_plane_data_grid.over_sampled[ - fit.dataset.grids.border_relocator.sub_border_slim -] - aplt.plot_array(array=fit.model_data, title="Plane 1 Image") @@ -396,22 +392,17 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( mapper = fit.inversion.cls_list_from(al.Mapper)[0] -border = mapper.source_plane_data_grid.over_sampled[ - fit.dataset.grids.border_relocator.sub_border_slim -] - aplt.plot_array(array=fit.model_data, title="Plane 1 Image") """ -Multi-galaxy modeling is rife for border effects and if you have multiple lens galaxies I heartily recommend you pay -a close eye to your source-plane borders! +Multi-galaxy modeling is rife with border effects, and if you have multiple lens galaxies I heartily recommend you +keep a close eye on your source-plane borders! -Care must also be taken when choosing the size of your mask. If you don't choose a big enough mask, the border won't -be able to relocate all of the demanigified image pixels to the border edge. +Care must also be taken when choosing the size of your mask. If you don't choose a big enough mask, the border won't +be able to relocate all of the demagnified image pixels to the border edge. -(The figures below look pretty horrible, because every ray-traced image coordinate is being plotted in the -source plane. Therefore, there are many black dots which overwhelm the figure. The point to focus on are the -edges of the grid, where one can see the relocations of these coordinates. +The fits below repeat the analysis with progressively larger circular masks, so you can see how the mask size +changes the border and the resulting model image. """ dataset = al.Imaging.from_fits( data_path=dataset_path / "data.fits", @@ -433,10 +424,6 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( mapper = fit.inversion.cls_list_from(al.Mapper)[0] -border = mapper.source_plane_data_grid.over_sampled[ - fit.dataset.grids.border_relocator.sub_border_slim -] - aplt.plot_array(array=fit.model_data, title="Plane 1 Image") @@ -454,10 +441,6 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( mapper = fit.inversion.cls_list_from(al.Mapper)[0] -border = mapper.source_plane_data_grid.over_sampled[ - fit.dataset.grids.border_relocator.sub_border_slim -] - aplt.plot_array(array=fit.model_data, title="Plane 1 Image") @@ -482,10 +465,6 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( mapper = fit.inversion.cls_list_from(al.Mapper)[0] -border = mapper.source_plane_data_grid.over_sampled[ - fit.dataset.grids.border_relocator.sub_border_slim -] - aplt.plot_array(array=fit.model_data, title="Plane 1 Image") @@ -511,10 +490,6 @@ def perform_fit_x2_lenses_with_source_galaxy_mask_and_border( mapper = fit.inversion.cls_list_from(al.Mapper)[0] -border = mapper.source_plane_data_grid.over_sampled[ - fit.dataset.grids.border_relocator.sub_border_slim -] - aplt.plot_array(array=fit.model_data, title="Plane 1 Image") diff --git a/scripts/chapter_4_pixelizations/tutorial_6_lens_modeling.py b/scripts/chapter_3_pixelizations/tutorial_7_lens_modeling.py similarity index 84% rename from scripts/chapter_4_pixelizations/tutorial_6_lens_modeling.py rename to scripts/chapter_3_pixelizations/tutorial_7_lens_modeling.py index fc65e92..78e9b50 100644 --- a/scripts/chapter_4_pixelizations/tutorial_6_lens_modeling.py +++ b/scripts/chapter_3_pixelizations/tutorial_7_lens_modeling.py @@ -1,17 +1,17 @@ """ -Tutorial 6: Lens Modeling +Tutorial 7: Lens Modeling ========================= -When modeling complex source's with parametric profiles, we quickly entered a regime where our non-linear search was +When modeling complex sources with parametric profiles, we quickly entered a regime where our non-linear search was faced with a parameter space of dimensionality N=20+ parameters. This made the model-fitting inefficient and likely to infer a local maxima. -Inversions do not suffer this problem, meaning they are a very a powerful tool for modeling strong lenses. Furthermore, -they have *more* freemdom than parametric light profiles because they do not relying on specific analytic light +Inversions do not suffer this problem, meaning they are a very powerful tool for modeling strong lenses. Furthermore, +they have *more* freedom than parametric light profiles because they do not rely on specific analytic light distributions and a symmetric profile shape. This will allow us to fit more complex mass models and ask ever more interesting scientific questions! -However, inversion do have some short comings that we need to be aware of before we use them for lens modeling. That`s +However, inversions do have some shortcomings that we need to be aware of before we use them for lens modeling. That's what we cover in this tutorial. __Contents__ @@ -100,8 +100,8 @@ def perform_fit_with_lens__source_galaxy(dataset, lens_galaxy, source_galaxy): The code below illustrates a systematic set of solutions called demagnified solutions, which negatively impact lens modeling using source pixelizations. -Since writing the code below, I have wrote a full readthedocs page illustrating the issue, which is linked too below. -I recommend you read this page first, to understand what a demagnified solution is, why its a problem and how we +Since writing the code below, I have written a full readthedocs page illustrating the issue, which is linked to below. +I recommend you read this page first, to understand what a demagnified solution is, why it is a problem and how we fix it. The code below should then build on this. https://pyautolens.readthedocs.io/en/latest/general/demagnified_solutions.html @@ -140,9 +140,9 @@ def perform_fit_with_lens__source_galaxy(dataset, lens_galaxy, source_galaxy): What happened!? This incorrect mass-model provides a really good fit to the image! The residuals and chi-squared-map are as good as the ones we saw in the previous tutorials. -How can an incorrect lens model provide such a fit? Well, as I'm sure you noticed, the source has been reconstructed -as a demagnified version of the image. Clearly, this is not a physical solution or a solution that we want our -non-linear search to find, but for inversion's the reality is these solutions eixst. +How can an incorrect lens model provide such a fit? Well, as I'm sure you noticed, the source has been reconstructed +as a demagnified version of the image. Clearly, this is not a physical solution or a solution that we want our +non-linear search to find, but for inversions the reality is these solutions exist. This is not necessarily problematic for lens modeling. Afterall, the source reconstruction above is extremely complex, it requires a lot of source pixels to fit the image accurately and its lack of smoothness will be heavily penalized @@ -179,19 +179,19 @@ def perform_fit_with_lens__source_galaxy(dataset, lens_galaxy, source_galaxy): print(correct_fit.log_evidence) """ -The `log_evidence` *is* lower. However, the difference in `log_evidence` is not *that large*. This could be a problem -for the non-linear search, as it will see many solutions in parameter space with high `log_evidence` values. Furthermore, -these solutions occupy a *large volumne* of parameter space (e.g. everywhere the lens model that is wrong). This makes -it easy for the non-linear search to get lost searching through these unphysical solutions and, unfortunately, inferring +The `log_evidence` *is* lower. However, the difference in `log_evidence` is not *that large*. This could be a problem +for the non-linear search, as it will see many solutions in parameter space with high `log_evidence` values. Furthermore, +these solutions occupy a *large volume* of parameter space (e.g. everywhere the lens model is wrong). This makes +it easy for the non-linear search to get lost searching through these unphysical solutions and, unfortunately, inferring an incorrect lens model (e.g. a local maxima). -There is no simple fix for this, and it is the price we pay for making the inversion has so much flexibility in how it -reconstructs the source's light. The solution to this problem? Search chaining. In fact, this is the problem that lead -us to initially conceive of search chaining! +There is no simple fix for this, and it is the price we pay for giving the inversion so much flexibility in how it +reconstructs the source's light. The solution to this problem? Search chaining, which we covered at the end of +chapter 2 (tutorials 9-11). In fact, this is the problem that led us to initially conceive of search chaining! The idea is simple, we write a pipeline that begins by modeling the source galaxy's light using a light profile, thereby -initializing the priors for the lens galaxy's light and mass. Then, when we switch to an `Inversion` in the next -search, the mass model starts in the correct regions of parameter space and does not get lost sampling these +initializing the priors for the lens galaxy's light and mass. Then, when we switch to an `Inversion` in the next +search, the mass model starts in the correct regions of parameter space and does not get lost sampling these incorrect solutions. The following paper discusses these solutions in more detail (https://arxiv.org/abs/2012.04665). @@ -202,8 +202,8 @@ def perform_fit_with_lens__source_galaxy(dataset, lens_galaxy, source_galaxy): simultaneously fit and subtract the lens galaxy's light using a light profile whilst reconstructing the source's light using an inversion. -To do this, all we have to do is give the lens galaxy a light profile and use the tracer and fit objects we are used -too:. +To do this, all we have to do is give the lens galaxy a light profile and use the tracer and fit objects we are +used to. """ dataset_name = "lens_sersic" dataset_path = Path("dataset") / "imaging" / dataset_name @@ -232,8 +232,6 @@ def perform_fit_with_lens__source_galaxy(dataset, lens_galaxy, source_galaxy): aplt.subplot_imaging_dataset(dataset=dataset) -aplt.subplot_fit_imaging_of_planes(fit=fit) - """ When fitting such an image we now want to include the lens's light in the analysis. Lets update our mask to be circular so that it includes the central regions of the image and lens galaxy. @@ -306,11 +304,11 @@ def perform_fit_with_lens__source_galaxy(dataset, lens_galaxy, source_galaxy): """ __Wrap Up__ -And with that, we're done. I'll end by pointing out a few things about what we've covered to get you thinking about -the next tutorial on adaption. - - - When the lens galaxy's light is subtracted perfectly it leaves no residuals. However, if it isn't subtracted - perfectly it does leave residuals, which will be fitted by the inversion. If the residual are significant this is - going to impact the source reconstruction negatively and can lead to some pretty nasty systematics. In the next - chapter, we'll learn how our adaptive analysis can prevent this residual fitting. +And with that, we're done. I'll end by pointing out a few things about what we've covered to get you thinking about +the adaption tutorials later in this chapter. + + - When the lens galaxy's light is subtracted perfectly it leaves no residuals. However, if it isn't subtracted + perfectly it does leave residuals, which will be fitted by the inversion. If the residuals are significant this is + going to impact the source reconstruction negatively and can lead to some pretty nasty systematics. In the adaption + tutorials at the end of this chapter, we'll learn how our adaptive analysis can prevent this residual fitting. """ diff --git a/scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py b/scripts/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.py similarity index 85% rename from scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py rename to scripts/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.py index 165526d..493fb5b 100644 --- a/scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py +++ b/scripts/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.py @@ -1,5 +1,5 @@ """ -Tutorial 7: Adaptive Pixelization +Tutorial 8: Adaptive Pixelization ================================= In this tutorial we will introduce a new `Pixelization` object, which uses an `Overlay` image-mesh and a `Delaunay` @@ -12,7 +12,7 @@ __Contents__ - **Initial Setup:** We'll use the same strong lensing data as the previous tutorial, where. -- **Advantages and Disadvatanges:** Lets think about the rectangular pixelization. +- **Advantages and Disadvantages:** Lets think about the rectangular pixelization. - **Image Mesh:** The Delaunay mesh is an irregular grid of pixels (or triangles) in the source-plane. - **Regularization:** On the rectangular grid, we regularized each source pixel with its 4 neighbors. - **Wrap Up:** Summary of the script and next steps. @@ -114,12 +114,12 @@ aplt.subplot_fit_imaging_of_planes(fit=fit) """ -__Advantages and Disadvatanges__ +__Advantages and Disadvantages__ -Lets think about the rectangular pixelization. Is this the optimal way to reconstruct our source? Are there features -in the source-plane that arn`t ideal? How do you think we could do a better job? +Lets think about the rectangular pixelization. Is this the optimal way to reconstruct our source? Are there features +in the source-plane that aren't ideal? How do you think we could do a better job? -There are a number of reasons the rectangular pixelization is not optimal, and is infact a pretty poor method to +There are a number of reasons the rectangular pixelization is not optimal, and it is in fact a pretty poor method to model strong lenses! So what is wrong with the grid? Well, lets think about the source reconstruction. @@ -130,10 +130,10 @@ aplt.subplot_fit_imaging_of_planes(fit=fit) """ -There is one clear problem, we are using only a small number of the total source pixels to reconstruct the source. The -majority of source pixels are located away from the source. By my estimate, we are using just 16 pixels (the central -4x4 grid) out of the 1600 pixels to actually fit the data! The remaining ~1500 pixels are doing nothing but fitting -noise. +There is one clear problem, we are using only a small number of the total source pixels to reconstruct the source. The +majority of source pixels are located away from the source. By my estimate, we are using just 16 pixels (the central +4x4 grid) out of the 784 pixels to actually fit the data! The remaining ~750 pixels are doing nothing but fitting +noise. This means that regularization is sub-optimal. In tutorial 4, we discussed how the Bayesian evidence of the regularization favours the simplest source solution. That is, the solution which fits the data using the fewest source @@ -149,7 +149,7 @@ __Image Mesh__ The Delaunay mesh is an irregular grid of pixels (or triangles) in the source-plane. We must first therefore determine -a set of (y,x) source-plane coordinates defining this grid, specifically where each triangle vertex is loated. +a set of (y,x) source-plane coordinates defining this grid, specifically where each triangle vertex is located. We do this using an `image_mesh`, which defines a method to determine a set of coordinates in the image-plane which are ray-traced to the source-plane. These traced coordinates are the triangle vertexes of our source-pixel mesh. @@ -164,7 +164,7 @@ image_plane_mesh_grid = image_mesh.image_plane_mesh_grid_from(mask=dataset.mask) """ -We can plot this grid over the image, to see that it is a coarse grid of (y,x) coordinates laid ove the image. +We can plot this grid over the image, to see that it is a coarse grid of (y,x) coordinates laid over the image. """ aplt.plot_array(array=dataset.data, title="Data") @@ -201,7 +201,7 @@ """ A closer inspection of the pixelization shows the improvement. -We are using fewer pixels than the rectangular grid (400, instead of 1600) and reconstructing the source is far +We are using fewer pixels than the rectangular grid (400, instead of 784) and reconstructing the source in far greater detail! """ aplt.subplot_fit_imaging_of_planes(fit=fit) @@ -217,13 +217,13 @@ __Wrap Up__ -The `Overlay` image-mesh and `Delaunay` mesh is still far from optimal. There are lots of source-pixels effectively f -itting just noise. We can achieve even better solutions if the central regions of the source were reconstructed using -more pixels and fewer source pixels are used in the outskirts of the source plane. +The `Overlay` image-mesh and `Delaunay` mesh is still far from optimal. There are lots of source-pixels effectively +fitting just noise. We could achieve even better solutions if the central regions of the source were reconstructed +using more pixels, and fewer source pixels were used in the outskirts of the source plane. -Tutorials 9, 10 and 11 show even more advanced and adaptive pixelizations which do just this, by adapting to the +Tutorials 10, 11 and 12 show even more advanced and adaptive pixelizations which do just this, by adapting to the source galaxy's morphology rather than the mass model magnification. -In the mean time, you may wish to experiment with using both Delaunay and Delaunay grids to fit -lenses which can be easily achieve by changing the input pixelization given to a pipeline. +In the mean time, you may wish to experiment with using both rectangular and Delaunay meshes to fit +lenses, which is easily achieved by changing the input pixelization given to a pipeline. """ diff --git a/scripts/chapter_4_pixelizations/tutorial_8_model_fit.py b/scripts/chapter_3_pixelizations/tutorial_9_model_fit.py similarity index 93% rename from scripts/chapter_4_pixelizations/tutorial_8_model_fit.py rename to scripts/chapter_3_pixelizations/tutorial_9_model_fit.py index 6d02e54..32c0ca4 100755 --- a/scripts/chapter_4_pixelizations/tutorial_8_model_fit.py +++ b/scripts/chapter_3_pixelizations/tutorial_9_model_fit.py @@ -1,5 +1,5 @@ """ -Tutorial 8: Model-Fit +Tutorial 9: Model-Fit ===================== You should now perform lens modeling using a pixelization, which is described fully in the example: diff --git a/scripts/chapter_3_search_chaining/README.md b/scripts/chapter_3_search_chaining/README.md deleted file mode 100644 index 2e2091f..0000000 --- a/scripts/chapter_3_search_chaining/README.md +++ /dev/null @@ -1,18 +0,0 @@ -In chapter 3, we introduce non-linear search chaining, whereby lens modeling pipelines are composed which each fit a -different lens model. - -**Colab** links to every tutorial are included. - -# Files - -- [Tutorial 1: Search Chaining](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb) — Breaking the lens modeling procedure into a chained sequence of model-fits. - -- [Tutorial 2: Prior Passing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb) — How the results of earlier searches are passed to later searches. - -- [Tutorial 3: Lens and Source](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_3_lens_and_source.ipynb) — Fitting the lens's light followed by its mass using chained searches. - -- [Tutorial 4: Two Lens galaxies](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.ipynb) — Modeling a strong lens with two lens galaxies using chained searches. - -- [Tutorial 5: Complex Source](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_5_complex_source.ipynb) — Using multiple light profiles to fit a complex and irregular source using chained searches. - -- [Tutorial 6: SLaM](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_3_search_chaining/tutorial_6_slam.ipynb) — Template pipelines for fitting lens model is standardized ways. diff --git a/scripts/chapter_3_search_chaining/introduction b/scripts/chapter_3_search_chaining/introduction deleted file mode 100755 index e359483..0000000 --- a/scripts/chapter_3_search_chaining/introduction +++ /dev/null @@ -1,12 +0,0 @@ -You are now familiar and have a clear understanding of lens modeling. In this chapter, we'll -introduce search chaining, a concept we mentioned in the previous chapter. Search chaining allows us to generically link -searches together, so that we can seamlessly navigate the complex non-linear parameter spaces that come when fitting -realistic lens models. - -With these pipelines, you'll be able to: - -1) Fit a lens mass model and source light model to an image of a strongly lensed source. -2) Additionally fit the lens galaxy's light, if it is present. -3) Write customized pipelines for strong lens systems with multiple lens galaxies or source galaxies. -4) Customize pipelines such that the priors on parameters during the fit are adjusted to provide a more robust or - efficient fit. \ No newline at end of file diff --git a/scripts/chapter_3_search_chaining/tutorial_3_lens_and_source.py b/scripts/chapter_3_search_chaining/tutorial_3_lens_and_source.py deleted file mode 100644 index 5b8d6ce..0000000 --- a/scripts/chapter_3_search_chaining/tutorial_3_lens_and_source.py +++ /dev/null @@ -1,359 +0,0 @@ -""" -Tutorial 3: Lens and Source -=========================== - -In this tutorial, we demonstrate search chaining using three searches to fit strong lens `Imaging` which includes the -lens galaxy's light. - -The crucial point to note is that for many lenses the lens galaxy's light can be fitted and subtracted reasonably -well before we attempt to fit the source galaxy. This makes sense, as fitting the lens's light (which is an elliptical -blob of light in the centre of the imaging) looks nothing like the source's light (which is a ring of light)! Formally, -we would say that these two model components (the lens's light and source's light) are not covariate. - -So, as a newly trained lens modeler, what does the lack of covariance between these parameters make you think? -Hopefully, you're thinking, why should I bother fitting the lens and source galaxy simultaneously? Surely we can -find the right regions of non-linear parameter space by fitting each separately first? This is what we're going to do -in this tutorial, using a pipeline composed of a modest 3 searches: - - 1) Fit the lens galaxy's light, ignoring the source. - 2) Fit the source-galaxy's light (and therefore lens galaxy's mass), ignoring the len`s light. - 3) Fit both simultaneously, using these results to initialize our starting location in parameter space. - -Of course, given that we do not care for the errors in searches 1 and 2, we will set up our non-linear search to -perform sampling as fast as possible! - -__Contents__ - -- **Dated Tutorial:** This example tutorial was written ~4 years ago, when **PyAutoLens** was in its infancy and had a. -- **Initial Setup:** we'll use strong lensing data, where. -- **Paths:** All three searches will use the same `path_prefix`, so we write it here to avoid repetition. -- **Notes:** We use linear light profiles througout this script, given that the model is quite complex and this. -- **Wrap Up:** Summary of the script and next steps. - -__Dated Tutorial__ - -This example tutorial was written ~4 years ago, when **PyAutoLens** was in its infancy and had a number of limitations: - - - The non-linear search used MultiNest or dynesty, which were less reliable (e.g. more likely to infer a local maxima - for complex lens models) and less efficient than Nautilus. - - - Linear light profiles and techniques like a Multi-Gaussian Expansion were not available. - -With all the new features added to **PyAutoLens** since, we no longer recommend that one breaks down the fitting of -the lens and source galaxy's light into separate searches, as perform in this search chaining example. Instead, we -would recommend you fit the lens and source simultaneously, using linear light profiles to make the model simpler -or a Multi-Gaussian Expansion. - -However, the example is still useful for demonstrating the core concepts of search chaining, which is still vital -for fitting complex lens model. Therefore, we recommend you still read through this tutorial and try to get a good -understanding of how search chaining works, but bear in mind that the example is a little dated and we now recommend -you fit the lens and source simultaneously! -""" - -from autolens import jax_wrapper # Sets JAX environment before other imports - -# from autolens import setup_notebook; setup_notebook() - -from pathlib import Path -import autofit as af -import autolens as al -import autolens.plot as aplt - -""" -__Initial Setup__ - -we'll use strong lensing data, where: - - - The lens galaxy's light is an `Sersic`. - - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. - - The source galaxy's light is an `Exponential`. - -This image was fitted throughout chapter 2. -""" -dataset_name = "lens_sersic" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if al.util.dataset.should_simulate(str(dataset_path)): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/simulator/lens_sersic.py"], - check=True, - ) - -dataset = al.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -""" -__Paths__ - -All three searches will use the same `path_prefix`, so we write it here to avoid repetition. -""" -path_prefix = Path("howtolens") / "chapter_3" / "tutorial_3_lens_and_source" - -""" -__Masking (Search 1)__ - -We need to choose our mask for the analysis. We use an annular mask that does not remove the source's light -from the fit, assuming the lens light model will still be sufficiently accurate to reveal the source in the -second search. -""" -mask = al.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0 -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Model + Search + Analysis + Model-Fit (Search 1)__ - -Search 1 fits a lens model where: - - - The lens galaxy's light is a linear `Sersic` bulge [6 parameters]. - - - The lens galaxy's mass and source galaxy are omitted. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=6. - -__Notes__ - -We use linear light profiles througout this script, given that the model is quite complex and this helps -simplify it. -""" -model_1 = af.Collection( - galaxies=af.Collection(lens=af.Model(al.Galaxy, redshift=0.5, bulge=al.lp.Sersic)), -) - -""" -The `info` attribute shows the model in a readable format. -""" -print(model_1.info) - -""" -__Search + Analysis + Model-Fit (Search 1)__ -""" -analysis_1 = al.AnalysisImaging(dataset=dataset) - -search_1 = af.Nautilus( - path_prefix=path_prefix, - name="search[1]_light[bulge]", - unique_tag=dataset_name, - n_live=75, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - -""" -__Result (Search 1)__ - -The results which are used for prior passing are summarised in the `info` attribute. -""" -print(result_1.info) - -""" -__Masking (Search 2)__ - -Search 2 we are only fitting the source's light, thus we can apply an annular mask that removes regions of the -image that contained only the lens's light. -""" -dataset = al.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -mask = al.Mask2D.circular_annular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - inner_radius=0.6, - outer_radius=2.4, -) - -dataset = dataset.apply_mask(mask=mask) - -""" -__Model + Search + Analysis + Model-Fit (Search 2)__ - -Search 2 fits a lens model where: - - - The lens galaxy's light is a linear `Sersic` bulge [Parameters fixed to results of search 1]. - - - The lens galaxy's total mass distribution is an `Isothermal` with `ExternalShear` [7 parameters]. - - - The source galaxy's light is a linear `Sersic` [6 parameters]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=13. - -Search 2, we fit the source-`galaxy's light and fix the lens light model to the model inferred in search 1, -ensuring the image we has the foreground lens subtracted. We do this below by passing the lens light as an `instance` -object. - -By passing an `instance`, we are telling **PyAutoLens** that we want it to pass the maximum log likelihood result of -that search and use those parameters as fixed values in the model. The model parameters passed as an `instance` are not -free parameters fitted for by the non-linear search, thus this reduces the dimensionality of the non-linear search -making model-fitting faster and more reliable. - -Thus, search 2 includes the lens light model from search 1, but it is completely fixed during the model-fit! - -We also use the centre of the `bulge` to initialize the priors on the lens's `mass`. -""" -mass = af.Model(al.mp.Isothermal) -mass.centre_0 = result_1.model.galaxies.lens.bulge.centre_0 -mass.centre_1 = result_1.model.galaxies.lens.bulge.centre_1 - -model_2 = af.Collection( - galaxies=af.Collection( - lens=af.Model( - al.Galaxy, - redshift=0.5, - bulge=result_1.instance.galaxies.lens.bulge, - mass=mass, - shear=al.mp.ExternalShear, - ), - source=af.Model(al.Galaxy, redshift=1.0, bulge=al.lp.Sersic), - ), -) - -analysis_2 = al.AnalysisImaging(dataset=dataset) - -search_2 = af.Nautilus( - path_prefix=path_prefix, - name="search[2]_mass[sie]_source[bulge]", - unique_tag=dataset_name, - n_live=100, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -""" -__Result (Search 2)__ - -The results which are used for prior passing are summarised in the `info` attribute. -""" -print(result_2.info) - -""" -__Masking (Search 3)__ - -Search 3 we fit the lens and source, therefore we will use a large circular mask. -""" -dataset = al.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.1, -) - -mask_radius = 3.0 - -mask = al.Mask2D.circular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - radius=mask_radius, -) - -dataset = dataset.apply_mask(mask=mask) - -""" -__Model + Search + Analysis + Model-Fit (Search 3)__ - -Search 3 fits a lens model where: - - - The lens galaxy's light is a linear `Sersic` bulge [6 Parameters: priors initialized from search 1]. - - - The lens galaxy's total mass distribution is an `Isothermal` with `ExternalShear` [7 parameters: priors - initialized from search 2]. - - - The source galaxy's light is a linear `Sersic` [6 parameters: priors initialized from search 2]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=23. - -There isn't a huge amount to say about this search, we have initialized the priors on all of our models parameters -and the only thing that is left to do is fit for all model components simultaneously, with slower Nautilus settings -that will give us more accurate parameter values and errors. -""" -model_3 = af.Collection( - galaxies=af.Collection( - lens=af.Model( - al.Galaxy, - redshift=0.5, - bulge=result_1.model.galaxies.lens.bulge, - mass=result_2.model.galaxies.lens.mass, - ), - source=af.Model( - al.Galaxy, redshift=1.0, bulge=result_2.model.galaxies.source.bulge - ), - ), -) - -analysis_3 = al.AnalysisImaging(dataset=dataset) - -search_3 = af.Nautilus( - path_prefix=path_prefix, - name="search[3]_light[bulge]_mass[sie]_source[bulge]", - unique_tag=dataset_name, - n_live=150, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_3 = search_3.fit(model=model_3, analysis=analysis_3) - -""" -__Result (Search 3)__ - -The final results are summarised in the `info` attribute. -""" -print(result_3.info) - -""" -__Wrap Up__ - -And there we have it, a sequence of searches that breaks modeling the lens and source galaxy into 3 simple searches. -This approach is much faster than fitting the lens and source simultaneously from the beginning. Instead of asking you -questions at the end of this chapter`s tutorials, I'm going to ask questions which I then answer. This will hopefully -get you thinking about how to approach pipeline writing. - - 1) Can this pipeline really be generalized to any lens? Surely the radii of the masks depends on the lens and source - galaxies? - -Whilst this is true, we chose mask radii above that are `excessive` and masks out a lot more of the image than just -the source (which, in terms of run-time, is desirable). Thus, provided you know the Einstein radius distribution of -your lens sample, you can choose mask radii that will masks out every source in your sample adequately (and even if -some of the source is still there, who cares? The fit to the lens galaxy will be okay). - -However, the template pipelines provided on the `autolens_workspace` simply use circular masks for every search and do -not attempt to use different masks for the lens light fit and source fit. This is to keep things simple (at the expense -of slower run times). It is up to you if you want to adapt these scripts to try and use more specific masking strategies. - -__Dated Tutorial__ - -In fact, we now strongly recommend that you do not change masks between each search when using search chaining. -This is because it is very fiddly, and can waste a lot of your time refining masks to ensure they are suitable for -each lens. We recommend you always just use a large circular mask which is big enough to include the entire lens and -source of all lenses in your sample. This will save you a lot of time and means lens modeling can be automated much -easier. - -Building on the discussion above, a known limitation of using a pipeline which fits the lens light first, then the -source, is that it will do a poor job deblending the lens and source light if the Einstein radius is low. This often -leads the mass model to infer incorrect solutions which fit residuals from the lens light subtraction. - -This is why, given all the improvements to autolens, we now recommend that you do not use this pipeline and instead -always begin by fitting the lens and source simultaneously. This can use linear light profiles of a Multi-Gaussian -Expansion. -""" diff --git a/scripts/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.py b/scripts/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.py deleted file mode 100644 index ff3c9a4..0000000 --- a/scripts/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.py +++ /dev/null @@ -1,329 +0,0 @@ -""" -Tutorial 4: Two Lens galaxies -============================= - -Up to now, all the images we've fitted had one lens galaxy. However, we saw in chapter 1 that our lens plane can -consist of multiple galaxies which each contribute to the strong lensing. Multi-galaxy systems are challenging to -model, because they add an extra 5-10 parameters to the non-linear search and, more problematically, the degeneracies -between the parameters of the mass profiles of the two galaxies can be severe. - -However, we can still break their analysis down using multiple searches and give ourselves a shot at getting a good -lens model. Here, we're going to fit a double lens system, fitting as much about each individual lens galaxy before -fitting them simultaneously. - -Up to now, I've put a focus on an analysis being general. The script we write in this example is going to be the -opposite, specific to the image we're modeling. Fitting multiple lens galaxies is really difficult and writing a -pipeline that we can generalize to many lenses isn't currently possible. - -__Contents__ - -- **Initial Setup:** we'll use new strong lensing data, where. -- **Mask:** Define the 2D mask applied to the dataset for the model-fit. -- **Paths:** All four searches will use the same `path_prefix`, so we write it here to avoid repetition. -- **Search Chaining Approach:** Looking at the image, there are two blobs of light corresponding to the two lens galaxies. -- **Wrap Up:** Summary of the script and next steps. - -""" - -from autolens import jax_wrapper # Sets JAX environment before other imports - -# from autolens import setup_notebook; setup_notebook() - -from pathlib import Path -import autofit as af -import autolens as al -import autolens.plot as aplt - -""" -__Initial Setup__ - -we'll use new strong lensing data, where: - - - There are two lens galaxy's whose `LightProfile`'s are both linear `Sersic`'s. - - There are two lens galaxy's whose `MassProfile`'s are both `Isothermal`'s - - The source galaxy's light is an `Exponential`. -""" -dataset_name = "x2_lens_galaxies" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if al.util.dataset.should_simulate(str(dataset_path)): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/simulator/lens_x2.py"], - check=True, - ) - -dataset = al.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.05, -) - -""" -__Mask__ - -We need to choose our mask for the analysis. Given the lens light is present in the image we'll need to include all -of its light in the central regions of the image, so lets use a circular mask. - -We'll use this mask in all four searches. -""" -mask_radius = 3.0 - -mask = al.Mask2D.circular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - radius=mask_radius, -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - - -""" -__Paths__ - -All four searches will use the same `path_prefix`, so we write it here to avoid repetition. -""" -path_prefix = Path("howtolens") / "chapter_3" / "tutorial_4_x2_lens_galaxies" - -""" -__Search Chaining Approach__ - -Looking at the image, there are two blobs of light corresponding to the two lens galaxies. The source's light is also -pretty complex and the arcs don't posses the rotational symmetry we're used to seeing up to now. -Multi-galaxy ray-tracing is just a lot more complicated, which means so is modeling it! - -So, how can we break the lens modeling up? As follows: - - 1) Fit and subtract the light of each lens galaxy individually. - 2) Use these results to initialize each lens galaxy's total mass distribution. - -So, with this in mind, we'll perform an analysis using searches: - - 1) Fit the light of the lens galaxy on the left of the image, at coordinates (0.0", -1.0"). - 2) Fit the light of the lens galaxy on the right of the image, at coordinates (0.0", 1.0"). - 3) Use this lens-subtracted image to fit the source's light. The mass profile centres of the two lens galaxies are - fixed to (0.0", -1.0") and (0.0", 1.0"). - 4) Fit all relevant parameters simultaneously, using priors from searches 1, 2 and 3. - -__Model + Search + Analysis + Model-Fit (Search 1)__ - -Search 1 fits a lens model where: - - - The left lens galaxy's light is a linear `Sersic` bulge with a fixed centre [4 parameters]. - - - The lens galaxy's mass and source galaxy are omitted. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=4. - -We fix the centre of its light to (0.0, -1.0), the pixel we know the left galaxy's light centre peaks. -""" -left_lens = af.Model(al.Galaxy, redshift=0.5, bulge=al.lp_linear.Sersic) -left_lens.bulge.centre_0 = 0.0 -left_lens.bulge.centre_1 = -1.0 - -model_1 = af.Collection(galaxies=af.Collection(left_lens=left_lens)) - -""" -The `info` attribute shows the model in a readable format. -""" -print(model_1.info) - -""" -__Search + Analysis + Model-Fit (Search 1)__ -""" -analysis_1 = al.AnalysisImaging(dataset=dataset) - -search_1 = af.Nautilus( - path_prefix=path_prefix, - name="search[1]__left_lens_light[bulge]", - unique_tag=dataset_name, - n_live=75, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - - -""" -__Model + Search + Analysis + Model-Fit (Search 2)__ - -Search 2 fits a lens model where: - - - The left lens galaxy's light is a linear `Sersic` bulge [0 parameters: fixed from search 1]. - - - The right lens galaxy's light is a linear `Sersic` bulge with a fixec centre [4 parameters]. - - - The lens galaxy's mass and source galaxy are omitted. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=4. - -We fix the centre of the right lens's light to (0.0, 1.0), the pixel we know the right galaxy's light centre peaks. - -We also pass the result of the `left_lens` from search ` as an `instance`, which should improve the fitting of the -right lens. -""" -right_lens = af.Model(al.Galaxy, redshift=0.5, bulge=al.lp_linear.Sersic) -right_lens.bulge.centre_0 = 0.0 -right_lens.bulge.centre_1 = 1.0 - -model_2 = af.Collection( - galaxies=af.Collection( - left_lens=result_1.instance.galaxies.left_lens, right_lens=right_lens - ), -) - -analysis_2 = al.AnalysisImaging(dataset=dataset) - -search_2 = af.Nautilus( - path_prefix=path_prefix, - name="search[2]__right_lens_light[bulge]", - unique_tag=dataset_name, - n_live=75, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -""" -__Model + Search + Analysis + Model-Fit (Search 3)__ - -Search 3 fits a lens model where: - - - The left lens galaxy's light is a linear `Sersic` bulge [0 parameters: fixed from search 1]. - - - The right lens galaxy's light is a linear `Sersic` bulge [0 parameters: fixed from search 2]. - - - The lens galaxy's mass is modeled using two `Isothermal` profiles whose centres are fixed to (0.0, -1.0) - and (0.0, 1.0) [6 parameters]. - - - The source galaxy's light is a linear `Exponential` [5 parameters]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=11. -""" -left_lens = af.Model( - al.Galaxy, - redshift=0.5, - bulge=result_1.instance.galaxies.left_lens.bulge, - mass=al.mp.Isothermal, -) - -right_lens = af.Model( - al.Galaxy, - redshift=0.5, - bulge=result_2.instance.galaxies.right_lens.bulge, - mass=al.mp.Isothermal, -) - -left_lens.mass.centre_0 = 0.0 -left_lens.mass.centre_1 = -1.0 -right_lens.mass.centre_0 = 0.0 -right_lens.mass.centre_1 = 1.0 - -model_3 = af.Collection( - galaxies=af.Collection( - left_lens=left_lens, - right_lens=right_lens, - source=af.Model(al.Galaxy, redshift=1.0, bulge=al.lp_linear.ExponentialCore), - ), -) - -analysis_3 = al.AnalysisImaging(dataset=dataset) - -search_3 = af.Nautilus( - path_prefix=path_prefix, - name="search[3]__mass_x2[sie]__source[exp]", - unique_tag=dataset_name, - n_live=100, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_3 = search_3.fit(model=model_3, analysis=analysis_3) - -""" -__Model + Search + Analysis + Model-Fit (Search 4)__ - -Search 4 fits a lens model where: - - - The left lens galaxy's light is a linear `Sersic` bulge [6 parameters: priors initialized from search 1]. - - - The right lens galaxy's light is a linear `Sersic` bulge [6 parameters: priors initialized from search 2]. - - - The lens galaxy's mass is modeled using two `Isothermal` profiles whose centres are fixed to (0.0, -1.0) - and (0.0, 1.0) [6 parameters: priors initialized from search 3]. - - - The source galaxy's light is a linear `Sersic` [6 parameters: priors initialized from search 3]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=24. - -We don't explicitly check the run-time, but the same principles discussed above apply here. -""" -left_lens = af.Model( - al.Galaxy, - redshift=0.5, - bulge=result_1.model.galaxies.left_lens.bulge, - mass=result_3.model.galaxies.left_lens.mass, -) - -right_lens = af.Model( - al.Galaxy, - redshift=0.5, - bulge=result_2.model.galaxies.right_lens.bulge, - mass=result_3.model.galaxies.right_lens.mass, -) - -source_bulge = af.Model(al.lp_linear.Sersic) - -source_bulge.take_attributes(result_3.model.galaxies.source.bulge) - -model_4 = af.Collection( - galaxies=af.Collection( - left_lens=left_lens, - right_lens=right_lens, - source=af.Model(al.Galaxy, redshift=1.0, bulge=source_bulge), - ), -) - -analysis_4 = al.AnalysisImaging(dataset=dataset) - -search_4 = af.Nautilus( - path_prefix=path_prefix, - name="search[4]_light_x2[bulge]_mass_x2[sie]_source[exp]", - unique_tag=dataset_name, - n_live=100, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_4 = search_4.fit(model=model_4, analysis=analysis_4) - -""" -__Wrap Up__ - -This pipeline takes a while to run, as is the nature of multi-galaxy modeling. Nevertheless, the techniques we've -learnt above can be applied to systems with even more galaxies, albeit the increases in parameters will slow down the -non-linear search. Here are some more Q&A`s - - 1) This system had two very similar lens galaxy's with comparable amounts of light and mass. How common is this? - Does it make it harder to model them? - -Typically, lenses with 2 galaxies have one massive galaxy (that makes up some 80%-90% of the overall light and mass), -accompanied by a smaller satellite. The satellite cannotbe ignored, it impacts the ray-tracing in a measureable way, -but it is a lot less degenerate with the main lens galaxy. This means we can often model the satellite with much -simpler profiles (e.g. spherical profiles). So yes, multi-galaxy systems can often be easier to model. - - 2) It got pretty confusing passing all those priors towards the end of the pipeline there, didn`t it? - -It does get confusing. This is why we made galaxies named objects, so that we could call them the `left_lens` and -`right_lens`. It still requires caution when writing the pipeline, but goes to show that if you name your galaxies -sensibly you should be able to avoid errors, or spot them quickly when you make them. -""" diff --git a/scripts/chapter_3_search_chaining/tutorial_5_complex_source.py b/scripts/chapter_3_search_chaining/tutorial_5_complex_source.py deleted file mode 100644 index 24e167e..0000000 --- a/scripts/chapter_3_search_chaining/tutorial_5_complex_source.py +++ /dev/null @@ -1,323 +0,0 @@ -""" -Tutorial 5: Complex Source -========================== - -Up to now, we've not paid much attention to the source galaxy's morphology. We've assumed its a single-component -exponential profile, which is a fairly crude assumption. A quick look at any image of a real galaxy reveals a -wealth of different structures that could be present: bulges, disks, bars, star-forming knots and so on. Furthermore, -there could be more than one source-galaxy! - -In this example, we'll explore how far we get fitting a complex source using a pipeline. Fitting complex source's is -an exercise in diminishing returns. Each light profile we add to our source model brings with it an extra 5-7, -parameters. If there are 4 components, or multiple galaxies, we are quickly entering the somewhat nasty regime of -30-40+ parameters in our non-linear search. Even with a pipeline, that is a lot of parameters to fit! - -__Contents__ - -- **Initial Setup:** we'll use new strong lensing data, where. -- **Paths:** All four searches will use the same `path_prefix`, so we write it here to avoid repetition. -- **Search Chaining Approach:** The source is clearly complex, with more than 4 peaks of light. -- **Run Times:** Profiling the expected run time of the model-fit. -- **Wrap Up:** Summary of the script and next steps. - -""" - -from autolens import jax_wrapper # Sets JAX environment before other imports - -# from autolens import setup_notebook; setup_notebook() - -from pathlib import Path -import autofit as af -import autolens as al -import autolens.plot as aplt - -""" -__Initial Setup__ - -we'll use new strong lensing data, where: - - - The lens galaxy's light is omitted. - - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. - - The source galaxy's light is four linear `Sersic`. -""" -dataset_name = "source_complex" -dataset_path = Path("dataset") / "imaging" / dataset_name - -""" -__Dataset Auto-Simulation__ - -If the dataset does not already exist on your system, it will be created by running the corresponding -simulator script. This ensures that all example scripts can be run without manually simulating data first. -""" -if al.util.dataset.should_simulate(str(dataset_path)): - import subprocess - import sys - - subprocess.run( - [sys.executable, "scripts/simulator/source_complex.py"], - check=True, - ) - -dataset = al.Imaging.from_fits( - data_path=dataset_path / "data.fits", - noise_map_path=dataset_path / "noise_map.fits", - psf_path=dataset_path / "psf.fits", - pixel_scales=0.05, -) - -mask_radius = 3.0 - -mask = al.Mask2D.circular( - shape_native=dataset.shape_native, - pixel_scales=dataset.pixel_scales, - radius=mask_radius, -) - -dataset = dataset.apply_mask(mask=mask) - -aplt.subplot_imaging_dataset(dataset=dataset) - -""" -__Paths__ - -All four searches will use the same `path_prefix`, so we write it here to avoid repetition. -""" -path_prefix = Path("howtolens") / "chapter_3" / "tutorial_4_complex_source" - -""" -__Search Chaining Approach__ - -The source is clearly complex, with more than 4 peaks of light. Through visual inspection of this image, we cannot state -with confidence how many sources of light there truly is! The data also omits he lens galaxy's light. This keep the -number of parameters down and therefore makes the searches faster, however we would not get such a luxury for a real -galaxy. - -To fit this lens with a complex source model, our approach is simply to fit the lens galaxy mass and source using -one light profile in the first search, and then add an additional light profile to each search. The mass model and -light profiles inferred in the previous search are then used to pass priors. - -__Run Times__ - -In this example we don't explicitly check run-times, for brevity. However, the same rules of thumb we discussed in the -previous tutorial still apply. - -For example, as we add more light profiles to the source model, the likelihood evaluation time will increase. As the -model becomes more complex, search chaining is key to ensuring run times stay lower. - -__Model + Search + Analysis + Model-Fit (Search 1)__ - - - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters]. - - - The source galaxy's light is a linear `Sersic` [6 parameters]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=11. -""" -model_1 = af.Collection( - galaxies=af.Collection( - lens=af.Model(al.Galaxy, redshift=0.5, mass=al.mp.Isothermal), - source=af.Model(al.Galaxy, redshift=1.0, bulge_0=al.lp_linear.Sersic), - ), -) - -""" -The `info` attribute shows the model in a readable format. -""" -print(model_1.info) - -""" -__Search + Analysis + Model-Fit (Search 1)__ -""" -analysis_1 = al.AnalysisImaging(dataset=dataset) - -search_1 = af.Nautilus( - path_prefix=path_prefix, - name="search[1]__mass[sie]__source_x1[bulge]", - unique_tag=dataset_name, - n_live=120, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_1 = search_1.fit(model=model_1, analysis=analysis_1) - -""" -__Model + Search + Analysis + Model-Fit (Search 2)__ - - - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters: priors initialized from - search 1]. - - - The source galaxy's light is two linear `Sersic` [12 parameters: first Sersic initialized from - search 1]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=17. -""" -model_2 = af.Collection( - galaxies=af.Collection( - lens=af.Model(al.Galaxy, redshift=0.5, mass=result_1.model.galaxies.lens.mass), - source=af.Model( - al.Galaxy, - redshift=1.0, - bulge_0=result_1.model.galaxies.source.bulge_0, - bulge_1=al.lp_linear.Sersic, - ), - ), -) - -analysis_2 = al.AnalysisImaging(dataset=dataset) - -search_2 = af.Nautilus( - path_prefix=path_prefix, - name="search[2]_mass[sie]_source_x2[bulge]", - unique_tag=dataset_name, - n_live=120, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_2 = search_2.fit(model=model_2, analysis=analysis_2) - -""" -__Model + Search + Analysis + Model-Fit (Search 3)__ - - - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters: priors initialized from - search 2]. - - - The source galaxy's light is three linear `Sersic` [18 parameters: first two Sersic's initialized from - search 2]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=21. -""" -model_3 = af.Collection( - galaxies=af.Collection( - lens=af.Model(al.Galaxy, redshift=0.5, mass=result_2.model.galaxies.lens.mass), - source=af.Model( - al.Galaxy, - redshift=1.0, - bulge_0=result_2.model.galaxies.source.bulge_0, - bulge_1=result_2.model.galaxies.source.bulge_1, - bulge_2=al.lp_linear.Sersic, - ), - ), -) - -analysis_3 = al.AnalysisImaging(dataset=dataset) - -search_3 = af.Nautilus( - path_prefix=path_prefix, - name="search[3]_mass[sie]_source_x3[bulge]", - unique_tag=dataset_name, - n_live=140, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_3 = search_3.fit(model=model_3, analysis=analysis_3) - -""" -__Model + Search + Analysis + Model-Fit (Search 4)__ - - - The lens galaxy's total mass distribution is an `Isothermal` [5 parameters: priors initialized from - search 4]. - - - The source galaxy's light is four linear `Sersic` [24 parameters: first three Sersic's initialized from - search 2]. - -The number of free parameters and therefore the dimensionality of non-linear parameter space is N=29. -""" -model_4 = af.Collection( - galaxies=af.Collection( - lens=af.Model(al.Galaxy, redshift=0.5, mass=result_3.model.galaxies.lens.mass), - source=af.Model( - al.Galaxy, - redshift=1.0, - bulge_0=result_3.model.galaxies.source.bulge_0, - bulge_1=result_3.model.galaxies.source.bulge_1, - bulge_2=result_3.model.galaxies.source.bulge_2, - bulge_3=al.lp_linear.Sersic, - ), - ), -) - -analysis_4 = al.AnalysisImaging(dataset=dataset) - -search_4 = af.Nautilus( - path_prefix=path_prefix, - name="search[4]_mass[sie]_source_x4[bulge]", - unique_tag=dataset_name, - n_live=160, - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations -) - -result_4 = search_4.fit(model=model_4, analysis=analysis_4) - -""" -__Wrap Up__ - -With four light profiles, we were still unable to produce a fit to the source that did not leave residuals. However, I -actually simulated the lens using a source with four light profiles. A `perfect fit` was therefore somewhere in -parameter space, but our search unfortunately was unable to locate this. - -Lets confirm this, by manually fitting the imaging data with the true input model. - -We cannot apply a mask to a dataset that was already masked, so we first reload the imaging from .fits. -""" -dataset = dataset.apply_mask( - mask=al.Mask2D.circular( - shape_native=dataset.shape_native, pixel_scales=dataset.pixel_scales, radius=3.0 - ) -) - -lens_galaxy = al.Galaxy( - redshift=0.5, - mass=al.mp.Isothermal( - centre=(0.0, 0.0), einstein_radius=1.6, ell_comps=(0.17647, 0.0) - ), -) - -source_galaxy = al.Galaxy( - redshift=1.0, - light_0=al.lp.Sersic( - centre=(0.1, 0.1), - ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0), - intensity=0.1, - effective_radius=1.0, - sersic_index=2.5, - ), - light_1=al.lp.Sersic( - centre=(0.8, 0.6), - ell_comps=al.convert.ell_comps_from(axis_ratio=0.5, angle=30.0), - intensity=0.2, - effective_radius=0.3, - sersic_index=3.0, - ), - light_2=al.lp.Sersic( - centre=(-0.3, 0.6), - ell_comps=al.convert.ell_comps_from(axis_ratio=0.3, angle=120.0), - intensity=0.6, - effective_radius=0.5, - sersic_index=1.5, - ), - light_3=al.lp.Sersic( - centre=(-0.3, -0.3), - ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=85.0), - intensity=0.4, - effective_radius=0.1, - sersic_index=2.0, - ), -) - -tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) - -true_fit = al.FitImaging(dataset=dataset, tracer=tracer) - -aplt.subplot_fit_imaging(fit=true_fit) -aplt.subplot_fit_imaging_of_planes(fit=true_fit) - -""" -And indeed, we see an improved residual-map, chi-squared-map, and so forth. - -If the source morphology is complex, there is no way we chain searches to fit it perfectly. The non-linear parameter -space simply becomes too complex. For this tutorial, this was true even though our source model could actually fit -the data perfectly. For real lenses, the source may be *even more complex* giving us even less hope of getting a -good fit. - -But fear not, **PyAutoLens** has you covered. In chapter 4, we'll introduce a completely new way to model the source -galaxy, which addresses the problem faced here. -""" diff --git a/scripts/chapter_3_search_chaining/tutorial_6_slam.py b/scripts/chapter_3_search_chaining/tutorial_6_slam.py deleted file mode 100755 index 3e627a5..0000000 --- a/scripts/chapter_3_search_chaining/tutorial_6_slam.py +++ /dev/null @@ -1,44 +0,0 @@ -""" -Tutorial 6: SLaM -================ - -You are now familiar with pipelines, in particular how we use them to break-down the lens modeling procedure -to provide more efficient and reliable model-fits. In the previous tutorials, you learnt how to write your own -pipelines, which can fit whatever lens model is of particular interest to your scientific study. - -However, for most lens models there are standardized approaches one can take to fitting them. For example, as we saw in -tutorial 3 of this chapter, an effective approach is to fit a model for the lens's light followed by a model for its -mass and the source. It would be wasteful for all **PyAutoLens** users to have to write their own pipelines to -perform the same tasks. - -For this reason, the `autolens_workspace` comes with a number of standardized pipelines, which fit common lens models -in ways we have tested are efficient and robust. These pipelines can be found the `chaining/pipelines` packages and -can be easily adapted to fit lens models with a range of different parameterizations and assumptions. - -__SLaM (Source, Light and Mass)__ - -A second set of template pipelines, called the **SLaM** (Source, Light and Mass) pipelines can be found in the folder -`autolens_workspace/slam`. These are similar in design to the pipelines, but are composed of a specific sequence of -pipelines: - - - `Source`: A pipeline that focuses on producing a robust model for the source's light, using simpler models for the - lens's light and mass. - - - `Light`: A pipeline that fits a complex lens light model (e.g. one with many components), using the initialized - source model to cleanly deblend the lens and source light. - - - `Mass`: A pipeline that fits a complex lens mass model, benefitting from the good models for the lens's light and - source. - -For fitting very complex lens models, for example ones which decompose its mass into its stellar and dark components, -the **SLaM** pipelines have been carefully crafted to do this in a reliable and automated way that is still efficient. -The **SLaM** pipelines also make fitting many different models to a single dataset efficient, as they reuse the results -of earlier searches (e.g. in the Source pipeline) to fit different models in the `Light` and `Mass` pipelines for the -lens's light and mass. - -__Wrap Up__ - -Whether you should use individual searches, pipelines, The SLaM pipelines or write your own model-fitting script -depends on the scope of your scientific analysis. I would advise you begin by trying to adapting the scripts in the -`autolens_workspace`to fit your data, and also try using the SLaM pipelines once you are a confident **PyAutoLens** user. -""" diff --git a/scripts/chapter_4_pixelizations/README.md b/scripts/chapter_4_pixelizations/README.md deleted file mode 100644 index 8dea793..0000000 --- a/scripts/chapter_4_pixelizations/README.md +++ /dev/null @@ -1,27 +0,0 @@ -In chapter 4, we use **Pixelizations** to reconstruct complex source galaxies on pixelized grids. - -**Colab** links to every tutorial are included. - -# Files - -- [Tutorial 1: Pixelizations](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb) — Creating a pixel-grid in the source-plane. - -- [Tutorial 2: Mappers](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_2_mappers.ipynb) — How a pixelization maps source-pixels to image-pixels. - -- [Tutorial 3: Inversions](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_3_inversions.ipynb) — Inverting the mappings to reconstruct the source's light. - -- [Tutorial 4: Bayesian Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb) — Smoothing the source within a Bayesian framework. - -- [Tutorial 5: Borders](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_5_borders.ipynb) — Preventing highly demagnified image-pixels ruining the inversion. - -- [Tutorial 6: Lens Modeling](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_6_lens_modeling.ipynb) — How to use inversions to fit a lens model. - -- [Tutorial 7: Adaptive Pixelization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb) — A Voronoi mesh which adapts to the mass model's magnification. - -- [Tutorial 8: Model Fit](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_8_model_fit.ipynb) — An example lens modeling pipeline which uses an inversion. - -- [Tutorial 9: Fit Problems](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_9_fit_problems.ipynb) — The shortcomings of our lens models and inversions. - -- [Tutorial 10: Brightness Adaption](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb) — Adapting the pixelization to the source's morphology. - -- [Tutorial 11: Adaptive Regularization](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_pixelizations/tutorial_11_adaptive_regularization.ipynb) — Adapting the regularization to the source's morphology. diff --git a/scripts/chapter_4_scaling_up_lensing/README.md b/scripts/chapter_4_scaling_up_lensing/README.md new file mode 100644 index 0000000..f059478 --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/README.md @@ -0,0 +1,18 @@ +In chapter 4, we scale lens modeling up beyond a single lens galaxy: extra galaxies, multi-galaxy lenses, +scaling relations, group and cluster scales, and weak lensing. + +**Colab** links to every tutorial are included. + +# Files + +- [Tutorial 1: Extra Galaxies](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.ipynb) — Handling nearby galaxies by scaling their light out of the fit or modeling them explicitly. + +- [Tutorial 2: Multi Galaxy](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.ipynb) — Modeling lenses with two or more co-dominant lens galaxies and the degeneracies this creates. + +- [Tutorial 3: Scaling Relation](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.ipynb) — Tying galaxy masses to their luminosities so model complexity stops growing with each galaxy. + +- [Tutorial 4: Group Scale](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_4_group_scale.ipynb) — Modeling group-scale lenses with truncated dPIE members and an optional group dark matter halo. + +- [Tutorial 5: Cluster Scale](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.ipynb) — Modeling cluster-scale lenses with point-source positions, the CSV interface and multi-plane ray tracing. + +- [Tutorial 6: Weak Lensing](https://colab.research.google.com/github/PyAutoLabs/HowToLens/blob/2026.8.4.1/notebooks/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.ipynb) — Fitting weak-lensing shear catalogues, which probe mass far beyond the strong lensing region. diff --git a/scripts/chapter_4_pixelizations/__init__.py b/scripts/chapter_4_scaling_up_lensing/__init__.py old mode 100755 new mode 100644 similarity index 100% rename from scripts/chapter_4_pixelizations/__init__.py rename to scripts/chapter_4_scaling_up_lensing/__init__.py diff --git a/scripts/chapter_4_scaling_up_lensing/introduction b/scripts/chapter_4_scaling_up_lensing/introduction new file mode 100644 index 0000000..6bb46fc --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/introduction @@ -0,0 +1,19 @@ +Every lens we have modeled so far had one lens galaxy and one source. Real strong lenses live in crowded +fields, and the most scientifically exciting systems are bigger: two or more lens galaxies, galaxy groups, +entire clusters, and mass distributions that only weak lensing can reach. + +In this chapter, we scale lens modeling up, step by step. You'll learn how we: + +1) Handle extra galaxies near a lens, by scaling their light out of the fit or modeling them explicitly. +2) Model systems with two or more co-dominant lens galaxies, and understand the degeneracies this creates. +3) Use scaling relations to tie galaxy masses to their luminosities, so model complexity stops growing + with every galaxy we add. +4) Model group-scale lenses, introducing truncated dPIE mass profiles and an optional group dark matter halo. +5) Model cluster-scale lenses, using point-source positions, a CSV interface for member catalogues and + multi-plane ray tracing. +6) Fit weak-lensing shear catalogues, which probe dark matter far beyond the strong lensing region. + +Much of this chapter's methodology (extra galaxies, multi-galaxy fields, scaling relations) applies equally +to non-lensing galaxy studies with **PyAutoGalaxy**. + +Sound good? I agree, so lets get on with it. diff --git a/scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py b/scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py new file mode 100644 index 0000000..fc4ce81 --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py @@ -0,0 +1,478 @@ +""" +Tutorial 1: Extra Galaxies +========================== + +Welcome to chapter 4 of **HowToLens**, where we scale up lens modeling beyond a single lens galaxy. + +In chapters 1 to 3, every strong lens we studied had the same simple anatomy: one lens galaxy, whose light and mass +we modeled, and one background source galaxy, whose lensed emission we fitted. This is the cleanest possible lensing +configuration, and it was the right place to learn the fundamentals of ray-tracing, non-linear searches and +pixelized source reconstructions. + +Real strong lenses are rarely this tidy. Galaxies live in crowded fields: a lens galaxy may have a faint companion a +few arc-seconds away, sit in a small group of comparably massive galaxies, or be embedded in a cluster containing +hundreds of members and a vast dark matter halo. The science of strong lensing scales up through this hierarchy, and +so must our modeling: + +- **Extra galaxies (this tutorial)**: a single dominant lens galaxy, with one or more nearby interloper galaxies + whose light and mass may contaminate the analysis. + +- **Multi-galaxy lenses (next tutorial)**: systems where two or more galaxies are co-equal lenses, and no single + galaxy can be called "the" lens. + +- **Scaling relations**: modeling many galaxies at once by tying their properties to their luminosity, so that + adding galaxies does not add free parameters. + +- **Group and cluster scales**: lenses whose deflectors are entire groups or clusters of galaxies, including a + shared dark matter halo. + +- **Weak lensing**: the statistical distortion of many background galaxies by foreground mass, extending lensing + beyond the strong regime entirely. + +Much of the methodology in this chapter is not specific to lensing. Fitting the light of extra galaxies, modeling +blended multi-galaxy fields and composing models via scaling relations are equally important for non-lensing galaxy +studies, and **PyAutoGalaxy** provides the same API for these tasks applied to galaxy morphology (without the +ray-tracing). + +In this first tutorial, we take the first step up in scale: a galaxy-scale lens with one extra galaxy nearby. We +will learn how to decide whether the extra galaxy matters, and the two approaches to dealing with it when it does: +removing its light from the data, or including it in the lens model. + +__Contents__ + +- **Initial Setup:** Load the imaging dataset of a lens with an extra galaxy and inspect the interloper. +- **Light Versus Mass:** The core decision: do the extra galaxy's light and / or mass affect the analysis? +- **Mask:** Define a circular mask large enough to include the extra galaxy's emission. +- **Approach 1 Noise Scaling:** Remove the extra galaxy's light from the fit by scaling its data and noise values. +- **Noise Scaling Fit:** Fit a lens model to the noise-scaled data, without the extra galaxy in the model. +- **Approach 2 Extra Galaxies Model:** Include the extra galaxy's light and mass in the lens model explicitly. +- **Extra Galaxy Centres:** Why the extra galaxy's centre is fixed to its observed light centre. +- **Extra Galaxies Fit:** Fit the lens model which includes the extra galaxy. +- **Which Approach When:** Guidance on choosing between noise scaling and explicit modeling. +- **Wrap Up:** Summary of the script and next steps. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +from pathlib import Path +import autolens as al +import autolens.plot as aplt +import autofit as af + +""" +__Initial Setup__ + +Lets load the `Imaging` dataset we'll fit in this tutorial. It is similar to the `lens_sersic` dataset used +throughout chapter 2, where: + + - The lens galaxy's light is an `Sersic`. + - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. + - The source galaxy's light is an `SersicCore`. + +However, there is one addition: an extra galaxy, with its own light (an `ExponentialSph`) and its own +mass (an `IsothermalSph`), located a few arc-seconds from the lens galaxy. +""" +dataset_name = "lens_extra_galaxy" +dataset_path = Path("dataset") / "imaging" / dataset_name + +""" +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it will be created by running the corresponding +simulator script. This ensures that all example scripts can be run without manually simulating data first. +""" +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/lens_extra_galaxy.py"], + check=True, + ) + +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +""" +When we plot the dataset, the familiar sight of a lens galaxy surrounded by a lensed source's ring of emission is +joined by a blob of light in the upper-right of the image. + +This is the extra galaxy. It is not part of the strong lens system we are studying: it is an interloper, a galaxy +that happens to lie close to the lens on the sky. Interlopers like this are extremely common in real imaging of +strong lenses, and every lens modeler has to decide what to do about them. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Light Versus Mass__ + +The decision of what to do about an extra galaxy hinges on one question, asked twice: + +**Does its light affect the analysis?** The extra galaxy's emission may extend into the region of the image we fit. +If it does, and we fit a model containing only the lens and source, the model has no component that can produce +this emission. The fit will respond by distorting the lens light, source light or mass model to soak it up, +biasing every parameter we infer. + +**Does its mass affect the analysis?** The extra galaxy's mass deflects light rays, just like the lens galaxy's +mass does. If the galaxy is close (in projection) to the lensed source's light, its deflections measurably perturb +the ray-tracing. A model without this mass will compensate by biasing the main lens galaxy's mass parameters, for +example its `einstein_radius` or `ell_comps`. + +These two effects are dealt with separately, which gives us two approaches: + +- **Approach 1 (remove its light)**: If only the light matters, we can remove the extra galaxy's emission from the + fit entirely, without adding anything to the model. The mass is ignored. + +- **Approach 2 (model it explicitly)**: If the mass matters (or the light blends too closely with the lensed source + to cleanly remove), we include the extra galaxy in the lens model, with its own light and / or mass profiles. + +We will now perform both, and at the end of the tutorial discuss when each is appropriate. + +__Mask__ + +We first define the circular mask used to fit the data. In chapter 2 we typically used a 2.6" - 3.0" mask, which +tightly contained the lens and lensed source. + +Here, we use a larger 4.0" mask, so that the region containing the extra galaxy is included in the fit. If we +simply shrank the mask to exclude the extra galaxy, we would also throw away pixels containing lensed source +emission, and the mask's hard edge could still cut through the extra galaxy's light. +""" +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=4.0, +) + +dataset = dataset.apply_mask(mask=mask) + +""" +Plotting the masked dataset confirms the extra galaxy's emission is inside the mask, and will therefore impact the +model-fit unless we do something about it. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Approach 1 Noise Scaling__ + +Our first approach removes the extra galaxy's light from the fit. + +The most obvious way to do this would be to mask the extra galaxy's pixels, removing them from the fit entirely. +However, removing pixels changes the fit in subtle ways: their coordinates are no longer used in the ray-tracing, +and for certain models (e.g. the pixelized source reconstructions of chapter 3) removing interior pixels creates +discontinuities in the pixelization that introduce unexpected systematics. + +Instead, we use **noise scaling**: the pixels stay in the fit, but their data values are set to zero and their +noise-map values are increased to very large values. A pixel with enormous noise contributes negligibly to the +likelihood, so the extra galaxy's light cannot influence the model, while the pixels themselves remain part of the +fit's geometry. + +To do this we need a mask of the extra galaxy's region. For real data, you would create this yourself by +inspecting the image (the workspace's `data_preparation` package includes a GUI for drawing it); for this simulated +dataset the simulator script has already output a `mask_extra_galaxies.fits` circle covering the extra galaxy. + +We reload the dataset first, because noise scaling must be applied before the circular mask. +""" +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +mask_extra_galaxies = al.Mask2D.from_fits( + file_path=dataset_path / "mask_extra_galaxies.fits", + pixel_scales=0.1, + invert=True, # Note that we invert the mask here as `True` means a pixel is scaled. +) + +dataset = dataset.apply_noise_scaling(mask=mask_extra_galaxies) + +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=4.0, +) + +dataset = dataset.apply_mask(mask=mask) + +""" +Plotting the dataset shows the extra galaxy's emission has vanished: its data values are zero and the +signal-to-noise of its pixels is effectively zero, so the fit will simply ignore that region of the image. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Noise Scaling Fit__ + +We now fit this noise-scaled dataset with a lens model that does **not** include the extra galaxy. The model is the +same one we used in chapter 2's linear profiles tutorial: a linear `Sersic` lens bulge, an `Isothermal` plus +`ExternalShear` mass model and a linear `SersicCore` source. +""" +# Lens: + +bulge = af.Model(al.lp_linear.Sersic) +mass = af.Model(al.mp.Isothermal) + +lens = af.Model( + al.Galaxy, redshift=0.5, bulge=bulge, mass=mass, shear=al.mp.ExternalShear +) + +# Source: + +source = af.Model(al.Galaxy, redshift=1.0, bulge=al.lp_linear.SersicCore) + +# Overall Lens Model: + +model = af.Collection(galaxies=af.Collection(lens=lens, source=source)) + +search = af.Nautilus( + path_prefix=Path("howtolens", "chapter_4"), + name="tutorial_1_extra_galaxies_noise_scaling", + unique_tag=dataset_name, + n_live=100, + n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations +) + +analysis = al.AnalysisImaging(dataset=dataset) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_1_extra_galaxies_noise_scaling" + " folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result_noise_scaling = search.fit(model=model, analysis=analysis) + +print("The search has finished run - you may now continue the notebook.") + +""" +Plotting the maximum log likelihood fit shows the lens and source are fitted well, with the noise-scaled region +contributing nothing to the residuals. + +The extra galaxy's light has been dealt with. However, note what this approach did **not** do: the extra galaxy's +mass is completely absent from the model. Its deflection of the source's light rays is unaccounted for, and the +inferred lens mass model will have absorbed that perturbation as best it can. For an extra galaxy that is small +and / or far from the lensed emission this bias is negligible; for one that is massive and close, it is not. +""" +aplt.subplot_fit_imaging(fit=result_noise_scaling.max_log_likelihood_fit) + +""" +__Approach 2 Extra Galaxies Model__ + +Our second approach includes the extra galaxy in the lens model, fitting its light (so we no longer need to remove +it from the data) and its mass (so its deflections are included in the ray-tracing). + +We reload the dataset and apply the 4.0" circular mask, but this time we do **not** apply noise scaling, because +the extra galaxy's emission is now something the model itself will fit. +""" +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=4.0, +) + +dataset = dataset.apply_mask(mask=mask) + +""" +__Extra Galaxy Centres__ + +To include the extra galaxy in the model, we input the centre of its light, as observed in the image. + +In principle, we could add the extra galaxy to the model with a completely free centre, and let the non-linear +search figure out where it is. In practice this is a bad idea: the extra parameters make parameter space more +complex, and fits commonly go wrong in characteristic ways. For example, the extra galaxy's light profile may +wander off and fit one of the lensed source's multiple images instead of the interloper, or its mass profile may +recentre itself onto the main lens galaxy and act as part of its mass distribution. + +Fixing each extra galaxy's light and mass centres to its observed light centre removes these failure modes and +keeps the model as simple as possible. The observed centre is an excellent estimate of the true centre, because +(unlike the lensed source) the extra galaxy's light is not distorted by lensing. + +For real data you would measure these centres from the image (the workspace's `data_preparation` package shows +how, including a GUI for marking them); for this simulated dataset the simulator has output them to a .json file, +which we load below. +""" +extra_galaxies_centres = al.Grid2DIrregular( + al.from_json(file_path=dataset_path / "extra_galaxies_centres.json") +) + +print(extra_galaxies_centres) + +""" +__Extra Galaxies Model Composition__ + +We compose the main lens and source model exactly as before. +""" +# Lens: + +bulge = af.Model(al.lp_linear.Sersic) +mass = af.Model(al.mp.Isothermal) + +lens = af.Model( + al.Galaxy, redshift=0.5, bulge=bulge, mass=mass, shear=al.mp.ExternalShear +) + +# Source: + +source = af.Model(al.Galaxy, redshift=1.0, bulge=al.lp_linear.SersicCore) + +""" +We now compose the model of the extra galaxy, using the dedicated `extra_galaxies` modeling API. + +For each extra galaxy centre (there is only one here, but the loop below scales to any number) we create a `Galaxy` +model with: + + - A linear `ExponentialSph` light profile, with its `centre` fixed to the observed centre [1 free + parameter: `effective_radius`, as the `intensity` is solved for by the linear inversion]. + + - An `IsothermalSph` mass profile, with its `centre` fixed to the observed centre [1 free + parameter: `einstein_radius`]. + +Extra galaxy mass profiles can wander to unphysically high `einstein_radius` values, degrading the fit, so we set +a `UniformPrior` with an upper limit of 0.5" to prevent this. The extra galaxy is much less massive than the main +lens (whose `einstein_radius` is around 1.6"), so this prior comfortably contains all plausible solutions. + +The extra galaxies are grouped into their own `af.Collection`, which is passed to the overall model via its +`extra_galaxies` input, alongside the `galaxies` collection containing the lens and source. This is the same +API used throughout the `autolens_workspace` for extra galaxies, and it is how **PyAutoLens** knows to include +these galaxies in the ray-tracing without treating them as the main lens or source. +""" +# Extra Galaxies: + +extra_galaxies_list = [] + +for extra_galaxy_centre in extra_galaxies_centres: + + # Extra Galaxy Light + + light = af.Model(al.lp_linear.ExponentialSph) + light.centre = extra_galaxy_centre + + # Extra Galaxy Mass + + mass = af.Model(al.mp.IsothermalSph) + mass.centre = extra_galaxy_centre + mass.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=0.5) + + # Extra Galaxy + + extra_galaxy = af.Model(al.Galaxy, redshift=0.5, light=light, mass=mass) + + extra_galaxies_list.append(extra_galaxy) + +extra_galaxies = af.Collection(extra_galaxies_list) + +# Overall Lens Model: + +model = af.Collection( + galaxies=af.Collection(lens=lens, source=source), extra_galaxies=extra_galaxies +) + +""" +The `info` attribute confirms the model includes the extra galaxy, with its fixed centres and its +free `effective_radius` and `einstein_radius` parameters. +""" +print(model.info) + +""" +__Extra Galaxies Fit__ + +We fit this model with the same search set up as before. The model has only two more free parameters than the +noise-scaling fit, thanks to the fixed centres and linear light profile, so the fit remains fast. +""" +search = af.Nautilus( + path_prefix=Path("howtolens", "chapter_4"), + name="tutorial_1_extra_galaxies_model", + unique_tag=dataset_name, + n_live=100, + n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations +) + +analysis = al.AnalysisImaging(dataset=dataset) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_1_extra_galaxies_model" + " folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result_extra_galaxies = search.fit(model=model, analysis=analysis) + +print("The search has finished run - you may now continue the notebook.") + +""" +Plotting the maximum log likelihood fit shows the extra galaxy's emission is now fitted by its own light profile, +and its mass has contributed deflections to the ray-tracing of the source. +""" +aplt.subplot_fit_imaging(fit=result_extra_galaxies.max_log_likelihood_fit) + +""" +The result's `info` shows the inferred `einstein_radius` of the extra galaxy, quantifying how much lensing power +the interloper contributes to the system. +""" +print(result_extra_galaxies.info) + +""" +__Which Approach When__ + +We have seen the two extremes of dealing with an extra galaxy. Choosing between them comes back to the +light-versus-mass question at the start of this tutorial, which in practice is settled by three properties of the +interloper: + +- **Distance from the lensed emission**: An extra galaxy whose light is well separated from the lensed source (as + in this tutorial) can be cleanly noise-scaled away. If its light blends into the source's arcs, noise scaling + would also delete source emission we need, and the light must be modeled instead. + +- **Brightness**: A faint interloper whose emission barely rises above the noise can often be ignored entirely, or + noise-scaled with no consequence. A bright one must be removed or modeled, or it will bias the fit. + +- **Mass and proximity to the lens**: The mass matters when the extra galaxy is massive enough, and close enough + (in projection) to the lensed source's light, for its deflections to measurably perturb the ray-tracing. A rough + rule of thumb is to compare its expected Einstein radius to the astrometric precision of the data: a small galaxy + several arc-seconds from the arcs can have its mass safely ignored, whereas one abutting the Einstein ring + cannot. When in doubt, fit both approaches and compare the inferred lens models: if the main lens's mass + parameters shift appreciably, the extra galaxy's mass matters. + +Intermediate options also exist and are fully supported by the API: you can noise-scale the light but still include +the mass profile in the model, or model the light but omit the mass. The `extra_galaxies` collection simply +contains whatever profiles you give it. + +__Wrap Up__ + +In this tutorial, we took the first step up in scale from the single lens galaxy of chapters 1 to 3, and learnt: + +1. Real lenses live in crowded fields, and interloping extra galaxies are the first complication real data throws + at us. + +2. Whether an extra galaxy matters hinges on two separate questions: does its **light** contaminate the region of + the image we fit, and does its **mass** perturb the ray-tracing of the source? + +3. Noise scaling removes an extra galaxy's light from the fit without adding model complexity, by zeroing its data + and inflating its noise, but leaves its mass unaccounted for. + +4. The `extra_galaxies` modeling API includes extra galaxies in the model with their own light and mass profiles, + with their centres fixed to the observed light centres to keep parameter space simple and well behaved. + +5. Which approach is appropriate depends on the interloper's distance from the lensed emission, its brightness, + and its mass's proximity to the lens. + +Throughout, the extra galaxy was a nuisance: something to remove or account for, so that our analysis of the +main lens remained accurate. In the next tutorial we meet systems where that framing breaks down entirely, because +a second galaxy is not a nuisance but a co-equal lens, with light and mass comparable to the first. There, no +single galaxy is "the" lens, and the model must treat them all on an equal footing. +""" diff --git a/scripts/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.py b/scripts/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.py new file mode 100644 index 0000000..84c05e3 --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.py @@ -0,0 +1,488 @@ +""" +Tutorial 2: Multi-Galaxy Lenses +=============================== + +In the previous tutorial, we learned how to deal with extra galaxies near a strong lens — nuisance objects whose +light contaminates the data but which play no meaningful role in the lensing itself. We removed their emission +from the analysis, or gave them a heavily restricted model, and the single dominant lens galaxy remained the +star of the show. + +In this tutorial, we meet systems where that picture breaks down entirely: **multi-galaxy lenses**, where two +(or more) galaxies of comparable mass both contribute significantly to the lensing of a single background source. +Neither galaxy is a minor perturber we can mask away or simplify — they are **co-dominant deflectors**, and every +one of them needs its own free light and mass model. + +How do such systems arise physically? There are two main channels: + +- **Merging galaxies**: two galaxies at the same redshift caught in the act of merging, or orbiting one another + in close proximity. Their projected separation is small compared to the Einstein radius of their combined mass, + so the source's light rays feel both galaxies' gravity at once. Because both deflectors are at one redshift, + ray tracing is single-plane: their deflection angles simply add. + +- **Chance line-of-sight alignments**: two physically unrelated galaxies at *different* redshifts that happen to + lie along the same line of sight. Light from the source is deflected first by the more distant galaxy, and the + already-deflected rays are deflected again by the nearer one. This is compound, multi-plane ray tracing — the + `Tracer` handles it natively by simply assigning each galaxy its redshift, and we will defer its details to + later in this chapter, where multi-plane lensing becomes the default at cluster scale. + +In this tutorial, both lens galaxies are at the same redshift, so we stay in the simpler single-plane regime and +focus on the modeling challenge that defines multi-galaxy lensing: the growth of the model's parameter space. + +__Contents__ + +- **Initial Setup:** Load the double lens galaxy dataset, simulating it first if it is not on disk. +- **Mask:** Define a mask which encloses the combined Einstein ring of both lens galaxies. +- **Over Sampling:** Centre the adaptive over sampling grid on every deflector, not just one. +- **Model:** Compose a lens model with a free light and mass model per deflector, and count its parameters. +- **Fixing the Mass Centres:** Fix each galaxy's mass centre to its observed light centre, and why this is standard. +- **Model Fit:** Fit the two-deflector model to the data with a non-linear search. +- **Result:** Inspect the combined critical curve and the fit to the data. +- **Mass Degeneracies:** The total mass is well constrained, but its split between the galaxies is not. +- **No Shared Halo:** What we are deliberately not yet assuming, and why that changes at group scale. +- **Three Lens Galaxies:** A triple-deflector system, and how the parameter count keeps growing. +- **Wrap Up:** Summary and the road to scaling relations. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +from pathlib import Path +import autofit as af +import autolens as al +import autolens.plot as aplt + +""" +__Initial Setup__ + +We begin with `Imaging` of a strong lens where **two** lens galaxies, separated by 2.0", both deflect the light +of a single background source: + + - The two lens galaxies' light are `Sersic` profiles, centred at (0.0", -1.0") and (0.0", 1.0"). + - The two lens galaxies' mass distributions are `Isothermal` profiles with comparable Einstein radii (1.0" and + 0.8") — this comparability is what makes them co-dominant. + - The source galaxy's light is an `ExponentialCoreSph`. + +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it will be created by running the corresponding +simulator script. This ensures that all example scripts can be run without manually simulating data first. +""" +dataset_name = "x2_lens_galaxies" +dataset_path = Path("dataset") / "imaging" / dataset_name + +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/lens_x2.py"], + check=True, + ) + +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +""" +When we plot the data, the difference from every lens we have modeled so far is obvious: there are two bright +lens galaxies, and the lensed source's arcs wrap around the *pair as a whole* rather than around either galaxy +individually. The mass distribution the source's light responds to is the sum of both galaxies' mass. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Mask__ + +We define a 3.0" circular mask. For a multi-galaxy lens, sizing the mask needs more care than at galaxy scale: +the Einstein radius that matters is that of the *combined* mass distribution, not either galaxy's individually. +A mask sized by eye from one galaxy's light would clip the arcs, which extend beyond both galaxies' centres. +""" +mask_radius = 3.0 + +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=mask_radius, +) + +dataset = dataset.apply_mask(mask=mask) + +""" +__Over Sampling__ + +We use the adaptive over sampling scheme introduced in earlier chapters, which evaluates the steep central +regions of the lens galaxies' light at high resolution. The one multi-galaxy specific point is that the adaptive +grid is centred on **every** deflector, not just one — each galaxy has its own steep central light profile +needing accurate evaluation, and `centre_list` takes as many centres as we give it. + +The two centres below are the observed centres of the two lens galaxies, which for this simulated dataset we +know exactly. For real data you would measure them from the image itself (the `autolens_workspace` provides a +GUI which writes them from mouse clicks). +""" +lens_centres = [(0.0, -1.0), (0.0, 1.0)] + +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=dataset.grid, + sub_size_list=[4, 2, 2], + radial_list=[0.3, 0.6], + centre_list=lens_centres, +) + +dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) + +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Model__ + +We now compose the lens model, and here the multi-galaxy regime shows its teeth. Every co-dominant deflector +needs its own free light and mass model, so the model has one `Galaxy` entry per deflector. + +We build each lens galaxy in a loop over the observed centres and store them as `lens_0`, `lens_1`, etc. This +list-based composition scales to any number of deflectors, and it is the same API the `autolens_workspace`'s +`multi_galaxy` package (and the group-scale examples above it) use — so nothing needs re-learning later. + +For each lens galaxy's light we use a Multi Gaussian Expansion (MGE) made of linear light profiles, built by the +utility function `al.model_util.mge_model_from` (this hides the longer `Basis` composition API we stepped through +in the linear profiles tutorial of chapter 2). For each lens galaxy's mass we use an `Isothermal` (SIE) profile, +initially with every parameter free. The source is also an MGE. + +The MGE choice matters more here than anywhere we have used it before. Each lens galaxy's 20 Gaussians add just +**4 non-linear parameters** (a shared centre and shared elliptical components — every `sigma` is fixed and every +`intensity` is solved for by the linear inversion). A single ordinary `Sersic` bulge would add 7 non-linear +parameters per galaxy, and would fit the blended, overlapping light of a close pair far less well. Because the +multi-galaxy regime multiplies whatever light model we choose by the number of deflectors, a light model that is +both cheap and flexible is close to essential — this is exactly where the MGE shines. + +Each Isothermal mass model adds 5 free parameters: its centre (2), elliptical components (2) and Einstein +radius (1). + +(The `autolens_workspace` multi-galaxy examples additionally include a single external shear for the whole +system, held in its own model entry at the system centre rather than attached to any one deflector — the shear +describes the tidal field of structure *outside* the system, so it belongs to no individual galaxy. We omit it +here to keep the parameter accounting simple.) +""" +lens_dict = {} + +for i, centre in enumerate(lens_centres): + + bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=20, + centre_prior_is_uniform=True, + centre=(centre[0], centre[1]), + sigma_min=dataset.pixel_scales[0] / 10.0, + ) + + mass = af.Model(al.mp.Isothermal) + + lens_dict[f"lens_{i}"] = af.Model( + al.Galaxy, + redshift=0.5, + bulge=bulge, + mass=mass, + ) + +bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=20, + gaussian_per_basis=1, + centre_prior_is_uniform=False, +) + +source = af.Model(al.Galaxy, redshift=1.0, bulge=bulge) + +model = af.Collection(galaxies=af.Collection(**lens_dict, source=source)) + +""" +The model's `info` shows `lens_0` and `lens_1` each carrying their own free mass model — the signature of the +multi-galaxy regime — and its `prior_count` gives the total number of free parameters. +""" +print(model.info) + +print(f"Free parameters (free mass centres): {model.prior_count}") + +""" +The count is **22**: each deflector contributes 4 (MGE light) + 5 (SIE mass) = 9 free parameters, and the MGE +source contributes the final 4. Compare this to the equivalent single-galaxy model, which has just 13 — one +co-dominant deflector added 9 parameters, and every further deflector will add 9 more. Model complexity compounds +fast in this regime, and with plain Sersic light profiles instead of MGEs it would compound faster still (12 per +deflector rather than 9). + +__Fixing the Mass Centres__ + +Before fitting, we make one change that is standard practice for multi-galaxy lenses: we **fix each galaxy's +mass centre to its observed light centre**, removing 2 free parameters per deflector. + +Why? With a single lens galaxy, the data usually constrains the mass centre well: the arcs pin down where the +deflection field is centred, and there is only one plausible culprit. With multiple deflectors this breaks down. +The source's light responds to the *summed* deflection field, so a small shift of one galaxy's mass centre can be +compensated by shifts of the other's centre, ellipticity or Einstein radius — the free centres become extremely +degenerate with one another. The search wanders these degeneracies, converging slowly and often settling in +unphysical corners of parameter space where one galaxy's mass has drifted far from any light. + +Fixing each mass centre to the galaxy's light centre resolves this at minimal cost. Light traces stellar mass, +and the stellar body sits at the bottom of the galaxy's potential well, so the light centre is an excellent +estimate of the mass centre. (Measuring a genuine offset between mass and light — which interacting pairs can +show at the kiloparsec level — is real science, but it is a *follow-up* fit performed after a robust model with +fixed centres has been found.) + +Assigning a tuple to the mass model's `centre` fixes it, so it is no longer a free parameter with a prior. +""" +lens_dict = {} + +for i, centre in enumerate(lens_centres): + + bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=20, + centre_prior_is_uniform=True, + centre=(centre[0], centre[1]), + sigma_min=dataset.pixel_scales[0] / 10.0, + ) + + mass = af.Model(al.mp.Isothermal) + mass.centre = (centre[0], centre[1]) + + lens_dict[f"lens_{i}"] = af.Model( + al.Galaxy, + redshift=0.5, + bulge=bulge, + mass=mass, + ) + +bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=20, + gaussian_per_basis=1, + centre_prior_is_uniform=False, +) + +source = af.Model(al.Galaxy, redshift=1.0, bulge=bulge) + +model = af.Collection(galaxies=af.Collection(**lens_dict, source=source)) + +""" +Printing the model's `info` again, each mass `centre` is now listed as a fixed value with no prior, and the +free parameter count has dropped from 22 to **18** — each SIE now contributes 3 free parameters instead of 5. +""" +print(model.info) + +print(f"Free parameters (fixed mass centres): {model.prior_count}") + +""" +__Model Fit__ + +We fit the model with the nested sampling algorithm `Nautilus`, as in previous chapters. Because this model has +more free parameters than the single-galaxy fits of chapter 2 (which used 100 live points), we raise `n_live` +to 200 — a multi-galaxy parameter space is more multi-modal, and too few live points risks converging on a local +maximum where the two galaxies' Einstein radii have been mis-apportioned. +""" +search = af.Nautilus( + path_prefix=Path("howtolens", "chapter_4"), + name="tutorial_2_multi_galaxy", + unique_tag=dataset_name, + n_live=200, + n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations +) + +analysis = al.AnalysisImaging(dataset=dataset) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_2_multi_galaxy" + " folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result = search.fit(model=model, analysis=analysis) + +""" +__Result__ + +The result's `info` shows the inferred model, with each deflector's parameters listed under its own `lens_0` / +`lens_1` entry. +""" +print(result.info) + +""" +The tracer subplot is worth a close look for a multi-galaxy lens: the critical curve is that of the *combined* +mass distribution, so it wraps around the pair as a whole rather than encircling either galaxy individually — +just as the arcs in the data do. +""" +aplt.subplot_tracer(tracer=result.max_log_likelihood_tracer, grid=result.grids.lp) + +aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) + +""" +__Mass Degeneracies__ + +The corner plot of the posterior is where the multi-galaxy regime reveals its most important lesson. Below we +plot the probability density function of every parameter; when you run this tutorial for real (rather than +skimming the generated output), focus on the panel showing `lens_0`'s Einstein radius against `lens_1`'s. +""" +aplt.corner_anesthetic(samples=result.samples) + +""" +You will see a strongly elongated, tilted degeneracy: the two Einstein radii are *anti-correlated*. When one +galaxy's Einstein radius goes up, the other's goes down, tracing out a narrow ridge in parameter space. + +The physical reason is the summed deflection field. The arcs constrain the **total** mass enclosed by the +combined Einstein ring extremely well — move along the ridge and the sum of the two galaxies' deflections barely +changes, so the fit to the data barely changes either. What the data constrains much more weakly is the **split** +of that mass between the two galaxies. The closer the pair (relative to the combined Einstein radius), the less +information the arcs carry about which galaxy the mass belongs to, and the longer the ridge grows. + +This has a real scientific cost. For a single-galaxy lens, the Einstein mass is one of the cleanest measurements +in astrophysics. For a multi-galaxy lens, the total is still clean, but per-galaxy quantities — each galaxy's +mass, its mass-to-light ratio, its dark matter fraction — inherit the full width of the degeneracy, and their +error bars can be many times larger than the total's. Any interpretation built on the split (e.g. comparing the +two galaxies' dark matter content) must honestly propagate this, which is why we fit these systems with a +sampler that maps the full posterior rather than an optimizer that returns a single best-fit point: the best-fit +point sits somewhere on the ridge and tells you nothing about the ridge's length. + +__No Shared Halo__ + +It is worth being explicit about an assumption we did *not* make. We modeled two galaxies, each with its own +untruncated `Isothermal` mass profile, and simply added their deflections. We did not assume the pair inhabits a +single, large, shared dark matter halo — and because there is no host halo in the model, we also made no +assumptions about tidal stripping or the truncation of each galaxy's individual halo. + +At this scale, that is the right call: a pair (or triple) of comparable galaxies has no dominant halo for its +members to be stripped by, and the data cannot demand one. But climb the mass ladder and it changes. At the +group scale, a dominant group-sized halo enters the model as an explicit choice, and the member galaxies orbiting +within it are tidally truncated by its potential — assumptions that reshape the entire mass model, as we will +see later in this chapter. + +__Three Lens Galaxies__ + +What happens when a third co-dominant deflector joins the system? Let us load a triple-galaxy lens, simulated +with three `Isothermal` mass profiles of comparable Einstein radii (0.9", 0.8" and 0.7") arranged in a triangle, +all lensing a single source. +""" +dataset_name = "x3_lens_galaxies" +dataset_path = Path("dataset") / "imaging" / dataset_name + +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/lens_x3.py"], + check=True, + ) + +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +We compose the three-deflector model with exactly the same loop as before — this is the payoff of the list-based +API, which needs no changes as the deflector count grows. Each mass centre is again fixed to its galaxy's +observed light centre; with three deflectors the centre degeneracies are even more severe than for the pair, so +the standard trick matters even more. +""" +lens_centres = [(0.9, 0.0), (-0.6, -0.9), (-0.6, 0.9)] + +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=mask_radius, +) + +dataset = dataset.apply_mask(mask=mask) + +lens_dict = {} + +for i, centre in enumerate(lens_centres): + + bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=20, + centre_prior_is_uniform=True, + centre=(centre[0], centre[1]), + sigma_min=dataset.pixel_scales[0] / 10.0, + ) + + mass = af.Model(al.mp.Isothermal) + mass.centre = (centre[0], centre[1]) + + lens_dict[f"lens_{i}"] = af.Model( + al.Galaxy, + redshift=0.5, + bulge=bulge, + mass=mass, + ) + +bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=20, + gaussian_per_basis=1, + centre_prior_is_uniform=False, +) + +source = af.Model(al.Galaxy, redshift=1.0, bulge=bulge) + +model = af.Collection(galaxies=af.Collection(**lens_dict, source=source)) + +print(model.info) + +print(f"Free parameters (three deflectors, fixed mass centres): {model.prior_count}") + +""" +The count is now **25**: even with fixed mass centres and cheap MGE light models, every deflector adds 7 free +parameters (4 light + 3 mass), and with free centres it would be 9. The trend is linear and unforgiving: + + - 1 deflector: 13 free parameters. + - 2 deflectors: 18 (this tutorial's fit). + - 3 deflectors: 25. + - 10 deflectors: ~74. A group-scale lens can easily have this many members. + +And the parameter count is only half the problem. As the deflectors multiply, so do the degeneracies between +their masses — every pair of galaxies gets its own version of the Einstein radius ridge we saw above, and the +posterior becomes progressively harder for any sampler to map. + +You are welcome to fit this three-galaxy model yourself — the search above works unchanged, though expect it to +take noticeably longer than the pair. Clearly, freeing every galaxy's mass cannot scale to the tens or hundreds +of member galaxies in groups and clusters. Something has to give. + +__Wrap Up__ + +In this tutorial, we modeled strong lenses with multiple co-dominant deflectors. Let's summarise what we've +learnt: + +- **Co-dominant deflectors**: When two or more galaxies of comparable mass both contribute significantly to the + lensing, every one of them needs its own free light and mass model — unlike the nuisance neighbours of the + previous tutorial, none can be masked away or simplified. + +- **Physical origins**: Multi-galaxy lenses arise from merging galaxies at one redshift (single-plane, deflections + add) or chance line-of-sight alignments at different redshifts (multi-plane ray tracing, covered later in this + chapter). + +- **Parameter growth**: Each deflector added 9 free parameters (4 for its MGE light, 5 for its SIE mass) — and the + MGE is what kept it that cheap, since its Gaussian intensities are solved for by linear algebra rather than sampled. + +- **Fixed mass centres**: Fixing each galaxy's mass centre to its observed light centre (removing 2 parameters per + deflector) is the standard trick for taming the severe centre degeneracies of multi-deflector models. + +- **Mass degeneracies**: The data constrains the total mass well but its split between the galaxies poorly, + producing an anti-correlated ridge between the deflectors' Einstein radii — and inflating the uncertainties on + any per-galaxy science. + +- **No shared halo**: At this scale we do not assume the galaxies inhabit one large dark matter halo, so no tidal + truncation assumptions enter the model — that framing arrives at group scale. + +The three-galaxy model made the trajectory clear: freeing every deflector cannot scale. In the next tutorial, we +introduce the tool that stops this growth in its tracks — scaling relations, which tie the masses of many +galaxies to their observed luminosities so that an entire population of deflectors costs almost no extra free +parameters. +""" diff --git a/scripts/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.py b/scripts/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.py new file mode 100644 index 0000000..b78aef9 --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.py @@ -0,0 +1,607 @@ +""" +Tutorial 3: Scaling Relations +============================= + +The previous tutorial ended on a warning: every deflector we add to a lens model brings its own free parameters. +Two galaxies were manageable, but each one cost us a mass profile's worth of dimensions, and the arithmetic only +gets worse. A group-scale lens may have ten member galaxies, a cluster hundreds. If every member keeps its own +free mass, the parameter space explodes — a 100-galaxy cluster with 5 free mass parameters per galaxy is a +500-dimensional model, which no non-linear search can sample and no dataset can constrain anyway. + +This tutorial introduces the tool that stops the explosion: the **scaling relation**. Instead of giving every +galaxy its own mass freedom, we set each galaxy's mass from something we can *measure* — its luminosity — via a +relation shared by the whole population. The relation itself has only a few global parameters (in this tutorial, +zero new ones!), so the dimensionality of the model stops growing with the number of galaxies. Ten members or a +hundred, the parameter space stays the same size. + +This pairs beautifully with a tool we already have. In chapter 2, we saw that a Multi-Gaussian Expansion (MGE) +fits a galaxy's light using linear algebra, so its intensities add no non-linear parameters. Put the two +together and the recipe for a many-galaxy lens model emerges: + +- **Light:** MGE for every galaxy — flexible light models at (almost) no non-linear cost. +- **Mass:** a scaling relation tying every galaxy's mass to its measured luminosity — a whole population of + mass profiles at no non-linear cost. + +The entire many-galaxy system then costs barely more than a single-lens fit. This tutorial builds that model on +the two-lens dataset from the previous tutorial, verifies the parameter counting explicitly, and is honest about +what a scaling relation assumes and when it can bite you. + +__Overview__ + +In this tutorial, we will: + +- Explain the physics of why luminosity predicts lensing mass (mass-follows-light and the Faber-Jackson relation). + +- Compose a lens model where one galaxy "anchors" the relation and the other galaxy's Einstein radius is tied to + it via the ratio of their measured luminosities. + +- Verify by explicit parameter counting that tied galaxies add zero free parameters, however many there are. + +- Fit the model and inspect how a tied parameter appears in the results. + +- Discuss the limitations: real galaxies scatter around any relation, and a member that deviates can bias the + whole lens model. + +__Contents__ + +- **Mass Follows Light:** The physics linking a galaxy's luminosity to its lensing mass. +- **Initial Setup:** Load the two-lens-galaxy imaging dataset from the previous tutorial. +- **Mask:** Define the 2D mask applied to the dataset for the model-fit. +- **Measured Luminosities:** The luminosities the relation needs, and where they come from. +- **The Anchor:** Identify the brightest galaxy, which anchors the relation. +- **Over Sampling:** Adaptive over-sampling centred on both lens galaxies. +- **Light Via MGE:** MGE light models for every galaxy, at a cost of two parameters each. +- **Scaling Relation:** Tie the fainter galaxy's Einstein radius to the anchor's. +- **Model:** Compose the model with the scaling tier as its own collection. +- **Parameter Counts:** Compare against the previous tutorial's per-galaxy model. +- **Scaling To Many Galaxies:** Show the model does not grow when 12 more members are tied. +- **Model Fit:** Fit the scaling-relation model with Nautilus. +- **Results:** How a tied parameter appears in the results, and how close the truth is to the relation. +- **Limitations:** Scatter, deviant members, and adding per-galaxy freedom selectively. +- **Lens Environments:** The same machinery models the environment of single-lens systems. +- **Wrap Up:** Summary of the script and next steps. + +__Mass Follows Light__ + +Why should a galaxy's luminosity tell us anything about its lensing mass? + +The starting point is that light traces stars and stars trace mass. A more luminous elliptical galaxy contains +more stars, sits in a deeper potential well, and its stars therefore move faster. This is quantified by the +**Faber-Jackson relation**, an empirical scaling law for elliptical galaxies which states that luminosity grows +steeply with the velocity dispersion \sigma of the stars: + + L ~ \sigma^4 + +Lensing connects \sigma to the deflection of light. For the isothermal mass profiles we have used throughout +these tutorials, the Einstein radius is set directly by the velocity dispersion: + + \theta_E = 4 \pi (\sigma / c)^2 (D_ls / D_s) + +where the D's are distances between observer, lens and source. So \theta_E ~ \sigma^2, and combining the two +scalings gives us a relation between the two things we care about — the Einstein radius we want to know and the +luminosity we can measure: + + \theta_E ~ \sigma^2 ~ (L^{1/4})^2 = L^{1/2} + +If one galaxy in a lens system has a measured luminosity L_anchor and Einstein radius \theta_E_anchor, every +other galaxy's Einstein radius follows from its own luminosity: + + \theta_E_i = \theta_E_anchor * (L_i / L_anchor)^{0.5} + +This is the scaling relation we will build into the model below. The exponent 0.5 is the Faber-Jackson value; +more sophisticated versions exist (the "fundamental plane" adds a dependence on galaxy size and surface +brightness, tightening the relation), but the Faber-Jackson form captures the essential physics and is the +standard workhorse for lens modeling. It is the same idea that cluster lensing studies have used for decades: +tie the hundreds of cluster member galaxies to their luminosities via a scaling relation, so the model stays +tractable. + +The crucial property is that only *ratios* of luminosity enter the relation. The absolute calibration, the +units, even the waveband cancel out (so long as all galaxies are measured consistently) — which is what makes +the relation so easy to apply in practice. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +import numpy as np +from pathlib import Path +import autofit as af +import autolens as al +import autolens.plot as aplt + +""" +__Initial Setup__ + +We use the same two-lens-galaxy imaging dataset as the previous tutorial, where: + + - Both lens galaxies' light are `Sersic` profiles. + - Both lens galaxies' mass distributions are `Isothermal` profiles. + - The source galaxy's light is an `ExponentialCoreSph`. +""" +dataset_name = "x2_lens_galaxies" +dataset_path = Path("dataset") / "imaging" / dataset_name + +""" +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it will be created by running the corresponding +simulator script. This ensures that all example scripts can be run without manually simulating data first. +""" +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/lens_x2.py"], + check=True, + ) + +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Mask__ + +We define a 3.0" circular mask, which contains both lens galaxies (centred at (0.0", -1.0") and (0.0", 1.0")) +and the lensed source emission. +""" +mask_radius = 3.0 + +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=mask_radius, +) + +dataset = dataset.apply_mask(mask=mask) + +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Measured Luminosities__ + +The scaling relation needs two inputs for every galaxy it touches: its centre and its luminosity. Neither is a +free parameter of the model — both are **measurements**, made from the imaging data before the mass model is +ever fitted. + +In a real analysis these come from a light-only fit performed first: for example, fitting an MGE to every +galaxy and integrating it to a total flux (the `autolens_workspace`'s scaling-relation SLaM pipeline does exactly +this). Because only luminosity *ratios* enter the relation, a magnitude catalogue from any survey works just as +well, converting via `L_i / L_anchor = 10 ** (0.4 * (m_anchor - m_i))`. + +In this tutorial we take a shortcut available only with simulated data: the simulator saved the true galaxies to +a `tracer.json` file, so we load it and integrate each lens galaxy's true light profile directly. This keeps the +tutorial self-contained, but keep in mind that on real data this step is a fit to the data, with its own +(small) uncertainties. +""" +tracer_true = al.from_json(file_path=dataset_path / "tracer.json") + +lens_galaxies_true = [ + galaxy for galaxy in tracer_true.galaxies if galaxy.redshift == 0.5 +] + +centres = [ + tuple(float(value) for value in galaxy.bulge.centre) + for galaxy in lens_galaxies_true +] + +luminosities = [ + galaxy.bulge.luminosity_within_circle_from(radius=mask_radius) + for galaxy in lens_galaxies_true +] + +print(f"Lens galaxy centres: {centres}") +print(f"Lens galaxy luminosities: {luminosities}") + +""" +__The Anchor__ + +The relation needs an "anchor": one galaxy whose Einstein radius is a free parameter of the model, off which +every other galaxy's Einstein radius hangs. We use the brightest galaxy, identified by `argmax` over the +measured luminosities — a measurement, not an assumption about which galaxy happens to be listed first. + +Anchoring on a galaxy the model is already fitting is the key trick: the anchor's `einstein_radius` is not a new +parameter, so the relation itself will add **zero** free parameters to the model. +""" +anchor_index = int(np.argmax(luminosities)) + +luminosity_anchor = luminosities[anchor_index] +centre_anchor = centres[anchor_index] + +tied_indexes = [i for i in range(len(centres)) if i != anchor_index] + +print(f"Anchor galaxy index: {anchor_index}, L_anchor = {luminosity_anchor:.4f}") + +""" +__Over Sampling__ + +As in the previous tutorial, we apply adaptive over-sampling centred on both lens galaxies, so the steeply +varying central light of each is computed accurately. +""" +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=dataset.grid, + sub_size_list=[4, 2, 2], + radial_list=[0.3, 0.6], + centre_list=centres, +) + +dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) + +""" +__Light Via MGE__ + +Every galaxy's light is modelled with a Multi-Gaussian Expansion, composed via the `al.model_util.mge_model_from` +convenience function (this wraps the `Basis` of linear `Gaussian` profiles we built by hand in chapter 2 into a +single call — the composition it returns is the same). + +Because the galaxy centres were measured above, we fix each MGE's centre to them (passing the measured centre as +both the `centre` and `centre_fixed` inputs, which fixes it rather than giving it a prior). The Gaussian +`sigma` values are fixed by the basis and the intensities are solved for by linear algebra, so each galaxy's +light costs just **two** non-linear parameters: its elliptical components. This is the "light adds (almost) no +parameters" half of the recipe from the introduction. + +The anchor galaxy also gets its mass: a free `Isothermal` profile with its centre fixed at the measured light +centre. Its `einstein_radius` is an ordinary free parameter — but it is also what the scaling relation below +hangs off. +""" +bulge_anchor = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=10, + centre=centre_anchor, + centre_fixed=centre_anchor, + sigma_min=dataset.pixel_scales[0] / 10.0, +) + +mass_anchor = af.Model(al.mp.Isothermal) +mass_anchor.centre = centre_anchor + +lens_anchor = af.Model( + al.Galaxy, + redshift=0.5, + bulge=bulge_anchor, + mass=mass_anchor, +) + +""" +The source galaxy is also an MGE, with a free centre (we have no pre-measured position for the unlensed source). +""" +source_bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=10, + gaussian_per_basis=1, + centre_prior_is_uniform=False, +) + +source = af.Model(al.Galaxy, redshift=1.0, bulge=source_bulge) + +""" +__Scaling Relation__ + +Now the relation itself. The fainter galaxy's mass is a spherical isothermal profile whose `einstein_radius` is +not given a prior — instead, it is *assigned an expression*: the anchor's `einstein_radius` multiplied by the +luminosity ratio raised to the Faber-Jackson exponent of 0.5. + +This single line is the whole trick. Because `mass_anchor.einstein_radius` is the model's own free parameter, +multiplying it by a fixed number produces a **derived quantity**, not a new parameter. Whenever the non-linear +search proposes an Einstein radius for the anchor, every tied galaxy's Einstein radius follows automatically. + +The tied galaxy keeps its MGE bulge (its light is inside the mask and must be fitted), so it costs two light +parameters and **zero** mass parameters. + +An honest aside before we continue: this dataset's two galaxies are comparably bright, and for a genuine system +of two co-dominant deflectors the previous tutorial's model — full mass freedom for both — is the right choice. +A galaxy contributing half the lensing deserves its own parameters. We tie one here so you can learn the +machinery on a familiar dataset; the regime where the relation genuinely earns its keep is a *population* of +fainter members, which we build towards below and which dominates the next tutorial. +""" +scaling_exponent = 0.5 + +scaling_galaxies_list = [] + +for i in tied_indexes: + bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=10, + centre=centres[i], + centre_fixed=centres[i], + sigma_min=dataset.pixel_scales[0] / 10.0, + ) + + mass = af.Model(al.mp.IsothermalSph) + mass.centre = centres[i] + mass.einstein_radius = ( + mass_anchor.einstein_radius + * (luminosities[i] / luminosity_anchor) ** scaling_exponent + ) + + scaling_galaxies_list.append( + af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass) + ) + +scaling_galaxies = af.Collection(scaling_galaxies_list) + +""" +__Model__ + +The model has two top-level collections: `galaxies` (the anchor and the source) and `scaling_galaxies` (the tied +population). This is not just presentational — `scaling_galaxies` is the collection name **PyAutoLens** expects a +scaling population in: the analysis appends it to the tracer's galaxy list when fitting, and results loaded back +via the aggregator restore it. It also keeps `model.info` readable, with the tied population visibly separate +from the freely modelled galaxies. + +Inspecting `model.info` below, note that the tied galaxies have no `einstein_radius` prior of their own — it is +listed as a function of the anchor's. +""" +model = af.Collection( + galaxies=af.Collection(lens=lens_anchor, source=source), + scaling_galaxies=scaling_galaxies, +) + +print(model.info) + +""" +__Parameter Counts__ + +Let's verify the claims above by counting, rather than believing. First we recompose the previous tutorial's +style of model on this dataset — every galaxy with its own free elliptical `Isothermal` mass (centres fixed at +the measured light centres, as above, so the comparison is like-for-like). +""" +per_galaxy_lens_dict = {} + +for i in range(len(centres)): + bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=10, + centre=centres[i], + centre_fixed=centres[i], + sigma_min=dataset.pixel_scales[0] / 10.0, + ) + + mass = af.Model(al.mp.Isothermal) + mass.centre = centres[i] + + per_galaxy_lens_dict[f"lens_{i}"] = af.Model( + al.Galaxy, redshift=0.5, bulge=bulge, mass=mass + ) + +model_per_galaxy = af.Collection( + galaxies=af.Collection(**per_galaxy_lens_dict, source=source), +) + +print(f"Free parameters, per-galaxy model (previous tutorial): {model_per_galaxy.prior_count}") +print(f"Free parameters, scaling-relation model: {model.prior_count}") + +""" +The scaling-relation model is smaller. To isolate exactly what the relation saves, we compose one more variant: +identical to the scaling-relation model in every way, except each tied galaxy's `einstein_radius` is freed with +a uniform prior instead of tied. The difference in parameter count must equal the number of tied galaxies — +one saved `einstein_radius` each. +""" +freed_galaxies_list = [] + +for i in tied_indexes: + bulge = al.model_util.mge_model_from( + mask_radius=mask_radius, + total_gaussians=10, + centre=centres[i], + centre_fixed=centres[i], + sigma_min=dataset.pixel_scales[0] / 10.0, + ) + + mass = af.Model(al.mp.IsothermalSph) + mass.centre = centres[i] + mass.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=2.0) + + freed_galaxies_list.append( + af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass) + ) + +model_freed = af.Collection( + galaxies=af.Collection(lens=lens_anchor, source=source), + scaling_galaxies=af.Collection(freed_galaxies_list), +) + +print(f"\nGalaxies tied by the relation: {len(scaling_galaxies_list)}") +print(f"Free parameters, tier tied: {model.prior_count}") +print(f"Free parameters, tier freed: {model_freed.prior_count}") + +assert model_freed.prior_count - model.prior_count == len(scaling_galaxies_list) + +""" +__Scaling To Many Galaxies__ + +Saving one parameter looks modest — the point is what happens as the population grows. Below we imagine this +lens had twelve additional member galaxies (we invent centres on a ring and plausible luminosities; this model +is composed for counting only, **not** fitted — our dataset contains no such galaxies!). + +Each imagined member is mass-only (faint distant perturbers whose light is negligible or outside the mask need +no light model — the mass-only counterpart of the intermediate options tutorial 1 discussed), with its Einstein +radius tied to the anchor by the same relation. We count the parameters of this 14-galaxy lens model, and of its nightmare per-galaxy twin. +""" +total_members = 12 + +member_centres = [ + (2.5 * float(np.sin(angle)), 2.5 * float(np.cos(angle))) + for angle in np.linspace(0.0, 2.0 * np.pi, total_members, endpoint=False) +] + +member_luminosities = [ + float(luminosity) for luminosity in np.linspace(0.05, 0.5, total_members) * luminosity_anchor +] + +members_tied_list = [] +members_freed_list = [] + +for centre, luminosity in zip(member_centres, member_luminosities): + mass_tied = af.Model(al.mp.IsothermalSph) + mass_tied.centre = centre + mass_tied.einstein_radius = ( + mass_anchor.einstein_radius + * (luminosity / luminosity_anchor) ** scaling_exponent + ) + + members_tied_list.append(af.Model(al.Galaxy, redshift=0.5, mass=mass_tied)) + + mass_freed = af.Model(al.mp.IsothermalSph) + mass_freed.centre = centre + mass_freed.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=2.0) + + members_freed_list.append(af.Model(al.Galaxy, redshift=0.5, mass=mass_freed)) + +model_many_tied = af.Collection( + galaxies=af.Collection(lens=lens_anchor, source=source), + scaling_galaxies=af.Collection(scaling_galaxies_list + members_tied_list), +) + +model_many_freed = af.Collection( + galaxies=af.Collection(lens=lens_anchor, source=source), + scaling_galaxies=af.Collection(scaling_galaxies_list + members_freed_list), +) + +print(f"Free parameters, 2-galaxy scaling model: {model.prior_count}") +print(f"Free parameters, 14-galaxy scaling model: {model_many_tied.prior_count}") +print(f"Free parameters, 14-galaxy per-galaxy-mass model: {model_many_freed.prior_count}") + +assert model_many_tied.prior_count == model.prior_count + +""" +Twelve more galaxies, zero more parameters. The tied model's dimensionality is *independent of the number of +galaxies* — this is the property that makes group- and cluster-scale lens modeling possible at all, and it is +why the assertion above is worth having in the script rather than in prose. +""" + +""" +__Model Fit__ + +We now fit the scaling-relation model to the data, using the familiar `Nautilus` search and `AnalysisImaging` +object. The parameter space is barely larger than a single-lens fit, which is the whole point. +""" +search = af.Nautilus( + path_prefix=Path("howtolens") / "chapter_4", + name="tutorial_3_scaling_relation", + unique_tag=dataset_name, + n_live=100, + n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations +) + +analysis = al.AnalysisImaging(dataset=dataset) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_3_scaling_relation" + " folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result = search.fit(model=model, analysis=analysis) + +print("The search has finished run - you may now continue the notebook.") + +""" +__Results__ + +In `result.info`, the tied galaxy has no `einstein_radius` entry of its own — it is reported as a derived +function of the anchor's, which is what a tied parameter looks like in the output. Its posterior uncertainty +comes entirely from the anchor's. +""" +print(result.info) + +aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) + +""" +The fit is good — but we should ask *why* it is good, because the answer is the honest heart of this tutorial. + +The tied model can only fit well if the true galaxies actually lie close to the relation we imposed. We can +check, because this is simulated data and we know the truth. Below we compare the fainter galaxy's true Einstein +radius against the value the relation predicts from the anchor's true Einstein radius and the luminosity ratio. +""" +einstein_radii_true = [ + float(galaxy.mass.einstein_radius) for galaxy in lens_galaxies_true +] + +for i in tied_indexes: + einstein_radius_predicted = einstein_radii_true[anchor_index] * ( + (luminosities[i] / luminosity_anchor) ** scaling_exponent + ) + + print(f"Tied galaxy {i}: true einstein_radius = {einstein_radii_true[i]:.3f}") + print(f"Tied galaxy {i}: relation predicts = {einstein_radius_predicted:.3f}") + +""" +The prediction lands within a few percent of the truth: this simulated pair happens to sit almost exactly on the +Faber-Jackson relation. That is why the tied fit succeeds. Real galaxies are not always so obliging. + +__Limitations__ + +A scaling relation is an assumption, and it is worth being clear-eyed about what we assumed: + +**Scatter.** Faber-Jackson is a statistical relation with real intrinsic scatter — at fixed luminosity, +galaxies show a spread in velocity dispersion, and therefore in Einstein radius. A tied model has no freedom to +absorb this scatter: it asserts every member sits exactly on the relation. For a population of faint members +whose individual lensing effects are small, the errors average down and this is fine. For any single galaxy +that matters a lot to the fit, it is a risk. + +**Deviant members.** Worse than scatter is a member that is systematically off the relation — a tidally +stripped satellite that has lost dark matter but kept its stars, a galaxy with an unusual mass-to-light ratio, +or a misclassified foreground/background interloper assigned a luminosity at the wrong distance. Because the +tied model *cannot* adjust that one galaxy, the non-linear search compensates the only way it can: by biasing +the parameters it does control — the anchor's mass, the source — dragging the whole lens model away from the +truth. One bad member can poison an otherwise excellent fit, and nothing in the residuals will neatly point at +the culprit. + +**The anchor itself.** Anchoring on the brightest galaxy assumes the anchor sits on the relation too. If it is +the anomalous one, every tied galaxy inherits its anomaly. + +The defence against all three is **selective freedom**. The tiers can be mixed within one model: keep the long +tail of faint members tied, but promote any galaxy that is bright, close to the lensed images, or visibly +suspicious to its own free mass profile — exactly as the extra galaxies were treated earlier in this chapter. A +middle option bounds a member's free Einstein radius using its luminosity (free, but only within a factor of the +relation's prediction). And the relation itself can be loosened: freeing `scaling_exponent` as a fit parameter +costs a single dimension *for the entire population* and is a useful systematics test. The `autolens_workspace`'s +`imaging/features/scaling_relation` example shows all of these tiers working together. + +__Lens Environments__ + +Everything above was framed around lenses with multiple deflectors, but the same machinery solves a problem we +met at the start of this chapter: the **environment** of an ordinary single-galaxy lens. + +The first tutorial of this chapter modelled a nearby extra galaxy individually, which was fine for one. +But a deep image of a real lens can reveal dozens of galaxies near the line of sight, each contributing a small +perturbation to the deflection field. Modeling them individually is hopeless; ignoring them entirely can bias +precision measurements. With a scaling relation they cost nothing: measure each galaxy's centre and luminosity +from the image, tie the lot to the main lens galaxy (which is the natural anchor — it is usually the brightest +galaxy in the field), and the model gains dozens of physically motivated perturbers with zero added parameters. + +__Wrap Up__ + +In this tutorial we learned: + +1. Luminosity predicts lensing mass, via mass-follows-light arguments and the Faber-Jackson relation + (`L ~ \sigma^4`, `\theta_E ~ \sigma^2`, hence `\theta_E ~ L^{0.5}`). + +2. A scaling relation replaces per-galaxy mass freedom with a shared relation anchored on a parameter the model + already fits, so tied galaxies add **zero** free parameters — verified by explicit parameter counting. + +3. Combined with MGE light profiles, a many-galaxy lens model stays low-dimensional: dimensionality is + independent of the number of galaxies. + +4. The relation is an assumption: scatter and deviant members can bias the whole model, and the remedy is + selective freedom — tie the faint majority, free the important few. + +5. The same machinery models the environments of single-lens systems, at no parameter cost. + +So far, we tied galaxy to galaxy. But the biggest lenses in the Universe — galaxy groups and clusters — contain +something the galaxies themselves cannot account for: a massive dark matter halo enveloping the whole system, +holding most of its mass. In the next tutorial we step up to group-scale lenses, where the scaling relation +becomes essential (there are simply too many member galaxies to free) and a dark matter halo joins the model as +a new component. There, we will also meet a different way to normalise the relation — a shared free +normalization at a fixed reference luminosity, rather than an anchor galaxy — and truncated mass profiles, which +describe members whose outer dark matter has been tidally stripped by the very halo we are adding. +""" diff --git a/scripts/chapter_4_scaling_up_lensing/tutorial_4_group_scale.py b/scripts/chapter_4_scaling_up_lensing/tutorial_4_group_scale.py new file mode 100644 index 0000000..792c13e --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/tutorial_4_group_scale.py @@ -0,0 +1,635 @@ +""" +Tutorial 4: Group Scale +======================= + +In the previous tutorials we took our first steps beyond the single lens galaxy: we included extra galaxies near +the lens in the model, we modeled systems where two or more galaxies of comparable mass share the lensing, and we +introduced scaling relations, which tie the mass of a galaxy to its light so that adding more galaxies to a model +does not mean adding more free parameters. + +In this tutorial we put all of those tools together and take the first big step up the "regime ladder" of strong +lensing: the **group scale**. + +In the standard cosmological model (Lambda Cold Dark Matter, or LCDM), galaxies do not live in isolation. They +form and evolve inside "dark matter halos", and these halos are themselves nested: small halos hosting single +galaxies merge and fall into larger halos hosting many. A **galaxy group** is the next rung above a single +galaxy in this hierarchy: a dark matter halo of roughly 10^13 to 10^14 solar masses hosting anywhere from a +handful to a few dozen galaxies. (Our own Milky Way lives in such a structure, the Local Group.) Above groups +sit **galaxy clusters**, halos of 10^14 solar masses and beyond hosting hundreds or thousands of galaxies, which +are the subject of the next tutorial. + +When a group acts as a strong lens, several of its galaxies contribute significantly to the deflection of the +source's light. The brightest galaxy of the group -- called the brightest group galaxy (BGG), or the brightest +cluster galaxy (BCG) at cluster scale -- typically sits near the centre of the group's dark matter halo and +dominates the lensing. The other group members perturb the lensed image around it. The group's shared dark +matter halo may also enter the mass model as its own component, often centred on the BGG where one exists -- +though as we will see, whether it is needed is an explicit modelling choice, not an automatic ingredient. + +__Groups vs Multi-Galaxy Lenses__ + +You may now be wondering how a "group-scale lens" differs from the multi-galaxy lenses of the previous tutorials, +and the honest answer is that the line between them is **blurry**. Every group is a multi-galaxy lens: it has +multiple galaxies whose mass matters for the lensing. But not every multi-galaxy lens is a group: two galaxies of +comparable mass that happen to sit close on the sky (or are mid-merger) share the lensing without being embedded +in a massive shared halo with a member population. + +The physically meaningful distinction is the **environment**: a group has a shared dark matter halo, a dominant +central galaxy, and members orbiting inside the host's potential. That environment leaves signatures in the mass +model, and the most important one is **tidal truncation**, which we introduce below. + +__Tidal Truncation and the dPIE Profile__ + +A galaxy orbiting inside a group does not keep its full dark matter halo. The tidal field of the host group +strips the outer, loosely bound parts of the member's halo, so the member's mass distribution is cut off -- or +"truncated" -- at a finite radius. The isothermal profiles we have used so far are a poor description of this: +their density falls as 1/r^2 forever, so their total mass diverges with radius. + +The standard truncated profile of group- and cluster-scale lensing is the **dPIE** (dual Pseudo-Isothermal +Elliptical) profile, available in **PyAutoLens** as `al.mp.dPIEMassSph`. It behaves isothermally at intermediate +radii but its density falls off much more steeply beyond a truncation radius, giving it a finite total mass -- +exactly the behaviour we expect for a tidally stripped group member. + +__Contents__ + +- **Initial Setup:** Load the simulated group-scale dataset (auto-simulating it if absent) and plot it. +- **Mask:** Define the 2D mask, which is larger than at galaxy scale because the group spans more sky. +- **Galaxy Centres:** Load the centres of the BGG and member galaxies from .json files. +- **The dPIE Profile:** Introduce the truncated dPIE mass profile and its parameters. +- **Fitting a Group:** Fit the data with a tracer containing the BGG, members and source. +- **A Group Halo?:** Add a group-scale dark matter halo to the tracer and see how it changes the fit. +- **Model Fit:** Compose and fit a group-scale lens model with individually modeled members. +- **Scaling Relation Members:** Tie the members' masses to their light so they cost almost no parameters. +- **The Group Scale Sweet Spot:** Why groups are the sweet spot of the familiar lens modeling toolkit. +- **Wrap Up:** Summary of the script and next steps. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +import matplotlib.pyplot as plt +import numpy as np +from pathlib import Path + +import autofit as af +import autolens as al +import autolens.plot as aplt + +""" +__Initial Setup__ + +We begin by loading the group-scale strong lens dataset `simple`, which was simulated with: + + - A main lens galaxy (the BGG) at (0.0", 0.0"), with a `SersicSph` light profile and an `IsothermalSph` mass + profile with a 4.0" Einstein radius. + - Two member galaxies at (3.5", 2.5") and (-4.4", -5.0"), with `SersicSph` light profiles and tidally + truncated `dPIEMassSph` mass profiles. + - A single source galaxy with a `SersicCore` light profile. +""" +dataset_name = "simple" +dataset_path = Path("dataset") / "group" / dataset_name + +""" +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it will be created by running the corresponding +simulator script. This ensures that all example scripts can be run without manually simulating data first. +""" +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/group.py"], + check=True, + ) + +dataset = al.Imaging.from_fits( + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + psf_path=dataset_path / "psf.fits", + pixel_scales=0.1, +) + +""" +When we plot the dataset, the difference from a galaxy-scale lens is immediately clear: the lensed arcs form a +much larger ring (the BGG's Einstein radius is 4.0", compared to the ~1.0" typical of a single galaxy), and two +smaller galaxies are visible away from the centre. Their light -- and, more importantly for the lensing, their +mass -- must be included in our analysis. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Mask__ + +We define a 7.5" circular mask, much larger than the 2.5"-3.0" masks of previous chapters. It must enclose the +full lensed arc system and the member galaxies, because all of them contribute light and mass to the region we +fit. +""" +mask_radius = 7.5 + +mask = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + radius=mask_radius, +) + +dataset = dataset.apply_mask(mask=mask) + +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Galaxy Centres__ + +Following the convention of the previous tutorials, the centres of the galaxies are loaded from .json files in +the dataset folder. For a group we distinguish two categories: + + - `main_lens_centres`: the main lens galaxies -- here just the BGG at (0.0", 0.0") -- which are modeled with + full freedom. + + - `member_centres`: the group members, which are modeled with restrictive assumptions (fixed centres, and + later a scaling relation) to keep the model dimensionality low. + +For real data these centres come from the observed light of each galaxy (e.g. via a click-based GUI or a +photometry catalogue), exactly as for the extra galaxies of the earlier tutorials. +""" +main_lens_centres = al.from_json(file_path=dataset_path / "main_lens_centres.json") + +member_centres = al.from_json(file_path=dataset_path / "extra_galaxies_centres.json") + +print(f"Main lens centres: {main_lens_centres}") +print(f"Member centres: {member_centres}") + +""" +__The dPIE Profile__ + +Before fitting anything, lets get to know the dPIE profile, since it is the workhorse mass profile of the group +and cluster regimes. + +The `dPIEMassSph` profile is parameterized in the convention of Lenstool (a long-established cluster lensing +code), which differs from the profiles we have met so far: + + - `sigma`: the fiducial velocity dispersion of the galaxy in km/s. This sets the overall lensing strength, + playing the role the `einstein_radius` played for the isothermal profile (for an isothermal sphere the two + are directly related: a bigger velocity dispersion means a bigger Einstein radius). + + - `r_core`: the core radius in arcseconds, within which the density profile flattens. It is common to fix + `r_core = 0.0` (a "vanishing core", where the profile has an exact analytic form), which we do throughout + this tutorial. + + - `r_cut`: the truncation radius in arcseconds -- the key new parameter. Outside `r_cut` the density falls off + steeply (as 1/r^4 rather than the isothermal 1/r^2), encoding the tidal stripping of the member's outer dark + matter halo and giving the profile a finite total mass. + + - `redshift_object`, `redshift_source`, `H0`, `Om0`: because `sigma` is a physical velocity, converting it to + deflection angles requires the lensing geometry and a cosmology. These are fixed inputs describing the lens + system, not parameters we fit. + +Lets make two dPIE profiles with identical `sigma` but different truncation radii -- one truncated at 8.0" and +one truncated so far out (1000.0") that it is effectively untruncated -- and compare their convergence profiles. +""" +dpie_truncated = al.mp.dPIEMassSph( + centre=(0.0, 0.0), + sigma=200.0, + r_core=0.0, + r_cut=8.0, + redshift_object=0.5, + redshift_source=1.0, +) + +dpie_untruncated = al.mp.dPIEMassSph( + centre=(0.0, 0.0), + sigma=200.0, + r_core=0.0, + r_cut=1000.0, + redshift_object=0.5, + redshift_source=1.0, +) + +""" +We evaluate the convergence of both profiles on a 1D line of radial coordinates, using a `Grid2DIrregular` so we +can choose the radii ourselves. +""" +radii = np.linspace(0.01, 20.0, 200) + +radial_grid = al.Grid2DIrregular([(0.0, radius) for radius in radii]) + +convergence_truncated = dpie_truncated.convergence_2d_from(grid=radial_grid) +convergence_untruncated = dpie_untruncated.convergence_2d_from(grid=radial_grid) + +plt.semilogy(radii, np.asarray(convergence_truncated), label="dPIE (r_cut = 8.0)") +plt.semilogy(radii, np.asarray(convergence_untruncated), label="dPIE (untruncated)") +plt.axvline(x=8.0, color="k", linestyle="--", label="r_cut") +plt.xlabel("Radius (arcseconds)") +plt.ylabel("Convergence") +plt.legend() +plt.show() +plt.close() + +""" +Inside `r_cut` the two profiles are nearly identical -- both behave isothermally. Beyond `r_cut`, the truncated +profile's convergence plummets while the untruncated one keeps its shallow isothermal decline. The truncated +member therefore contributes far less mass at large radii, which is exactly what tidal stripping does. + +This is why the dPIE, and not the isothermal profile, is the standard choice for group and cluster members: the +extra galaxies of a galaxy-scale lens have no host environment stripping them, so untruncated isothermal +profiles are fine there, but a member orbiting inside a shared group halo is physically truncated whether or +not the model includes that halo as an explicit component. + +For scale: a member with `sigma = 200.0` km/s at these redshifts has an Einstein radius of ~0.7", roughly a +fifth of the BGG's -- a perturber, not a co-dominant lens. + +__Fitting a Group__ + +Now lets fit the data. Following the fitting tutorials of earlier chapters, we first build a tracer from +galaxies whose light and mass profiles match the true values used to simulate the data, and fit it with the +`FitImaging` object. + +The tracer contains four galaxies: the BGG, the two members and the source. The fit handles all of them +simultaneously, summing the deflection field of every mass profile to ray-trace the source's light. +""" +bgg = al.Galaxy( + redshift=0.5, + bulge=al.lp.SersicSph( + centre=(0.0, 0.0), intensity=0.7, effective_radius=2.0, sersic_index=4.0 + ), + mass=al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=4.0), +) + +member_0 = al.Galaxy( + redshift=0.5, + bulge=al.lp.SersicSph( + centre=(3.5, 2.5), intensity=0.9, effective_radius=0.8, sersic_index=3.0 + ), + mass=al.mp.dPIEMassSph( + centre=(3.5, 2.5), + sigma=200.0, + r_core=0.0, + r_cut=8.0, + redshift_object=0.5, + redshift_source=1.0, + ), +) + +member_1 = al.Galaxy( + redshift=0.5, + bulge=al.lp.SersicSph( + centre=(-4.4, -5.0), intensity=1.866, effective_radius=0.8, sersic_index=3.0 + ), + mass=al.mp.dPIEMassSph( + centre=(-4.4, -5.0), + sigma=240.0, + r_core=0.0, + r_cut=13.3, + redshift_object=0.5, + redshift_source=1.0, + ), +) + +source = al.Galaxy( + redshift=1.0, + bulge=al.lp.SersicCore( + centre=(0.0, 0.1), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0), + intensity=3.0, + effective_radius=0.4, + sersic_index=1.0, + ), +) + +tracer = al.Tracer(galaxies=[bgg, member_0, member_1, source]) + +fit = al.FitImaging(dataset=dataset, tracer=tracer) + +aplt.subplot_fit_imaging(fit=fit) + +""" +Because this is the true model, the residuals are consistent with noise and the log likelihood is as high as it +can be for this dataset. Note that nothing about the `FitImaging` object changed: the same fitting machinery we +learnt at galaxy scale simply receives a tracer with more galaxies in it. +""" +print(f"Log likelihood of the true model: {fit.log_likelihood}") + +""" +__A Group Halo?__ + +You may have noticed that the model above contains no group-scale dark matter halo, despite this tutorial +opening with the statement that a group is defined by one. Two things are going on here. + +First, the BGG's `IsothermalSph` has a 4.0" Einstein radius -- far larger than the ~1.0" of an isolated galaxy +of its luminosity. A mass profile centred on the BGG cannot tell "stars and dark matter belonging to the BGG" +apart from "group dark matter centred on the BGG": a single isothermal profile at the group centre soaks up +both. So the group's halo is partly hidden inside the BGG's mass profile already. + +Second, whether to add a *separate* halo component to the model is an **explicit modelling choice**, and this +dataset was deliberately simulated without one. Lets see what happens if we add one anyway. We add a fifth +galaxy carrying only mass -- an `IsothermalSph` centred on the BGG, the conventional choice for a group halo +where a BGG exists -- and refit. +""" +group_halo = al.Galaxy( + redshift=0.5, + mass=al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=1.5), +) + +tracer_with_halo = al.Tracer(galaxies=[bgg, group_halo, member_0, member_1, source]) + +fit_with_halo = al.FitImaging(dataset=dataset, tracer=tracer_with_halo) + +aplt.subplot_fit_imaging(fit=fit_with_halo) + +""" +The fit is now visibly worse -- the extra 1.5" Einstein radius of mass at the centre over-deflects the source's +light, displacing the model arcs from the observed ones -- and the log likelihood drops accordingly. +""" +print(f"Log likelihood with a group halo added: {fit_with_halo.log_likelihood}") + +""" +On real data the same experiment runs in reverse: fit the group with and without an explicit halo component and +let the data decide (via the residuals and the Bayesian evidence introduced in earlier chapters). Some groups +are well described by their galaxies alone, with the central galaxy's profile absorbing the halo; others -- +typically those with the largest Einstein radii or image configurations no galaxy-mass model can reproduce -- +require a dominant halo component, often centred on the BGG. + +__Model Fit__ + +We now perform a proper model-fit with a non-linear search, as a scientist would for a group whose true +parameters are unknown. We compose the model as follows: + + - The BGG's light is a linear `SersicSph` (its centre free) and its mass an `IsothermalSph` [7 parameters]. + + - Each member's light is a linear `SersicSph` with its centre fixed to the observed centre, and its mass a + `dPIEMassSph` with a free `sigma`, its centre fixed, `r_core = 0.0` and `r_cut` fixed to a fiducial 10.0" + [3 parameters per member, 6 total]. + + - The source's light is a linear `SersicCore` [6 parameters]. + +Fixing the member centres to their observed light and fixing `r_cut` follows the reasoning of the extra +galaxies tutorial: the data rarely constrains a perturber's truncation radius, so we spend our parameter budget +on the quantity that matters most -- each member's overall mass, via `sigma`. The redshifts and cosmology of +the dPIE are pinned to their known values, since they are properties of the lens system, not parameters to +sample. +""" +# BGG: + +bulge = af.Model(al.lp_linear.SersicSph) + +mass = af.Model(al.mp.IsothermalSph) + +bgg_model = af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass) + +# Member Galaxies: + +member_list = [] + +for centre in member_centres: + + bulge = af.Model(al.lp_linear.SersicSph) + bulge.centre = (centre[0], centre[1]) + + mass = af.Model(al.mp.dPIEMassSph) + mass.centre = (centre[0], centre[1]) + mass.sigma = af.UniformPrior(lower_limit=0.0, upper_limit=300.0) + mass.r_core = 0.0 # vanishing core -- fixed; the dPIE is analytic at r_core = 0 + mass.r_cut = 10.0 # truncation fixed at a fiducial radius + mass.redshift_object = 0.5 + mass.redshift_source = 1.0 + mass.H0 = 67.66 # pinned: model constants, not parameters to sample + mass.Om0 = 0.30966 + + member_list.append(af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass)) + +# Source: + +source_model = af.Model(al.Galaxy, redshift=1.0, bulge=af.Model(al.lp_linear.SersicCore)) + +# Overall Lens Model: + +model = af.Collection( + galaxies=af.Collection(lens=bgg_model, source=source_model), + extra_galaxies=af.Collection(member_list), +) + +""" +Printing the model's `info` confirms its composition: the BGG and source under `galaxies`, and the two members +under `extra_galaxies` with only their `sigma`, `effective_radius` and `sersic_index` free. + +The total number of free parameters is N=19 -- take note of this number, as it is central to the discussion at +the end of this tutorial. +""" +print(model.info) + +print(f"Total free parameters: {model.prior_count}") + +""" +Before fitting, we set up the adaptive over-sampling scheme at the centre of every galaxy in the group (not +just the BGG), following the standard workspace approach. As in previous tutorials, the details are not +important yet -- just note that at group scale every galaxy centre needs it. +""" +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=dataset.grid, + sub_size_list=[4, 2, 2], + radial_list=[0.3, 0.6], + centre_list=list(main_lens_centres) + list(member_centres), +) + +dataset = dataset.apply_over_sampling(over_sample_size_lp=over_sample_size) + +""" +We fit the model with the `Nautilus` non-linear search, using the same `AnalysisImaging` object as every +imaging fit so far -- another sign that the group scale does not require new machinery. +""" +search = af.Nautilus( + path_prefix=Path("howtolens") / "chapter_4", + name="tutorial_4_group_scale", + unique_tag=dataset_name, + n_live=100, + n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations +) + +analysis = al.AnalysisImaging(dataset=dataset) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_4_group_scale" + " folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result = search.fit(model=model, analysis=analysis) + +print("The search has finished run - you may now continue the notebook.") + +""" +The result's `info` and maximum likelihood fit show the model recovers the group: the BGG's Einstein radius, +the members' velocity dispersions and the source's properties are all inferred from the data. +""" +print(result.info) + +aplt.subplot_fit_imaging(fit=result.max_log_likelihood_fit) + +""" +__Scaling Relation Members__ + +The model above gave every member its own free `sigma`. With two members that costs two parameters -- harmless. +But groups can host dozens of members, and one free mass parameter per member quickly bloats the model, slows +the search, and asks the data to constrain masses it barely feels individually. + +The previous tutorial's solution applies directly here: tie the members' masses to their light through a +scaling relation. More luminous galaxies are more massive, so we write each member's `sigma` and `r_cut` as +power laws of its luminosity, normalized by a single shared free parameter: + + - `sigma = sigma_ref * (L / L_ref) ** 0.25` + - `r_cut = r_cut_ref * (L / L_ref) ** 0.7` + +These exponents follow the convention of modern cluster lensing analyses, and `L_ref` is a fixed reference +luminosity. The 0.25 is the previous tutorial's Faber-Jackson relation recast for velocity dispersion: +`L ~ sigma^4` inverts to `sigma ~ L^0.25`, and since `theta_E ~ sigma^2` this reproduces the `theta_E ~ L^0.5` +scaling we anchored there. Note also the change of normalization: rather than hanging the relation off an +anchor galaxy the model already fits, we normalise it with a shared free parameter at a fixed reference +luminosity. Now the *entire member population's mass* costs one free parameter (`sigma_ref`), no matter how +many members the group has, and each member's observed luminosity acts as a physically motivated prior on its +mass. + +For real data the luminosities come from your photometry catalogue (only luminosity ratios enter the model, so +any consistent units work). Here we compute them from the true member light profiles, using an aperture +luminosity within 3.0". +""" +luminosity_list = [ + member_0.bulge.luminosity_within_circle_from(radius=3.0), + member_1.bulge.luminosity_within_circle_from(radius=3.0), +] + +print(f"Member luminosities: {luminosity_list}") + +reference_luminosity = luminosity_list[0] + +sigma_ref = af.UniformPrior(lower_limit=0.0, upper_limit=300.0) +r_cut_ref = 8.0 # fixed, like r_cut in the previous model + +member_list = [] + +for centre, luminosity in zip(member_centres, luminosity_list): + + luminosity_ratio = float(luminosity) / float(reference_luminosity) + + bulge = af.Model(al.lp_linear.SersicSph) + bulge.centre = (centre[0], centre[1]) + + mass = af.Model(al.mp.dPIEMassSph) + mass.centre = (centre[0], centre[1]) + mass.sigma = sigma_ref * luminosity_ratio**0.25 + mass.r_core = 0.0 + mass.r_cut = r_cut_ref * luminosity_ratio**0.7 + mass.redshift_object = 0.5 + mass.redshift_source = 1.0 + mass.H0 = 67.66 + mass.Om0 = 0.30966 + + member_list.append(af.Model(al.Galaxy, redshift=0.5, bulge=bulge, mass=mass)) + +model = af.Collection( + galaxies=af.Collection(lens=bgg_model, source=source_model), + scaling_galaxies=af.Collection(member_list), +) + +""" +Note that the tied members now live in the `scaling_galaxies` collection — the home the previous tutorial +introduced for a scaling-relation population — whereas the individually-modelled members above sat in +`extra_galaxies`, like tutorial 1's interlopers. Both collections are included in the ray-tracing; the split +keeps `model.info` readable and tells the results machinery which galaxies are a tied population. + +The model's `info` shows that both members' `sigma` values now trace back to the single shared `sigma_ref` +prior, and the parameter count has dropped to N=18. + +A drop of one parameter looks unremarkable -- until you scale it up. A group with 20 members modeled +individually would need 20 free mass parameters; with the scaling relation it still needs exactly one. (In +practice the members' light is handled the same way, using the Multi Gaussian Expansion from earlier in the +lectures, whose fixed-centre members add almost no free parameters either -- so entire member populations can +be added to a group model almost for free.) + +This dataset was in fact simulated with members that obey this exact scaling relation, so the model remains the +true model. +""" +print(model.info) + +print(f"Total free parameters: {model.prior_count}") + +""" +We fit this model with an identical search set up. +""" +search = af.Nautilus( + path_prefix=Path("howtolens") / "chapter_4", + name="tutorial_4_group_scale_scaling_relation", + unique_tag=dataset_name, + n_live=100, + n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations +) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/" + "tutorial_4_group_scale_scaling_relation folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result_scaling = search.fit(model=model, analysis=analysis) + +print("The search has finished run - you may now continue the notebook.") + +print(result_scaling.info) + +aplt.subplot_fit_imaging(fit=result_scaling.max_log_likelihood_fit) + +""" +__The Group Scale Sweet Spot__ + +Step back and consider what this tutorial did *not* require: no new fitting machinery, no new analysis object, +no new data format. The `FitImaging`, `AnalysisImaging`, linear light profiles and `Nautilus` search of the +galaxy-scale chapters handled the group unchanged -- we just put more galaxies in the tracer, reached for the +truncated dPIE profile, and used scaling relations to keep the member population cheap. Even the pixelized +source reconstructions of the previous chapter work at group scale exactly as they do at galaxy scale, and for +real groups with complex arcs they are the recommended source model. + +This is why the group scale is the **sweet spot** of strong lens modeling. The full toolkit of galaxy-scale +modeling still applies, and the model dimensionality stays low: our individually-modeled-member fit had N=19 +free parameters and the scaling-relation fit N=18. Real group models are of this order -- a few tens of free +parameters at most (N < 30 or so) -- comfortably within reach of the non-linear searches we already know, with +run times of minutes to hours. + +Push the system any richer, however, and this toolkit breaks down. A cluster hosts hundreds of members and +lenses many sources at once; its image is too large and complex to fit pixel-by-pixel with a tracer's light +model, and its model would need hundreds of parameters. Cluster-scale modeling therefore switches to a +different tool set -- fitting the *positions* of the multiple images of many point-like sources, with member +catalogues driving the mass model -- which is the subject of the next tutorial. + +__Wrap Up__ + +In this tutorial, we took the tools of the previous tutorials to the group scale. Lets recap what we learnt: + +- **Groups in LCDM**: A galaxy group is a dark matter halo of roughly 10^13 to 10^14 solar masses hosting a + handful to a few dozen galaxies, with the brightest group galaxy (BGG) typically near its centre. The group's + halo may enter the mass model as its own component, often centred on the BGG -- but whether it is needed is + an explicit modelling choice, tested by fitting with and without it. + +- **Groups vs multi-galaxy lenses**: The line is blurry. All groups are multi-galaxy lenses, but not all + multi-galaxy lenses are groups -- the group's defining feature is the shared halo environment its members + orbit within. + +- **Tidal truncation and the dPIE**: Members orbiting in the group's potential have their outer dark matter + tidally stripped, so their profiles are truncated. The `dPIEMassSph` profile encodes this via its `r_cut` + truncation radius, with `sigma` setting the lensing strength and `r_core` the (usually vanishing) core. + +- **Fitting and modeling groups**: The galaxy-scale machinery works unchanged -- a tracer with more galaxies, + members with fixed centres and free `sigma`, and a `Nautilus` search over N=19 parameters. + +- **Scaling relations**: Tying member `sigma` and `r_cut` to luminosity via power laws with one shared free + normalization means the whole member population costs a single parameter, however many members there are. + +- **The sweet spot**: Groups are the largest systems the familiar toolkit -- including pixelized source + reconstruction -- still handles, with model dimensionality staying below a few tens of parameters. + +In the next tutorial we climb the final rung of the ladder to galaxy clusters, where tens to hundreds of member +galaxies lens many sources simultaneously. There the toolkit changes: we will model the positions of multiply +imaged point sources rather than every pixel of the data, and drive the mass model from member catalogues via +a csv interface built on the same scaling relations we used here. +""" diff --git a/scripts/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.py b/scripts/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.py new file mode 100644 index 0000000..65b9495 --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.py @@ -0,0 +1,708 @@ +""" +Tutorial 5: Cluster Scale +========================= + +Throughout this chapter we have been climbing a ladder of scale: from a single lens galaxy with an extra +galaxy nearby, to multi-galaxy lenses, to galaxy groups whose members share a common dark matter halo. + +This tutorial climbs the final rung: **galaxy clusters**, the most massive gravitationally-bound structures in +the Universe. A cluster-scale strong lens contains: + +- **30+ to hundreds of lens galaxies**: a brightest cluster galaxy (BCG) and bright satellites, plus a large + population of lower-mass member galaxies, all embedded in... + +- **A cluster-scale dark matter halo of more than 10^14 solar masses** (the most massive exceed 10^15 solar + masses) — hundreds of times more massive than the halo of a single lens galaxy. + +- **Many multiply-imaged background sources** — often tens of them — and, crucially, these sources typically + all sit at **different redshifts**. Light from each source therefore traverses a different path through the + Universe, and modeling them together requires **multi-plane ray tracing**, which we explain below. + +The science a cluster lens delivers is different from the galaxy-scale lensing of earlier chapters: + +- **Extreme magnification**: near a cluster's critical curves, magnifications of tens to hundreds are common + (compared to the factors of a few of galaxy-scale lensing). Clusters act as natural telescopes, letting us + study intrinsically faint galaxies in the early Universe that would otherwise be invisible — many of the + most distant galaxies known were discovered behind lensing clusters. + +- **Mapping the cluster's mass**: every multiply-imaged source constrains the cluster's total mass + distribution along a different line of sight. With many sources, the mass model pins down the shape of the + cluster's dark matter halo and the substructure within it — the granularity of dark matter on sub-cluster + scales — providing some of the sharpest tests of the cold dark matter model. + +The modeling toolkit also changes at this scale. Fitting the extended arcs of every source, with a light +profile or pixelization each, is computationally prohibitive when there are tens of sources and hundreds of +member galaxies. The standard approach instead fits only the **positions** of each source's multiple images — +point-source modeling — and drives the mass model from spreadsheet-editable **CSV catalogues**, because +writing Python model code for hundreds of galaxies by hand is not sensible either. + +__Contents__ + +- **Multi-Plane Ray Tracing:** What it means for sources at different redshifts to be lensed together. +- **Dataset:** Load the simulated cluster's CCD imaging and inspect the true multi-plane tracer. +- **Point Source Modeling:** Why cluster modeling fits multiple-image positions rather than extended arcs. +- **Point Datasets:** The per-source multiple-image positions, loaded from a single hand-editable CSV. +- **The CSV Interface:** The named-galaxy model CSVs and the scaling-member catalogue CSV. +- **Point Solver:** The solver which finds the image-plane multiple images of a point source. +- **Model:** Compose the four-tier cluster model (main lenses, scaling members, host halo, sources). +- **Analysis + Factor Graph:** One analysis per source dataset, combined into a single global fit. +- **Search:** Configure the Nautilus non-linear search. +- **Model Fit:** Run the fit. +- **Results:** A thorough tour of results access for a cluster fit — tracer, per-member profiles, + solved source-plane positions, magnifications and image-plane validation. +- **Customization:** Where to go for the full point-source customization options. +- **Wrap Up:** Summary and the hand-off to weak lensing. + +__Multi-Plane Ray Tracing__ + +Every lens system in the tutorials so far had two planes: an image plane (the lens galaxies, all at one +redshift) and a source plane (the lensed galaxy). The lens equation was applied once, mapping image-plane +coordinates to source-plane coordinates via the deflection angles: + +$\beta = \theta - \alpha(\theta)$ + +A cluster breaks this picture, because its sources sit at different redshifts. Consider our simulated cluster, +which has lens galaxies at $z = 0.5$ and two sources, at $z = 1.0$ and $z = 2.0$: + +- Light from the $z = 1.0$ source is deflected by the $z = 0.5$ cluster and travels to us: the familiar + two-plane system. + +- Light from the $z = 2.0$ source first passes the $z = 1.0$ plane, then the $z = 0.5$ cluster plane. If any + galaxy at $z = 1.0$ had mass, it would deflect this light too — the deflections of successive planes + accumulate, each applied at the position the ray has reached by that plane. + +This is **multi-plane ray tracing**: the lens equation is applied recursively, plane by plane in redshift +order. There is a second, subtler effect: the strength of a deflection depends on the distances between the +observer, the deflector and the source. The same cluster bends light from a $z = 2.0$ source through a larger +effective angle than light from a $z = 1.0$ source, because the geometry of the light paths differs. The +recursive lens equation accounts for this by scaling each plane's deflections by ratios of cosmological +(angular diameter) distances. + +The maths is more bookkeeping than new physics, and the good news is that the `Tracer` handles all of it +automatically: give it galaxies at three or more redshifts and it groups them into planes, orders them, scales +their deflections and applies the recursion. Every source's redshift must simply be *correct* — hardcoding the +wrong redshift silently produces the wrong multi-plane geometry, which is why the dataset CSVs below carry a +redshift for every source. +""" + +import importlib.util +import sys + +if importlib.util.find_spec("jax") is None: + print( + "Skipping this tutorial: it requires the `jax` package (used to " + "accelerate the multi-plane point-source solves), which is not " + "installed (install with `pip install autolens[optional]`)." + ) + sys.exit(0) + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +import numpy as np +from pathlib import Path + +import autofit as af +import autolens as al +import autolens.plot as aplt + +""" +__Dataset__ + +We use a simulated cluster whose scale is kept deliberately small so it runs quickly, but which contains every +ingredient of the cluster regime: 2 main lens galaxies (a BCG and a satellite), 10 lower-mass member galaxies +on a luminosity-mass scaling relation, a 10^15.3 solar-mass dark matter halo, and 2 multiply-imaged sources at +different redshifts (z = 1.0 and z = 2.0). A real cluster simply has more of everything — and we will see that +scaling this model up to hundreds of members does not add a single free parameter. +""" +dataset_name = "simple" +dataset_path = Path("dataset") / "cluster" / dataset_name + +""" +__Dataset Auto-Simulation__ + +If the dataset does not already exist on your system, it will be created by running the corresponding +simulator script. This ensures that all example scripts can be run without manually simulating data first. +""" +if al.util.dataset.should_simulate(str(dataset_path)): + import subprocess + import sys + + subprocess.run( + [sys.executable, "scripts/simulator/cluster.py"], + check=True, + ) + +""" +We first load and plot the cluster's CCD imaging. Note the pixel scale of 0.2" and the sheer size of the +field: 100" x 100", compared to the ~6" fields of the galaxy-scale tutorials. The Einstein radius of a +10^15 solar-mass cluster is ~20-30", so its multiple images and arcs are spread across a region hundreds of +times larger in area than a galaxy-scale lens. + +The imaging is loaded for visualization only — as we discuss next, the model is not fitted to these pixels. +""" +data = al.Array2D.from_fits(file_path=dataset_path / "data.fits", pixel_scales=0.2) + +aplt.plot_array(array=data, title="Cluster CCD Imaging") + +""" +The simulator saved the true `Tracer` used to make this dataset, which we can load to inspect the multi-plane +structure described above. The tracer has three planes: the lens plane at z = 0.5 (holding all 13 lens-plane +galaxies — 2 main galaxies, 10 scaling members and the dark matter halo) and one source plane per source. +""" +tracer_true = al.from_json(file_path=dataset_path / "tracer.json") + +print(f"Number of planes: {len(tracer_true.planes)}") +print(f"Plane redshifts: {[float(plane.redshift) for plane in tracer_true.planes]}") +print(f"Galaxies in lens plane: {len(tracer_true.planes[0])}") + +""" +__Point Source Modeling__ + +In every previous tutorial we fitted the data pixel-by-pixel: a model image of the lens and source light was +computed, convolved with the PSF and compared to every image pixel. For a cluster this becomes computationally +prohibitive: + +- Every one of the tens of sources needs its own light model (or pixelization), each adding parameters (or an + expensive linear inversion) to the fit. + +- The image is enormous — our modest simulated cluster already spans 500 x 500 pixels — and the deflection + angles of hundreds of member galaxies would need evaluating at every one of them, for every likelihood + evaluation. + +The standard approach in cluster lensing therefore simplifies the data dramatically: each multiply-imaged +source is reduced to the **positions of its multiple images** — the (y,x) coordinates of the brightest pixel +of each image, measured from the imaging. The model then only has to answer a much cheaper question: does this +mass model ray-trace all of a source's observed image positions back to the same source-plane location? + +This is **point-source modeling**. We are deliberately discarding the information in the arcs' extended light +in exchange for a likelihood that is fast enough to evaluate with hundreds of galaxies and tens of sources. +(For lensed quasars and supernovae — genuine point sources — nothing is even discarded.) Extended-source +cluster modeling, where individual arcs are fitted at pixel level with the pixelizations of the previous +chapter, does exist in **PyAutoLens** as a specialised follow-up analysis — see the `autolens_workspace` +cluster examples — but position fitting is the workhorse that published cluster mass models are built on. + +__Point Datasets__ + +The positions of each source's multiple images are stored in a single CSV file, `point_datasets.csv`, with +one row per observed multiple image: + + - `name`: the source identifier (e.g. `point_0`). All rows sharing a `name` belong to the same source. + - `y`, `x`: the image-plane position of the multiple image, in arc-seconds. + - `positions_noise`: the positional uncertainty in arc-seconds — how precisely the image's centroid is + measured (for space-telescope imaging ~0.005", set by PSF-fitting precision, not the pixel scale). + - `redshift`: the source redshift, which every row of a group must share. This column is what makes the + multi-plane geometry reproducible from the file alone. + +This CSV is the recommended input format for cluster datasets because it is hand-editable: measuring your own +cluster's multiple images means filling in a spreadsheet, not writing Python. + +`al.list_from_csv` loads the file into a list of `PointDataset` objects, one per source. +""" +dataset_list = al.list_from_csv(file_path=dataset_path / "point_datasets.csv") + +for dataset in dataset_list: + print("Point Dataset Info:") + print(dataset.info) + print(f"Redshift: {dataset.redshift}") + +""" +Plotting each dataset shows just how sparse point-source data is: a handful of (y,x) coordinates per source. +That a full cluster mass model can be constrained by so few numbers is because each coordinate is measured to +milli-arcsecond precision, and because every source at its own redshift probes the mass distribution +differently. +""" +for dataset in dataset_list: + aplt.plot_grid( + grid=al.Grid2DIrregular(np.atleast_2d(dataset.positions)), + title=dataset.name, + ) + +""" +__The CSV Interface__ + +With hundreds of member galaxies, composing the lens model in Python — one `af.Model(al.Galaxy)` per galaxy, +as in every previous tutorial — stops being sensible. The cluster workflow therefore defines the model in CSV +files as well, which the simulator wrote alongside the data: + + - `mass.csv`: every individually-modelled mass profile — the two main lens galaxies' `dPIEMassSph` profiles + and the host halo's `NFWMCRLudlowSph`. + - `light.csv`: the light profiles (not used in point-source modeling, since light does not lens). + - `point.csv`: the source galaxies' `Point` components. + +Each row of these CSVs carries: + + - `galaxy`: a galaxy name (e.g. `lens_0`, `host_halo`, `source_0`). Rows sharing a name compose into one + galaxy. + - `attr_name`: the attribute the profile is bound to on the galaxy (e.g. `mass`, `dark`, `point_0`). + - `profile_class`: the profile class name (e.g. `dPIEMassSph`), looked up in `al.mp` / `al.lp` / `al.ps`. + - The profile's constructor parameters as columns (tuples like `centre` split into `y` / `x`); parameters a + row's class does not use are left blank. + - `redshift`: the galaxy's redshift. + +The lower-mass member population lives in a fourth, simpler catalogue, `scaling_galaxies.csv`, with one row +per member and columns `y, x, luminosity`. No profile class, no mass parameters — because, as the previous +tutorial showed, the members' masses are not free parameters but derive from their luminosities via a scaling +relation. The catalogue is exactly what an observer measures: where each member is and how bright it is. + +This is the key scaling property of the CSV interface: modeling a richer cluster means appending rows to +`scaling_galaxies.csv` (and, for another individually-modelled galaxy or halo, to `mass.csv`) — the Python +below does not change, and the number of free parameters does not grow with the member count. + +We load the mass and point families (`al.galaxy_models_from_csv`) and the member catalogue +(`al.galaxy_table_from_csv`): +""" +mass_table = al.galaxy_models_from_csv( + file_path=dataset_path / "mass.csv", family="mass" +) +point_table = al.galaxy_models_from_csv( + file_path=dataset_path / "point.csv", family="point" +) + +scaling_galaxies_table = al.galaxy_table_from_csv( + file_path=dataset_path / "scaling_galaxies.csv" +) +scaling_galaxies_centres = scaling_galaxies_table.centres +scaling_galaxies_luminosity_list = scaling_galaxies_table.luminosities + +print(f"Scaling galaxies in catalogue: {len(scaling_galaxies_luminosity_list)}") + +""" +In a real analysis the member centres and luminosities come from light fits to the imaging data (or an +external photometric catalogue), and the main-galaxy centres in `mass.csv` are pinned to the observed light +centres. Fixing the centres to observed values is essential at cluster scale: a handful of multiple-image +positions cannot constrain hundreds of free centre coordinates, but the imaging measures them for free. + +__Point Solver__ + +Point-source modeling needs one new piece of machinery: given a mass model and a source-plane (y,x) +coordinate, where in the image plane do its multiple images appear? Answering this means solving the lens +equation *backwards*, which has no analytic solution. + +The `PointSolver` solves it numerically: it tiles the image plane with triangles, ray-traces each triangle's +corners to the source plane, keeps the triangles that land on the source-plane coordinate, and subdivides +them repeatedly until the image positions are located to a precision of `pixel_scale_precision`. + +The solver's grid must span the full 100" cluster field — the multiple images sit tens of arc-seconds from +the cluster centre. The `magnification_threshold` discards heavily demagnified images (e.g. the faint central +image of a typical lens configuration, which real observations do not detect). +""" +grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=1.0, # The pixel-scale converts pixel units to arc-seconds. +) + +solver = al.PointSolver.for_grid( + grid=grid, pixel_scale_precision=0.001, magnification_threshold=0.1 +) + +""" +__Model__ + +We now compose the cluster model, which has four tiers of component — the same four tiers every published +cluster mass model is built from: + + - **Main lens galaxies (2)**: the BCG and satellite, modelled individually with `dPIEMassSph` profiles. As + the previous tutorial on group-scale lenses discussed, the dPIE is the standard truncated profile of this + regime: members orbiting in the cluster's shared potential have their outer dark matter tidally stripped, + so their mass genuinely ends at a finite truncation radius `r_cut`. Each main galaxy has free `sigma` (its + fiducial velocity dispersion, setting the lens strength) and free `r_cut`, with its centre fixed to the + observed light centre [4 free parameters]. + + - **Scaling-tier members (10)**: `dPIEMassSph` profiles with centres fixed from the catalogue and `sigma` / + `r_cut` derived from each member's luminosity via the scaling relation introduced earlier in this chapter: + sigma = sigma_ref * (L / L_ref)^0.25 and r_cut = r_cut_ref * (L / L_ref)^0.7. Only the normalization + `sigma_ref` — the velocity dispersion of a member at the reference luminosity — is free. Whether the + catalogue holds 10 members or 300, the tier costs [1 free parameter]. + + - **Host dark matter halo (1)**: a standalone galaxy carrying the cluster's `NFWMCRLudlowSph` halo, centred + on the BCG, with a free total mass `mass_at_200` [1 free parameter]. + + - **Source galaxies (2)**: parameter-free `PointSolved` components. Rather than sampling each source's + (y,x) source-plane centre as free parameters, the fit solves for the centre analytically — the + precision-weighted mean of the back-traced image positions. With tens of sources this matters: it removes + two parameters per source from the search [0 free parameters]. + +**Total: N = 6 free parameters** — for a model containing 13 galaxies and 2 sources. This is the punchline of +the cluster workflow: catalogues and scaling relations decouple the model's physical richness from the +dimensionality of the non-linear search. + +`al.galaxy_af_models_from_csv_tables` converts the CSV tables into a dictionary of `af.Model(Galaxy)` objects +keyed by galaxy name, with every CSV value fixed as a default. We then selectively promote parameters to free +priors — exactly the `af.Model` customization API used throughout the earlier chapters, just applied to +models built from files instead of Python. +""" +redshift_lens = 0.5 +source_redshifts = [dataset.redshift for dataset in dataset_list] + +galaxy_models = al.galaxy_af_models_from_csv_tables(mass_table, point_table) + +""" +The main lens galaxies get free dPIE `sigma` / `r_cut`; their centres and redshifts stay fixed at the CSV +values, and `r_core` stays fixed at the CSV's 0.0 (the vanishing-core convention discussed in the previous +tutorial). The cosmology constants `H0` / `Om0` are pinned explicitly: they are model *constants*, not +parameters to sample — left unset they would inherit priors from the configuration files and float. +""" +for name in ("lens_0", "lens_1"): + galaxy_models[name].mass.sigma = af.UniformPrior( + lower_limit=50.0, upper_limit=600.0 + ) + galaxy_models[name].mass.r_cut = af.UniformPrior(lower_limit=2.0, upper_limit=40.0) + galaxy_models[name].mass.H0 = 67.66 + galaxy_models[name].mass.Om0 = 0.30966 + +""" +The host halo gets a free `mass_at_200`, spanning the full range of cluster masses with a log-uniform prior +(mass scales spanning orders of magnitude are sampled in log space, as we saw for intensities in earlier +chapters). +""" +galaxy_models["host_halo"].dark.mass_at_200 = af.LogUniformPrior( + lower_limit=10**14.5, upper_limit=10**16.0 +) + +""" +Each source's `point_i` component is swapped for the parameter-free `al.ps.PointSolved`. The name pairing +(`point_0`, `point_1`) is what links each model component to the `PointDataset` with the same name — in a +multi-source cluster this pairing is what guarantees every source's positions are fitted by the correct +component. +""" +for i, dataset in enumerate(dataset_list): + setattr(galaxy_models[f"source_{i}"], f"point_{i}", af.Model(al.ps.PointSolved)) + +""" +The scaling tier is composed in a loop over the catalogue, exactly as in the previous tutorial: the single +shared prior `scaling_sigma_ref` is defined once outside the loop, and each member's `sigma` / `r_cut` derive +from it via that member's luminosity ratio. The truth value used by the simulator is sigma_ref = 85 km/s. +""" +scaling_sigma_ref = af.UniformPrior(lower_limit=0.0, upper_limit=200.0) +scaling_sigma_exponent = 0.25 # alpha (Faber-Jackson) +scaling_gamma = 0.2 # mass-to-light tilt, fixed +scaling_rcut_exponent = 1.0 + scaling_gamma - 2.0 * scaling_sigma_exponent # 0.7 + +reference_luminosity = 1.0 +scaling_r_core_fixed = 0.0 +scaling_r_cut_ref_fixed = 5.0 + +scaling_galaxies_list = [] +for centre, luminosity in zip( + scaling_galaxies_centres, scaling_galaxies_luminosity_list +): + luminosity_ratio = luminosity / reference_luminosity + + mass = af.Model(al.mp.dPIEMassSph) + mass.centre = tuple(centre) + mass.sigma = scaling_sigma_ref * luminosity_ratio**scaling_sigma_exponent + mass.r_core = scaling_r_core_fixed + mass.r_cut = scaling_r_cut_ref_fixed * luminosity_ratio**scaling_rcut_exponent + mass.redshift_object = redshift_lens + mass.redshift_source = max(source_redshifts) + mass.H0 = 67.66 + mass.Om0 = 0.30966 + + scaling_galaxies_list.append(af.Model(al.Galaxy, redshift=redshift_lens, mass=mass)) + +scaling_galaxies = af.Collection(scaling_galaxies_list) + +""" +The named galaxies and the scaling tier combine into the overall model. Printing `model.info` confirms the +parameter count: for all its 15 galaxies, the search explores just 6 dimensions. +""" +model = af.Collection( + galaxies=af.Collection(**galaxy_models), + scaling_galaxies=scaling_galaxies, +) + +print(model.info) + +""" +__Analysis + Factor Graph__ + +Each source's dataset gets its own `AnalysisPoint`, which defines the log likelihood for that source: how +well do the model's predictions match this source's observed image positions? + +`fit_positions_cls=al.FitPositionsSourceSolved` selects the **solved source-plane** likelihood — the default +cluster configuration. Rather than forward-solving the lens equation for the model's image positions on every +likelihood evaluation (accurate but expensive), it ray-traces the *observed* positions back to the source +plane and measures how tightly they converge around the analytically-solved source centre, with a weighting +that maps source-plane scatter back to image-plane precision. No lens-equation solve per evaluation makes +cluster-scale inference fast; the `PointSolver` is reserved for validating the final model below. + +The analyses are then combined through a **factor graph**, the same multi-dataset machinery used for +multi-wavelength imaging: each analysis becomes a factor sharing the one global model, and the total log +likelihood is the sum over the per-source likelihoods. This is what lets every source — each at its own +redshift, each with its own multi-plane geometry — constrain the same cluster mass model simultaneously. +""" +analysis_list = [ + al.AnalysisPoint( + dataset=dataset, + solver=solver, + fit_positions_cls=al.FitPositionsSourceSolved, + use_jax=True, + ) + for dataset in dataset_list +] + +analysis_factor_list = [ + af.AnalysisFactor(prior_model=model, analysis=analysis) + for analysis in analysis_list +] + +factor_graph = af.FactorGraphModel(*analysis_factor_list, use_jax=True) + +""" +__Search__ + +We fit the model with Nautilus. The parameter space is only 6-dimensional, so a modest number of live points +suffices; a richer cluster with several individually-modelled galaxies would warrant more. +""" +search = af.Nautilus( + path_prefix=Path("howtolens", "chapter_4"), + name="tutorial_5_cluster_scale", + unique_tag=dataset_name, + n_live=100, + n_batch=50, + iterations_per_quick_update=10000, +) + +""" +__Model Fit__ + +We pass the factor graph's global model and the factor graph itself (as the analysis) to the search. Because +the likelihood involves no pixel-level computation, a full cluster point-source fit takes minutes — this is +the payoff of point-source modeling, given that a pixel-level fit of the same system would take days. +""" +print( + """ + The non-linear search has begun running. + + This Jupyter notebook cell will progress once the search has completed - this could take a few minutes! + + On-the-fly updates every iterations_per_quick_update are printed to the notebook. + """ +) + +result_list = search.fit(model=factor_graph.global_prior_model, analysis=factor_graph) + +print("The search has finished run - you may now continue the notebook.") + +""" +__Results__ + +A cluster fit produces the most complex results object we have met: many galaxies across multiple planes, +tiers whose parameters are derived rather than sampled, and one result per source dataset. This section tours +how to pull out each quantity you will actually need — it is worth working through slowly, because navigating +results is half the practical skill of cluster modeling. + +A factor-graph fit returns one `Result` per dataset (in the same order as `dataset_list`). All results share +the same global model — and therefore the same samples and maximum likelihood model — but each carries its own +dataset and its own maximum likelihood `FitPointDataset`. +""" +print(f"Number of results (one per source dataset): {len(result_list)}") + +result = result_list[0] + +""" +__Results: The Maximum Likelihood Instance__ + +The `max_log_likelihood_instance` is the model instance with the highest likelihood found. Its structure +mirrors the model composition above: named galaxies under `.galaxies`, the scaling tier under +`.scaling_galaxies`. We can read off the best-fit parameters of every tier. +""" +instance = result.max_log_likelihood_instance + +print("Max Log Likelihood Model:") +print(f"BCG sigma: {instance.galaxies.lens_0.mass.sigma:.1f} km/s") +print(f"BCG r_cut: {instance.galaxies.lens_0.mass.r_cut:.2f} arcsec") +print(f"Satellite sigma: {instance.galaxies.lens_1.mass.sigma:.1f} km/s") +print(f"Halo mass_at_200: {instance.galaxies.host_halo.dark.mass_at_200:.2e} solar masses") + +""" +__Results: Per-Member Profiles__ + +The scaling tier's single free parameter was `sigma_ref`, but the instance contains the fully-realised +`dPIEMassSph` of every member — the scaling relation has been applied, so each member carries the `sigma` and +`r_cut` its luminosity implies. This is how you would tabulate the inferred mass of every member galaxy in a +real cluster analysis. +""" +for i, member in enumerate(instance.scaling_galaxies): + print( + f"Member {i}: centre={member.mass.centre}, " + f"sigma={member.mass.sigma:.1f} km/s, r_cut={member.mass.r_cut:.2f} arcsec" + ) + +""" +__Results: The Tracer__ + +`max_log_likelihood_tracer` assembles every galaxy of the instance — named tiers and scaling members alike — +into the best-fit multi-plane `Tracer`. This is the object to use for any lensing calculation with the +best-fit mass model: convergence maps, deflection fields, critical curves and caustics, all computed exactly +as in the earlier chapters, but now for the full cluster. +""" +tracer = result.max_log_likelihood_tracer + +print(f"Tracer planes: {[float(plane.redshift) for plane in tracer.planes]}") +print(f"Galaxies in lens plane: {len(tracer.planes[0])}") + +aplt.subplot_tracer(tracer=tracer, grid=grid) + +""" +__Results: Solved Source-Plane Positions__ + +Each result's `max_log_likelihood_fit` is the `FitPointDataset` of that source. Because we fitted with the +solved source-plane likelihood, the fit's `positions` object carries the analytically-solved source-plane +centre — the model's inferred true position of the source, before lensing. Its `residual_map` lists how far +each back-traced image lands from that centre (in the source plane): for a good model these residuals are +tiny, since all images of a source originate from the same point. +""" +for result in result_list: + fit = result.max_log_likelihood_fit + + print(f"Source '{result.max_log_likelihood_fit.dataset.name}':") + print(f" Solved source-plane centre: {np.asarray(fit.positions.source_plane_coordinate)}") + print(f" Source-plane residuals: {np.asarray(fit.positions.residual_map)}") + print(f" Log likelihood: {float(fit.positions.log_likelihood):.2f}") + +""" +We can perform the same back-tracing manually with the tracer, which makes explicit what the fit just did: +ray-trace each source's observed image positions through the multi-plane lens equation to that source's own +plane. `plane_index_via_redshift_from` maps a source redshift to its plane, and `traced_grid_2d_list_from` +returns the positions traced to every plane — the multi-plane generalisation of the two-plane ray tracing of +chapter 1. +""" +for dataset in dataset_list: + plane_index = tracer.plane_index_via_redshift_from(redshift=dataset.redshift) + traced_grids = tracer.traced_grid_2d_list_from(grid=dataset.positions) + + print(f"Source '{dataset.name}' (z={dataset.redshift}, plane {plane_index}):") + print(f" Back-traced source-plane positions: {traced_grids[plane_index].in_list}") + +""" +__Results: Magnifications__ + +The magnification of each multiple image tells us how much the cluster brightens the source at that position — +the "natural telescope" number. It is computed from the local distortion of the multi-plane deflection field +(its Hessian) at each observed position, via the `LensCalc` object. `plane_j` selects the source's plane, so +each source's magnifications are evaluated through its own multi-plane chain. + +At cluster scale, images near the critical curves can reach magnifications of tens to hundreds — this is +precisely the quantity you would quote when using the cluster to study an intrinsically faint background +galaxy, since the source's true luminosity is the observed luminosity divided by the magnification. +""" +for dataset in dataset_list: + plane_index = tracer.plane_index_via_redshift_from(redshift=dataset.redshift) + + lens_calc = al.LensCalc.from_tracer( + tracer=tracer, use_multi_plane=True, plane_j=plane_index + ) + magnifications = lens_calc.magnification_2d_via_hessian_from(grid=dataset.positions) + + print(f"Source '{dataset.name}' image magnifications: {np.abs(np.asarray(magnifications))}") + +""" +__Results: Image-Plane Validation__ + +The solved source-plane likelihood never solves the lens equation forwards, so it cannot tell us whether the +best-fit model predicts the *right number* of multiple images — a model that would produce an extra image, or +lose an observed one, looks the same to it. The standard final check is therefore to forward-solve the +best-fit model with the `PointSolver` and compare the predicted image positions to the observed ones, +per source. +""" +for result, dataset in zip(result_list, dataset_list): + fit = result.max_log_likelihood_fit + + source_plane_centre = tuple(np.asarray(fit.positions.source_plane_coordinate)) + + model_positions = solver.solve( + tracer=tracer, + source_plane_coordinate=source_plane_centre, + plane_redshift=dataset.redshift, + ) + + print(f"Source '{dataset.name}':") + print(f" Observed positions ({len(dataset.positions)}): {dataset.positions.in_list}") + print(f" Model positions ({len(model_positions)}): {model_positions.in_list}") + + aplt.plot_grid( + grid=model_positions, + title=f"Model Multiple Images: {dataset.name}", + ) + +""" +__Results: Samples and Errors__ + +Everything above used the maximum likelihood model, but the `Samples` object gives the full posterior — the +same API as chapter 2, unchanged by the cluster's complexity. Of particular scientific interest here are the +inferred halo mass and the scaling-relation normalization `sigma_ref` (whose truth value in the simulator was +85 km/s): the marginalised errors on these are what a cluster paper reports. +""" +samples = result_list[0].samples + +median_pdf = samples.median_pdf() + +print("Median PDF Model:") +print(f"Halo mass_at_200: {median_pdf.galaxies.host_halo.dark.mass_at_200:.2e}") +print(f"BCG sigma: {median_pdf.galaxies.lens_0.mass.sigma:.1f} km/s") + +upper_3_sigma = samples.values_at_upper_sigma(sigma=3.0) +lower_3_sigma = samples.values_at_lower_sigma(sigma=3.0) + +print("BCG sigma 3-sigma interval:") +print(f" {lower_3_sigma.galaxies.lens_0.mass.sigma:.1f}" + f" - {upper_3_sigma.galaxies.lens_0.mass.sigma:.1f} km/s") + +aplt.corner_anesthetic(samples=samples) + +""" +__Customization__ + +This tutorial used the default point-source setup throughout: `Point` / `PointSolved` source components, the +solved source-plane likelihood, and default `PointSolver` settings. Point-source modeling has a rich set of +options beyond these — image-plane likelihoods and their pairing schemes, free source centres, fitting the +fluxes of the multiple images, fitting time delays (the observable behind lensed-quasar cosmology) — and a +dedicated workspace guide covers them extensively: + + - `autolens_workspace/scripts/point_source`: the galaxy-scale point-source examples (lensed quasars and + supernovae), including the `fit.py` guide to every fit variant and the `features` folder (fluxes, time + delays, multiple sources). + - `autolens_workspace/scripts/guides/point_source_pairing.py`: the full matrix of position-fitting schemes + and when to use each. + - `autolens_workspace/scripts/cluster`: the cluster-scale workflow this tutorial is built on, including a + fit to real Hubble Space Telescope data of the cluster Abell 2744 with 188 catalogue members and 7 + sources, the CSV-schema guide (`csv_api.py`) and a step-by-step walkthrough of the point-source + likelihood (`likelihood_function.py`). + +__Wrap Up__ + +This tutorial reached the top of the strong-lensing mass ladder. Lets recap what we learnt: + +- **Clusters**: 30+ to hundreds of lens galaxies inside a dark matter halo of more than 10^14 solar masses, + lensing many background sources at once. Their science is extreme magnification — using the cluster as a + natural telescope onto the faint early Universe — and mapping the cluster's dark matter distribution. + +- **Multi-plane ray tracing**: cluster sources sit at different redshifts, so the lens equation is applied + recursively through the planes, with each plane's deflections scaled by cosmological distance ratios. The + `Tracer` handles this automatically, provided every source's redshift is set correctly. + +- **Point-source modeling**: fitting the extended light of every source is computationally prohibitive at + this scale, so the standard workflow fits the positions of each source's multiple images, reducing the + likelihood to source-plane geometry that evaluates in milliseconds. + +- **The CSV interface**: the data (`point_datasets.csv`), the individually-modelled galaxies (`mass.csv` / + `point.csv`) and the member catalogue (`scaling_galaxies.csv`) are all spreadsheet-editable files. Scaling + the model to a richer cluster is a row-append, not a code change. + +- **Six parameters, fifteen galaxies**: fixed observed centres, a scaling relation for the member population + and solved source centres decouple the model's physical richness from the dimensionality of the search. + +- **Results access**: per-member realised profiles, the multi-plane tracer, solved source-plane centres, + per-image magnifications and forward-solved image positions — the complete toolkit for interpreting a + cluster fit. + +Strong lensing — multiple images, arcs, Einstein rings — has carried us from single galaxies to the most +massive structures in the Universe. But a cluster's gravity does not stop deflecting light at the radius +where multiple images form. Far beyond it, every background galaxy is still subtly sheared — distorted by a +percent or less — and by measuring those distortions statistically across thousands of galaxies, the +cluster's mass can be mapped out to its edges. That is **weak lensing**, a different regime with different +data and different statistics, and it is where the next tutorial takes us. +""" diff --git a/scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py b/scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py new file mode 100644 index 0000000..755cfd6 --- /dev/null +++ b/scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py @@ -0,0 +1,457 @@ +""" +Tutorial 6: Weak Lensing +======================== + +Every tutorial in this series so far — indeed, every fit performed in all four chapters of **HowToLens** — has been +a *strong* lensing analysis. Strong lensing occurs when a background galaxy lies so close (in projection) to a +foreground mass that its light is bent into multiple images, arcs or a complete Einstein ring. These dramatic +features are what we simulated, fitted and modeled, from the single galaxy-scale lenses of chapters 1 and 2 up to +the group-scale and cluster-scale systems earlier in this chapter. + +Strong lensing is spectacular, but it is also rare and spatially confined. Multiple images only form inside (or very +near) the Einstein radius, so a strong-lensing analysis constrains the mass distribution only within that region — +a few arc-seconds for a galaxy, tens of arc-seconds for a cluster core. Outside it, the arcs and multiple images +simply do not exist, and everything we have built so far has nothing to fit. + +Yet the gravity of the foreground mass does not stop at the Einstein radius. *Every* background galaxy, however far +from the lens centre, has its shape slightly distorted — sheared — by the foreground mass it sits behind. This is +**weak lensing**: a percent-level stretching of each background galaxy's observed ellipticity, tangentially aligned +around the foreground mass. + +The catch is that we cannot see this distortion in any single galaxy. Galaxies are not intrinsically round: a +typical galaxy has a random unlensed ellipticity of ~0.25 per component, whereas the weak-lensing shear it receives +is ~0.01-0.1. The signal is buried an order of magnitude below the intrinsic "shape noise" of any one object. + +The solution is statistical. The intrinsic ellipticities of different galaxies are randomly oriented and average to +zero, whereas the lensing shear is coherent — every galaxy near a given point on the sky is sheared the same way. +Average the measured shapes of many background galaxies and the random part cancels while the lensing part remains. +This is the fundamental contrast with everything the series has done so far: strong lensing extracts an exquisite +signal from a single special system, weak lensing extracts a faint signal from the ensemble of many ordinary ones. + +This changes what the data even *is*. There is no image to fit, no PSF to convolve, no mask, no pixelized source +reconstruction. A weak-lensing dataset is a **shear catalogue**: a table of background-galaxy sky positions, each +with a measured shear estimate (two ellipticity components) and a per-galaxy uncertainty. Fitting one means +comparing a mass model's predicted shear field, evaluated at those positions, against the measured values. + +In this tutorial we simulate a weak-lensing shear catalogue around a cluster-scale lens (entirely in memory — no +files are written), visualize its shear field, fit it with a mass model whose parameters we know, and then infer +those parameters with a non-linear search. You will recognise every step: the `Tracer`, `Fit`, `Analysis` and +search workflow of the whole series, applied to a completely different kind of data. + +This tutorial closes both this chapter and the four core chapters of **HowToLens**. + +__Contents__ + +- **Shear Catalogues:** What a weak-lensing dataset is and where the catalogue comes from in a real analysis. +- **Mass Scales:** Why weak lensing is a cluster-scale tool, tying to the mass scales of the previous tutorials. +- **Ray Tracing:** Build a `Tracer` for a cluster-scale lens, the mass whose shear field we will measure. +- **Source Galaxy Positions:** Draw background galaxies in an annulus outside the strong-lensing core. +- **Simulate:** Create a `WeakDataset` in memory with `SimulatorShearYX`, adding realistic shape noise. +- **Visualize:** Plot the shear field as a whisker map and the dataset's 2x2 summary mosaic. +- **Mass Map:** Invert the shear catalogue into a model-independent convergence map (Kaiser-Squires). +- **Fitting:** Fit the catalogue with the true mass model via `FitWeak` and inspect its goodness-of-fit. +- **Model Fit:** Infer the mass model from the shear catalogue alone with `AnalysisWeak` and a non-linear search. +- **Result:** The inferred model, the fit mosaic and the tangential shear profile. +- **Joint Strong and Weak Lensing:** How PyAutoLens expects weak lensing to be used — combined with strong lensing. +- **Wrap Up:** The end of the chapter, and of the core HowToLens lectures. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +import numpy as np +from pathlib import Path + +import autofit as af +import autolens as al +import autolens.plot as aplt + +""" +__Shear Catalogues__ + +Before we simulate anything, lets be clear about what a real weak-lensing dataset looks like, because it is unlike +any data this series has fitted before. + +A **shear catalogue** is a table with one row per background galaxy, containing: + + - The galaxy's sky position (for us, (y, x) arc-second offsets from a chosen centre). + - A measured shear estimate (gamma_1, gamma_2) — in practice the galaxy's measured ellipticity components, which + estimate the lensing shear because the intrinsic shape averages to zero over many galaxies. + - A per-galaxy uncertainty, combining the intrinsic shape dispersion (~0.25 per component) with the measurement + error, added in quadrature. + +Producing this catalogue from raw telescope images is a substantial data-reduction task in its own right: galaxies +must be detected, stars separated from galaxies, the PSF measured and its blurring of galaxy shapes corrected +(the PSF makes every galaxy look rounder, diluting the shear signal), and calibration biases characterised. +Dedicated shape-measurement pipelines exist for exactly this job, and entire survey collaborations are built +around getting it right. + +**All of that is outside PyAutoLens's scope.** PyAutoLens assumes the shear catalogue has already been produced +upstream by such a pipeline, and starts where the catalogue ends: fitting mass models to the measured shears. This +division of labour is why the weak-lensing API is so lean — no PSF, no mask, no image — the hard observational +systematics were dealt with before the data reached us. + +__Mass Scales__ + +This chapter has climbed a ladder of mass scales, and weak lensing is its top rung. The strength of the statistical +shear signal around a lens scales with its mass, so whether weak lensing is even measurable depends on where on the +ladder we stand: + + - **A single galaxy** (Einstein radius ~1-2", the lenses of chapters 1-3): the shear imprinted on background + galaxies beyond the Einstein radius is far too weak to detect around any one system — there are too few + background galaxies carrying too little shear. (Surveys recover it only by *stacking* thousands of lens + galaxies, a technique called galaxy-galaxy lensing that averages over the lens population.) + + - **A group** (Einstein radius ~4-10", the scale of the group tutorial earlier in this chapter): the signal is + marginal — detectable for well-observed individual groups, but noisy. + + - **A cluster** (Einstein radius ~10-30"+, the previous tutorial's scale): the signal is strong enough to measure + around a *single* system. Hundreds to thousands of background galaxies within a few arc-minutes each carry a + |gamma| ~ 0.01-0.2 shear, and their average traces the cluster's mass out to radii of arc-minutes — ten or more + times further out than the strong-lensing core. + +This reach is the scientific point. The dark matter halo of a cluster extends to Mpc scales, far beyond where arcs +form. Weak lensing is the only lensing probe of that outer halo, which is why it is the tool of choice for cluster +mass measurements, and it is why this tutorial simulates a cluster-scale lens. + +__Ray Tracing__ + +We begin exactly as the series always has: with a `Tracer`. The lens is a cluster-scale `Isothermal` mass profile +with an Einstein radius of 25.0" — the same order as the previous tutorial's cluster, and a mass for which the +weak shear signal is genuinely measurable. The source galaxy carries no light profile: weak lensing measures the +lens's shear field at the background galaxies' positions, so their appearance is irrelevant — they are pure probes, +included only to give the `Tracer` its source-plane redshift. + +Note also what the lens galaxy lacks: no light profile. A shear catalogue contains no image of the lens, so there +is no lens light to model — another simplification compared to every imaging fit in this series. +""" +lens_galaxy = al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=25.0, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0), + ), +) + +source_galaxy = al.Galaxy(redshift=1.0) + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) + +""" +__Source Galaxy Positions__ + +Real weak-lensing measurements avoid the strong-lensing core: inside it the shear is no longer weak (the linear +shear approximation breaks down), and cluster-member galaxies contaminate the background sample. We therefore draw +our background galaxies in an **annulus**, from an inner radius of 50" (twice the Einstein radius, safely into the +weak regime) to an outer radius of 200" (over 3 arc-minutes), distributed uniformly in area. + +At 50" from this lens the shear is |gamma| ~ 0.25, and by 200" it has fallen to |gamma| ~ 0.06 — every galaxy is a +weak probe whose individual distortion is at or below the shape noise. With 1500 galaxies (a deep-survey source +density of ~45 per square arc-minute) the *ensemble* nonetheless carries a decisive signal, exactly as in a real +cluster analysis. + +The positions are stored as an `al.Grid2DIrregular` of (y, x) coordinates — the same irregular-grid structure the +point-source tutorials used, because a catalogue of sky positions is not a uniform pixel grid. +""" +rng = np.random.default_rng(1) + +n_galaxies = 1500 +radius_inner = 50.0 # arc-seconds — inside this we are in the strong-lensing core. +radius_outer = 200.0 # arc-seconds — the edge of our simulated weak-lensing field. + +radii = np.sqrt(rng.uniform(radius_inner**2.0, radius_outer**2.0, n_galaxies)) +phi = rng.uniform(0.0, 2.0 * np.pi, n_galaxies) + +positions = al.Grid2DIrregular( + values=np.stack([radii * np.sin(phi), radii * np.cos(phi)], axis=1) +) + +""" +__Simulate__ + +The `SimulatorShearYX` object simulates a weak-lensing shear catalogue from a tracer: its `via_tracer_from` method +evaluates the tracer's shear field at every galaxy position (by differentiating the deflection-angle field) and +adds Gaussian shape noise to each measurement. + +A `noise_sigma` of 0.25 per shear component is the standard intrinsic shape dispersion of real galaxies — this is +the dominant noise source in weak lensing, and it is what makes the per-galaxy signal invisible: the noise on each +measurement is several times larger than the shear it contains. + +Unlike every previous tutorial, we do not write this dataset to disk and reload it. The whole catalogue is a few +thousand numbers, so we keep it in memory and use it directly — which also mirrors how you might resimulate +catalogues on the fly when testing survey sensitivities. (This repo's `scripts/simulator/weak_lensing.py` — the +script chapter 1's data tutorial auto-ran to give you your first glimpse of a shear catalogue — shows the +disk-based version, with the same JSON output pattern as the imaging simulators.) +""" +dataset_name = "weak_lensing" + +simulator = al.SimulatorShearYX(noise_sigma=0.25, seed=1) + +dataset = simulator.via_tracer_from( + tracer=tracer, + grid=positions, + name=dataset_name, +) + +print(dataset.info) + +""" +__Visualize__ + +Weak-lensing data has its own visual language. The shear field is drawn as a **whisker map**: at each galaxy +position, a line segment whose length is the shear magnitude |gamma| and whose orientation is the shear's position +angle. The segments are deliberately *headless* (no arrowheads), because shear is a spin-2 quantity — rotating a +shear by 180 degrees maps it back onto itself, so a whisker has an orientation but no direction, and an arrowhead +would suggest information the data does not contain. + +Around a massive lens, the whiskers align *tangentially* — each traces a small piece of a circle around the lens +centre, the statistical echo of the rings and arcs of strong lensing. Squint at the plot below and you can see it, +though the shape noise makes it far from obvious galaxy-by-galaxy: this is what a signal an order of magnitude +below the per-object noise looks like. +""" +aplt.plot_shear_yx_2d(shear_yx=dataset.shear_yx) + +""" +`aplt.subplot_weak_dataset` summarises the full dataset as a 2x2 mosaic: the whisker map, the per-galaxy noise-map, +the shear magnitude |gamma| and the position angle phi at every galaxy. +""" +aplt.subplot_weak_dataset(dataset=dataset) + +""" +__Mass Map__ + +A remarkable property of weak lensing is that the shear catalogue can be inverted directly into a map of the +convergence `kappa` — the dimensionless projected mass density we have used throughout the series — *without +assuming any mass model at all*. In Fourier space, shear and convergence are related algebraically, so two FFTs +turn the catalogue into a "dark matter map". This is the Kaiser-Squires technique, and it produced some of the most +famous images in cosmology, such as the mass map of the Bullet Cluster showing dark matter offset from the +colliding gas. + +`aplt.plot_convergence_map` bins the catalogue onto a regular grid, applies a small Gaussian smoothing (per-cell +shears are shape-noise dominated) and plots the reconstruction. For our simulated cluster it peaks at the lens +centre at (0.0", 0.0"). Two caveats: the map's mean is unconstrained (the mass-sheet degeneracy, which chapter 3 +met in its strong-lensing form) and FFT periodicity produces edge artefacts — for quantitative masses we fit a +mass model, which is what the rest of this tutorial does. +""" +aplt.plot_convergence_map( + shear_yx=dataset.shear_yx, + shape_native=(30, 30), + smoothing_sigma_pixels=1.0, +) + +""" +__Fitting__ + +We now fit the catalogue, following the same pattern as every fit in this series: build a model `Tracer`, pass it +with the dataset to a `Fit` object, and inspect residuals and the log likelihood. For weak lensing the fit object +is `FitWeak`, and because we simulated the data ourselves we can hand it the *true* tracer and see what a perfect +model looks like. + +`FitWeak` evaluates the model tracer's shear field at the dataset's galaxy positions and compares it with the +measured shears, assuming each component is independently Gaussian-distributed around the model with the +per-galaxy noise. Each galaxy contributes **two** independent data points (gamma_1 and gamma_2), so the number of +degrees of freedom is 2 * n_galaxies, and for a good fit whose residuals are pure shape noise the chi-squared +should be close to that number. + +Compare this to the imaging fits of chapter 1: there, the fit convolved a model image with the PSF and compared +tens of thousands of pixels; here there is no convolution, no mask and only a few thousand numbers. A weak-lensing +likelihood is orders of magnitude cheaper — which is precisely why it is so inexpensive to add to a strong-lensing +analysis, as discussed at the end of this tutorial. +""" +fit = al.FitWeak(dataset=dataset, tracer=tracer) + +print() +print("Fit Summary (true model)") +print("------------------------") +print(f"n_galaxies : {dataset.n_galaxies}") +print(f"degrees_of_freedom: {2 * dataset.n_galaxies}") +print(f"chi_squared : {fit.chi_squared:.3f}") +print(f"log_likelihood : {fit.log_likelihood:.3f}") + +""" +As always, an incorrect mass model produces a worse fit. Halving the Einstein radius halves the predicted shear +everywhere, leaving coherent tangential residuals across the field, and the log likelihood drops accordingly. +""" +tracer_wrong = al.Tracer( + galaxies=[ + al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=12.5, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0), + ), + ), + source_galaxy, + ] +) + +fit_wrong = al.FitWeak(dataset=dataset, tracer=tracer_wrong) + +print() +print(f"log_likelihood (true model) : {fit.log_likelihood:.3f}") +print(f"log_likelihood (wrong model): {fit_wrong.log_likelihood:.3f}") + +""" +`aplt.subplot_fit_weak` visualises a weak-lensing fit as a 2x2 mosaic: the observed shear field, the model shear +field, the two overlaid on one axes (data in black, model in red), and the per-galaxy chi-squared map. For the true +model the overlay's residuals are short and randomly oriented — the shape-noise floor — and the chi-squared map is +spatially uniform. +""" +aplt.subplot_fit_weak(fit=fit) + +""" +__Model Fit__ + +In a real analysis we do not know the mass model — we infer it. The workflow is the one you have used since +chapter 2: compose a model with `af.Model` and `af.Collection`, choose a non-linear search, create an `Analysis` +object and fit. For weak lensing the analysis object is `AnalysisWeak`, whose `log_likelihood_function` builds a +`Tracer` from each sampled parameter set, evaluates its shear field at the catalogue positions and returns the +Gaussian log likelihood — the same `FitWeak` machinery we just used by hand. + +The model is the lens's `Isothermal` mass [5 parameters]; the source galaxy again carries no components. Note the +tiny parameter space — no lens light, no source light, no shear-as-nuisance-parameter (the shear field *is* the +data here) — compared to the N=20+ models of the imaging tutorials. + +One adjustment is needed, familiar from the group and cluster tutorials: the default priors are tuned for +galaxy-scale lenses (Einstein radii of 0"-8", centres within a fraction of an arc-second). Our cluster's 25" +Einstein radius lies entirely outside that prior, so we widen the priors to cluster scales — always match your +priors to the mass scale of the system you are fitting. +""" +mass = af.Model(al.mp.Isothermal) +mass.einstein_radius = af.UniformPrior(lower_limit=0.0, upper_limit=60.0) +mass.centre.centre_0 = af.GaussianPrior(mean=0.0, sigma=20.0) +mass.centre.centre_1 = af.GaussianPrior(mean=0.0, sigma=20.0) + +lens = af.Model(al.Galaxy, redshift=0.5, mass=mass) + +source = af.Model(al.Galaxy, redshift=1.0) + +model = af.Collection(galaxies=af.Collection(lens=lens, source=source)) + +print(model.info) + +""" +We fit the model with the nested sampling algorithm Nautilus, as throughout the series. A weak-lensing likelihood +evaluation takes milliseconds (one shear-field evaluation at 1500 positions plus a chi-squared sum), so this +5-parameter fit completes in minutes on an ordinary CPU — a refreshing contrast to the imaging fits of the earlier +chapters. +""" +search = af.Nautilus( + path_prefix=Path("howtolens", "chapter_4"), + name="tutorial_6_weak_lensing", + unique_tag=dataset_name, + n_live=100, + iterations_per_quick_update=5000, +) + +analysis = al.AnalysisWeak(dataset=dataset) + +print( + "The non-linear search has begun running - checkout the workspace/output/howtolens/chapter_4/tutorial_6_weak_lensing" + " folder for live output of the results, images and model." + " This Jupyter notebook cell with progress once search has completed - this could take some time!" +) + +result = search.fit(model=model, analysis=analysis) + +print("The search has finished run - you may now continue the notebook.") + +""" +__Result__ + +The result's `info` attribute confirms the search recovers the input mass model — Einstein radius, ellipticity and +centre — from nothing but the statistically-averaged shapes of background galaxies. No arc, no ring, no multiple +images: the ensemble alone constrains the cluster's mass. +""" +print(result.info) + +aplt.subplot_fit_weak(fit=result.max_log_likelihood_fit) + +""" +Cluster weak-lensing measurements are conventionally presented as the **tangential shear profile**: the mean +tangential shear gamma_t of background galaxies in radial bins about the lens centre, which traces the projected +mass profile. `aplt.plot_shear_profile` bins our catalogue and, given the fit, overlays the model's profile as a +line. + +The plot also shows the *cross* component gamma_x — the shear rotated by 45 degrees. Gravitational lensing produces +no cross component at leading order, so cross points scattering around zero is the standard "B-mode" null test of a +weak-lensing measurement: a systematic contaminating the shapes would show up there. +""" +aplt.plot_shear_profile( + result.max_log_likelihood_fit, + centre=(0.0, 0.0), + bins=8, +) + +""" +__Joint Strong and Weak Lensing__ + +We fitted this catalogue on its own, and weak-lensing-only fits of real catalogues are a fully supported PyAutoLens +workflow. But the way PyAutoLens really expects weak lensing to be used is **jointly with strong lensing**, because +the two probes are perfectly complementary: + + - **Strong lensing** (everything before this tutorial) pins the mass in the inner region with exquisite precision — + the positions of multiple images and arcs nail the Einstein mass and the inner density profile — but says + nothing beyond the Einstein radius. + + - **Weak lensing** (this tutorial) constrains the mass profile out to arc-minute radii — the outskirts of the dark + matter halo — but its centre and inner profile are only weakly pinned by the field's geometry. + +Fit both datasets simultaneously with a single mass model — one analysis for the imaging or point-source data, one +`AnalysisWeak` for the shear catalogue, combined exactly as the multi-dataset fits earlier in this chapter combined +their analyses — and the model is anchored at the centre by strong lensing while its outer profile is measured by +weak lensing. This hybrid is the state of the art for cluster mass measurements, used across systems like the +Frontier Fields clusters, and PyAutoLens's shared `Tracer` infrastructure makes it natural: the same mass profiles +that ray-traced arcs in the previous tutorial produce the shear field fitted here. + +The workspace's `autolens_workspace/*/weak` package has the complete treatment: + + - `weak/start_here.py`: fits a real shear catalogue of the merging cluster Abell 2744 ("Pandora's Cluster") — + download, quality cuts, mass map, NFW halo fit. + - `weak/simulator.py`, `weak/fit.py`, `weak/modeling.py`: the disk-based versions of this tutorial's workflow. + - `weak/likelihood_function.py`: a step-by-step walkthrough of the weak-lensing likelihood. + - `weak/features/strong_lensing`: the joint strong-plus-weak fits described above. + +__Wrap Up__ + +This tutorial completed the chapter's journey up the mass ladder, and with it the four core chapters of +**HowToLens**. Lets take stock of how far we have come. + +Chapter 1 built lensing from first principles: grids, light and mass profiles, ray tracing, the lens equation, and +fitting imaging data with a `Tracer`. Chapter 2 turned fitting into inference — non-linear searches, priors, +Bayesian model comparison — so we could *infer* lens models rather than guess them. Chapter 3 replaced analytic +sources with pixelized reconstructions, letting the data itself reveal the source's morphology. + +This chapter then scaled those tools up through every regime nature offers, and in this tutorial: + +- **Weak Lensing:** Outside the strong-lensing region every background galaxy is still weakly sheared; the signal + is invisible per galaxy beneath intrinsic shape noise but emerges statistically over an ensemble. + +- **Shear Catalogues:** The weak-lensing dataset is a catalogue of galaxy positions, measured shear estimates and + uncertainties, produced upstream by a dedicated shape-measurement pipeline — PyAutoLens fits mass models to it + and leaves the shape measurement to those pipelines. + +- **Mass Scales:** The statistical signal is hopeless around a single galaxy, marginal around a group and + measurable around a cluster — and it probes the dark matter halo to radii far beyond the reach of arcs. + +- **The Same Workflow:** `SimulatorShearYX`, `FitWeak` and `AnalysisWeak` slot into the identical + simulate-fit-model pattern the series has used since chapter 1, and a non-linear search recovered the cluster's + mass from galaxy shapes alone. + +- **Joint Fits:** Strong lensing pins the inner mass, weak lensing measures the outskirts; fitting both with one + model is the expected PyAutoLens workflow for cluster and group science. + +And that is the series: from a single lens galaxy, to lenses with extra galaxies, to multi-galaxy lenses, to +scaling relations that keep many-galaxy models tractable, to group-scale and cluster-scale systems, and finally +beyond the Einstein radius altogether into the weak-lensing regime. You have every conceptual tool the modern +lensing literature uses, and you have used each one on data. + +Where next? The `autolens_workspace` is the destination for real science. Its `imaging`, `point_source`, `group`, +`cluster` and `weak` packages hold the production-ready versions of everything taught here — including the +`start_here.py` scripts that are each topic's canonical, always-current reference — plus the features this series +could only gesture at: multi-wavelength fits, interferometry, automated SLaM pipelines and more. Take your own +data there, and good luck with your science! +""" diff --git a/scripts/chapter_optional/tutorial_searches.py b/scripts/chapter_optional/tutorial_searches.py index 35d2289..4bb465d 100644 --- a/scripts/chapter_optional/tutorial_searches.py +++ b/scripts/chapter_optional/tutorial_searches.py @@ -137,7 +137,7 @@ unique_tag=dataset_name, n_live=400, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) analysis = al.AnalysisImaging(dataset=dataset) @@ -170,7 +170,7 @@ unique_tag=dataset_name, n_live=75, n_batch=50, # GPU batching and VRAM use explained in chapter 2 tutorial 2. - iterations_per_quick_update=2500, # Outpuers Notebook visualization of max likelihood model every N iterations + iterations_per_quick_update=2500, # Outputs Notebook visualization of max likelihood model every N iterations ) print( @@ -205,7 +205,7 @@ error estimates (perhaps this is our final lens model fit before we publish the results in a paper), these extra iterations are acceptable. -However, we often don't care about the errors. For example, in the previous tutorial when chaining searches, the only +However, we often don't care about the errors. For example, in chapter 2's search chaining tutorial, the only result we used from the fit performed in the first search was the maximum log likelihood model, omitting the errors entirely! Its seems wasteful to use a nested sampling algorithm like Nautilus to map out the entirity of parameter space when we don't use this information! @@ -308,7 +308,7 @@ __Wrap Up__ This tutorial showed how the same lens model can be fitted using different non-linear searches (e.g. `Nautilus`, -`Emcee`, `PySwarms`). Each search explores parameter space in a different way, and the best choice depends on the +`Emcee`, `Zeus`). Each search explores parameter space in a different way, and the best choice depends on the dimensionality and complexity of the model you are fitting. For the vast majority of lens models in **PyAutoLens**, the default nested sampling search `Nautilus` is recommended, diff --git a/scripts/simulator/cluster.py b/scripts/simulator/cluster.py new file mode 100644 index 0000000..e9989d7 --- /dev/null +++ b/scripts/simulator/cluster.py @@ -0,0 +1,529 @@ +""" +Simulator: Cluster +================== + +This script simulates a strong lens on the 'cluster' scale: 2 main lens galaxies (a brightest cluster +galaxy and a satellite), 10 lower-mass cluster member galaxies on a luminosity-mass scaling relation, +a cluster-scale dark matter halo not tied to any individual galaxy, and 2 multiply-imaged background +source galaxies at *different* redshifts (z = 1.0 and z = 2.0) — a genuine multi-plane lens. + +It is used to illustrate cluster-scale lens modeling in the HowToLens lecture series. + +Cluster modeling uses the *point source* API: rather than fitting the extended arc light of each +lensed source, only the image-plane positions of its multiple images are fitted. This script therefore +simulates point-source datasets (one per source) alongside CCD imaging, which in real datasets is used +to measure those positions and to visually confirm the lens configuration. + +__Contents__ + +- **Dataset Paths:** The `dataset_path` describes where the simulated cluster dataset is output to. +- **Redshifts:** The lens redshift and the two distinct source redshifts of the multi-plane system. +- **Galaxy Centres:** The centres of the main lens galaxies, scaling members, halo and sources. +- **Grids:** The imaging grid (with adaptive over sampling) and a coarse visualization grid. +- **Main Lens Galaxies:** The 2 individually-modelled cluster members, each a `SersicSph` light and `dPIEMassSph` mass. +- **Scaling Member Galaxies:** 10 lower-mass members whose masses derive from a luminosity-mass scaling relation. +- **Host Dark Matter Halo:** A standalone `NFWMCRLudlowSph` halo with `mass_at_200 = 10^15.3`. +- **Source Galaxies:** The 2 multi-plane background sources, each a `SersicCore` light + a `Point` component. +- **Ray Tracing:** Combine all galaxies into a single multi-plane `Tracer`. +- **Point Solver:** Solve the lens equation for the image-plane multiple-image positions of each source. +- **Point Datasets:** Collect the per-source positions (with noise) into `PointDataset` objects. +- **Combined CSV:** Write all point datasets to a single hand-editable CSV. +- **Scaling Galaxies CSV:** Write the scaling-member centres and luminosities to `scaling_galaxies.csv`. +- **Model CSVs:** Write the truth model to `mass.csv` + `light.csv` + `point.csv` (the named-galaxy CSV API). +- **Tracer json:** Save the true `Tracer` in the dataset folder as a .json file. +- **Imaging:** Simulate CCD imaging of the cluster and output it to .fits files. +- **Visualize:** Output subplots of the point datasets, tracer and imaging to the dataset path. + +__Model__ + +This script simulates a 'cluster-scale' strong lens where: + + - The 2 main lens galaxies have `SersicSph` light profiles and `dPIEMassSph` mass profiles. + - The 10 scaling-tier member galaxies have `dPIEMassSph` masses set by a luminosity-mass scaling relation. + - The cluster's dark matter halo is a standalone `NFWMCRLudlowSph` profile. + - The 2 source galaxies have `SersicCore` light profiles and `Point` components at z = 1.0 and z = 2.0. + +__Start Here Notebook__ + +If any code in this script is unclear, refer to the `autolens_workspace/*/cluster/simulator.ipynb` notebook. +""" + +import importlib.util +import sys + +if importlib.util.find_spec("jax") is None: + print( + "Skipping this simulator: it requires the `jax` package, which is not " + "installed (install with `pip install autolens[optional]`)." + ) + sys.exit(0) + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +import jax +import jax.numpy as jnp +import numpy as np +from pathlib import Path + +import autolens as al +import autolens.plot as aplt + +""" +__Dataset Paths__ + +The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a +descriptive name. The cluster dataset is output to `howtolens/dataset/cluster/simple`, which the +tutorial on cluster-scale lensing loads via the auto-simulation guard. +""" +dataset_type = "cluster" +dataset_name = "simple" +dataset_path = Path("dataset", dataset_type, dataset_name) + +""" +__Redshifts__ + +All lens galaxies and the host dark matter halo sit at the lens redshift z = 0.5. The two sources sit +at *different* redshifts (z = 1.0 and z = 2.0), making this a genuine multi-plane lens: light from the +z = 2.0 source passes through the z = 1.0 plane on its way to us, so the `Tracer` must ray-trace +through every plane in redshift order. +""" +redshift_lens = 0.5 +source_redshifts = [1.0, 2.0] + +""" +__Galaxy Centres__ + +The host halo is anchored at the cluster centre (the origin); the two main galaxies are the central +brightest cluster galaxy (BCG) and a satellite offset to the upper-right. The 10 scaling-member +centres sit at radii of 5-15" from the centre — inside the strongly-lensed region of the host halo but +clear of the cores of the two main galaxies. The source centres are chosen so both sources land in the +strongly-lensed region and are multiply imaged. +""" +main_lens_centres = [ + (0.0, 0.0), # BCG at cluster centre + (10.0, 8.0), # satellite member +] + +scaling_galaxies_centres = [ + (5.5, -6.5), + (-7.5, 3.0), + (12.0, -5.0), + (-4.0, -9.0), + (3.0, 13.0), + (-14.0, 4.0), + (15.0, 9.0), + (-9.0, -12.0), + (8.5, 5.5), + (-6.5, 11.0), +] + +scaling_galaxies_luminosities = [ + 0.40, + 0.32, + 0.25, + 0.20, + 0.16, + 0.13, + 0.10, + 0.08, + 0.06, + 0.05, +] + +host_halo_centre = (0.0, 0.0) + +source_centres = [ + (0.3, 0.5), + (-0.8, 1.2), +] + +""" +__Grids__ + +The typical Einstein radius of a 10^15 solar-mass halo is ~20-30" and the member galaxies span ~30" +across, so the field must be large (100" x 100") to capture the multiple images and arcs — far bigger +than the galaxy-scale fields of earlier simulators. + +Two grids are used: a high-resolution `imaging_grid` for rendering the CCD image, with adaptive over +sampling around the centre of every cluster member, and a coarse `viz_grid` used only for the +visualization plots at the end of the script (which do not need full resolution). +""" +imaging_grid = al.Grid2D.uniform( + shape_native=(500, 500), + pixel_scales=0.2, +) + +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=imaging_grid, + sub_size_list=[32, 8, 2], + radial_list=[0.3, 0.6], + centre_list=main_lens_centres + scaling_galaxies_centres, +) + +imaging_grid = imaging_grid.apply_over_sampling(over_sample_size=over_sample_size) + +viz_grid = al.Grid2D.uniform(shape_native=(200, 200), pixel_scales=0.5) + +""" +__Main Lens Galaxies__ + +The 2 individually-modelled cluster members. Each has a `SersicSph` light profile (used for the CCD +imaging — the light plays no role in point-source modeling) and a `dPIEMassSph` mass profile, the +standard truncated mass profile for cluster members. Its parameters are Lenstool's native ones: + + - `sigma` (km/s): the fiducial velocity dispersion, setting the lens strength. + - `r_core` (arcsec): the core radius, fixed to 0 here (the standard vanishing-core convention). + - `r_cut` (arcsec): the truncation radius, beyond which the density falls off rapidly — the tidal + stripping physics of galaxies orbiting inside a cluster's shared potential. +""" +main_lens_dpie_params = [ + # (r_core, r_cut, sigma) per galaxy — arcsec, arcsec, km/s + (0.0, 20.0, 330.0), # BCG — strongest + (0.0, 12.0, 210.0), # satellite +] + +main_lens_sersic_params = [ + # (intensity, effective_radius, sersic_index) + (1.5, 3.0, 4.0), # BCG — bright and extended + (0.8, 1.5, 3.5), # satellite +] + +main_lens_galaxies = [] +for centre, (r_core, r_cut, sigma), (intensity, effective_radius, sersic_index) in zip( + main_lens_centres, main_lens_dpie_params, main_lens_sersic_params +): + bulge = al.lp.SersicSph( + centre=centre, + intensity=intensity, + effective_radius=effective_radius, + sersic_index=sersic_index, + ) + mass = al.mp.dPIEMassSph( + centre=centre, + sigma=sigma, + r_core=r_core, + r_cut=r_cut, + redshift_object=redshift_lens, + redshift_source=max(source_redshifts), + ) + main_lens_galaxies.append(al.Galaxy(redshift=redshift_lens, bulge=bulge, mass=mass)) + +""" +__Scaling Member Galaxies__ + +The 10 lower-mass cluster members are modelled collectively via a luminosity-mass scaling relation — +the convention used by essentially every published cluster strong-lensing analysis: + + sigma_i = sigma_ref * (L_i / L_ref) ** 0.25 + r_cut_i = r_cut_ref * (L_i / L_ref) ** 0.7 + r_core_i = 0 + +where `L_ref` is a fixed reference luminosity and `sigma_ref` is the fiducial velocity dispersion of a +galaxy at that reference luminosity. The truth value simulated here is `sigma_ref = 85.0` km/s — the +value the tutorial's model-fit seeks to recover with a single free parameter for the whole tier. +""" +scaling_sigma_ref_truth = 85.0 +scaling_sigma_exponent = 0.25 # alpha (Faber-Jackson) +scaling_gamma = 0.2 # mass-to-light tilt, fixed +scaling_rcut_exponent = 1.0 + scaling_gamma - 2.0 * scaling_sigma_exponent # 0.7 +reference_luminosity = 1.0 +scaling_r_core = 0.0 # vanishing core — fixed, never scaled +scaling_r_cut_ref = 5.0 + +scaling_galaxies = [] +for centre, luminosity in zip(scaling_galaxies_centres, scaling_galaxies_luminosities): + bulge = al.lp.SersicSph( + centre=centre, + intensity=luminosity, + effective_radius=0.8, + sersic_index=3.0, + ) + luminosity_ratio = luminosity / reference_luminosity + mass = al.mp.dPIEMassSph( + centre=centre, + sigma=scaling_sigma_ref_truth * luminosity_ratio**scaling_sigma_exponent, + r_core=scaling_r_core, + r_cut=scaling_r_cut_ref * luminosity_ratio**scaling_rcut_exponent, + redshift_object=redshift_lens, + redshift_source=max(source_redshifts), + ) + scaling_galaxies.append(al.Galaxy(redshift=redshift_lens, bulge=bulge, mass=mass)) + +""" +__Host Dark Matter Halo__ + +A standalone galaxy holding the cluster's dark matter halo. It has no light profile — it sits in the +tracer solely to contribute mass. `NFWMCRLudlowSph` is parameterised by the physical halo mass within +r_200 (`mass_at_200 = 10^15.3`, ~2 x 10^15 solar masses) and the redshifts; its concentration follows +from a concentration-mass relation. The `redshift_source` is anchored to the *furthest* source +(z = 2.0), the multi-plane convention used throughout (deflections are normalized to the final plane). +""" +host_halo = al.mp.NFWMCRLudlowSph( + centre=host_halo_centre, + mass_at_200=10**15.3, + redshift_object=redshift_lens, + redshift_source=max(source_redshifts), +) + +host_halo_galaxy = al.Galaxy(redshift=redshift_lens, dark=host_halo) + +""" +__Source Galaxies__ + +The 2 background sources at *different* redshifts. Each carries a `SersicCore` light profile (so the +lensed arcs appear in the CCD imaging) and a `Point` component whose multiple-image positions we solve +for below — those positions are the data the tutorial's model-fit uses. + +The attribute name of each `Point` (`point_0`, `point_1`) is important: it is the name that pairs each +model component to its `PointDataset` during modeling. +""" +source_galaxies = [] +for i, (centre, src_z) in enumerate(zip(source_centres, source_redshifts)): + bulge = al.lp.SersicCore( + centre=centre, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0 + 30.0 * i), + intensity=2.0, + effective_radius=0.3, + sersic_index=1.0, + ) + point = al.ps.Point(centre=centre) + source_galaxies.append( + al.Galaxy(redshift=src_z, bulge=bulge, **{f"point_{i}": point}) + ) + +""" +__Ray Tracing__ + +Combine the main lens galaxies, scaling members, host halo and sources into a single tracer. With +sources at distinct redshifts, the tracer automatically performs multi-plane ray tracing. +""" +tracer = al.Tracer( + galaxies=main_lens_galaxies + + scaling_galaxies + + [host_halo_galaxy] + + source_galaxies +) + +""" +__Point Solver__ + +The `PointSolver` solves the lens equation for the image-plane positions of each point source, by +ray-tracing triangles from the image plane to the source plane and iteratively refining those that +contain the source centre. At cluster scale (13 lens galaxies, multi-plane ray tracing) this solve is +expensive, so we accelerate it with JAX: `use_jax=True` plus a `jax.jit` wrapper around the solve call +compiles the triangle-refinement kernel once and reuses it for both sources. + +The solver's grid must span the full cluster field, because the multiple images sit at radii of +~20-30" from the cluster centre — far outside the arc-second scale grids of galaxy-scale lensing. + +`plane_redshift` is per-source and must be passed: it tells the solver which source plane to solve +for. Without it, the solver defaults to the tracer's final plane, which would silently place the +z = 1.0 source's images as if it sat at z = 2.0. +""" +from autolens.jax import register_tracer_classes + +register_tracer_classes(tracer) + +solver = al.PointSolver.for_grid( + grid=al.Grid2D.uniform(shape_native=(400, 400), pixel_scales=0.25), + pixel_scale_precision=0.001, + magnification_threshold=0.1, + use_jax=True, +) + + +def jitted_solve_for(plane_redshift): + @jax.jit + def jitted_solve(tracer, source_plane_coordinate): + return solver.solve( + tracer=tracer, + source_plane_coordinate=source_plane_coordinate, + plane_redshift=plane_redshift, + ).array + + return jitted_solve + + +positions_list = [] +for i, (src_centre, src_z) in enumerate(zip(source_centres, source_redshifts)): + coord = jnp.asarray(src_centre) + jitted_solve = jitted_solve_for(float(src_z)) + raw = np.asarray(jitted_solve(tracer, coord)) + finite = ~(np.isinf(raw).any(axis=1) | np.isnan(raw).any(axis=1)) + positions_list.append(al.Grid2DIrregular(raw[finite])) + + print(f"point_{i} (z={src_z}): {int(finite.sum())} multiple images solved") + +""" +__Point Datasets__ + +One `PointDataset` per source. The `name` (`point_0`, `point_1`) pairs each dataset with the matching +`Point` component in the lens model during modeling, and the `redshift` records each source's +redshift so the multi-plane geometry can be rebuilt when the dataset is loaded. + +The position uncertainty is 0.005" (5 mas), the centroid precision achievable by PSF fitting on +space-telescope imaging — not the imaging pixel scale. +""" +position_noise = 0.005 + +dataset_list = [] +for i, positions in enumerate(positions_list): + dataset = al.PointDataset( + name=f"point_{i}", + positions=positions, + positions_noise_map=position_noise, + redshift=source_redshifts[i], + ) + dataset_list.append(dataset) + +for i, dataset in enumerate(dataset_list): + al.output_to_json( + obj=dataset, + file_path=dataset_path / f"point_dataset_{i}.json", + ) + +""" +__Combined CSV__ + +For cluster workflows with many sources, a single CSV with one row per observed multiple image — +grouped by source `name`, with `y`, `x`, `positions_noise` and `redshift` columns — is far easier to +edit in a spreadsheet than many per-source files. `al.output_to_csv` writes every dataset into one +file, which the tutorial loads back with `al.list_from_csv`. +""" +al.output_to_csv( + datasets=dataset_list, + file_path=dataset_path / "point_datasets.csv", +) + +""" +__Scaling Galaxies CSV__ + +The scaling-tier members are written to `scaling_galaxies.csv` — one row per member carrying its +centre and luminosity (`y, x, luminosity`). This is the catalogue the tutorial's scaling relation +consumes: scaling a real cluster up to hundreds of members is a CSV-level edit, with the model's +free-parameter count unchanged. +""" +al.galaxy_table_to_csv( + centres=scaling_galaxies_centres, + luminosities=scaling_galaxies_luminosities, + file_path=dataset_path / "scaling_galaxies.csv", +) + +""" +__Model CSVs__ + +Write the truth model out as three family-level CSVs — `mass.csv`, `light.csv`, `point.csv` — keyed +by galaxy name. The tutorial loads these with `al.galaxy_models_from_csv` and composes them into +`af.Model` galaxies ready for the non-linear search. See `autolens_workspace/*/cluster/csv_api.py` +for the full schema walkthrough. +""" +mass_profiles = { + **{f"lens_{i}": {"mass": g.mass} for i, g in enumerate(main_lens_galaxies)}, + "host_halo": {"dark": host_halo_galaxy.dark}, +} + +light_profiles = { + **{f"lens_{i}": {"bulge": g.bulge} for i, g in enumerate(main_lens_galaxies)}, + **{f"source_{i}": {"bulge": g.bulge} for i, g in enumerate(source_galaxies)}, +} + +point_profiles = { + f"source_{i}": {f"point_{i}": getattr(g, f"point_{i}")} + for i, g in enumerate(source_galaxies) +} + +redshifts_by_galaxy = { + **{f"lens_{i}": redshift_lens for i in range(len(main_lens_galaxies))}, + "host_halo": redshift_lens, + **{f"source_{i}": z for i, z in enumerate(source_redshifts)}, +} + +al.galaxy_models_to_csv( + profiles_by_galaxy=mass_profiles, + file_path=dataset_path / "mass.csv", + family="mass", + redshifts=redshifts_by_galaxy, +) + +al.galaxy_models_to_csv( + profiles_by_galaxy=light_profiles, + file_path=dataset_path / "light.csv", + family="light", + redshifts=redshifts_by_galaxy, +) + +al.galaxy_models_to_csv( + profiles_by_galaxy=point_profiles, + file_path=dataset_path / "point.csv", + family="point", + redshifts=redshifts_by_galaxy, +) + +""" +__Tracer json__ + +Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass +profiles and galaxies are safely stored and available to check how the dataset was simulated in the +future. + +This can be loaded via the method `tracer = al.from_json()`. +""" +al.output_to_json( + obj=tracer, + file_path=Path(dataset_path, "tracer.json"), +) + +""" +__Imaging__ + +Strong lens clusters come with imaging data, used to *measure* the point positions and to visually +confirm the lens configuration. Although modeling is point-source only, we simulate CCD imaging so +the dataset looks like a realistic cluster observation. +""" +psf = al.Convolver.from_gaussian( + shape_native=(11, 11), sigma=0.2, pixel_scales=imaging_grid.pixel_scales +) + +simulator = al.SimulatorImaging( + exposure_time=300.0, + psf=psf, + background_sky_level=0.1, + add_poisson_noise_to_data=True, +) + +dataset = simulator.via_tracer_from(tracer=tracer, grid=imaging_grid) + +aplt.fits_imaging( + dataset=dataset, + data_path=dataset_path / "data.fits", + psf_path=dataset_path / "psf.fits", + noise_map_path=dataset_path / "noise_map.fits", + overwrite=True, +) + +""" +__Visualize__ + +Output .png plots of the per-source point datasets, the tracer, and the simulated imaging to the +dataset path. +""" +for pd in dataset_list: + aplt.subplot_point_dataset( + dataset=pd, output_path=dataset_path, output_format="png" + ) + +aplt.subplot_imaging_dataset(dataset=dataset) +aplt.plot_array(array=dataset.data, title="Data") + +aplt.subplot_tracer( + tracer=tracer, grid=viz_grid, output_path=dataset_path, output_format="png" +) +aplt.subplot_galaxies_images( + tracer=tracer, grid=viz_grid, output_path=dataset_path, output_format="png" +) + +""" +The dataset can be viewed in the folder `howtolens/dataset/cluster/simple`. +""" diff --git a/scripts/simulator/group.py b/scripts/simulator/group.py new file mode 100644 index 0000000..f65dc76 --- /dev/null +++ b/scripts/simulator/group.py @@ -0,0 +1,310 @@ +""" +Simulator: Group Scale Lens +=========================== + +This script simulates `Imaging` of a 'group-scale' strong lens, which is used in chapter 4 of the **HowToLens** +lectures to illustrate lens modeling at the group scale. + +The group consists of a single main lens galaxy -- the brightest group galaxy (BGG) -- and two smaller member +galaxies nearby, whose mass contributes significantly to the ray-tracing and is therefore included in the strong +lens model. A single source galaxy is lensed by the combined mass of all three galaxies. + +__Model__ + +This script simulates `Imaging` of a 'group-scale' strong lens where: + + - The main lens galaxy's light is a `SersicSph` and its total mass distribution is an `IsothermalSph`. + - The two member galaxies have `SersicSph` light profiles and tidally truncated `dPIEMassSph` mass profiles. + - The source galaxy's light is a `SersicCore`. + +The member galaxies are simulated to follow the light-mass scaling relation used in the group-scale modeling +tutorial: `sigma = sigma_ref * (L / L_ref) ** 0.25` and `r_cut = r_cut_ref * (L / L_ref) ** 0.7`, where `L` is +each member's luminosity. The second member is 2.07 times more luminous than the first, so its velocity +dispersion is 1.2 times higher (240 km/s vs 200 km/s) and its truncation radius 1.67 times larger. + +__Contents__ + +- **Dataset Paths:** The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a. +- **Grid:** Define the 2D grid of (y,x) coordinates the simulated image is evaluated on. +- **Galaxy Centres:** Define the centres of the main lens galaxy and member galaxies. +- **Over Sampling:** Set up the adaptive over-sampling scheme, centred on every galaxy in the group. +- **PSF / Simulator:** The Point Spread Function and simulator defining the observational properties of the data. +- **Main Lens Galaxy:** The brightest group galaxy (BGG) at the origin (0.0", 0.0"). +- **Member Galaxies:** The two smaller group members, with tidally truncated `dPIEMassSph` mass profiles. +- **Source Galaxy:** The source galaxy whose lensed image we simulate. +- **Ray Tracing:** Use all galaxies to set up a tracer, which generates the image that is simulated. +- **Output:** Output the simulated dataset to the dataset path as .fits files. +- **Visualize:** Output a subplot of the simulated dataset and the tracer's quantities to the dataset path. +- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file. +- **Centre JSON Files:** Save the centres of the main lens galaxy and member galaxies as .json files. +- **Positions:** Solve for the multiple-image positions of the lensed source and save them as a .json file. + +__Start Here Notebook__ + +If any code in this script is unclear, refer to the `autolens_workspace/*/group/simulator.ipynb` notebook. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +from pathlib import Path +import autolens as al +import autolens.plot as aplt + +""" +__Dataset Paths__ + +The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name. + +The dataset is output to `dataset/group/simple`. +""" +dataset_type = "group" +dataset_name = "simple" +dataset_path = Path("dataset", dataset_type, dataset_name) + +""" +__Grid__ + +Define the 2D grid of (y,x) coordinates the simulated image is evaluated on. + +A group-scale lens spans a wider area of sky than a galaxy-scale lens, because it contains multiple lens +galaxies which are separated by many arc-seconds. The grid is therefore larger (250 x 250 pixels, or 25.0") +than those used in previous chapters. +""" +grid = al.Grid2D.uniform( + shape_native=(250, 250), + pixel_scales=0.1, +) + +""" +__Galaxy Centres__ + +Define the centres of the main lens galaxy and member galaxies. These are used for over-sampling and are also +output to .json files so that the modeling tutorial can load them. +""" +main_lens_centres = [(0.0, 0.0)] +member_centres = [(3.5, 2.5), (-4.4, -5.0)] + +""" +__Over Sampling__ + +Over sampling evaluates the light profiles on a higher resolution grid in their bright central regions, to +ensure the calculation is accurate. + +The adaptive over-sampling scheme below is applied at the centre of every galaxy in the group, not just the +main lens galaxy. +""" +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=grid, + sub_size_list=[32, 8, 2], + radial_list=[0.3, 0.6], + centre_list=main_lens_centres + member_centres, +) + +grid = grid.apply_over_sampling(over_sample_size=over_sample_size) + +""" +__PSF / Simulator__ + +Simulate a simple Gaussian PSF for the image, and create the simulator for the imaging data, which defines the +exposure time, background sky, noise levels and psf. +""" +psf = al.Convolver.from_gaussian( + shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales +) + +simulator = al.SimulatorImaging( + exposure_time=300.0, + psf=psf, + background_sky_level=0.1, + add_poisson_noise_to_data=True, +) + +""" +__Main Lens Galaxy__ + +The main lens galaxy is the brightest group galaxy (BGG), located at the origin (0.0", 0.0"). It has a spherical +Sersic light profile and an isothermal mass profile with a large Einstein radius of 4.0", reflecting that it +dominates the mass of the group. +""" +lens_galaxy = al.Galaxy( + redshift=0.5, + bulge=al.lp.SersicSph( + centre=(0.0, 0.0), intensity=0.7, effective_radius=2.0, sersic_index=4.0 + ), + mass=al.mp.IsothermalSph(centre=(0.0, 0.0), einstein_radius=4.0), +) + +""" +__Member Galaxies__ + +The two member galaxies are smaller galaxies orbiting within the group. They have spherical Sersic light +profiles and tidally truncated `dPIEMassSph` mass profiles (vanishing core `r_core = 0.0` and a finite +truncation radius `r_cut`), reflecting that their outer dark matter has been stripped by the tides of the +group's potential. + +The members follow the light-mass scaling relation used in the modeling tutorial: the second member is 2.07 +times more luminous than the first (`intensity` of 1.866 vs 0.9, with identical `effective_radius` and +`sersic_index`), so its `sigma` is a factor 2.07 ** 0.25 = 1.2 higher (240 km/s vs 200 km/s) and its `r_cut` +a factor 2.07 ** 0.7 = 1.67 larger (13.3" vs 8.0"). + +The `dPIEMassSph` profile is parameterized in Lenstool's native convention -- `sigma` (fiducial velocity +dispersion, km/s), `r_core` and `r_cut` (arcsec) -- and converts these to a lensing strength internally using +the object/source redshifts and cosmology (`H0` / `Om0`, which default to Planck-like values). +""" +member_galaxy_0 = al.Galaxy( + redshift=0.5, + bulge=al.lp.SersicSph( + centre=(3.5, 2.5), intensity=0.9, effective_radius=0.8, sersic_index=3.0 + ), + mass=al.mp.dPIEMassSph( + centre=(3.5, 2.5), + sigma=200.0, + r_core=0.0, + r_cut=8.0, + redshift_object=0.5, + redshift_source=1.0, + ), +) + +member_galaxy_1 = al.Galaxy( + redshift=0.5, + bulge=al.lp.SersicSph( + centre=(-4.4, -5.0), intensity=1.866, effective_radius=0.8, sersic_index=3.0 + ), + mass=al.mp.dPIEMassSph( + centre=(-4.4, -5.0), + sigma=240.0, + r_core=0.0, + r_cut=13.3, + redshift_object=0.5, + redshift_source=1.0, + ), +) + +member_galaxies = [member_galaxy_0, member_galaxy_1] + +""" +__Source Galaxy__ + +The source galaxy whose lensed image we simulate. It uses a cored Sersic profile, whose light changes gradually +in its central regions and can therefore be evaluated accurately without adaptive over-sampling. +""" +source_galaxy = al.Galaxy( + redshift=1.0, + bulge=al.lp.SersicCore( + centre=(0.0, 0.1), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0), + intensity=3.0, + effective_radius=0.4, + sersic_index=1.0, + ), +) + +""" +__Ray Tracing__ + +Use all galaxies to set up a tracer, which will generate the image for the simulated `Imaging` dataset. + +The tracer combines the main lens galaxy, the member galaxies and the source galaxy. +""" +tracer = al.Tracer(galaxies=[lens_galaxy] + member_galaxies + [source_galaxy]) + +""" +Lets look at the tracer`s image, this is the image we'll be simulating. +""" +aplt.plot_array(array=tracer.image_2d_from(grid=grid), title="Image") + +""" +Pass the simulator a tracer, which creates the image which is simulated as an imaging dataset. +""" +dataset = simulator.via_tracer_from(tracer=tracer, grid=grid) + +""" +Plot the simulated `Imaging` dataset before outputting it to fits. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Output__ + +Output the simulated dataset to the dataset path as .fits files. +""" +aplt.fits_imaging( + dataset=dataset, + data_path=dataset_path / "data.fits", + psf_path=dataset_path / "psf.fits", + noise_map_path=dataset_path / "noise_map.fits", + overwrite=True, +) + +""" +__Visualize__ + +Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset path as .png files. +""" +aplt.subplot_imaging_dataset(dataset=dataset) +aplt.plot_array(array=dataset.data, title="Data") + +aplt.subplot_tracer( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) +aplt.subplot_galaxies_images( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) + +""" +__Tracer json__ + +Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and +galaxies are safely stored and available to check how the dataset was simulated in the future. + +This can be loaded via the method `tracer = al.from_json()`. +""" +al.output_to_json( + obj=tracer, + file_path=Path(dataset_path, "tracer.json"), +) + +""" +__Centre JSON Files__ + +Save the centres of the main lens galaxy and member galaxies as .json files. These are loaded by the group-scale +modeling tutorial to set up the lens model (e.g. fixing the centres of the member galaxies). +""" +al.output_to_json( + obj=al.Grid2DIrregular(main_lens_centres), + file_path=Path(dataset_path, "main_lens_centres.json"), +) + +al.output_to_json( + obj=al.Grid2DIrregular(member_centres), + file_path=Path(dataset_path, "extra_galaxies_centres.json"), +) + +""" +__Positions__ + +Solve for the multiple-image positions of the lensed source galaxy, which can be used as input to group-scale +modeling to help the non-linear search converge (and are the primary observable of cluster-scale modeling, +introduced in the next chapter 4 tutorial). +""" +solver = al.PointSolver.for_grid( + grid=al.Grid2D.uniform(shape_native=(500, 500), pixel_scales=0.1), + pixel_scale_precision=0.001, + magnification_threshold=0.01, +) + +positions = solver.solve( + tracer=tracer, source_plane_coordinate=source_galaxy.bulge.centre +) + +al.output_to_json( + obj=positions, + file_path=dataset_path / "positions.json", +) + +""" +The dataset can be viewed in the folder `dataset/group/simple`. +""" diff --git a/scripts/simulator/interferometer.py b/scripts/simulator/interferometer.py new file mode 100644 index 0000000..9c388e6 --- /dev/null +++ b/scripts/simulator/interferometer.py @@ -0,0 +1,193 @@ +""" +Simulator: Interferometer +========================= + +This script simulates `Interferometer` data of a 'galaxy-scale' strong lens, as would be observed by a radio or +sub-mm interferometer like ALMA or the JVLA. + +Unlike CCD imaging, an interferometer does not observe an image of the lens. It measures "visibilities", which are +the Fourier transform of the sky brightness sampled at a set of points in the "uv-plane", where each point +corresponds to the separation of a pair of antennas in the array. + +It is used to illustrate interferometer data in the HowToLens lecture series. HowToLens does not cover +interferometry beyond this glimpse; the `autolens_workspace/scripts/interferometer` package is the dedicated +resource for uv-plane lens modeling. + +__Contents__ + +- **Dataset Paths:** The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a. +- **Simulate:** Simulate the image using a (y,x) real-space grid and a synthetic set of uv-plane baselines. +- **Ray Tracing:** Setup the lens galaxy's mass and source galaxy light for this simulated lens. +- **Output:** Output the simulated dataset to the dataset path as .fits files. +- **Visualize:** Output a subplot of the simulated dataset's dirty images and the tracer's quantities. +- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass. + +__Model__ + +This script simulates `Interferometer` data of a 'galaxy-scale' strong lens where: + + - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. + - The source galaxy's light is a `SersicCore`. + +__Start Here Notebook__ + +If any code in this script is unclear, refer to the `autolens_workspace/*/interferometer/simulator.ipynb` notebook. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +from pathlib import Path +import numpy as np +import autolens as al +import autolens.plot as aplt + +""" +__Dataset Paths__ + +The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name. +""" +dataset_type = "interferometer" +dataset_name = "simple" +dataset_path = Path("dataset", dataset_type, dataset_name) + +""" +__Simulate__ + +For interferometer data, the strong lens's image is evaluated in real space on a (y,x) grid and then Fourier +transformed to the uv-plane, where it is compared to the observed visibilities. + +Interferometers do not observe galaxies in a way where over sampling is necessary, therefore the grid uses no +over sampling. +""" +grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.1, +) + +""" +To perform the Fourier transform we need the wavelengths of the baselines, which describe where in the uv-plane +each visibility samples the sky's Fourier transform. + +For real data these are determined by the layout of the interferometer's antennas. The `autolens_workspace` +bundles baselines of real instruments (e.g. the Square Mile Array (SMA) and ALMA). For this tutorial dataset we +instead draw a small synthetic set of baselines from a Gaussian distribution in the uv-plane, with a scale +comparable to the SMA's (a few hundred visibilities), keeping the simulation fast and self-contained. +""" +rng = np.random.default_rng(1) + +total_visibilities = 200 + +uv_wavelengths = rng.normal(loc=0.0, scale=1.0e5, size=(total_visibilities, 2)) + +""" +To simulate the interferometer dataset we first create a simulator, which defines the exposure time, noise levels +and Fourier transform method used in the simulation. + +We use the `TransformerDFT`, an exact Discrete Fourier Transform which is fast for datasets with a low number of +visibilities like this one. +""" +simulator = al.SimulatorInterferometer( + uv_wavelengths=uv_wavelengths, + exposure_time=300.0, + noise_sigma=1000.0, + transformer_class=al.TransformerDFT, +) + +""" +__Ray Tracing__ + +Setup the lens galaxy's mass and source galaxy light for this simulated lens. + +The lens galaxy has no light profile, because at the radio and sub-mm wavelengths an interferometer observes the +foreground lens galaxy typically emits negligibly. +""" +lens_galaxy = al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=1.6, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0), + ), + shear=al.mp.ExternalShear(gamma_1=0.05, gamma_2=0.05), +) + +source_galaxy = al.Galaxy( + redshift=1.0, + bulge=al.lp.SersicCore( + centre=(0.0, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0), + intensity=10.0, + effective_radius=1.0, + sersic_index=2.5, + ), +) + +""" +Use these galaxies to setup a tracer, which will generate the image for the simulated interferometer dataset. +""" +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) + +""" +Lets look at the tracer`s image, this is the image we'll be simulating. +""" +aplt.plot_array(array=tracer.image_2d_from(grid=grid), title="Image") + +""" +Pass the simulator a tracer, which creates the ray-traced image plotted above and simulates it as an +interferometer dataset. +""" +dataset = simulator.via_tracer_from(tracer=tracer, grid=grid) + +""" +Plot the simulated interferometer dataset's dirty images before outputting it to fits. +""" +aplt.subplot_interferometer_dirty_images(dataset=dataset) + +""" +__Output__ + +Output the simulated dataset to the dataset path as .fits files. +""" +aplt.fits_interferometer( + dataset=dataset, + data_path=dataset_path / "data.fits", + noise_map_path=dataset_path / "noise_map.fits", + uv_wavelengths_path=dataset_path / "uv_wavelengths.fits", + overwrite=True, +) + +""" +__Visualize__ + +Output a subplot of the simulated dataset's dirty images and the tracer's quantities to the dataset path as +.png files. +""" +aplt.subplot_interferometer_dirty_images( + dataset=dataset, output_path=dataset_path, output_format="png" +) + +aplt.subplot_tracer( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) +aplt.subplot_galaxies_images( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) + +""" +__Tracer json__ + +Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and galaxies +are safely stored and available to check how the dataset was simulated in the future. + +This can be loaded via the method `tracer = al.from_json()`. +""" +al.output_to_json( + obj=tracer, + file_path=Path(dataset_path, "tracer.json"), +) + +""" +The dataset can be viewed in the folder `dataset/interferometer/simple`. +""" diff --git a/scripts/simulator/lens_extra_galaxy.py b/scripts/simulator/lens_extra_galaxy.py new file mode 100644 index 0000000..2d7d20a --- /dev/null +++ b/scripts/simulator/lens_extra_galaxy.py @@ -0,0 +1,270 @@ +""" +Simulator: Lens With Extra Galaxy +================================= + +This script simulates `Imaging` of a 'galaxy-scale' strong lens which is identical to the `lens_sersic` dataset +simulated for chapter 2 (lens light + mass + lensed source), but with one extra galaxy located a few arc-seconds +from the lens galaxy. + +The extra galaxy has its own light profile, which blends into the outskirts of the image, and its own mass profile, +which contributes to the lensing of the source. Chapter 4's first tutorial uses this dataset to illustrate the two +approaches to dealing with such interloper galaxies: removing their light via noise scaling, or including them in +the lens model explicitly. + +It is used to illustrate scaling up lens modeling in the HowToLens lecture series. + +__Contents__ + +- **Model:** Compose the lens model fitted to the data. +- **Dataset Paths:** The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a. +- **Simulate:** Simulate the image using a (y,x) grid with the adaptive over sampling scheme. +- **Ray Tracing:** Setup the lens galaxy's light, mass and source galaxy light for this simulated lens. +- **Extra Galaxy:** Include one extra galaxy, whose light and mass must be masked or modeled in the tutorial. +- **Output:** Output the simulated dataset to the dataset path as .fits files. +- **Mask Extra Galaxy:** Build and save `mask_extra_galaxies.fits` so the tutorial can load it directly. +- **Visualize:** Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset. +- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass. +- **Extra Galaxy Centre:** Output the centre of the extra galaxy to a .json file for use in the lens model. + +__Model__ + +This script simulates `Imaging` of a 'galaxy-scale' strong lens where: + + - The lens galaxy's light profile is an `Sersic`. + - The lens galaxy's total mass distribution is an `Isothermal` and `ExternalShear`. + - The source galaxy's light is an `Sersic`. + - There is one extra galaxy whose light is near the lens and whose mass perturbs the lensed source's emission. + +__Start Here Notebook__ + +If any code in this script is unclear, refer to the `autolens_workspace/*/imaging/simulator.ipynb` notebook. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +from pathlib import Path + +import numpy as np + +import autolens as al +import autolens.plot as aplt + +""" +__Dataset Paths__ + +The `dataset_type` describes the type of data being simulated and `dataset_name` gives it a descriptive name. +""" +dataset_type = "imaging" +dataset_name = "lens_extra_galaxy" +dataset_path = Path("dataset", dataset_type, dataset_name) + +""" +__Simulate__ + +Simulate the image using a (y,x) grid with the adaptive over sampling scheme. + +This simulated lens has an extra galaxy offset from the main lens galaxy centre of (0.0", 0.0"). The adaptive over +sampling grid has both centres input to account for this. +""" +extra_galaxy_centre = (1.5, 2.5) + +grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.1, +) + +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=grid, + sub_size_list=[32, 8, 2], + radial_list=[0.3, 0.6], + centre_list=[(0.0, 0.0), extra_galaxy_centre], +) + +grid = grid.apply_over_sampling(over_sample_size=over_sample_size) + +""" +Simulate a simple Gaussian PSF for the image. +""" +psf = al.Convolver.from_gaussian( + shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales +) + +""" +Create the simulator for the imaging data, which defines the exposure time, background sky, noise levels and psf. +""" +simulator = al.SimulatorImaging( + exposure_time=300.0, + psf=psf, + background_sky_level=0.1, + add_poisson_noise_to_data=True, +) + +""" +__Ray Tracing__ + +Setup the lens galaxy's light, mass and source galaxy light for this simulated lens. + +These are identical to the `lens_sersic` dataset simulated for chapter 2, so that the only difference between the +two datasets is the extra galaxy included below. +""" +lens_galaxy = al.Galaxy( + redshift=0.5, + bulge=al.lp.Sersic( + centre=(0.0, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0), + intensity=1.0, + effective_radius=0.8, + sersic_index=4.0, + ), + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=1.6, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=45.0), + ), + shear=al.mp.ExternalShear(gamma_1=0.05, gamma_2=0.05), +) + +source_galaxy = al.Galaxy( + redshift=1.0, + bulge=al.lp.SersicCore( + centre=(0.0, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=60.0), + intensity=4.0, + effective_radius=0.1, + sersic_index=1.0, + ), +) + +""" +__Extra Galaxy__ + +Include one extra galaxy, which is offset a few arc-seconds from the lens galaxy. + +Its light is an `ExponentialSph` profile, which blends into the image near the lensed source's emission, and its +mass is an `IsothermalSph` profile, which perturbs the ray-tracing of the source's light. + +Note that its redshift is the same as the main lens galaxy, which is not necessarily the case in real observations. +If it were at a different redshift, the tools for masking or modeling the extra galaxy are equipped to handle this, +with mass modeling leading to multi-plane ray-tracing being performed. +""" +extra_galaxy = al.Galaxy( + redshift=0.5, + light=al.lp.ExponentialSph( + centre=extra_galaxy_centre, intensity=2.0, effective_radius=0.5 + ), + mass=al.mp.IsothermalSph(centre=extra_galaxy_centre, einstein_radius=0.15), +) + +""" +Use these galaxies to setup a tracer, which will generate the image for the simulated `Imaging` dataset. +""" +tracer = al.Tracer(galaxies=[lens_galaxy, extra_galaxy, source_galaxy]) + +""" +Lets look at the tracer`s image, this is the image we'll be simulating. +""" +aplt.plot_array(array=tracer.image_2d_from(grid=grid), title="Image") + +""" +Pass the simulator a tracer, which creates the image which is simulated as an imaging dataset. +""" +dataset = simulator.via_tracer_from(tracer=tracer, grid=grid) + +""" +Plot the simulated `Imaging` dataset before outputting it to fits. +""" +aplt.subplot_imaging_dataset(dataset=dataset) + +""" +__Output__ + +Output the simulated dataset to the dataset path as .fits files. +""" +aplt.fits_imaging( + dataset=dataset, + data_path=dataset_path / "data.fits", + psf_path=dataset_path / "psf.fits", + noise_map_path=dataset_path / "noise_map.fits", + overwrite=True, +) + +""" +__Mask Extra Galaxy__ + +Build and output a `mask_extra_galaxies.fits` covering the extra galaxy's region, so that the chapter 4 tutorial +which uses this dataset can load the mask directly without a separate data-preparation step. + +The circle is sized to ~3x the galaxy's `effective_radius`, which comfortably covers the light extent of the +`ExponentialSph` profile used above. The geometry is derived from the same centre + radius defined for the extra +galaxy in this script, so it stays in sync with any future tweak to those values. + +`Mask2D.circular` honours the `PYAUTO_SMALL_DATASETS=1` env var (caps to 16x16 at 0.6"/px), so the mask +automatically shrinks alongside the small-dataset image and never raises an out-of-bounds error. +""" +extra_galaxies_mask = np.zeros(dataset.shape_native, dtype=bool) + +circle = al.Mask2D.circular( + shape_native=dataset.shape_native, + pixel_scales=dataset.pixel_scales, + centre=extra_galaxy_centre, + radius=3.0 * 0.5, + invert=True, # True inside the circle (i.e. masked region) +) +extra_galaxies_mask = np.logical_or(extra_galaxies_mask, circle.native) + +mask_extra_galaxies = al.Mask2D( + mask=extra_galaxies_mask, + pixel_scales=dataset.pixel_scales, +) + +aplt.fits_array( + array=mask_extra_galaxies, + file_path=dataset_path / "mask_extra_galaxies.fits", + overwrite=True, +) + +""" +__Visualize__ + +Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset path as .png files. +""" + +aplt.subplot_imaging_dataset(dataset=dataset) +aplt.plot_array(array=dataset.data, title="Data") + +aplt.subplot_tracer( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) +aplt.subplot_galaxies_images( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) + +""" +__Tracer json__ + +Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and galaxies +are safely stored and available to check how the dataset was simulated in the future. + +This can be loaded via the method `tracer = al.from_json()`. +""" +al.output_to_json( + obj=tracer, + file_path=Path(dataset_path, "tracer.json"), +) + +""" +__Extra Galaxy Centre__ + +Output the centre of the extra galaxy to a .json file, so that it can be used to set up the lens model in the +chapter 4 tutorial. +""" +al.output_to_json( + obj=al.Grid2DIrregular(values=[extra_galaxy_centre]), + file_path=dataset_path / "extra_galaxies_centres.json", +) + +""" +The dataset can be viewed in the folder `autolens_workspace/dataset/imaging/lens_extra_galaxy`. +""" diff --git a/scripts/simulator/lens_x2.py b/scripts/simulator/lens_x2.py index 661b1a0..00d83ad 100644 --- a/scripts/simulator/lens_x2.py +++ b/scripts/simulator/lens_x2.py @@ -31,9 +31,9 @@ - The lens galaxy's light is two `Sersic`'s. - The lens galaxy's mass distribution is two `Isothermal`'s. - - The source galaxy's light is an `Sersic`. + - The source galaxy's light is an `ExponentialCoreSph`. -This dataset is used in chapter 3 of the **HowToLens** lectures. +This dataset is used in chapter 4 of the **HowToLens** lectures. __Start Here Notebook__ diff --git a/scripts/simulator/lens_x3.py b/scripts/simulator/lens_x3.py new file mode 100644 index 0000000..a778fef --- /dev/null +++ b/scripts/simulator/lens_x3.py @@ -0,0 +1,219 @@ +""" +Simulator: Lens x3 +================== + +This script simulates `Imaging` of a 'galaxy-scale' lens where there are three lens galaxies, each with their own +light and mass profiles, which all contribute significantly to the lensing of a single background source. + +Strong lenses with this complex mass distribution are more challenging to model than those with one or two lens +galaxies, because every additional deflector adds its own free light and mass parameters to the model. + +A system with three co-dominant lens galaxies and no shared dark-matter halo is exactly what the +`autolens_workspace`'s `multi_galaxy` package is dedicated to — its `start_here` and `modeling` examples are +directly applicable to this lens. (Systems where a dominant group-scale halo joins the galaxies belong one rung +up the regime ladder, in the `group` package.) + +This dataset is modeled in HowToLens chapter 4 and is used to illustrate how the parameter count of a lens model +grows with every co-dominant deflector, motivating the scaling relations introduced later in that chapter. + +__Contents__ + +- **Model:** Compose the lens model fitted to the data. +- **Dataset Paths:** The `dataset_type` describes the type of data being simulated (in this case, `Imaging` data) and. +- **Simulate:** Simulate the image using a (y,x) grid with the adaptive over sampling scheme. +- **Ray Tracing:** Setup the lens galaxies' light, mass and source galaxy light for this simulated lens. +- **Output:** Output the simulated dataset to the dataset path as .fits files. +- **Visualize:** Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset. +- **Tracer json:** Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass. + +__Model__ + +This script simulates `Imaging` of a 'galaxy-scale' strong lens where: + + - The lens galaxies' light are three `Sersic`'s. + - The lens galaxies' mass distributions are three `Isothermal`'s. + - The source galaxy's light is an `ExponentialCoreSph`. + +This dataset is used in chapter 4 of the **HowToLens** lectures. + +__Start Here Notebook__ + +If any code in this script is unclear, refer to the `autolens_workspace/*/imaging/simulator.ipynb` notebook. +""" + +from pathlib import Path +import autolens as al +import autolens.plot as aplt + +""" +__Dataset Paths__ + +The `dataset_type` describes the type of data being simulated (in this case, `Imaging` data) and `dataset_name` +gives it a descriptive name. +""" +dataset_type = "imaging" +dataset_name = "x3_lens_galaxies" +dataset_path = Path("dataset", dataset_type, dataset_name) + +""" +__Simulate__ + +Simulate the image using a (y,x) grid with the adaptive over sampling scheme. + +This simulated lens has three galaxies whose centres are offset from (0.0", 0.0"), forming a triangle around the +origin. The adaptive over sampling grid has all three centres input to account for this. +""" +grid = al.Grid2D.uniform( + shape_native=(100, 100), + pixel_scales=0.1, +) + +over_sample_size = al.util.over_sample.over_sample_size_via_radial_bins_from( + grid=grid, + sub_size_list=[32, 8, 2], + radial_list=[0.3, 0.6], + centre_list=[(0.9, 0.0), (-0.6, -0.9), (-0.6, 0.9)], +) + +grid = grid.apply_over_sampling(over_sample_size=over_sample_size) + +""" +Simulate a simple Gaussian PSF for the image. +""" +psf = al.Convolver.from_gaussian( + shape_native=(11, 11), sigma=0.1, pixel_scales=grid.pixel_scales +) + +""" +Create the simulator for the imaging data, which defines the exposure time, background sky, noise levels and psf. +""" +simulator = al.SimulatorImaging( + exposure_time=300.0, + psf=psf, + background_sky_level=0.1, + add_poisson_noise_to_data=True, +) + +""" +__Ray Tracing__ + +Setup the lens galaxies' light, mass and source galaxy light for this simulated lens. + +The image plane is made of three separate lens galaxies, whose Einstein radii (0.9", 0.8" and 0.7") are +deliberately comparable — no single galaxy dominates the deflection of the source's light, which is what makes +every one of them a co-dominant deflector that must be modeled individually. + +All three galaxies are at the same redshift, so ray tracing is single-plane and their deflection fields simply add. +""" +lens_galaxy_0 = al.Galaxy( + redshift=0.5, + bulge=al.lp.Sersic( + centre=(0.9, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=30.0), + intensity=0.1, + effective_radius=0.7, + sersic_index=3.0, + ), + mass=al.mp.Isothermal( + centre=(0.9, 0.0), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.85, angle=30.0), + einstein_radius=0.9, + ), +) + +lens_galaxy_1 = al.Galaxy( + redshift=0.5, + bulge=al.lp.Sersic( + centre=(-0.6, -0.9), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.7, angle=120.0), + intensity=0.1, + effective_radius=0.6, + sersic_index=2.5, + ), + mass=al.mp.Isothermal( + centre=(-0.6, -0.9), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=120.0), + einstein_radius=0.8, + ), +) + +lens_galaxy_2 = al.Galaxy( + redshift=0.5, + bulge=al.lp.Sersic( + centre=(-0.6, 0.9), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=60.0), + intensity=0.1, + effective_radius=0.5, + sersic_index=3.5, + ), + mass=al.mp.Isothermal( + centre=(-0.6, 0.9), + ell_comps=al.convert.ell_comps_from(axis_ratio=0.9, angle=60.0), + einstein_radius=0.7, + ), +) + +source_galaxy = al.Galaxy( + redshift=1.0, + bulge=al.lp.ExponentialCoreSph( + centre=(0.05, 0.15), intensity=0.2, effective_radius=0.5, radius_break=0.025 + ), +) + +""" +Use these galaxies to setup a tracer, which will generate the image for the simulated `Imaging` dataset. +""" +tracer = al.Tracer( + galaxies=[lens_galaxy_0, lens_galaxy_1, lens_galaxy_2, source_galaxy] +) + +""" +Pass the simulator a tracer, which creates the image which is simulated as an imaging dataset. +""" +dataset = simulator.via_tracer_from(tracer=tracer, grid=grid) + +""" +__Output__ + +Output the simulated dataset to the dataset path as .fits files. +""" +aplt.fits_imaging( + dataset=dataset, + data_path=dataset_path / "data.fits", + psf_path=dataset_path / "psf.fits", + noise_map_path=dataset_path / "noise_map.fits", + overwrite=True, +) + +""" +__Visualize__ + +Output a subplot of the simulated dataset, the image and the tracer's quantities to the dataset path as .png files. +""" + +aplt.subplot_imaging_dataset(dataset=dataset) +aplt.plot_array(array=dataset.data, title="Data") + +aplt.subplot_tracer( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) +aplt.subplot_galaxies_images( + tracer=tracer, grid=grid, output_path=dataset_path, output_format="png" +) + +""" +__Tracer json__ + +Save the `Tracer` in the dataset folder as a .json file, ensuring the true light profiles, mass profiles and galaxies +are safely stored and available to check how the dataset was simulated in the future. + +This can be loaded via the method `tracer = al.from_json()`. +""" +al.output_to_json( + obj=tracer, + file_path=Path(dataset_path, "tracer.json"), +) + +""" +The dataset can be viewed in the folder `autolens_workspace/dataset/imaging/x3_lens_galaxies`. +""" diff --git a/scripts/simulator/weak_lensing.py b/scripts/simulator/weak_lensing.py new file mode 100644 index 0000000..c5739bf --- /dev/null +++ b/scripts/simulator/weak_lensing.py @@ -0,0 +1,170 @@ +""" +Simulator: Weak Lensing +======================= + +This script simulates a weak gravitational lensing shear catalogue. Unlike the imaging simulators (which produce +a 2D image of the lensed source) the weak-lensing simulator produces a *catalogue* of (gamma_2, gamma_1) shear +measurements at the (y, x) positions of a population of background source galaxies. + +The lens is a **cluster-scale** mass (Einstein radius 25", the regime where weak lensing is actually used) and the +background galaxies are placed in an **annulus outside the strong-lensing core** — this is where real weak-lensing +measurements are made, so the simulated shears are genuinely weak (|gamma| ~ 0.06-0.25) rather than the order-unity +shears found among galaxies projected near the Einstein radius. + +The shear computation itself comes from `Tracer.shear_yx_2d_via_hessian_from`, which differentiates the +deflection-angle field. On top of that the simulator adds Gaussian shape noise per galaxy (the dominant noise +source in real weak-lensing data — each galaxy has a random unlensed ellipticity around 0.2-0.4 per component). + +It is used to illustrate weak lensing data in the HowToLens lecture series. + +__Contents__ + +- **Dataset Paths:** The `dataset_type` and `dataset_name` define the on-disk output folder. +- **Ray Tracing:** Build a Tracer from a cluster-scale Isothermal lens galaxy. +- **Source Positions:** Draw background source galaxies in an annulus outside the strong-lensing core. +- **Simulator:** Construct a `SimulatorShearYX` with the desired shape-noise level and random seed. +- **Output:** Save the simulated `WeakDataset` and the `Tracer` to JSON. +- **Visualize:** Plot the shear field and the dataset subplot mosaic via `aplt`. + +__Start Here Notebook__ + +If any code in this script is unclear, refer to the `autolens_workspace/*/weak/simulator.ipynb` notebook. +""" + +from autolens import jax_wrapper # Sets JAX environment before other imports + +# from autolens import setup_notebook; setup_notebook() + +from pathlib import Path + +import numpy as np + +import autolens as al +import autolens.plot as aplt + +""" +__Dataset Paths__ + +The `dataset_type` describes the type of data being simulated (in this case, a weak-lensing shear catalogue) and +`dataset_name` gives it a descriptive name. They define the folder the dataset is output to on your hard-disk: + + - The shear catalogue will be output to `/dataset/weak_lensing/simple/dataset.json`. + - The tracer used to simulate the dataset will be output alongside as `tracer.json`. +""" +dataset_type = "weak_lensing" +dataset_name = "simple" + +dataset_path = Path("dataset") / dataset_type / dataset_name + +""" +__Ray Tracing__ + +We define the lens galaxy's mass distribution as a **cluster-scale** `Isothermal` profile with an Einstein radius +of 25" (no external shear, no source light — weak-lensing measurements are sensitive to the shear field induced +by the lens mass alone). A 25" Einstein radius corresponds to a very massive cluster (velocity dispersion of order +1200-1400 km/s) — the mass scale on which weak lensing is the tool of choice, because the shear signal extends to +the many-arc-minute radii where strong lensing has no features. + +Because the source-galaxy positions are an irregular catalogue rather than a 2D pixel grid, this simulator +does not need PSF convolution, over-sampling, or background-sky modelling — those are all imaging-specific +concerns. +""" +lens_galaxy = al.Galaxy( + redshift=0.5, + mass=al.mp.Isothermal( + centre=(0.0, 0.0), + einstein_radius=25.0, + ell_comps=al.convert.ell_comps_from(axis_ratio=0.8, angle=45.0), + ), +) + +source_galaxy = al.Galaxy(redshift=1.0) + +tracer = al.Tracer(galaxies=[lens_galaxy, source_galaxy]) + +""" +__Source Positions__ + +Real weak-lensing measurements avoid the strong-lensing core, where the linear-shear approximation breaks down +and cluster-member galaxies contaminate the sample. We therefore draw the background galaxies in an **annulus** +between an inner radius of 50" (~2 Einstein radii, safely into the weak regime) and an outer radius of 200" +(~3.3 arc-minutes), uniformly in area. + +This is the physically genuine weak-lensing geometry: at 50" the shear is |gamma| ~ 0.25 and by 200" it has +fallen to |gamma| ~ 0.06, so every galaxy is a *weak* probe whose individual shear is well below the 0.25 shape +noise. The signal lives in the ensemble of 1500 galaxies (a deep-survey source density of ~45 / arc-minute^2), +exactly as in a real cluster weak-lensing analysis. + +We build the positions as an `al.Grid2DIrregular` of (y, x) coordinates and pass them to the simulator's +`via_tracer_from` method. +""" +rng = np.random.default_rng(1) + +n_galaxies = 1500 +radius_inner = 50.0 # arc-seconds — inside this we are in the strong-lensing core. +radius_outer = 200.0 # arc-seconds — the edge of the simulated weak-lensing field. + +radii = np.sqrt(rng.uniform(radius_inner**2.0, radius_outer**2.0, n_galaxies)) +phi = rng.uniform(0.0, 2.0 * np.pi, n_galaxies) + +positions = al.Grid2DIrregular( + values=np.stack([radii * np.sin(phi), radii * np.cos(phi)], axis=1) +) + +""" +__Simulator__ + +`SimulatorShearYX` takes a shape-noise level and an optional random seed. A `noise_sigma` of 0.25 is a typical +per-component shape-noise value for a weak-lensing survey; reduce it to 0.0 to inspect the noise-free shear field. + +`via_tracer_from` evaluates the tracer's shear at the supplied (y, x) positions and adds the shape noise. +""" +simulator = al.SimulatorShearYX(noise_sigma=0.25, seed=1) + +dataset = simulator.via_tracer_from( + tracer=tracer, + grid=positions, + name=dataset_name, +) + +""" +__Output__ + +Save the simulated `WeakDataset` and the `Tracer` to the dataset folder as JSON, ensuring the inputs to the +simulation are reproducible and inspectable later. +""" +dataset_path.mkdir(parents=True, exist_ok=True) + +al.output_to_json(obj=dataset, file_path=dataset_path / "dataset.json") +al.output_to_json(obj=tracer, file_path=dataset_path / "tracer.json") + +""" +__Visualize__ + +The shear field is visualised with `matplotlib.quiver` rendered as *headless line segments* +(`headwidth=0, headlength=0, headaxislength=0`) — the standard weak-lensing convention, because shear is a +spin-2 quantity and a 180-degree rotation maps it back to itself, so an arrowhead would suggest a +directionality the data does not have. + +`aplt.subplot_weak_dataset` produces a 2x2 mosaic combining the shear field, the per-galaxy noise map, the +shear magnitude `|gamma|`, and the position angle `phi`. `aplt.plot_shear_yx_2d` writes a single-panel +quiver of the shear field alone — useful for high-resolution figures where the mosaic is too dense. +""" +aplt.subplot_weak_dataset( + dataset=dataset, + output_path=dataset_path, + output_format="png", +) + +aplt.plot_shear_yx_2d( + shear_yx=dataset.shear_yx, + output_path=dataset_path, + output_format="png", +) + +print(dataset.info) +print(f"Wrote dataset to {dataset_path}") + +""" +The dataset can be viewed in the folder `dataset/weak_lensing/simple`. +""" diff --git a/workspace_index.json b/workspace_index.json index 91e127e..7fd32f2 100644 --- a/workspace_index.json +++ b/workspace_index.json @@ -3,6 +3,7 @@ "contents": [ "Directories", "Dataset", + "Dataset Auto-Simulation", "Subplots", "Plot Customization", "Overlays", @@ -21,7 +22,11 @@ "Light Profiles", "One Dimension Projection", "Galaxies", - "Units" + "Units", + "Wrap Up", + "Advanced Topics", + "Other Unit Conversion", + "Over Sampling" ], "cross_refs": [ "/guides/advanced/over_sampling.ipynb" @@ -39,7 +44,8 @@ "Ray Tracing Images", "Galaxies", "Tracer", - "Mappings" + "Mappings", + "Wrap Up" ], "cross_refs": [], "notebook": "notebooks/chapter_1_introduction/tutorial_2_ray_tracing.ipynb", @@ -62,26 +68,45 @@ "notebook": "notebooks/chapter_1_introduction/tutorial_3_more_ray_tracing.ipynb", "path": "scripts/chapter_1_introduction/tutorial_3_more_ray_tracing.py", "summary": "We'll now reinforce the ideas that we learnt about ray-tracing in the previous tutorial and introduce the following new concepts:", - "title": "Tutorial 5: More Ray Tracing" + "title": "Tutorial 3: More Ray Tracing" }, { "contents": [ + "Initial Setup", + "Point Sources", + "Point Source Tracer", + "The Lens Equation", + "Point Solver", + "Multiple Images and Critical Curves", + "Magnifications", + "Time Delays", + "Extended Versus Point Computations", "Wrap Up" ], "cross_refs": [], "notebook": "notebooks/chapter_1_introduction/tutorial_4_point_sources.ipynb", "path": "scripts/chapter_1_introduction/tutorial_4_point_sources.py", - "summary": "This tutorial is not wrriten yet, but will explain how point source lensing works.", + "summary": "In the previous tutorials, the background sources we lensed were galaxies: extended objects whose light spreads over many thousands of light years. When lensed, their light is warped into the arcs and Einstein rings we produced with the `Tracer`, spread across many pixels of the image.", "title": "Tutorial 4: Point Sources" }, { "contents": [ + "Initial Setup", + "Cosmological Distances", + "The Lens Equation", + "Convergence", + "Deflection Angles", + "The Lensing Potential", + "Shear and Magnification", + "Critical Curves and Caustics", + "Einstein Radius", + "Time Delays", "Wrap Up" ], "cross_refs": [], "notebook": "notebooks/chapter_1_introduction/tutorial_5_lensing_formalism.ipynb", "path": "scripts/chapter_1_introduction/tutorial_5_lensing_formalism.py", - "summary": "This tutorial is not wrriten yet, but will explain what all the different lens quantities are and give a more formal description of them.", + "summary": "This tutorial is the equations lecture of **HowToLens**.", "title": "Tutorial 5: Lensing Formalism" }, { @@ -92,9 +117,14 @@ "Background Sky", "Simulator", "Output", + "Interferometer Data", + "Weak Lensing Data", "Wrap Up" ], - "cross_refs": [], + "cross_refs": [ + "scripts/simulator/interferometer.py", + "scripts/simulator/weak_lensing.py" + ], "notebook": "notebooks/chapter_1_introduction/tutorial_6_data.ipynb", "path": "scripts/chapter_1_introduction/tutorial_6_data.py", "summary": "In the last tutorials, we use tracers to create images of strong lenses. However, those images don't accurately represent what we would observe through a telescope.", @@ -102,7 +132,9 @@ }, { "contents": [ - "Dataset & Mask", + "Dataset", + "Dataset Auto-Simulation", + "Mask", "Masked Grid", "Fitting", "Incorrect Fit", @@ -114,7 +146,7 @@ ], "notebook": "notebooks/chapter_1_introduction/tutorial_7_fitting.ipynb", "path": "scripts/chapter_1_introduction/tutorial_7_fitting.py", - "summary": "In previous tutorials, we used light profiles to create simulated images of tracer and visualized how these images would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope.", + "summary": "In previous tutorials, we used light profiles to create simulated images of a tracer and visualized how these images would appear when captured by a CCD detector on a telescope like the Hubble Space Telescope.", "title": "Tutorial 7: Fitting" }, { @@ -130,22 +162,66 @@ "notebook": "notebooks/chapter_1_introduction/tutorial_8_summary.ipynb", "path": "scripts/chapter_1_introduction/tutorial_8_summary.py", "summary": "In this chapter, we have learnt that:", - "title": "Tutorial 9: Summary" + "title": "Tutorial 8: Summary" + }, + { + "contents": [ + "Initial Setup", + "Dataset Auto-Simulation", + "Model", + "Search", + "Result (Search 1)", + "Prior Passing", + "Result", + "Wrap Up", + "Detailed Explanation Of Prior Passing", + "EXAMPLE" + ], + "cross_refs": [ + "autolens_workspace/scripts/guides/modeling/chaining.py" + ], + "notebook": "notebooks/chapter_2_lens_modeling/tutorial_10_prior_passing.ipynb", + "path": "scripts/chapter_2_lens_modeling/tutorial_10_prior_passing.py", + "summary": "In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two non-linear searches. This used an initial search to fit a simple lens model, whose results were used to tune and initialize the priors of a more complex lens model that was fitted by the second search.", + "title": "Tutorial 10: Prior Passing" + }, + { + "contents": [ + "Search Chaining In The Workspace", + "SLaM (Source, Light and Mass)", + "Wrap Up" + ], + "cross_refs": [ + "autolens_workspace/scripts/guides/modeling/chaining.py", + "autolens_workspace/scripts/guides/modeling/slam_start_here.py", + "autolens_workspace/scripts/imaging/features/multi_gaussian_expansion/slam.py", + "autolens_workspace/scripts/imaging/features/pixelization/slam.py", + "slam.py" + ], + "notebook": "notebooks/chapter_2_lens_modeling/tutorial_11_slam.ipynb", + "path": "scripts/chapter_2_lens_modeling/tutorial_11_slam.py", + "summary": "In the previous two tutorials, we learnt how search chaining breaks a lens model-fit into a sequence of simpler non-linear searches, and how prior passing carries the results of each search into the next. Together, they give us the flexibility to juggle the dimensionality, priors and settings of each search \u2014 the three drivers of run-time we met in tutorial 8 \u2014 whilst still fitting a complex and realistic lens model at the end.", + "title": "Tutorial 11: SLaM" }, { "contents": [ "Overview", "Parameter Space", + "Non-Linear Search", "Search Types", "Deeper Background", "PyAutoFit", "Initial Setup", + "Dataset Auto-Simulation", "Mask", "Model", "Priors", "Analysis", "Searches", + "Maximum Likelihood Estimation (MLE)", + "Markov Chain Monte Carlo (MCMC)", "Nested Sampling", + "What is The Best Search To Use?", "Wrap Up" ], "cross_refs": [ @@ -153,13 +229,14 @@ ], "notebook": "notebooks/chapter_2_lens_modeling/tutorial_1_non_linear_search.ipynb", "path": "scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py", - "summary": "The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a strong lens using a telescope like the Hubble Space Telescope, and seek to learn about the lens galaxy, source galaxy and the Universe from these observations. With **PyAutoLen**, we seek to learn about the lenses may and ray-tracing, asking questions like how big is the lens galaxy and what does the unlensed source galaxy look like?", + "summary": "The starting point for most scientific analysis conducted by an Astronomer is that they have observations of a strong lens using a telescope like the Hubble Space Telescope, and seek to learn about the lens galaxy, source galaxy and the Universe from these observations. With **PyAutoLens**, we seek to learn about the lens's mass and ray-tracing, asking questions like how big is the lens galaxy and what does the unlensed source galaxy look like?", "title": "Tutorial 1: Non-linear Search" }, { "contents": [ "PyAutoFit", "Initial Setup", + "Dataset Auto-Simulation", "Mask", "Model", "Search", @@ -168,6 +245,7 @@ "Analysis", "VRAM Use", "Run Times", + "Model-Fit", "Result Info", "Output Folder", "Unique Identifier", @@ -187,8 +265,10 @@ { "contents": [ "Initial Setup", + "Dataset Auto-Simulation", "Mask", "Model", + "Search + Analysis", "Run Time", "Result", "Global and Local Maxima", @@ -203,11 +283,15 @@ { "contents": [ "Initial Setup", + "Dataset Auto-Simulation", "Mask", "Prior Tuning", "Run Time", "Result", - "Discussion" + "Discussion", + "Approach 2: Reducing Complexity", + "Approach 3: Look Harder", + "Wrap Up" ], "cross_refs": [ "chapter_optional/tutorial_searches.ipynb" @@ -220,6 +304,7 @@ { "contents": [ "Initial Setup", + "Dataset Auto-Simulation", "Mask", "Linear Light Profiles", "Run Time", @@ -247,7 +332,9 @@ { "contents": [ "Initial Setup", + "Dataset Auto-Simulation", "Mask", + "Model + Analysis", "Run Time", "Search", "Discussion", @@ -263,6 +350,7 @@ { "contents": [ "Initial Setup", + "Dataset Auto-Simulation", "Tracer & Fit", "Samples", "Workspace", @@ -277,13 +365,15 @@ }, { "contents": [ - "Algorithmic Optimization", + "Searching Non-linear Parameter Space", + "Cost Per Evaluation", + "JAX", + "Run Time Estimation", "Data Quantity", "Wrap Up" ], "cross_refs": [ - "//numba.py", - "chapter_optional/tutorial_searches.ipynb" + "chapter_optional/tutorial_searches.py" ], "notebook": "notebooks/chapter_2_lens_modeling/tutorial_8_need_for_speed.ipynb", "path": "scripts/chapter_2_lens_modeling/tutorial_8_need_for_speed.py", @@ -293,7 +383,9 @@ { "contents": [ "Initial Setup", + "Dataset Auto-Simulation", "Model", + "Search + Analysis", "Result", "Prior Passing", "Run Time", @@ -301,77 +393,26 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_3_search_chaining/tutorial_1_search_chaining.ipynb", - "path": "scripts/chapter_3_search_chaining/tutorial_1_search_chaining.py", - "summary": "In chapter 2, we learnt how to perform lens modeling using a non-linear search. In all of the tutorials, we fitted the data using just one non-linear search. In this chapter, we introduce a technique called 'non-linear search chaining', fits a lens model using a sequence of non-linear searches. The initial searches fit simpler lens models whose parameter spaces can be more accurately and efficiently sampled. The results of this search are then passed to later searches which fit lens models of gradually increasing complexity.", - "title": "Tutorial 1: Search Chaining" - }, - { - "contents": [ - "Initial Setup", - "Model", - "Search", - "Prior Passing", - "Result", - "Wrap Up", - "Detailed Explanation Of Prior Passing", - "EXAMPLE" - ], - "cross_refs": [], - "notebook": "notebooks/chapter_3_search_chaining/tutorial_2_prior_passing.ipynb", - "path": "scripts/chapter_3_search_chaining/tutorial_2_prior_passing.py", - "summary": "In the previous tutorial, we used non-linear search chaining to break the model-fitting procedure down into two non-linear searches. This used an initial search to fit a simple lens model, whose results were used to tune and initialize the priors of a more complex lens model that was fitted by the second search.", - "title": "Tutorial 2: Prior Passing" - }, - { - "contents": [ - "Dated Tutorial", - "Initial Setup", - "Paths", - "Notes", - "Wrap Up" - ], - "cross_refs": [], - "notebook": "notebooks/chapter_3_search_chaining/tutorial_3_lens_and_source.ipynb", - "path": "scripts/chapter_3_search_chaining/tutorial_3_lens_and_source.py", - "summary": "In this tutorial, we demonstrate search chaining using three searches to fit strong lens `Imaging` which includes the lens galaxy's light.", - "title": "Tutorial 3: Lens and Source" + "notebook": "notebooks/chapter_2_lens_modeling/tutorial_9_search_chaining.ipynb", + "path": "scripts/chapter_2_lens_modeling/tutorial_9_search_chaining.py", + "summary": "Throughout this chapter, we have fitted the data using just one non-linear search. The previous tutorial discussed the run-time cost of doing this: with a single search, the dimensionality of the model, the priors on its parameters and the search settings must all be juggled at once, leaving us little flexibility to trade them off against one another.", + "title": "Tutorial 9: Search Chaining" }, { "contents": [ "Initial Setup", "Mask", - "Paths", - "Search Chaining Approach", - "Wrap Up" - ], - "cross_refs": [], - "notebook": "notebooks/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.ipynb", - "path": "scripts/chapter_3_search_chaining/tutorial_4_x2_lens_galaxies.py", - "summary": "Up to now, all the images we've fitted had one lens galaxy. However, we saw in chapter 1 that our lens plane can consist of multiple galaxies which each contribute to the strong lensing. Multi-galaxy systems are challenging to model, because they add an extra 5-10 parameters to the non-linear search and, more problematically, the degeneracies between the parameters of the mass profiles of the two galaxies can be severe.", - "title": "Tutorial 4: Two Lens galaxies" - }, - { - "contents": [ - "Initial Setup", - "Paths", - "Search Chaining Approach", - "Run Times", + "Simulator", + "Fitting", + "Fit Problems", + "Discussion", "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_3_search_chaining/tutorial_5_complex_source.ipynb", - "path": "scripts/chapter_3_search_chaining/tutorial_5_complex_source.py", - "summary": "Up to now, we've not paid much attention to the source galaxy's morphology. We've assumed its a single-component exponential profile, which is a fairly crude assumption. A quick look at any image of a real galaxy reveals a wealth of different structures that could be present: bulges, disks, bars, star-forming knots and so on. Furthermore, there could be more than one source-galaxy!", - "title": "Tutorial 5: Complex Source" - }, - { - "contents": [], - "cross_refs": [], - "notebook": "notebooks/chapter_3_search_chaining/tutorial_6_slam.ipynb", - "path": "scripts/chapter_3_search_chaining/tutorial_6_slam.py", - "summary": "You are now familiar with pipelines, in particular how we use them to break-down the lens modeling procedure to provide more efficient and reliable model-fits. In the previous tutorials, you learnt how to write your own pipelines, which can fit whatever lens model is of particular interest to your scientific study.", - "title": "Tutorial 6: SLaM" + "notebook": "notebooks/chapter_3_pixelizations/tutorial_10_fit_problems.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_10_fit_problems.py", + "summary": "To begin, make sure you have read tutorials 4 and 5 carefully, as a clear understanding of how the Bayesian evidence works is key to understanding the adaption tutorials that close this chapter!", + "title": "Tutorial 10: Fit Problems" }, { "contents": [ @@ -383,10 +424,10 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_10_brightness_adaption.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_10_brightness_adaption.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_11_brightness_adaption.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_11_brightness_adaption.py", "summary": "In the previous tutorial we motivated our need to adapt the pixelization to the source's morphology, such that source pixels congregates in the source's brightest regions regardless of where the source is located in the source-plane.", - "title": "Tutorial 10: Brightness Adaption" + "title": "Tutorial 11: Brightness Adaption" }, { "contents": [ @@ -396,10 +437,10 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_11_adaptive_regularization.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_11_adaptive_regularization.py", - "summary": "In tutorial 7, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source demand different levels of regularization, motivating a regularization scheme which adapts to the reconstructed source's surface brightness.", - "title": "Tutorial 11: Adaptive Regularization" + "notebook": "notebooks/chapter_3_pixelizations/tutorial_12_adaptive_regularization.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_12_adaptive_regularization.py", + "summary": "In tutorial 10, we discussed why the `Constant` regularization scheme was sub-optimal. Different regions of the source demand different levels of regularization, motivating a regularization scheme which adapts to the reconstructed source's surface brightness.", + "title": "Tutorial 12: Adaptive Regularization" }, { "contents": [ @@ -408,10 +449,10 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_1_pixelizations.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_1_pixelizations.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_1_pixelizations.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_1_pixelizations.py", "summary": "In the previous chapters, we used light profiles to model the light of a strong lens's source galaxy, where the light profile was an analytic description of how the luminosity varies as a function of radius. In this chapter, we are instead going to reconstruct the source's light on a pixel-grid, and in this tutorial we will learn how to create a source-plane pixelization.", - "title": "Tutorial 1: pixelizations" + "title": "Tutorial 1: Pixelizations" }, { "contents": [ @@ -421,9 +462,9 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_2_mappers.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_2_mappers.py", - "summary": "In the previous tutorial, we used a pixelization to create made a `Mapper`. However, it was not clear what a `Mapper` does, why it was called a mapper and whether it was mapping anything at all!", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_2_mappers.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_2_mappers.py", + "summary": "In the previous tutorial, we used a pixelization to create a `Mapper`. However, it was not clear what a `Mapper` does, why it was called a mapper and whether it was mapping anything at all!", "title": "Tutorial 2: Mappers" }, { @@ -437,8 +478,8 @@ "cross_refs": [ "/imaging/features/pixelization/likelihood_function.ipynb" ], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_3_inversions.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_3_inversions.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_3_inversions.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_3_inversions.py", "summary": "In the previous two tutorials, we introduced:", "title": "Tutorial 3: Inversions" }, @@ -452,11 +493,45 @@ "Detailed Description" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_4_bayesian_regularization.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_4_bayesian_regularization.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_4_bayesian_regularization.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_4_bayesian_regularization.py", "summary": "So far, we have:", "title": "Tutorial 4: Bayesian Regularization" }, + { + "contents": [ + "Initial Setup", + "Mesh Shape", + "Ray Tracing", + "Border Relocation", + "Source Pixel Centres", + "Interpolation", + "Mapper", + "Mapping Matrix", + "Blurred Mapping Matrix", + "Data Vector", + "Curvature Matrix", + "Unregularized Solve", + "Regularization Matrix", + "Source Reconstruction", + "Image Reconstruction", + "Likelihood Function", + "Chi Squared", + "Regularization Term", + "Complexity Terms", + "Noise Normalization Term", + "Log Evidence", + "Fit", + "Wrap Up" + ], + "cross_refs": [ + "/imaging/features/pixelization/likelihood_function.ipynb" + ], + "notebook": "notebooks/chapter_3_pixelizations/tutorial_5_bayesian_formalism.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_5_bayesian_formalism.py", + "summary": "In tutorials 1 to 4, we built an intuition for how pixelized source reconstruction works: pixelizations place a pixel-grid in the source-plane, mappers pair source-pixels with image-pixels, inversions solve for the source-pixel fluxes that best fit the data, and regularization smooths the solution within a Bayesian framework.", + "title": "Tutorial 5: Bayesian Formalism" + }, { "contents": [ "Initial Setup", @@ -464,10 +539,10 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_5_borders.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_5_borders.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_6_borders.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_6_borders.py", "summary": "In the previous tutorials, the source-plane pixel grid perfectly mapped over the traced image-pixel $(y,x)$ coordinates in the source plane. If these pixels mapped to a larger area in the source plane, its pixel-grid would automatically increase its size so as to cover every source-plane coordinate.", - "title": "Tutorial 5: Borders" + "title": "Tutorial 6: Borders" }, { "contents": [ @@ -478,50 +553,173 @@ "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_6_lens_modeling.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_6_lens_modeling.py", - "summary": "When modeling complex source's with parametric profiles, we quickly entered a regime where our non-linear search was faced with a parameter space of dimensionality N=20+ parameters. This made the model-fitting inefficient and likely to infer a local maxima.", - "title": "Tutorial 6: Lens Modeling" + "notebook": "notebooks/chapter_3_pixelizations/tutorial_7_lens_modeling.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_7_lens_modeling.py", + "summary": "When modeling complex sources with parametric profiles, we quickly entered a regime where our non-linear search was faced with a parameter space of dimensionality N=20+ parameters. This made the model-fitting inefficient and likely to infer a local maxima.", + "title": "Tutorial 7: Lens Modeling" }, { "contents": [ "Initial Setup", - "Advantages and Disadvatanges", + "Advantages and Disadvantages", "Image Mesh", "Regularization", "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_7_adaptive_pixelization.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_8_adaptive_pixelization.py", "summary": "In this tutorial we will introduce a new `Pixelization` object, which uses an `Overlay` image-mesh and a `Delaunay` mesh.", - "title": "Tutorial 7: Adaptive Pixelization" + "title": "Tutorial 8: Adaptive Pixelization" }, { "contents": [], "cross_refs": [ "/imaging/features/pixelization/modeling.py" ], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_8_model_fit.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_8_model_fit.py", + "notebook": "notebooks/chapter_3_pixelizations/tutorial_9_model_fit.ipynb", + "path": "scripts/chapter_3_pixelizations/tutorial_9_model_fit.py", "summary": "You should now perform lens modeling using a pixelization, which is described fully in the example:", - "title": "Tutorial 8: Model-Fit" + "title": "Tutorial 9: Model-Fit" }, { "contents": [ "Initial Setup", + "Light Versus Mass", "Mask", - "Simulator", - "Fitting", - "Fit Problems", - "Discussion", + "Approach 1 Noise Scaling", + "Noise Scaling Fit", + "Approach 2 Extra Galaxies Model", + "Extra Galaxy Centres", + "Extra Galaxies Fit", + "Which Approach When", + "Wrap Up" + ], + "cross_refs": [], + "notebook": "notebooks/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.ipynb", + "path": "scripts/chapter_4_scaling_up_lensing/tutorial_1_extra_galaxies.py", + "summary": "Welcome to chapter 4 of **HowToLens**, where we scale up lens modeling beyond a single lens galaxy.", + "title": "Tutorial 1: Extra Galaxies" + }, + { + "contents": [ + "Initial Setup", + "Mask", + "Over Sampling", + "Model", + "Fixing the Mass Centres", + "Model Fit", + "Result", + "Mass Degeneracies", + "No Shared Halo", + "Three Lens Galaxies", "Wrap Up" ], "cross_refs": [], - "notebook": "notebooks/chapter_4_pixelizations/tutorial_9_fit_problems.ipynb", - "path": "scripts/chapter_4_pixelizations/tutorial_9_fit_problems.py", - "summary": "To begin, make sure you have read the `introduction` file carefully, as a clear understanding of how the Bayesian evidence works is key to understanding this chapter!", - "title": "Tutorial 9: Fit Problems" + "notebook": "notebooks/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.ipynb", + "path": "scripts/chapter_4_scaling_up_lensing/tutorial_2_multi_galaxy.py", + "summary": "In the previous tutorial, we learned how to deal with extra galaxies near a strong lens \u2014 nuisance objects whose light contaminates the data but which play no meaningful role in the lensing itself. We removed their emission from the analysis, or gave them a heavily restricted model, and the single dominant lens galaxy remained the star of the show.", + "title": "Tutorial 2: Multi-Galaxy Lenses" + }, + { + "contents": [ + "Mass Follows Light", + "Initial Setup", + "Mask", + "Measured Luminosities", + "The Anchor", + "Over Sampling", + "Light Via MGE", + "Scaling Relation", + "Model", + "Parameter Counts", + "Scaling To Many Galaxies", + "Model Fit", + "Results", + "Limitations", + "Lens Environments", + "Wrap Up" + ], + "cross_refs": [], + "notebook": "notebooks/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.ipynb", + "path": "scripts/chapter_4_scaling_up_lensing/tutorial_3_scaling_relation.py", + "summary": "The previous tutorial ended on a warning: every deflector we add to a lens model brings its own free parameters. Two galaxies were manageable, but each one cost us a mass profile's worth of dimensions, and the arithmetic only gets worse. A group-scale lens may have ten member galaxies, a cluster hundreds. If every member keeps its own free mass, the parameter space explodes \u2014 a 100-galaxy cluster with 5 free mass parameters per galaxy is a 500-dimensional model, which no non-linear search can sample and no dataset can constrain anyway.", + "title": "Tutorial 3: Scaling Relations" + }, + { + "contents": [ + "Initial Setup", + "Mask", + "Galaxy Centres", + "The dPIE Profile", + "Fitting a Group", + "A Group Halo?", + "Model Fit", + "Scaling Relation Members", + "The Group Scale Sweet Spot", + "Wrap Up" + ], + "cross_refs": [], + "notebook": "notebooks/chapter_4_scaling_up_lensing/tutorial_4_group_scale.ipynb", + "path": "scripts/chapter_4_scaling_up_lensing/tutorial_4_group_scale.py", + "summary": "In the previous tutorials we took our first steps beyond the single lens galaxy: we included extra galaxies near the lens in the model, we modeled systems where two or more galaxies of comparable mass share the lensing, and we introduced scaling relations, which tie the mass of a galaxy to its light so that adding more galaxies to a model does not mean adding more free parameters.", + "title": "Tutorial 4: Group Scale" + }, + { + "contents": [ + "Multi-Plane Ray Tracing", + "Dataset", + "Point Source Modeling", + "Point Datasets", + "The CSV Interface", + "Point Solver", + "Model", + "Analysis + Factor Graph", + "Search", + "Model Fit", + "Results", + "Customization", + "Wrap Up" + ], + "cross_refs": [ + "autolens_workspace/scripts/guides/point_source_pairing.py", + "csv_api.py", + "fit.py", + "likelihood_function.py" + ], + "notebook": "notebooks/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.ipynb", + "path": "scripts/chapter_4_scaling_up_lensing/tutorial_5_cluster_scale.py", + "summary": "Throughout this chapter we have been climbing a ladder of scale: from a single lens galaxy with an extra galaxy nearby, to multi-galaxy lenses, to galaxy groups whose members share a common dark matter halo.", + "title": "Tutorial 5: Cluster Scale" + }, + { + "contents": [ + "Shear Catalogues", + "Mass Scales", + "Ray Tracing", + "Source Galaxy Positions", + "Simulate", + "Visualize", + "Mass Map", + "Fitting", + "Model Fit", + "Result", + "Joint Strong and Weak Lensing", + "Wrap Up" + ], + "cross_refs": [ + "scripts/simulator/weak_lensing.py", + "start_here.py", + "weak/fit.py", + "weak/likelihood_function.py", + "weak/modeling.py", + "weak/simulator.py", + "weak/start_here.py" + ], + "notebook": "notebooks/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.ipynb", + "path": "scripts/chapter_4_scaling_up_lensing/tutorial_6_weak_lensing.py", + "summary": "Every tutorial in this series so far \u2014 indeed, every fit performed in all four chapters of **HowToLens** \u2014 has been a *strong* lensing analysis. Strong lensing occurs when a background galaxy lies so close (in projection) to a foreground mass that its light is bent into multiple images, arcs or a complete Einstein ring. These dramatic features are what we simulated, fitted and modeled, from the single galaxy-scale lenses of chapters 1 and 2 up to the group-scale and cluster-scale systems earlier in this chapter.", + "title": "Tutorial 6: Weak Lensing" }, { "contents": [ @@ -535,6 +733,98 @@ "summary": "Up to now, we've always used the non-linear search Nautilus and not considered the input parameters that control its sampling. In this tutorial, we'll consider how we can change these setting to balance finding the global maxima solution with fast run time.", "title": "Tutorial: Alternative Searches" }, + { + "contents": [ + "Dataset Paths", + "Redshifts", + "Galaxy Centres", + "Grids", + "Main Lens Galaxies", + "Scaling Member Galaxies", + "Host Dark Matter Halo", + "Source Galaxies", + "Ray Tracing", + "Point Solver", + "Point Datasets", + "Combined CSV", + "Scaling Galaxies CSV", + "Model CSVs", + "Tracer json", + "Imaging", + "Visualize" + ], + "cross_refs": [ + "/cluster/csv_api.py", + "/cluster/simulator.ipynb" + ], + "notebook": "notebooks/simulator/cluster.ipynb", + "path": "scripts/simulator/cluster.py", + "summary": "This script simulates a strong lens on the 'cluster' scale: 2 main lens galaxies (a brightest cluster galaxy and a satellite), 10 lower-mass cluster member galaxies on a luminosity-mass scaling relation, a cluster-scale dark matter halo not tied to any individual galaxy, and 2 multiply-imaged background source galaxies at *different* redshifts (z = 1.0 and z = 2.0) \u2014 a genuine multi-plane lens.", + "title": "Simulator: Cluster" + }, + { + "contents": [ + "Dataset Paths", + "Grid", + "Galaxy Centres", + "Over Sampling", + "PSF / Simulator", + "Main Lens Galaxy", + "Member Galaxies", + "Source Galaxy", + "Ray Tracing", + "Output", + "Visualize", + "Tracer json", + "Centre JSON Files", + "Positions" + ], + "cross_refs": [ + "/group/simulator.ipynb" + ], + "notebook": "notebooks/simulator/group.ipynb", + "path": "scripts/simulator/group.py", + "summary": "This script simulates `Imaging` of a 'group-scale' strong lens, which is used in chapter 4 of the **HowToLens** lectures to illustrate lens modeling at the group scale.", + "title": "Simulator: Group Scale Lens" + }, + { + "contents": [ + "Dataset Paths", + "Simulate", + "Ray Tracing", + "Output", + "Visualize", + "Tracer json" + ], + "cross_refs": [ + "/interferometer/simulator.ipynb" + ], + "notebook": "notebooks/simulator/interferometer.ipynb", + "path": "scripts/simulator/interferometer.py", + "summary": "This script simulates `Interferometer` data of a 'galaxy-scale' strong lens, as would be observed by a radio or sub-mm interferometer like ALMA or the JVLA.", + "title": "Simulator: Interferometer" + }, + { + "contents": [ + "Model", + "Dataset Paths", + "Simulate", + "Ray Tracing", + "Extra Galaxy", + "Output", + "Mask Extra Galaxy", + "Visualize", + "Tracer json", + "Extra Galaxy Centre" + ], + "cross_refs": [ + "/imaging/simulator.ipynb" + ], + "notebook": "notebooks/simulator/lens_extra_galaxy.ipynb", + "path": "scripts/simulator/lens_extra_galaxy.py", + "summary": "This script simulates `Imaging` of a 'galaxy-scale' strong lens which is identical to the `lens_sersic` dataset simulated for chapter 2 (lens light + mass + lensed source), but with one extra galaxy located a few arc-seconds from the lens galaxy.", + "title": "Simulator: Lens With Extra Galaxy" + }, { "contents": [ "Model", @@ -572,6 +862,24 @@ "summary": "This script simulates `Imaging` of a 'galaxy-scale' lens where there are two lens galaxies, each with their own light and mass profiles.", "title": "Simulator: Lens x2" }, + { + "contents": [ + "Model", + "Dataset Paths", + "Simulate", + "Ray Tracing", + "Output", + "Visualize", + "Tracer json" + ], + "cross_refs": [ + "/imaging/simulator.ipynb" + ], + "notebook": "notebooks/simulator/lens_x3.ipynb", + "path": "scripts/simulator/lens_x3.py", + "summary": "This script simulates `Imaging` of a 'galaxy-scale' lens where there are three lens galaxies, each with their own light and mass profiles, which all contribute significantly to the lensing of a single background source.", + "title": "Simulator: Lens x3" + }, { "contents": [ "Model", @@ -631,5 +939,22 @@ "path": "scripts/simulator/source_complex.py", "summary": "This script simulates `Imaging` of a 'galaxy-scale' strong lens where the source galaxy's light is more complex than other examples, being composed of 4 Sersics.", "title": "Simulator: Source Complex" + }, + { + "contents": [ + "Dataset Paths", + "Ray Tracing", + "Source Positions", + "Simulator", + "Output", + "Visualize" + ], + "cross_refs": [ + "/weak/simulator.ipynb" + ], + "notebook": "notebooks/simulator/weak_lensing.ipynb", + "path": "scripts/simulator/weak_lensing.py", + "summary": "This script simulates a weak gravitational lensing shear catalogue. Unlike the imaging simulators (which produce a 2D image of the lensed source) the weak-lensing simulator produces a *catalogue* of (gamma_2, gamma_1) shear measurements at the (y, x) positions of a population of background source galaxies.", + "title": "Simulator: Weak Lensing" } ]