Including "anatomy of an ASDF file tutorial by perrygreenfield · Pull Request #2 · asdf-format/tutorials

perrygreenfield · 2020-09-16T18:07:08Z

This supercedes PR #2 in that it adds a new tutorial that creates simpler ASDF files than the JWST one and highlights the basic capabilities available. Rather than change the original one (I still feel one that deals with real data is useful), I made one with a different focus and is not based on a practical data file. I have tried to incorporate comments from @nden, @eslavich, and @sosey into this one or the previous one. It's always possible I missed something though.

I now use the nbstripout script that @sosey recommended.

Some questions though:

as I'm developing these, should they go into the same PR even when previous ones aren't yet merged?
I would like to refer to other tutorials from a tutorial. What is the best way of doing that?

eslavich · 2020-09-16T18:45:53Z

I would like to refer to other tutorials from a tutorial. What is the best way of doing that?

I just figured this out while I was chipping away at the CRDS tutorials, if you use a relative link in the markdown:

[Some other notebook](some_other_notebook.ipynb)

clicking the link will pop open a new browser tab pointed to some_other_notebook.ipynb relative to the current notebook server directory.

perrygreenfield · 2020-09-16T18:55:25Z

With respect to the git directory path? If it is in the same directory, no other path info is needed, but otherwise it takes relative path specifications (if it ever comes to that) such as ../beginning_tutorials/How_to_turn_on_your_computer.ipynb

perrygreenfield · 2020-09-16T18:57:04Z

But I also wonder if the github location is the target (well, I suppose it doesn't matter so long as the same relative file structure is maintained, wherever they end up).

sosey · 2020-09-16T19:55:01Z

cool - definitely good to have the jwst real data tutorial, but also the basic one.

nden

I've reviewed the WCS notebook, comments inside.

We should specify software requirements in the beginning of the notebook - gwcs 0.14, asdf 2.7.1, astropy 4.0.1

I noticed the notebook does not mention the common WCS interface which we are trying to promote but actually do not use for JWST. Not sure if we need a section on this.

nden · 2020-09-21T20:52:01Z

+    "# ASDF and World Coordinate Systems\n",
+    "\n",
+    "The handling of World Coordinate Systems (WCS) is inextricably bound to \n",
+    "digital observational data.t For example, given a pixel in the data array \n",


typo - extra t

nden · 2020-09-21T20:54:02Z

+    "FITS has had a series of papers and resulting standards to deal with\n",
+    "WCS issues. Generally speaking, these have presumed mostly issues of \n",
+    "idealized detectors and optics, where distortion is not a significant\n",
+    "factor. There was much focus on handling many kinds of map projecions\n",


projections

nden · 2020-09-21T21:02:22Z

+    "space telescopes \n",
+    "such as HST, JWST or NGRST, the second is wasteful of disk space\n",
+    "while being inflexible to adjustments (more on that later). Since\n",
+    "most spactrographs use effectively the pixel translation approach,\n",


spectrographs

nden · 2020-09-22T17:53:47Z

+   "source": [
+    "This compound model is sufficiently complicated to require some detailed explanation (but not nearly as complicated as some JWST spectral WCS models!). First a brief explanation of the notation. Inputs and outputs indicate that this model takes two input values (or two arrays of values and generates two output values, i.e., two input coordinates and two output coordinates. Expression is the expression used to combine astropy models into a more complex compound model. To keep the expression concise, each term is replaced by a number and the model associated with that number is listed below. The two operators in particular that need special explanation are & and |. The & operator means that two functions are joined by adding dimensionality. If a Shift function shifts the value of the input by the given value, the joining two Shift functions shift the first input by the first value, and the second input by the second value. It is a way of making two 1D functions a 2D function, albeit with each input coordinate treated independently of the other. The | operator feeds the output of the preceding function into the next. The number of outputs of the preceding must match the number of inputs of the following function. Finally, the Mapping function acts as a switchboard between inputs and output. For example, Mapping(0, 1, 0, 1) takes two inputs and generates 4 outputs, where outputs 0, and 2 are both equal to the first input, and outputs 1 and 3 are both equal to the second input.\n",
+    "\n",
+    "Now to summarize in words what this transform does: shifts both input values first by 1 and then by -1024.5. This to make the coordinates relative to the center of the detector and cente of that pixel. The next step splits the two input values so that both go to two different 2D polynomials (each of which produces only one value, so the net result is two output value. Then these values go through a similar split and to a second polynomial that is effectively applying a small rotation, followed by a shift in the V2,v3 coordinate system.\n",


typos:

center,
values in the net result is two output values
capitalize V3

nden · 2020-09-22T17:58:21Z

+   "source": [
+    "from astropy.modeling.models import Scale\n",
+    "va_plate_scale_correction = 1.00008\n",
+    "#transform3 = (Scale(va_plate_scale_correction) & Scale(va_plate_scale_correction)) | transform2\n",


Should transform3 be commented out or did you mean a different expression for it.

Gives me:

NameError: name 'transform3' is not defined

I intended to temporarily comment it out (to redo some other calculation), but lots of temporary things last long than we realize.

eslavich · 2020-09-23T18:08:24Z

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The lines appearing before the line starting with `---` are used to indicate that it is an ASDF file, the versions of the appropriate components, and lastly an indication of where tags are defined. We will ignore tags for now. The actual yaml starts with the `---` and the end of the yaml is indicated by the `...`, which in this case is the end of the file. The yaml starts with the indication of the libraries involved and the homepage (the `asdf_library, homepage, and history` items. The user content starts with the `target` entry. While this is a very simple file, it is apparent that the first `name` key that appears is associated with `target` and that the second `name` key that appears is associated with `proposer` by dint of the indentation and proximity.\n",


The yaml starts with the indication of the libraries involved and the homepage (the asdf_library, homepage, and history items.

homepage is actually nested under asdf_library, it just looks independent from the way the notebook renders the content.

Also, missing close parenthesis there...

eslavich · 2020-09-23T18:11:42Z

+    "## Tags\n",
+    "\n",
+    "Tags are the mechanism asdf uses to indicate what follows is to be interpreted a special way, and in Python this usually means it should be converted to a specific Python object. There is\n",
+    "only one kind of tag used in this example and it is to indicate that the following items are \n",


There is only one kind of tag used in this example

This might be a confusing statement, since core/asdf-1.1.0, core/software-1.0.0, and core/extension_metadata-1.0.0 are also visible.

added these in the text

eslavich · 2020-09-23T18:13:23Z

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "af2.info()"


my_very_lucky_numbers is hidden in the info() output

added keyword arg to show all info

eslavich · 2020-09-23T18:14:46Z

+    "- equivalent of keywords can be very long (between 128 and 1024 characters depending on the library)\n",
+    "- values lengths are not restricted, and are not restricted to be simple values, values may be dictionaries or lists\n",
+    "- tag feature allows libraries to handle specific contents in a custom way (more later)\n",
+    "- references (addressed in a separate tutorial)\n",


references (addressed in a separate tutorial)

Looks like this notebook includes a section on references after all?

Yeah, I changed that comment in the intro

eslavich · 2020-09-23T18:16:04Z

+   "outputs": [],
+   "source": [
+    "# Save to a new file\n",
+    "af2.write_to('tut_with_data2.asdf')\n",


I ran into an error here:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd3 in position 3206: invalid continuation byte

Changed the brittle slice to prevent that

eslavich · 2020-09-23T20:14:31Z

+    "\n",
+    "## Modifying the WCS\n",
+    "\n",
+    "This is a somewhat contrived example, but it is to illustrate that one can make customizations to a WCS fairly easily. In this particular case one could change any of the parameters in the transforms, and save the to an ASDF file without any concern for how to do that. Or one can modify the existing transform by adding new elements to the transform. So supposing the existing plate scale used does not include velocity aberration, then one would want to include that as well as a plate scale change, and perhaps not change  the existing plate scale application to be clear there are two different scales applied. \n",


and save the to an ASDF file

Missing a word here

put it back in

eslavich · 2020-09-23T20:15:31Z

+   "source": [
+    "from astropy.modeling.models import Scale\n",
+    "va_plate_scale_correction = 1.00008\n",
+    "#transform3 = (Scale(va_plate_scale_correction) & Scale(va_plate_scale_correction)) | transform2\n",


Gives me:

NameError: name 'transform3' is not defined

eslavich · 2020-09-23T20:16:37Z

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The result is fairly extensive and not nearly as easy to interpret as the python printed version. Nevertheless, it demonstrates ASDF is capable of saving arbitrarily complex expressions without the user needed to understand the details of saving reading those expressions."


without the user needed to understand the details of saving reading those expressions.

Grammar needs some adjustment here

It isn't BNF compatible?

eslavich · 2020-09-23T20:21:10Z

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "af.info()"


One of the cells must have gotten lost, this fails with:

NameError: name 'af' is not defined

I hunted it down and dragged it back home (along with a stern talking to).

eslavich · 2020-09-23T20:23:27Z

+    "For example:\n",
+    "\n",
+    "```\n",
+    "asdftool info --max-rows 100 myfile.asdf\n",


We could demonstrate this in a code cell like this:

!asdftool info {filepath}

Well, I didn't know about that trick. On the other hand, it doesn't work since it apparently loads the file through jwst instead.

eslavich

The exercises seem reasonable to me. I didn't attempt any in ASDF_and_World_Coordinate_Systems.ipynb but did complete the others.

eslavich · 2020-11-19T16:45:55Z

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Exercise 2"


Exercise 2 seems to be missing? Or is it a zen thing?

Yes. The person is expected to think of their own exercise. Actually. I had one there and deleted it as not good enough and I think I had intended to replace it, but hadn't .

eslavich · 2020-11-19T16:47:38Z

+    "Of course Python 3.6 or later must be installed, along with ipython, jupyter, numpy, astropy, gwcs, and asdf. We generally recommend installing Python using miniconda since conda allows easy cloning of different Python environments ([miniconda download](https://docs.conda.io/en/latest/miniconda.html) and using pip install for the rest after activating the miniconda environment. For example, at the shell level:\n",
+    "\n",
+    "`\n",
+    "pip install ipython\n",


Should we add a requirements.txt that includes these?

If that is the normal solution, yes, I'll do that.

Though I was chastised on slack for recommending pip. For most things it seems to work fine. I probably wouldn't use in an operational case. Should I also suggest setting up a special environment just for the tutorial so that pip doesn't mess up some existing installation? Yes, probably.

eslavich · 2020-11-19T16:50:38Z

+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "af.search(value='crds').format(max_rows=None)"


This exercise comes pre-solved

eslavich · 2020-11-19T17:06:45Z

+   "source": [
+    "## Exercise 4\n",
+    "\n",
+    "Do a prime factorization of the FITS hash value."


2 * 2 * 5 * 7 * 199 * 20c5 * 2f59e53f80563fe87303d25b5feb8e529d8ee04619504fabf1e051999

Smartass :-)

nden

Other than Exercises 2 missing from the WCS notebook I think this looks good and can be merged.

perrygreenfield added 6 commits August 31, 2020 16:35

added first version of introduction

d427017

adding first version of basics of reading asdf files

3b5d8ec

testing nbstripout filter

bb489d6

added anatomy tutorial

d3a0b99

minor fixes to the basic tutorial

52dd2e5

change object to map in description in basic tutorial

0c65940

perrygreenfield added 2 commits September 18, 2020 14:58

adding partial error tutorial and gwcs tutorial

a85ef3b

added Ed's list of possible errors

4d1c71a

nden reviewed Sep 22, 2020

View reviewed changes

eslavich reviewed Sep 23, 2020

View reviewed changes

perrygreenfield added 4 commits September 30, 2020 16:26

addressed comments

020fcf9

strip output

94ea17d

added some exercises

9d5f6d1

fix merge conflicts

9326bd1

eslavich reviewed Nov 19, 2020

View reviewed changes

nden approved these changes Nov 30, 2020

View reviewed changes

minor edits to tutorials

2abd7f7

perrygreenfield merged commit 8628818 into asdf-format:master Dec 2, 2020

Conversation

perrygreenfield commented Sep 16, 2020

Uh oh!

eslavich commented Sep 16, 2020

Uh oh!

perrygreenfield commented Sep 16, 2020

Uh oh!

perrygreenfield commented Sep 16, 2020

Uh oh!

sosey commented Sep 16, 2020

Uh oh!

nden left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eslavich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment