029_Python_at_the_Large_Hadron_Collider_LHC.vtt

WEBVTT

00:00:00.001 --> 00:00:03.820
The largest machine ever built is the Large Hadron Collider at CERN.

00:00:03.820 --> 00:00:10.100
Its primary goal was the discovery of the Higgs boson, the fundamental particle which gives all objects mass.

00:00:10.100 --> 00:00:18.140
The LHC team actually achieved this audacious goal in 2012, winning them the Nobel Prize in physics in the process.

00:00:18.140 --> 00:00:25.220
Today on Talk Python to Me, Kyle Cranmer is here to share how Python was at the core of this amazing achievement.

00:00:25.760 --> 00:00:31.640
This is episode number 29, recorded Thursday, September 24th, 2015.

00:00:51.520 --> 00:01:05.240
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

00:01:05.240 --> 00:01:09.360
This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.

00:01:09.360 --> 00:01:15.820
Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via at Talk Python.

00:01:16.520 --> 00:01:19.780
This episode is brought to you by Hired and CodeShip.

00:01:19.780 --> 00:01:25.120
Thank them for supporting the show on Twitter via at Hired underscore HQ and at CodeShip.

00:01:25.120 --> 00:01:31.060
I don't have much news to share this week, but I am both honored and thrilled to bring you this episode,

00:01:31.060 --> 00:01:32.800
and I can't wait for you to listen to it.

00:01:32.800 --> 00:01:34.940
So, let's get right to the interview.

00:01:34.940 --> 00:01:37.160
Let me introduce Kyle.

00:01:37.160 --> 00:01:44.340
Kyle Cranmer is an American physicist and professor at New York University at the Center for Cosmology and Particle Physicists

00:01:44.340 --> 00:01:48.340
and affiliated faculty member at NYU's Center for Data Science.

00:01:48.340 --> 00:01:54.480
He is an experimental particle physicist working primarily on the Large Hadron Collider based in Geneva, Switzerland.

00:01:54.480 --> 00:02:00.540
Cranmer popularized a collaborative statistical modeling approach and developed statistical modeling,

00:02:00.540 --> 00:02:06.060
which was used extensively for the discovery of the Higgs boson at the LHC in July 2012.

00:02:06.060 --> 00:02:08.300
Kyle, welcome to the show.

00:02:08.300 --> 00:02:10.180
Thank you. Thank you. It's a pleasure to be here.

00:02:10.780 --> 00:02:17.340
Yeah, we have some amazing science and programming to talk about today, so I'm really excited to dig into all these topics with you.

00:02:17.340 --> 00:02:20.020
Yeah, no, I'm excited to see where it goes.

00:02:20.020 --> 00:02:21.600
Yeah, for sure.

00:02:21.600 --> 00:02:26.640
So, let's, you know, we're going to talk about the Large Hadron Collider,

00:02:26.640 --> 00:02:31.540
about using Python for scientific research and all those sorts of things,

00:02:31.580 --> 00:02:33.620
as well as some other cool projects that you've got going on.

00:02:33.620 --> 00:02:41.500
But people like to know how folks like you and your position kind of got started and they like to hear the background.

00:02:41.500 --> 00:02:47.580
So, maybe we could start with, you know, what got you interested in physics and what got you interested in programming and how do you get to where you are?

00:02:48.020 --> 00:02:53.200
I've been interested in, you know, in physics, you know, since I was a kid, not really knowing that that's what it was called.

00:02:53.200 --> 00:03:00.860
But later on, you know, I think, I guess it was in high school is when I really realized that it was physics that I wanted to do.

00:03:01.460 --> 00:03:09.420
I grew up in Arkansas and, you know, Arkansas is not exactly known for, like, leading the tide of physicists and computer scientists in the world.

00:03:09.420 --> 00:03:15.300
But they had started a special math and science high school that it was public school, but you actually lived there.

00:03:15.300 --> 00:03:21.600
And when I was there, I was just surrounded by all sorts of people, kind of the nerds and geeks of Arkansas.

00:03:21.600 --> 00:03:23.260
And it was really a special time.

00:03:23.260 --> 00:03:29.980
So, during that time, I, you know, got even more into physics, but it's also, that's when I was first exposed to serious programming.

00:03:30.880 --> 00:03:41.940
So, actually, even Python, like in 95, actually 94, I guess, I had a friend that was into early web things and he was playing with Zope and the Zope, you know, object database.

00:03:41.940 --> 00:03:49.560
And so, I started working with him and did some early web projects and that was kind of my first exposure to Python.

00:03:49.560 --> 00:03:51.320
So, that was a long time ago.

00:03:51.320 --> 00:03:58.060
And then that's, it's funny how much those experiences kind of keep being revisited today.

00:03:58.060 --> 00:04:00.620
Yeah, I'm sure you keep coming back to that.

00:04:00.780 --> 00:04:06.780
You know, basically, programming these days seems like a required skill to be a physicist.

00:04:06.780 --> 00:04:14.060
Yeah, no, well, it depends on what you do, but definitely for what we do, programming is a required skill.

00:04:14.060 --> 00:04:22.400
And unfortunately, it can, you know, for people that don't have those strengths, it really takes away from their ability to try to do the physics that they want to do.

00:04:23.060 --> 00:04:31.020
So, you know, so for incoming graduate students, you usually see a pretty big divide between people that are, have some programming skills and don't.

00:04:31.020 --> 00:04:36.800
And usually, the people that don't will catch up a little bit later, but, you know, you lose time and that's unfortunate.

00:04:36.800 --> 00:04:37.500
Right.

00:04:37.560 --> 00:04:43.920
I'm sure it's like a huge scramble, like, oh my gosh, I got to learn all this programming stuff too because, you know, we have projects or whatever, right?

00:04:44.400 --> 00:04:44.940
Right, right.

00:04:44.940 --> 00:04:56.180
It also colors a lot of the flavor about how we approach computing because somehow you have this enormous computing problem that you need to deal with and you would like to do it as nicely as possible.

00:04:56.180 --> 00:05:02.720
But it also can't be too fancy or the bulk of the physicists might not be able to understand what's going on.

00:05:02.720 --> 00:05:10.860
You have older physicists from the Fortran days and you have younger physicists that maybe never took any, you know, programming courses, like any serious programming courses.

00:05:10.860 --> 00:05:16.800
So things have to be somehow kept simple but still work for the difficult problems.

00:05:16.800 --> 00:05:18.800
And so it's a difficult balance to strike.

00:05:18.800 --> 00:05:19.860
Yeah, I'm sure.

00:05:20.660 --> 00:05:24.820
So let's talk a little bit about what you guys are doing at the Large Hadron Collider.

00:05:24.820 --> 00:05:30.000
And first of all, you know, congratulations on the Higgs boson discovery.

00:05:30.000 --> 00:05:30.580
That's amazing.

00:05:30.580 --> 00:05:31.720
Oh, thank you.

00:05:31.720 --> 00:05:35.580
No, it was, yeah, many, many years of work.

00:05:35.580 --> 00:05:39.480
And when it finally came, it was a huge treat.

00:05:39.480 --> 00:05:39.940
I don't know.

00:05:39.940 --> 00:05:44.500
It's funny to have such a big thing like that happen fairly early in your career.

00:05:44.500 --> 00:05:47.180
It's like, now what?

00:05:47.180 --> 00:05:47.960
Yeah.

00:05:47.960 --> 00:05:53.800
So, but yeah, so at the LHC, you know, we have this huge collider that's in Switzerland.

00:05:53.800 --> 00:05:57.220
It's about 17 miles around and underground.

00:05:57.220 --> 00:06:03.580
There are all these super connecting magnets that help protons go, you know, bend them in a circle at essentially the speed of light.

00:06:03.580 --> 00:06:06.320
And they're colliding together all the time.

00:06:06.420 --> 00:06:11.480
And they smack into each other and they make, you know, a lot of new particles that come flying out.

00:06:11.480 --> 00:06:13.660
And they hit our detector.

00:06:13.660 --> 00:06:17.280
And our detector, you can think of sort of like a digital camera.

00:06:17.280 --> 00:06:19.100
You know, it's like basically a bunch of pixels.

00:06:19.100 --> 00:06:22.860
And the particles smack into it and you get an image.

00:06:22.860 --> 00:06:24.500
But it's a 3D image.

00:06:24.500 --> 00:06:25.680
So it's a 3D detector.

00:06:25.980 --> 00:06:29.380
And the detector is like the size of a 12-story building.

00:06:29.380 --> 00:06:30.580
So, yeah.

00:06:30.580 --> 00:06:43.520
I think that, you know, when you just hear about particle colliders and especially LHC, you have maybe this idea of like a tube where things are shooting around.

00:06:43.520 --> 00:06:45.080
And, you know, how big does the tube have to be?

00:06:45.080 --> 00:06:45.860
Not that big.

00:06:46.120 --> 00:06:53.260
But the actual experiments, the collectors, I was blown away when I learned about how big they are.

00:06:53.260 --> 00:06:54.800
Like you said, 12 stories.

00:06:54.800 --> 00:06:55.640
These things are huge.

00:06:55.640 --> 00:06:56.280
Yeah.

00:06:56.280 --> 00:06:56.860
No, they are.

00:06:56.860 --> 00:07:04.960
And the range of scales is pretty crazy because we have to be able to track where these particles go very precisely.

00:07:04.960 --> 00:07:09.140
So, like, close to where they interact, you know, we're measuring things at the, like, micron level.

00:07:09.680 --> 00:07:14.480
And then they fly out over, like, the size of a building.

00:07:14.480 --> 00:07:18.260
And we're still measuring where they're going at this very precise level.

00:07:18.260 --> 00:07:20.920
But, you know, it's just this gargantuan thing.

00:07:20.920 --> 00:07:24.280
So you have to align it properly and all sorts of challenges there.

00:07:24.280 --> 00:07:30.820
And there are about 100 million, you know, well, a few hundred million electronic readouts coming out of this beast.

00:07:30.820 --> 00:07:35.580
So, you know, it's like a 100-megapixel camera or something like that.

00:07:35.580 --> 00:07:38.760
And we're taking 40 million photos every second.

00:07:38.760 --> 00:07:41.300
That's a stunning amount of data.

00:07:41.300 --> 00:07:42.800
That is a stunning amount of data.

00:07:42.800 --> 00:07:56.140
And so if you, you know, we have to slap special electronics, like, straight onto the detector to be able to start preprocessing it and compressing it and sort of, you know, coming up with some way to deal with the data volume.

00:07:56.140 --> 00:08:04.340
Because, you know, it's something, it's, you know, there are just totally staggering numbers about the data flow that's coming straight out of the detector.

00:08:04.340 --> 00:08:06.640
So how do you capture and store that?

00:08:06.700 --> 00:08:11.900
Do you store that, like, on hardware right on, like, Atlas in the machines?

00:08:11.900 --> 00:08:15.400
Or do you, like, get that into, like, a cluster of servers?

00:08:15.400 --> 00:08:16.440
Or what happens?

00:08:16.440 --> 00:08:17.440
Right, right.

00:08:17.540 --> 00:08:27.320
So we have a kind of a hierarchical online real-time system for tossing away, you know, the majority of the data.

00:08:27.320 --> 00:08:35.000
So we have to actually, we write algorithms that have to look at the data and real-time decide, does this look interesting or not?

00:08:35.000 --> 00:08:41.500
And so we go from the sort of 40 million a second through this, like, three levels of filtering down.

00:08:41.500 --> 00:08:47.300
And then we get to the point that we save something like a few hundred of these collisions every second.

00:08:48.000 --> 00:08:55.260
And that turns into, you know, several petabytes a year of data that we actually analyze later.

00:08:55.260 --> 00:08:56.660
That's amazing.

00:08:56.660 --> 00:08:59.800
It's got to be a little stressful to work on that initial filtering algorithm.

00:08:59.800 --> 00:09:03.240
Because what if you threw away the Higgs boson before you discovered it, right?

00:09:03.240 --> 00:09:04.140
That's right.

00:09:04.140 --> 00:09:04.320
Yeah.

00:09:04.320 --> 00:09:08.620
No, people, we always worry that we're, you know, kind of throwing the baby out with the bathwater.

00:09:09.580 --> 00:09:13.620
And sorry about the living in New York here.

00:09:13.620 --> 00:09:14.740
Yeah, no worries.

00:09:14.740 --> 00:09:18.540
So the, yeah, we call that thing the trigger.

00:09:18.540 --> 00:09:21.700
And, you know, that's something that I worked on a bit.

00:09:21.700 --> 00:09:29.000
It's true that, like, if we don't find anything else in this next run of the LHC, you know, a lot of people will think exactly that.

00:09:29.000 --> 00:09:33.960
That maybe, you know, the way the trigger was configured, we were throwing away the interesting stuff.

00:09:33.960 --> 00:09:36.000
But luckily, we're not stuck to that.

00:09:36.000 --> 00:09:38.360
You know, we can go and we can change it and things like that.

00:09:38.360 --> 00:09:40.200
But that is the worry.

00:09:40.200 --> 00:09:41.120
Sure.

00:09:41.120 --> 00:09:41.500
Yeah.

00:09:41.500 --> 00:09:45.540
I mean, you still have the time spent and the energy and all that, right?

00:09:45.540 --> 00:09:46.140
Yeah.

00:09:46.140 --> 00:09:46.660
Yeah, sure.

00:09:46.660 --> 00:09:48.560
You can rerun it, of course.

00:09:48.560 --> 00:09:54.180
But, you know, you got to, I suspect time is a valuable thing on that machine.

00:09:54.180 --> 00:09:55.640
Yeah.

00:09:55.640 --> 00:09:56.280
No, for sure.

00:09:56.280 --> 00:09:57.540
It's expensive to run.

00:09:57.540 --> 00:09:59.300
So, absolutely.

00:09:59.300 --> 00:10:00.740
Yeah.

00:10:00.740 --> 00:10:07.660
So, you know, I have a lot of listeners who are scientists and physicists and data science and so on.

00:10:07.660 --> 00:10:12.180
But a lot of them who are probably not.

00:10:12.180 --> 00:10:23.200
And so, I wanted to make a movie recommendation and a book recommendation just for people to, you know, if they want to kind of set the stage and learn the background, you know, as part of this whole thing we're talking about.

00:10:23.200 --> 00:10:26.780
I wanted to recommend the Particle Fever documentary.

00:10:26.780 --> 00:10:27.680
Have you seen this?

00:10:28.380 --> 00:10:28.880
Yeah, yeah.

00:10:28.880 --> 00:10:30.780
I actually have a credit in that movie.

00:10:30.780 --> 00:10:35.180
I worked with them quite a bit.

00:10:35.180 --> 00:10:45.020
And at one point there was a scene that was shot in my office but ended up having to cut it because it didn't really fit well with the, you know, it was a good choice.

00:10:45.120 --> 00:10:46.300
But it was painful.

00:10:46.300 --> 00:10:47.780
But they were nice.

00:10:47.780 --> 00:11:00.740
I worked with them a fair amount and got to go to the, like, you know, the opening in Sheffield at a documentary film festival and hang out with the producers and the whole crew.

00:11:00.740 --> 00:11:03.660
But it's a great film.

00:11:04.020 --> 00:11:08.340
I think it definitely, it's good for a non-physics audience also.

00:11:08.340 --> 00:11:09.500
It's not a technical film.

00:11:09.500 --> 00:11:18.260
It just basically captures what it's like to be inside one of these experiments and the sort of stress and the, you know, the drama associated to it.

00:11:18.260 --> 00:11:21.520
I think it's really one of the best science documentaries ever.

00:11:21.520 --> 00:11:22.900
I absolutely agree with you.

00:11:22.900 --> 00:11:30.540
I think it really captures the excitement, the imagination, the drama in a way that, you know, anybody could appreciate.

00:11:30.540 --> 00:11:32.440
And so, I definitely recommend people watch that.

00:11:32.600 --> 00:11:36.560
It's available for streaming on Netflix and iTunes and other places.

00:11:36.560 --> 00:11:44.980
And then the other thing is the book called Present at the Creation, Discovering the Higgs Boson by Amir Axel.

00:11:44.980 --> 00:11:45.960
I messed up his name.

00:11:45.960 --> 00:11:47.400
But that's also really good.

00:11:47.400 --> 00:11:51.580
So, people who are out there and want to learn more about what we're talking about, I think I recommend those.

00:11:51.580 --> 00:11:52.920
Okay, great.

00:11:52.920 --> 00:11:54.640
Yeah, I actually haven't read that second book.

00:11:54.640 --> 00:11:58.000
Yeah, I really enjoyed that book as well.

00:11:58.000 --> 00:12:00.420
It predates the Higgs Boson.

00:12:00.420 --> 00:12:02.160
So, it's like a lot of anticipation.

00:12:02.480 --> 00:12:02.800
So, that's cool.

00:12:02.800 --> 00:12:03.020
I see.

00:12:03.020 --> 00:12:03.940
Okay, great.

00:12:03.940 --> 00:12:11.320
Maybe we could talk a little bit about, like, the really big picture of software at the LHC.

00:12:11.320 --> 00:12:14.480
Because there's not just one team and there's not just one experiment.

00:12:14.480 --> 00:12:15.620
There's how many collectors?

00:12:15.620 --> 00:12:16.700
Are there seven collectors?

00:12:16.700 --> 00:12:17.920
Right.

00:12:17.920 --> 00:12:23.140
So, well, there are two really big kind of multipurpose particle detectors, Atlas and CMS.

00:12:23.140 --> 00:12:24.260
And I'm on Atlas.

00:12:24.640 --> 00:12:31.440
And those two experiments have, you know, in the neighborhood, a little bit more than 3,000 physicists working on them.

00:12:32.060 --> 00:12:35.340
So, you know, so it's a, there are big groups of people.

00:12:35.340 --> 00:12:45.620
And then there are two other experiments that are, you know, slightly smaller in scale, but they do, you know, and slightly more specialized in terms of the physics that they do.

00:12:45.940 --> 00:12:51.020
And then there are several other smaller dedicated experiments that are quite a bit smaller.

00:12:51.580 --> 00:12:55.820
And so, I don't, you know, it depends on how you count a little bit.

00:12:55.820 --> 00:13:05.520
But usually, you know, there's sort of the two big multipurpose detectors and two other more specialized ones that are the dominant, like, LHC experiments.

00:13:05.520 --> 00:13:06.560
Okay, cool.

00:13:06.560 --> 00:13:15.260
And so, maybe from, like, the higher level or larger scale, like, the thing that actually runs the machine down into the experiments, down into more, like, the data processing details.

00:13:15.400 --> 00:13:18.680
Could you give us a picture, like, what the software looks like there, what you guys are doing?

00:13:18.680 --> 00:13:19.820
Sure, yeah.

00:13:19.820 --> 00:13:23.600
So, I mean, it's mainly, you know, we have a whole bunch of collisions.

00:13:23.600 --> 00:13:31.180
And each collision, you know, if you think of what this metaphor of it being like an image, you know, it's like a pipeline for doing a bunch of image processing, you know.

00:13:31.180 --> 00:13:39.360
And you're looking for, you're trying to find the collisions that, you know, maybe have evidence of some new particle.

00:13:39.360 --> 00:13:50.400
So, you have lots of teams of people that are looking for different things, and each of those teams will develop a little pipeline to process the data to try to, you know, to search for what they want.

00:13:50.400 --> 00:14:00.120
Also, to put it into perspective a little bit, we had a quadrillion, you know, a couple quadrillion collisions total at the LHC.

00:14:00.120 --> 00:14:08.180
And when we discovered the Higgs, it was, you know, of the order of 100 or 1,000 of those collisions that were the interesting ones.

00:14:08.460 --> 00:14:11.180
So, it's a huge needle in a haystack problem.

00:14:11.180 --> 00:14:17.420
But it's also not really like a data mining kind of just generally looking for something weird in the data.

00:14:17.420 --> 00:14:30.080
We have theories that tell us, you know, what to look for, which is good because there's such small little deviations in the data that it would be basically impossible to find if you didn't have a good guide.

00:14:30.320 --> 00:14:41.280
And then this processing chain, because there's so much data and performance is such an issue, most of it, well, several years ago, the decision was to write most of the software in C++.

00:14:41.280 --> 00:14:44.880
You know, C++ has also evolved a ton during the time.

00:14:44.880 --> 00:14:49.900
Are you using like, are people using like C++ 11 and those types of things?

00:14:50.140 --> 00:14:50.640
Right.

00:14:50.640 --> 00:14:59.880
So, the different experiments kind of, you know, move to these new, you know, new standards and new computing technologies kind of at different paces.

00:14:59.880 --> 00:15:01.540
There's a lot of worry.

00:15:01.540 --> 00:15:07.800
You know, it's generally a pretty conservative attitude, you know, but we are making those kinds of transitions.

00:15:08.580 --> 00:15:14.040
But, you know, you just, it has to go through a lot of vetting before we make a big jump like that.

00:15:14.040 --> 00:15:33.520
We also usually have a very homogeneous computing environment in terms of like operating systems and things like that because, you know, we run into issues where, you know, you don't want to have to be worrying about like floating point arithmetic in your kernel or something, you know, when you're, because it's, so, so we just tried it.

00:15:33.520 --> 00:15:42.420
Yeah, so it's a little bit funny, you know, that CERN was responsible for sort of developing the web browser, right, you know, and HTML and things like that.

00:15:42.420 --> 00:15:45.700
And so, they had this huge win of, you know, where the web was born.

00:15:45.700 --> 00:15:51.420
And then that was followed by this idea of like, okay, we had the web, now we're going to have grid computing.

00:15:51.420 --> 00:15:53.760
And there was a lot of money poured into it.

00:15:53.760 --> 00:15:59.160
And the promise of the grid basically turned into what has happened with the cloud.

00:15:59.560 --> 00:16:09.400
And, but in, you know, and then within IMG physics, we do have the grid, but it's kind of like a huge global, you know, batch system in some sense.

00:16:09.400 --> 00:16:20.020
So, it tends to be, you know, more uniform and things like that than what, you know, at first people were working really hard to be able to work over very heterogeneous computing environments.

00:16:20.020 --> 00:16:24.320
But, you know, that all evolved over, you know, more than a decade.

00:16:24.320 --> 00:16:25.960
Yeah, I'm sure.

00:16:25.960 --> 00:16:29.740
I used to do a lot of work in sort of scientific computing and visualization.

00:16:29.740 --> 00:16:35.920
And it's super hard to do reproducibility and checking stuff, you know.

00:16:35.920 --> 00:16:36.620
Right.

00:16:36.620 --> 00:16:41.940
If you've got a sufficiently complicated series of mathematical steps, you can apply to something.

00:16:42.180 --> 00:16:46.680
You know, like, if it's so complicated, how do you know when you're right or not?

00:16:46.680 --> 00:16:46.880
Right.

00:16:46.880 --> 00:16:52.420
You know, how do you know when you're discovering something new versus, oh, it's like I expected or whatever, right?

00:16:52.420 --> 00:16:53.020
Right.

00:16:53.020 --> 00:17:01.880
Well, we're working a lot right now on trying to address the sort of reproducibility, you know, issues kind of specific and the challenges associated to our field.

00:17:02.460 --> 00:17:05.960
And there are a lot of challenges because there's so much data and software is very complicated.

00:17:05.960 --> 00:17:06.940
Yeah.

00:17:06.940 --> 00:17:13.640
So the core algorithms tend to all be C++, but they're, you know, they're organized into lots of, you know, lots of different tools.

00:17:13.640 --> 00:17:19.480
And, you know, you have a way of kind of composing this pipeline between different processing algorithms.

00:17:19.480 --> 00:17:29.960
And in the end, the configuration of that thing is such a beast that that's the first place where you see Python happening is that we have a way of kind of, you know, doing introspection on all of the tools.

00:17:30.040 --> 00:17:33.820
And then we just represent their configuration in terms of Python objects.

00:17:33.820 --> 00:17:40.400
And then there's a whole separate layer of, you know, of computing, which I mean, of programming, which is just essentially the configuration.

00:17:40.400 --> 00:17:55.600
And that includes both this trigger, that online system that's tossing out the data, as well as the, you know, the people that are analyzing the data, how they, you know, configure all these tools to be able to process the kajillions of events into something that's more manageable.

00:17:55.600 --> 00:17:57.980
Now, that sounds really interesting.

00:17:57.980 --> 00:18:05.860
When I was doing some research, it seemed like one of the major pieces used in Atlas was this thing called Athena.

00:18:05.860 --> 00:18:07.080
Right.

00:18:07.080 --> 00:18:07.860
That's right.

00:18:07.860 --> 00:18:16.880
That's the kind of name of the C++ framework that we use that also includes the way that it builds the Python bindings for configuring all the tools.

00:18:17.580 --> 00:18:24.620
And yeah, so that's, yeah, I've spent more hours than I'd like to admit doing programming in that framework.

00:18:24.620 --> 00:18:35.920
But then what's also interesting, I think a lot of your audience will find interesting, is that once you've used that huge, heavyweight data processing pipeline, usually you get to something quite a bit smaller.

00:18:36.260 --> 00:18:41.000
And that's where a lot of the more interactive and exploratory part of the data analysis happens.

00:18:41.000 --> 00:18:47.900
And at that stage, a lot of people, well, people stop using things like Athena for the most part.

00:18:48.140 --> 00:18:53.440
And that's where you start using, see people using Python a lot more in terms of data analysis.

00:18:53.440 --> 00:19:00.340
And so it's an interesting transition because people are always arguing about where do you make that swap, you know?

00:19:00.340 --> 00:19:01.360
Sure.

00:19:01.360 --> 00:19:01.920
Yeah.

00:19:02.640 --> 00:19:05.140
I suspect you guys probably do a lot of IPython.

00:19:05.140 --> 00:19:05.980
Is that true?

00:19:05.980 --> 00:19:08.700
Well, you would think that more people would.

00:19:08.700 --> 00:19:26.420
I guess part of it is that it's still, even at that stage, you still have so much data to process that the kinds of things that people end up wanting to do are, you know, well suited to having like, you know, programs that, you know, that look really like programs that run.

00:19:26.420 --> 00:19:33.140
And they might be Python based, but, you know, you kind of sort of batch systemy, you run over this thing, and then you get some results and look at them.

00:19:33.140 --> 00:19:35.940
There are times when you're doing something very interactive.

00:19:35.940 --> 00:19:51.920
And so years ago, the team at CERN that makes this tool called Root, which is like kind of the dominant data analysis package in high energy physics, came up with something like an interpreter because you want to sit there and have this feedback loop, right?

00:19:51.920 --> 00:19:55.740
You know, like the end where you can, you know, type commands, see plots.

00:19:55.740 --> 00:19:58.560
And that was actually done amazingly.

00:19:58.560 --> 00:20:01.940
They wrote a C++ interpreter many, many years ago.

00:20:01.940 --> 00:20:08.760
And so you actually write these commands in C++, and then they're interpreted and executed on the fly.

00:20:08.760 --> 00:20:11.120
That's actually pretty interesting by itself, isn't it?

00:20:11.120 --> 00:20:12.260
It is interesting.

00:20:12.260 --> 00:20:17.700
It, of course, had all sorts of issues, and C++ wasn't really meant for doing that, but it worked practically.

00:20:18.360 --> 00:20:27.960
And now they've gone through and they have a much heavier duty version of this interpreter that's based on, you know, Cling and more modern, like, compiling, compiler technologies and things.

00:20:27.960 --> 00:20:32.660
But Python, obviously, is another way to go with that, which is nice.

00:20:42.740 --> 00:20:45.460
This episode is brought to you by Hired.

00:20:45.460 --> 00:20:51.940
Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.

00:20:51.940 --> 00:21:01.100
Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company.

00:21:01.100 --> 00:21:07.460
Typically, candidates receive five or more offers in just the first week, and there are no obligations, ever.

00:21:07.460 --> 00:21:09.540
Sounds pretty awesome, doesn't it?

00:21:09.540 --> 00:21:11.600
Well, did I mention there's a signing bonus?

00:21:11.600 --> 00:21:15.680
Everyone who accepts a job from Hired gets a $2,000 signing bonus.

00:21:15.680 --> 00:21:20.040
And as Talk Python listeners, it gets way sweeter.

00:21:20.040 --> 00:21:27.600
Use the link Hired.com slash Talk Python To Me, and Hired will double the signing bonus to $4,000.

00:21:27.600 --> 00:21:29.300
Opportunity's knocking.

00:21:29.300 --> 00:21:32.920
Visit Hired.com slash Talk Python To Me and answer the call.

00:21:32.920 --> 00:21:45.300
So people started moving to the Python.

00:21:45.300 --> 00:21:52.220
Well, some people started moving to the Python way of doing things, I don't know, whatever, you know, eight years ago or something like that.

00:21:52.220 --> 00:21:55.760
But the field is kind of split between, you know, which way.

00:21:55.760 --> 00:21:59.640
And then since then, like, things like IPython and the IPython notebook have come around.

00:21:59.640 --> 00:22:03.980
I think that that's great, especially from the point of view of this, like, reproducibility.

00:22:03.980 --> 00:22:07.020
So we're working now that we give tons of talks.

00:22:07.020 --> 00:22:09.220
If you go to CERN, we have this agenda system.

00:22:09.220 --> 00:22:15.040
And you can see that there are, like, hundreds of thousands of presentations happening within these experiments every year.

00:22:15.040 --> 00:22:16.280
That's excellent.

00:22:16.680 --> 00:22:23.340
Yeah, so one of the things, but they're always, like, PowerPoint or, you know, Keynote or whatever, or LaTeX-based PDF presentations.

00:22:23.340 --> 00:22:32.720
And you read about what someone's doing, but it's not very handy for, like, trying to have reproducibility or for another graduate student to pick up where someone left off.

00:22:32.720 --> 00:22:45.600
So we have this effort now to try to make it so that the agenda system can, you know, can basically display notebooks directly so people can upload their IPython notebook directly and visualize it.

00:22:45.600 --> 00:22:49.480
And then if someone else thinks it's interesting, they can, you know, download it and execute it.

00:22:49.940 --> 00:23:04.880
There are efforts about trying to make it so that the whole computing environment associated to that notebook can be, you know, packaged up so that, because they usually aren't just standalone Python, Python notebooks with, like, SciPy dependency.

00:23:04.880 --> 00:23:06.140
They have a bunch of dependencies.

00:23:06.140 --> 00:23:11.000
So if you can package that all up, that's very handy.

00:23:11.000 --> 00:23:12.760
So there are tools like Binder now.

00:23:12.760 --> 00:23:14.560
There's a tool called Everware.

00:23:14.940 --> 00:23:21.060
And previously, there's something like SageMath, which all allowed you to sort of execute a notebook, you know.

00:23:21.060 --> 00:23:26.900
But the problem was how do you get all these, you know, these software dependencies packaged up?

00:23:26.900 --> 00:23:28.460
And now that problem is starting to be solved.

00:23:28.460 --> 00:23:29.200
Right.

00:23:29.200 --> 00:23:30.360
Oh, that's really excellent.

00:23:30.360 --> 00:23:41.260
Because I can imagine you guys have so much data and maybe these back-end systems you've got to reach into to actually work with the data that you're trying to, you know, do physics on.

00:23:41.260 --> 00:23:41.900
That's right.

00:23:41.900 --> 00:23:48.540
You can't just take the program and hand it out, you know, like, oh, and here's our, you know, 50 gigs of data and you've got to get it this way, right?

00:23:48.540 --> 00:23:49.120
Right.

00:23:49.120 --> 00:23:57.080
And not only that, there's also things like databases that say, like, how was the detector aligned on Friday, you know, November 25th or something.

00:23:57.080 --> 00:24:03.460
So there are all these databases involved that you have to connect to for the software to run.

00:24:03.640 --> 00:24:06.760
And that's also through tons of authentication layers.

00:24:06.760 --> 00:24:09.700
So it's a huge pain in the butt, basically.

00:24:09.700 --> 00:24:12.560
But people are solving it.

00:24:12.560 --> 00:24:15.880
And I think that will be a huge change.

00:24:15.880 --> 00:24:23.720
And the Project Jupyter people, you know, luckily had this great foresight to separate the notebook from the background kernel.

00:24:23.920 --> 00:24:29.700
So we're actually also writing kernel based on this C++ interpreter of root.

00:24:29.700 --> 00:24:34.980
So it still looks like notebooks and all the display and everything is the same.

00:24:34.980 --> 00:24:39.720
But in the background, instead of Python, it's the C++ interpreter, which is, you know, interesting.

00:24:40.260 --> 00:24:44.440
Yeah, I mean, that certainly opens it up to a much wider audience.

00:24:44.440 --> 00:24:53.180
Like, you're saying, like, the group that's working directly with Athena and so on, they can just, you know, possibly start using IPython or what do you call them?

00:24:53.180 --> 00:24:54.600
Call them Jupyter notebooks now?

00:24:54.600 --> 00:24:54.960
I don't know.

00:24:54.960 --> 00:24:56.840
I'm not really sure what the naming is.

00:24:56.840 --> 00:25:00.860
Yeah, the front end, kind of the language agnostic part is now Project Jupyter.

00:25:01.980 --> 00:25:18.480
But it's great because we have people like Fernando Perez, who's, you know, leading this effort as part of this advisory board for a project that we got, a grant we got from the National Science Foundation to try to take the tools that have been developed in Iron View Physics, which are mainly very siloed.

00:25:18.480 --> 00:25:20.040
You know, it's like we're trying to solve our problem.

00:25:20.040 --> 00:25:20.960
It's a very hard problem.

00:25:20.960 --> 00:25:22.600
And we don't have a lot of extra time or money.

00:25:22.600 --> 00:25:23.160
Right.

00:25:23.160 --> 00:25:25.520
And then, but now we've done some nice things.

00:25:25.520 --> 00:25:30.800
So let's try to open that up, make it more interoperable with, like, the scientific Python world.

00:25:31.460 --> 00:25:33.160
And it's definitely a two-way street.

00:25:33.160 --> 00:25:35.260
There are lots of other great tools out there that we don't use.

00:25:35.260 --> 00:25:39.080
So we're working on improving the interoperability of all of these things.

00:25:39.080 --> 00:25:41.840
Yeah, I think that's going to be good for science all over.

00:25:41.840 --> 00:25:45.460
And the Jupyter guys just got a huge grant.

00:25:45.460 --> 00:25:50.460
I'm not sure all the folks that contributed, but it was millions, like six million or something like that.

00:25:50.460 --> 00:25:50.960
Do you remember?

00:25:50.960 --> 00:25:57.620
I don't know the number, but they rightfully have been getting some support because they're doing some great things.

00:25:57.620 --> 00:26:01.180
And yeah, I'm really happy to see that.

00:26:01.180 --> 00:26:07.960
So, you know, people will start building C++ and, you know, I guess Ruby and maybe imagine Fortran.

00:26:07.960 --> 00:26:08.420
I don't know.

00:26:08.420 --> 00:26:11.000
That probably is important somewhere in science.

00:26:11.000 --> 00:26:13.080
But I try to not touch that stuff.

00:26:13.080 --> 00:26:13.780
Right.

00:26:14.680 --> 00:26:18.700
You talked about this sort of uniform computing environment.

00:26:18.700 --> 00:26:23.840
And I know just where you're coming from with that, like, just the whole reproducibility and setup and everything.

00:26:23.840 --> 00:26:27.180
Can you talk to, like, are you using Linux?

00:26:27.180 --> 00:26:28.140
What distribution?

00:26:28.140 --> 00:26:29.760
What do things look like there?

00:26:30.260 --> 00:26:30.540
Right.

00:26:30.540 --> 00:26:32.220
So it's definitely Linux-based.

00:26:32.220 --> 00:26:39.040
CERN has a distribution called Scientific Linux that they maintain.

00:26:39.040 --> 00:26:44.240
I've kind of stopped following all the ins and outs of it, to be honest.

00:26:44.240 --> 00:26:50.640
But, you know, at some point it kind of was a derivative of some Red Hat-type package way back in the day.

00:26:50.640 --> 00:26:51.780
But since then it's evolved.

00:26:51.780 --> 00:26:54.840
And now I'm not even sure which distribution it's closest to.

00:26:55.640 --> 00:27:04.600
And then there's been quite a bit of emphasis in virtualization technologies and, you know, OpenStack-related things.

00:27:04.600 --> 00:27:08.080
And what we're starting to see more use of now are also Docker images.

00:27:08.080 --> 00:27:17.380
So I have a student who's working on, you know, making, you know, Docker images that have our very kind of specific computing environment,

00:27:17.380 --> 00:27:21.300
specifically for the issue of reproducibility.

00:27:21.720 --> 00:27:31.800
There, you know, I think you also haven't really seen in high-energy physics, but we're starting to see now as, like, more and more, you know, web-based tools and things like that.

00:27:31.800 --> 00:27:32.800
So you have web services.

00:27:32.800 --> 00:27:38.980
And we have a project going on where we're trying to make not analyses just reproducible but reinterpretable.

00:27:38.980 --> 00:27:44.080
So, you know, you had a team of people analyzing the LHC data, and they were asking a certain question.

00:27:44.080 --> 00:27:47.780
But you can reuse that analysis pipeline to answer other questions.

00:27:47.960 --> 00:27:56.700
So we're trying to wrap up this very heavy computational, you know, pipeline with a very simple web interface, you know, web APIs and things,

00:27:56.700 --> 00:28:04.140
so that people can submit, you know, requests to the system, and they'll be processed through all of this infrastructure.

00:28:04.140 --> 00:28:06.940
And then you'll, you know, come out with a very simple answer.

00:28:06.940 --> 00:28:09.020
That's really cool.

00:28:09.340 --> 00:28:10.460
Yeah, I think it'll be great.

00:28:10.460 --> 00:28:21.940
It also, it addresses a lot of issues with reproducibility when you have, when you can't just, like, say, you know, the typical open data route.

00:28:21.940 --> 00:28:23.400
Like, here it is, do it again.

00:28:23.400 --> 00:28:31.560
Because, you know, there are so many steps involved, and the, you know, the configuration of everything is so heavy that basically almost no one can do that.

00:28:32.160 --> 00:28:35.640
But if the experiments host it, host that service, it's very valuable.

00:28:35.640 --> 00:28:36.000
So.

00:28:36.000 --> 00:28:37.280
Yeah, that makes sense.

00:28:37.280 --> 00:28:45.880
I suspect if it's sufficiently complicated and has enough configuration parameters and variation, like, even the original researchers couldn't reproduce it if they didn't have the details, right?

00:28:45.880 --> 00:28:46.400
Yeah.

00:28:46.400 --> 00:28:47.080
Well, I mean.

00:28:47.080 --> 00:28:50.620
I mean, without redoing the work from scratch, literally.

00:28:50.620 --> 00:28:50.920
Yeah.

00:28:51.360 --> 00:28:52.320
No, exactly.

00:28:52.320 --> 00:28:55.180
I mean, the goal is that we should all be able to do it in practice.

00:28:55.180 --> 00:28:57.880
That's, you know, rarely, rarely checked.

00:28:57.880 --> 00:29:04.200
But that's what we're working on now is to try to make it so we can more confidently say, yes, we can actually reproduce this stuff.

00:29:04.200 --> 00:29:05.500
That's super cool.

00:29:05.500 --> 00:29:08.660
I hadn't even thought about Docker, but that makes perfect sense.

00:29:08.660 --> 00:29:21.180
I mean, I always think of Docker as, here's the way I'm going to, like, horizontally scale my web app so I can do that easier or, like, you know, get higher density on servers and web data center type places.

00:29:21.180 --> 00:29:23.960
But it makes really good sense for scientific computing, doesn't it?

00:29:23.960 --> 00:29:24.920
Yeah, yeah.

00:29:24.920 --> 00:29:26.420
So we have, you know, these experiments.

00:29:26.420 --> 00:29:30.340
At this point, you know, each of the big experiments has put out, you know, several hundred papers.

00:29:30.340 --> 00:29:32.760
So you have, like, thousands of scientific results.

00:29:32.760 --> 00:29:46.440
And now, if associated to each one of these results, you have some sort of, like, Docker image and you can, you know, on demand spin up this service that you want to reproduce or reinterpret what was done.

00:29:46.440 --> 00:29:50.240
You end up making a very powerful, high-level scientific tool.

00:29:50.240 --> 00:29:53.760
So it's been an idea for several years.

00:29:53.760 --> 00:29:57.860
But now, with, you know, these tools that are around, it's really, you know, possible.

00:29:58.120 --> 00:30:00.340
And so, and it's starting to happen.

00:30:00.340 --> 00:30:02.080
Yeah, I think that's fantastic.

00:30:02.080 --> 00:30:07.580
And are you guys, I think, considering or actively putting these into the official Docker repository?

00:30:07.580 --> 00:30:18.080
So right now, that's the sort of model is that that's going to be hosted at CERN, partially because, you know, it's a lot of disk space.

00:30:18.080 --> 00:30:20.380
And we're just working closely with the CERN IT people.

00:30:20.380 --> 00:30:24.180
And there's a lot of trust between the CERN computing and IT and the experiments.

00:30:24.180 --> 00:30:26.760
So that's how that's being developed now.

00:30:27.540 --> 00:30:35.500
But, and some of these things are sensitive, you know, like, yeah, these are big international collaborations.

00:30:35.500 --> 00:30:42.620
And different countries are kind of at different places in terms of their attitudes about open science and things like that.

00:30:42.620 --> 00:30:48.680
So some people still want to keep these things closed, but they're willing to entertain the idea of hosting the service.

00:30:48.680 --> 00:30:52.400
So that's a big political discussion.

00:30:53.000 --> 00:30:56.920
Actually, the more I think about it, that it starts to bring me back to when I was in grad school.

00:30:56.920 --> 00:30:58.380
Yes, I can see that.

00:30:58.380 --> 00:31:09.620
So I suspect once a paper is published in a peer-reviewed journal and approved, pretty much everybody would be pretty happy to have their stuff public.

00:31:09.620 --> 00:31:19.440
But as you're developing that, right, before you've declared the Higgs boson to be found, for example, you would necessarily want to give all your algorithms away and let other people take a shot at it, right?

00:31:19.440 --> 00:31:24.980
You want to keep that until you're ready to publish your papers and publicly do your results, right?

00:31:24.980 --> 00:31:26.620
Yeah, no, that's true.

00:31:26.760 --> 00:31:37.260
And also, you know, I mean, the experiments also worry a lot about, you know, you have to be kind of careful about the scientific message that's coming out of these experiments.

00:31:37.260 --> 00:31:40.800
I mean, they're very, very expensive international projects.

00:31:40.800 --> 00:31:46.480
And you don't want people, like, claiming this and that and adding a lot of noise and drama.

00:31:46.480 --> 00:31:53.080
So, you know, we work very hard to get lots and lots of internal cross-checks of things and then make sure everyone's on the same page.

00:31:53.080 --> 00:31:56.160
And then we want to have kind of one unified voice from an experiment.

00:31:56.160 --> 00:31:58.940
And then we have another experiment to check it.

00:31:58.940 --> 00:32:01.720
But we don't want too much noise.

00:32:01.720 --> 00:32:08.440
So, you know, that's a lot of motivation for keeping these things kind of internal until they're ready.

00:32:08.440 --> 00:32:09.820
Yeah, absolutely.

00:32:09.820 --> 00:32:16.460
I mean, it took several years for the Higgs boson analysis to finally declare, hey, we found it, right?

00:32:17.460 --> 00:32:20.160
Yeah, that had a lot to do with just collecting enough data.

00:32:20.160 --> 00:32:28.240
Once we had enough data, the process was sort of streamlined enough that it was really a matter of like a week and a half or something.

00:32:28.240 --> 00:32:36.440
It was about a week and a half that we went from the last data that we had to the talk that, you know, where the discovery claim was made.

00:32:36.440 --> 00:32:40.760
So, but that was in some sense just like adding the last bits of data.

00:32:40.760 --> 00:32:42.820
But, yeah.

00:32:42.820 --> 00:32:44.660
Yeah, yeah, very cool.

00:32:45.060 --> 00:32:50.300
So, I want to ask you a few more things about LHC and CERN and particle physics.

00:32:50.300 --> 00:32:52.460
And then we could talk about space a little bit maybe.

00:32:52.460 --> 00:32:53.320
Oh, sure.

00:32:53.320 --> 00:32:53.600
Yeah.

00:32:53.600 --> 00:33:04.320
So, you said one of your goals there is that you guys are trying to move towards a more sort of dedicated professional software role there at CERN.

00:33:04.320 --> 00:33:05.520
Can you talk about that a little?

00:33:05.520 --> 00:33:06.880
Oh, right.

00:33:06.960 --> 00:33:13.180
So, I think, you know, the physicists, you know, need to write, obviously, a lot of code and be proficient to be able to do the science they want.

00:33:13.180 --> 00:33:18.800
And, but you also, there's a lot of infrastructure for the processing that's needed.

00:33:18.800 --> 00:33:30.960
So, you know, typically what happened was physicists that were very strong on computing kind of specialized on that and then, you know, became sort of software professionals with a physics background.

00:33:31.520 --> 00:33:34.740
And that's, that model has worked, you know, surprisingly well.

00:33:35.320 --> 00:33:44.640
There are some people, you know, relatively few people that are, don't really have a physics background that are really more just, you know, software professionals.

00:33:44.640 --> 00:33:56.960
But separately, there's been, you know, CERN has had like an IT department that deals with actual computing infrastructure that's going on, but has also developed several different tools that were more services.

00:33:56.960 --> 00:33:57.460
Sure.

00:33:57.540 --> 00:33:59.840
Because you guys have a lot of computers, a lot of network.

00:33:59.840 --> 00:34:01.720
How many computers do you have there?

00:34:01.720 --> 00:34:04.100
Oh, the CERN, I should know the number.

00:34:04.100 --> 00:34:06.520
Hundreds, like a hundred thousand or something, right?

00:34:06.520 --> 00:34:07.580
Like really a lot.

00:34:07.580 --> 00:34:08.600
It's a lot, yeah.

00:34:08.600 --> 00:34:16.440
There's a, there's a great YouTube video that I'll, that you can post that associated to it that has a wonderful overview of the whole processing.

00:34:16.440 --> 00:34:17.320
Oh, yeah, that'd be awesome.

00:34:17.320 --> 00:34:21.100
And lots of little factoids that's nicely produced.

00:34:21.100 --> 00:34:24.860
But I'm not going to say, I won't say a bunch of wrong numbers, so.

00:34:24.860 --> 00:34:25.760
Sure.

00:34:25.760 --> 00:34:25.860
Sure.

00:34:25.860 --> 00:34:42.120
This episode is brought to you by CodeShip.

00:34:42.120 --> 00:34:50.960
CodeShip has launched organizations, create teams, set permissions for specific team members, and improve collaboration in your continuous delivery workflow.

00:34:51.560 --> 00:34:56.860
Maintain centralized control over your organization's projects and teams with CodeShip's new organizations plan.

00:34:56.860 --> 00:35:02.400
And as Talk Python listeners, you can save 20% off any premium plan for the next three months.

00:35:02.400 --> 00:35:05.920
Just use the code TALKPYTHON, all caps, no spaces.

00:35:06.480 --> 00:35:11.660
Check them out at CodeShip.com and tell them thanks for supporting the show on Twitter where they're at CodeShip.

00:35:11.660 --> 00:35:20.760
But the, but there were people that were working on, on that side.

00:35:20.760 --> 00:35:33.340
And then also just general services for like, you know, like the system we use to manage all the papers and the, and the comments to the papers and things like that.

00:35:33.340 --> 00:35:39.720
And like an agenda system where people put up their talks and, you know, schedule meetings and the video services and blah, blah, blah, blah, blah.

00:35:40.160 --> 00:35:47.920
And, and those, the people that were working on that have started branching out into the kinds of services they provide.

00:35:47.920 --> 00:35:54.520
So, so I guess one example that's made me interesting is the very first website in the United States was called Spires.

00:35:54.880 --> 00:36:03.840
And it is, preceded this, the archive where like scientists put the versions of their papers before they're published, this preprint server.

00:36:03.840 --> 00:36:09.680
So the, the Spires was basically just a database of all of the, high energy physics literature.

00:36:09.680 --> 00:36:13.620
You know, like who wrote what papers and who cited whom and da, da, da, da, da.

00:36:13.620 --> 00:36:23.300
And, and so it was like a, it was the first website in the United States was where physicists would go to see where, you know, where the, who's citing who and which papers are about certain topics.

00:36:23.500 --> 00:36:26.640
And that has evolved into something new called Inspire.

00:36:26.640 --> 00:36:29.360
And that's a, you know, an international effort.

00:36:29.360 --> 00:36:42.800
Um, but the, the software technology that that's based on, which is mainly Flask based now and, you know, Pythonic, based things, are, has evolved into a set of different tools that are pretty nice.

00:36:42.800 --> 00:36:47.440
So one of them is a data repository where, this is not solving actually CERN's problem.

00:36:47.440 --> 00:36:49.080
It's like the long tail of science.

00:36:49.080 --> 00:36:58.140
So all these small experiments that are out there that have some data, they can put their data onto these servers and then they can refer to that data when they write their papers.

00:36:58.940 --> 00:37:19.620
So there's a service called Zenodo and Zenodo started working with GitHub and made it so that people that now, not just data that you want to point to when you're writing a paper, but the software of your paper, you can, there's a little web hook in, in GitHub that whenever you make a new release, we'll push a copy of the, of the code to the service of Zenodo.

00:37:19.620 --> 00:37:23.340
So, and they'll, they'll, they'll, the, the jargon is meant a DOI.

00:37:23.340 --> 00:37:28.640
So it's a digital object identifier, which the publishing industry knows about how to use and things like that.

00:37:28.640 --> 00:37:34.280
And it points to a specific version of the, of the code that was used and you can download that version of the code.

00:37:34.280 --> 00:37:37.060
So it's sort of out of the realm of version control.

00:37:37.060 --> 00:37:47.140
It's more, it's just that GitHub is not provided, like people could delete their GitHub repo, but once you get a DOI, it's kind of a, a trust relationship that that code is exactly.

00:37:47.140 --> 00:37:49.060
that code is always going to be there at that link.

00:37:49.060 --> 00:37:49.500
Right.

00:37:49.500 --> 00:37:51.720
Almost like escrow for your code.

00:37:51.720 --> 00:37:53.200
Yeah, exactly.

00:37:53.200 --> 00:37:57.080
So it's redundant in a, in a sense, but you know, there's a link back obviously to the repository.

00:37:57.080 --> 00:38:12.160
So you can see how it's evolved, but, but this is the, the, you know, the basic common denominator for all of the publishing industry and how to track citations to, to different, you know, research outputs, whether they be papers or data or software.

00:38:12.160 --> 00:38:15.580
So that, so that connection has been, you know, is nice.

00:38:15.580 --> 00:38:23.640
And now there are actually thousands of, pieces of code that have been pushed from GitHub to Zenodo where they, and been given these DOIs.

00:38:23.880 --> 00:38:34.700
And the same, that is also, you know, Flask based and, and they have been very nice about how they break up the project, into different, you know, Flask components and things like that.

00:38:34.820 --> 00:38:44.540
And now we're reusing that same kind of infrastructure to try to build these tools for doing a reproducibility and, reinterpretations and things like that.

00:38:44.540 --> 00:38:59.940
So there's a, there's a new effort that's evolving that has a, you know, much more of a like modern, you know, Pythonic programming, you know, web services mentality, which is not necessarily coming from within the experiments, but coming from within the CERN lab.

00:39:00.140 --> 00:39:06.640
And, and I think it's going to change how the experiments sort of approach these problems, in the next few years.

00:39:06.640 --> 00:39:07.480
That's really cool.

00:39:07.480 --> 00:39:10.800
It seems like a huge step down the right path for reproducibility.

00:39:10.800 --> 00:39:14.520
Yeah, no, I think it's very, very important, for what we're doing.

00:39:14.520 --> 00:39:15.780
Yeah, of course.

00:39:15.780 --> 00:39:16.160
Of course.

00:39:16.160 --> 00:39:16.760
Okay.

00:39:16.760 --> 00:39:26.960
The last thing I wanted to ask you about, was this project called Recast that you guys have going that, not you necessarily, but folks at CERN have an LHC of going.

00:39:27.480 --> 00:39:33.540
So Recast is the system that I was referring to that is, for trying to make the analyses reinterpretable.

00:39:33.540 --> 00:39:46.260
Um, and that's the one where we bring in these sort of Docker images for the, for the different analysis pipelines and try to make this web service, for being able to either reproduce or reinterpret, one of the published LHC analyses.

00:39:46.260 --> 00:39:54.700
So that one is, is definitely within CERN, but using a lot of these more modern, you know, web service, type approaches.

00:39:54.700 --> 00:40:00.580
There was another, another project that maybe, you also wanted to talk about, which was the space one.

00:40:00.580 --> 00:40:01.480
I don't know if that's what.

00:40:01.480 --> 00:40:02.260
Yeah, absolutely.

00:40:02.260 --> 00:40:02.900
So.

00:40:02.900 --> 00:40:05.340
Or I don't know if there's a, we're running out of time.

00:40:05.340 --> 00:40:07.240
No, no, that, yeah, no, we definitely have time.

00:40:07.240 --> 00:40:08.100
Let's talk about that one.

00:40:08.100 --> 00:40:09.660
And that's, Kreyfis.

00:40:09.660 --> 00:40:10.240
Yeah.

00:40:10.340 --> 00:40:10.680
That's right.

00:40:10.680 --> 00:40:11.420
That's right.

00:40:11.420 --> 00:40:16.080
So, Kreyfis is this project, which is, we started up, which is a lot of fun.

00:40:16.080 --> 00:40:20.480
Uh, and it's a very, very sort of small team of people right now.

00:40:20.480 --> 00:40:23.520
So it's, it's kind of the opposite of these LHC experiments.

00:40:23.520 --> 00:40:30.540
Um, and our basic idea is that we, we want to try to detect very, you know, ultra high energy cosmic rays.

00:40:30.540 --> 00:40:34.900
So these are particles coming from outer space that have much higher energies than the LHC.

00:40:34.900 --> 00:40:39.500
And they smack the atmosphere and create a shower of new particles.

00:40:39.500 --> 00:40:42.800
So you get, you get tons and tons of particles hitting the ground.

00:40:42.800 --> 00:40:54.400
And the idea is that if you get a bunch of people running an app on their phone, the camera of your phone, the little CMOS sensor in your camera is like a miniature particle detector.

00:40:54.400 --> 00:40:59.180
So if one of these particles happens to fly through your camera, it will light your camera up.

00:40:59.180 --> 00:41:09.480
And so basically the idea is that if you get a bunch of people at night running this app, like while their phone's plugged in and they're sleeping, and you see a bunch of these phones light up at the same time,

00:41:09.480 --> 00:41:14.820
then that's, like an indication they just, you just got hit by a high energy cosmic ray.

00:41:15.220 --> 00:41:17.260
And, and, these things are very rare.

00:41:17.260 --> 00:41:24.160
So if you want to see them, like the, one of the best shots you have is to try to cover as much of the earth with a detector as you can.

00:41:24.160 --> 00:41:35.780
And, and it looks like if we have, you know, in the order of a million people running this app, which is a big number, but you know, not crazy big number, that we might be able to do some like really cutting edge science.

00:41:35.780 --> 00:41:42.040
And, and be able to study like the very highest energy particles that people have ever seen in the universe.

00:41:42.040 --> 00:41:49.020
So, so it's a really fun, you know, like citizen science project and it has a lot of data science challenges.

00:41:49.020 --> 00:41:50.260
I'm sure it does.

00:41:50.260 --> 00:41:51.140
That's, that's really cool.

00:41:51.140 --> 00:41:55.480
So does it run on the major phone platforms, iPhone, Android, things like that?

00:41:55.480 --> 00:41:56.280
Yeah.

00:41:56.280 --> 00:42:01.360
So we have an Android app and an, and an iPhone app that they're both kind of in beta right now.

00:42:01.440 --> 00:42:06.120
Um, and, and so they collect data and, you know, one phone can't say anything.

00:42:06.120 --> 00:42:15.220
So they're uploading their data about, sort of when they saw flashes and, and where they were when they saw the flashes to, you know, a central server.

00:42:15.220 --> 00:42:26.540
And, we've, we've just, so it's been fun because we, the people working on this all kind of have a LHC type of background, but we now get a fresh set of choices about what we're going to use.

00:42:26.540 --> 00:42:39.700
So, so, you know, we have Elasticsearch and Django and Spark and, you know, you know, just much, you know, more kind of modern stack of, of, of tools for being able to process the data.

00:42:39.820 --> 00:42:46.900
And then we're working on trying to make it so that the final stages of the data analysis are, are more like IPython notebooks.

00:42:46.900 --> 00:42:53.080
And that we have this ability to, not just view the notebook, but to launch the notebook and recompute it.

00:42:53.080 --> 00:43:03.060
So that's using things like SageMath or this binder project, and, and, and Everware, which have, you know, with Docker, all of the computing environment.

00:43:03.260 --> 00:43:09.740
And this is going to, I think, be just awesome for doing outreach because they're, this has really grabbed the public's imagination.

00:43:09.740 --> 00:43:13.640
We got covered by NPR and Wired and some people.

00:43:13.640 --> 00:43:15.220
And so the, and IFL science.

00:43:15.220 --> 00:43:22.080
So we have like almost a hundred thousand people that have signed up and we're, we haven't even launched the app yet.

00:43:22.080 --> 00:43:22.980
That's really cool.

00:43:22.980 --> 00:43:25.280
So if people want to get started, how do they do it?

00:43:25.280 --> 00:43:27.160
Is that a thing they can do yet?

00:43:27.160 --> 00:43:28.080
Or do we have to wait a while?

00:43:28.760 --> 00:43:30.820
Um, we're going to have to wait a little bit longer.

00:43:30.820 --> 00:43:34.680
Uh, we're, we're working as fast as we can, with our ragtag team.

00:43:34.680 --> 00:43:42.440
But the, but if you, you can sign up for it right now, if you go to the website, it's, crayfish.io.

00:43:42.440 --> 00:43:44.580
So it's like crayfish without the H.

00:43:44.580 --> 00:43:45.060
Right.

00:43:45.060 --> 00:43:47.120
So see, yeah, I'll definitely add a link.

00:43:47.120 --> 00:43:50.220
C R A Y F I S dot IO.

00:43:50.220 --> 00:43:51.120
That's right.

00:43:51.120 --> 00:43:51.540
That's right.

00:43:51.540 --> 00:43:52.320
Fantastic.

00:43:52.600 --> 00:44:02.360
So this, you know, reminds me a little bit of the, the whole protein folding stuff that people try to do with like running it on idle computers and the study at home.

00:44:02.360 --> 00:44:06.400
But, but it seems even cooler because this is actually the science.

00:44:06.400 --> 00:44:07.060
That's right.

00:44:07.060 --> 00:44:07.440
Yeah.

00:44:07.440 --> 00:44:08.300
Happening.

00:44:08.300 --> 00:44:11.260
Not just the post analysis computational bits.

00:44:11.260 --> 00:44:12.000
That's right.

00:44:12.000 --> 00:44:12.160
Yeah.

00:44:12.160 --> 00:44:15.280
I think there's sort of three types of citizen science projects like this.

00:44:15.280 --> 00:44:23.960
There are ones like study at home and the protein folding where you're basically donating compute cycles, you know, to solve a problem, but you're not really involved in otherwise.

00:44:23.960 --> 00:44:33.440
Um, then there are projects like, there's a great platform called the Zooniverse, which I encourage people to go check out if you like these things.

00:44:33.440 --> 00:44:35.120
So they are the very interactive.

00:44:35.120 --> 00:44:44.860
You're like looking at pictures of galaxies and trying to, to classify them or, you know, like pictures of the ocean and trying to say if you see certain things in them.

00:44:44.920 --> 00:44:52.620
And it's, it's a lot like mechanical Turk, you know, you, so you, you're farming out, these things that computers might not be great at.

00:44:52.620 --> 00:44:56.960
We don't have, we're not good at writing algorithms for doing and having humans do it.

00:44:56.960 --> 00:45:00.480
Um, and this one, this project is neither of those.

00:45:00.480 --> 00:45:02.520
The citizen science is actually the instrument.

00:45:02.520 --> 00:45:04.860
I mean, you are, you are doing the science.

00:45:04.860 --> 00:45:10.340
And, so that's, that's, yeah, I think, you know, that makes our project really unique.

00:45:10.340 --> 00:45:11.100
Yeah.

00:45:11.100 --> 00:45:11.700
That's really awesome.

00:45:11.700 --> 00:45:15.840
So I encourage everyone out there to go sign up, participate.

00:45:15.840 --> 00:45:16.920
I'll definitely do it.

00:45:16.920 --> 00:45:17.800
This is exciting.

00:45:17.800 --> 00:45:18.180
Great.

00:45:18.180 --> 00:45:18.600
Thank you.

00:45:18.600 --> 00:45:19.700
Yeah.

00:45:19.700 --> 00:45:25.400
And then you said this is somewhat, done with a, a grant from Amazon web services.

00:45:25.400 --> 00:45:26.600
That's right.

00:45:26.600 --> 00:45:26.800
Yeah.

00:45:26.840 --> 00:45:31.080
So, so we talked to Amazon and they thought it was an exciting, you know, an exciting project.

00:45:31.080 --> 00:45:43.800
And so they, they've been great about trying to support some, in different, you know, scientific projects that, that, need some computing, that, you know, have, that are still in a prototyping stage.

00:45:44.040 --> 00:45:50.580
And, so they, yeah, they've given us a grant for some of the, you know, AWS, sort of budget.

00:45:50.580 --> 00:45:57.520
And, and we're using that for, you know, collecting the data and some of our, you know, you know, offline processing of the data.

00:45:57.520 --> 00:46:07.240
Uh, and, and, and we try to make it so that there's a kind of a feedback loop so that the, you know, the Django server is showing stats on what the current status of the network is.

00:46:07.240 --> 00:46:11.820
Um, and, so it's, it, that's been very nice of them.

00:46:11.820 --> 00:46:23.960
And also Yandex, which is like a sort of the, the Russian Google is, has a group of people that are data scientists, and part of a lab that they run that are very well integrated at CERN.

00:46:23.960 --> 00:46:33.680
And some of them have joined our project and they're, you know, helping us, you know, build the, the, the, sort of offline spark based, you know, data processing system and things like that.

00:46:33.680 --> 00:46:39.160
So it's a lot of fun to have, you know, these different mix of people and a very fresh project.

00:46:39.160 --> 00:46:40.100
We move very fast.

00:46:40.100 --> 00:46:40.520
It's nice.

00:46:40.520 --> 00:46:40.880
Yeah.

00:46:40.880 --> 00:46:41.440
How exciting.

00:46:41.440 --> 00:46:45.960
I mean, I know how it is to pick up a project that's been around for 10 or 20 years.

00:46:45.960 --> 00:46:52.400
It's kind of stale in the technology and what you can do because you don't want to break stuff, but it's nice to start over and do exciting things, right?

00:46:52.400 --> 00:46:53.380
Absolutely.

00:46:53.380 --> 00:46:53.740
Yeah.

00:46:53.740 --> 00:46:54.720
It's a breath of fresh air.

00:46:54.720 --> 00:46:55.540
Cool.

00:46:55.540 --> 00:46:57.220
So we're kind of getting short on time.

00:46:57.220 --> 00:47:00.080
Let me ask you just a few more questions before we wrap things up.

00:47:00.080 --> 00:47:08.840
What do you think the most amazing or interesting thing about the Large Hadron Collider and all the stuff you guys did there that people might not really know about?

00:47:08.840 --> 00:47:10.140
Oh boy.

00:47:10.140 --> 00:47:18.960
Part of it, I think that's really great about the whole project is just that, you know, it, it, we have people from all around the world, right?

00:47:18.960 --> 00:47:26.000
And they're not all employed by one, you know, one, you know, central, you know, corporation or something like that.

00:47:26.000 --> 00:47:28.180
And so, and we're all contributing.

00:47:28.180 --> 00:47:36.140
So I guess in that sense, it has a lot of the flavor of like a successful open science, I mean, it's open software project where people are contributing from all around.

00:47:36.400 --> 00:47:45.400
And I think the other part is that, you know, everyone should feel proud about it because, you know, it's, you know, it's not a huge amount of your tax dollars.

00:47:45.400 --> 00:47:52.920
You know, if you look at like, I forget for every thousand dollars of taxes you pay, it's just a few cents or something is going to this kind of project.

00:47:52.920 --> 00:47:54.740
But, you know, everyone made this happen.

00:47:54.740 --> 00:47:59.320
And I think everyone in the world should be proud of the fact that this, this happened.

00:47:59.320 --> 00:48:02.060
Yeah, it's definitely a global effort, which is awesome.

00:48:02.060 --> 00:48:03.480
It is a global effort.

00:48:03.480 --> 00:48:10.840
And it's, you know, this is a, I mean, it's a really big accomplishment what's happened in terms of our understanding of the universe.

00:48:10.840 --> 00:48:18.220
So, and there's obviously all of the software and computing and the technical challenges of pulling it off.

00:48:18.220 --> 00:48:22.920
But, you know, it's a big multi-decade global thing.

00:48:22.920 --> 00:48:25.100
And, you know, I don't know, humanity should be proud of itself.

00:48:25.100 --> 00:48:27.220
Yeah, absolutely.

00:48:27.220 --> 00:48:27.960
That's great.

00:48:27.960 --> 00:48:36.040
How do you see Python in science sort of evolving, growing, changing, shrinking?

00:48:36.040 --> 00:48:39.960
I mean, right now, I think it's growing very fast.

00:48:40.080 --> 00:48:47.220
I mean, you know, the whole scientific Python side of things and, you know, distributions that are working, working better.

00:48:47.220 --> 00:48:52.160
And GitHub is changing, you know, how people are doing science very, very quickly.

00:48:52.160 --> 00:49:01.980
If you look in, like, the astronomy and astrophysics community, there have been, you know, some big efforts to create new analysis platforms.

00:49:01.980 --> 00:49:17.040
There's a great project called AstroPy, which is a, you know, just a poster boy for, you know, for different, for, you know, kind of open science mentality of the community contributing to making the tools they need to do the science that they want.

00:49:18.040 --> 00:49:25.340
And the, that's working pretty well because, you know, a lot of those experiments, they have a lot of new experiments coming up and they want a fresh start.

00:49:26.120 --> 00:49:35.180
So, you know, we have this issue that we have a lot of legacy code and, you know, a lot of, you know, a lot of inertia about how we've done things.

00:49:35.260 --> 00:49:43.040
But we're, we're trying to move things into a more, you know, a less siloed software picture that looks more like that model.

00:49:43.040 --> 00:49:53.320
And, and Python is a big, I think Python is like the dominant kind of language of people that have that mentality and at least in, you know, particle physics and astrophysics.

00:49:53.320 --> 00:49:55.680
Yeah, that, that's my feeling as well.

00:49:55.680 --> 00:49:57.560
But obviously I do a lot less science than you do.

00:49:57.560 --> 00:49:59.000
So that's, that's great to hear.

00:50:00.200 --> 00:50:12.880
Yeah, I mean, I think the other place that you see it is just that to the extent that data science is attached to doing sciences, as opposed to like business analytics, you know, you see just a big influx of Python and R.

00:50:12.880 --> 00:50:25.860
And so I think that, you know, there are a lot of, a lot of other scientific fields that are undergoing a similar, that are undergoing a similar transformation, but, but maybe a little bit heavier on the R side than on the Python side.

00:50:25.860 --> 00:50:28.420
Yeah, it's definitely down to those two, I think.

00:50:28.820 --> 00:50:31.300
And modern analysis for the most part.

00:50:31.300 --> 00:50:31.940
Right.

00:50:31.940 --> 00:50:34.980
Kyle, thank you for being on the show and taking the time.

00:50:34.980 --> 00:50:38.340
I know you're super busy and this, this is really, really interesting.

00:50:38.340 --> 00:50:39.340
Thanks for sharing it with everyone.

00:50:39.340 --> 00:50:40.140
No, absolutely.

00:50:40.140 --> 00:50:41.020
Thank you for inviting me.

00:50:41.020 --> 00:50:41.380
It's great.

00:50:41.380 --> 00:50:42.060
Yeah, you bet.

00:50:42.060 --> 00:50:42.860
Take care.

00:50:42.860 --> 00:50:43.120
Bye.

00:50:43.120 --> 00:50:47.180
This has been another episode of Talk Python to Me.

00:50:47.180 --> 00:50:51.380
Today's guest was Kyle Cranmer and this episode has been sponsored by Hired and CodeShip.

00:50:51.380 --> 00:50:53.280
Thank you guys for supporting the show.

00:50:53.280 --> 00:50:55.920
Hired wants to help you find your next big thing.

00:50:55.920 --> 00:51:03.940
Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $4,000.

00:51:03.940 --> 00:51:07.480
CodeShip wants you to always keep shipping.

00:51:07.480 --> 00:51:11.340
Check them out at CodeShip.com and thank them on Twitter via at CodeShip.

00:51:11.600 --> 00:51:13.300
Don't forget the discount code for listeners.

00:51:13.300 --> 00:51:13.860
It's easy.

00:51:13.860 --> 00:51:14.720
Talk Python.

00:51:14.720 --> 00:51:15.420
All caps.

00:51:15.420 --> 00:51:16.120
No spaces.

00:51:16.120 --> 00:51:23.220
You can find the links from today's show at talkpython.fm/episodes slash show slash 29.

00:51:23.220 --> 00:51:25.500
Be sure to subscribe to the show.

00:51:25.500 --> 00:51:27.780
Open your favorite podcatcher and search for Python.

00:51:27.780 --> 00:51:28.960
We should be right at the top.

00:51:28.960 --> 00:51:33.140
You can also find the iTunes and direct RSS feeds in the footer of the website.

00:51:33.900 --> 00:51:38.280
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

00:51:38.280 --> 00:51:41.700
You can hear the entire song at talkpython.fm.

00:51:41.700 --> 00:51:43.840
This is your host, Michael Kennedy.

00:51:43.840 --> 00:51:44.880
Thanks for listening.

00:51:44.880 --> 00:51:46.900
Smix, take us out of here.

00:51:47.900 --> 00:51:49.040
Staying with my voice.

00:51:49.040 --> 00:51:50.840
There's no norm that I can fill within.

00:51:50.840 --> 00:51:52.060
Haven't been sleeping.

00:51:52.060 --> 00:51:53.660
I've been using lots of rest.

00:51:53.660 --> 00:51:56.500
I'll pass the mic back to who rocked it best.

00:51:56.500 --> 00:52:07.960
Developers, Developers, Developers, Developers, Developers, Developers, Developers, Developers,

00:52:07.960 --> 00:52:08.460
you

00:52:08.460 --> 00:52:10.460
.