From 49c11e2e2243e03a872e86e8fe51b6b8d05f59d7 Mon Sep 17 00:00:00 2001 From: Manikantagit Date: Wed, 20 Oct 2021 23:46:27 +0530 Subject: [PATCH] 338-cibuildwheel-scikit-hep.txt----- updated --- transcripts/338-cibuildwheel-scikit-hep.txt | 370 ++++++++++---------- 1 file changed, 184 insertions(+), 186 deletions(-) diff --git a/transcripts/338-cibuildwheel-scikit-hep.txt b/transcripts/338-cibuildwheel-scikit-hep.txt index 27ed8e58..e6a7435d 100644 --- a/transcripts/338-cibuildwheel-scikit-hep.txt +++ b/transcripts/338-cibuildwheel-scikit-hep.txt @@ -12,31 +12,31 @@ 00:00:34 On this episode, you'll meet Henry Schreiner. -00:00:36 He's developing tools for the next era of the large Hadron Collider and is an admin of Scikit. +00:00:36 He's developing tools for the next era of the Large Hadron Collider and is an admin of Scikit. -00:00:42 Hep, of course, CI build wheel is central to that process. +00:00:42 Hep, of course, cibuild wheel is central to that process. -00:00:47 This is Talk Python episode 338, recorded October 14, 2021. +00:00:47 This is Talk Python to Me episode 338, recorded October 14, 2021. -00:01:05 Welcome to Talk Python, a weekly podcast on Python. +00:01:05 Welcome to Talk Python to Me, a weekly podcast on Python. 00:01:08 This is your host, Michael Kennedy. -00:01:10 Follow me on Twitter where I'm at Mckinnedy and keep up with the show and listen to past episodes at Talk Python FM and follow the show on Twitter via at Talk Python. +00:01:10 Follow me on Twitter where I'm @mkennedy and keep up with the show and listen to past episodes at 'Talk Python.FM' and follow the show on Twitter via @talkpython. -00:01:19 We started streaming most of our episodes live on YouTube, subscribe to our YouTube channel over at Talk Python FM YouTube to get notified about upcoming shows and be part of that episode. +00:01:19 We started streaming most of our episodes live on YouTube, subscribe to our YouTube channel over at 'TalkPython.FM/youtube' to get notified about upcoming shows and be part of that episode. 00:01:31 Hey there. I have some exciting news to share before we jump into the interview. 00:01:34 We have a new course over at Talk Python. -00:01:36 Htmx Plus Flask modern Python Web apps Hold the JavaScript HTMX is one of the hottest properties in web development today, and for good reason, you might even remember all the stuff we talked about with Carson Gross back on episode 321 HTMX along with the libraries and techniques we introduced in our new course, will have you writing the best Python Web apps you've ever written, clean, fast and interactive all without that front end overhead. +00:01:36 Htmx Plus Flask modern Python Web apps Hold the JavaScript. HTMX is one of the hottest properties in web development today, and for good reason, you might even remember all the stuff we talked about with Carson Gross back on episode 321 HTMX along with the libraries and techniques we introduced in our new course, will have you writing the best Python Web apps you've ever written, clean, fast and interactive all without that front end overhead. 00:02:02 If you're a Python Web developer that has wanted to build more dynamic interactive apps, but don't want to or can't write a significant portion of your app enriched front end JavaScript frameworks, you'll absolutely love HTMX. -00:02:15 Check it out over at Talkbython FM HTMX or just click the link in your podcast player show notes. +00:02:15 Check it out over at 'Talkbython.FM/htmx or just click the link in your podcast player show notes. -00:02:21 Now let's get onto that interview, Henry, welcome to Talk by phone to me. +00:02:21 Now let's get onto that interview, Henry, welcome to Talk Python to me. 00:02:25 Thank you. @@ -68,13 +68,13 @@ 00:03:22 Well, before we get into all that stuff, though, let's start with your story. How do you get into programming in Python? -00:03:27 I started with a little bit of programming that my dad taught me. He was a physicist, and I remember it was C Plus Plus and sort of taught the way you teach Java, all objects and classes just a little bit. +00:03:27 I started with a little bit of programming that my dad taught me. He was a physicist, and I remember it was C++ and sort of taught the way you teach Java, all objects and classes just a little bit. -00:03:41 And then when I started at College and I wanted to take classes, and I took a couple of classes again in C Plus Plus, I just really loved objects and classes. +00:03:41 And then when I started at college and I wanted to take classes, and I took a couple of classes again in C++, I just really loved objects and classes. 00:03:52 Unfortunately, the courses didn't actually cover that much, but the book did. So I really got into that. -00:03:57 And then for Python, actually, right. When I started College, I started using this program called Blender. +00:03:57 And then for Python, actually, right when I started College, I started using this program called Blender. 00:04:02 Oh, yeah. @@ -88,7 +88,7 @@ 00:04:13 So I knew it had this built in language called Python. So that made me really want to learn Python. -00:04:17 And then when I went to research experience for undergraduates at Northwestern University in Chicago, and when I was there, we had this cluster that we were working on, this was in Salt State physics, material physics. +00:04:17 And then when I went to research experience for undergraduates at Northwestern University in Chicago, and when I was there, we had this cluster that we were working on, this was in Solid State Physics, material physics. 00:04:32 And we would launch these simulations on the cluster. @@ -114,7 +114,7 @@ 00:05:29 You can sort of do things that normally I think the programming gets in the way more, but Python tends to stay out. It looks more like pseudocode, so you can do more and learn more. -00:05:38 And eventually you can go do it in C Plus or something. +00:05:38 And eventually you can go do it in C++ or something. 00:05:41 Yeah. @@ -124,15 +124,15 @@ 00:05:45 Sometimes you do need to go do it in some other language, and sometimes you don't. -00:05:48 I think the stuff at CERN and LHC has an interesting exchange between C Plus Plus and maybe some more Python. +00:05:48 I think the stuff at CERN and LHC has an interesting exchange between C++ and maybe some more Python. 00:05:56 And whatnot so that'll be fun to talk about. 00:05:59 Yeah. -00:05:59 We've been C Plus Plus originally, but Python is really showing up in a lot more places, and there's been a lot of movement in that direction. +00:05:59 We've been C++ originally, but Python is really showing up in a lot more places, and there's been a lot of movement in that direction. -00:06:08 There's been some really interesting things that have come out. A lot of interesting things have come out of the Lac computing wise, as well as awesome. +00:06:08 There's been some really interesting things that have come out. A lot of interesting things have come out of the LAC computing wise, as well as awesome. 00:06:14 Yeah. @@ -146,9 +146,7 @@ 00:06:29 You talked about your dad teaching a little programming. -00:06:31 If people are out there and they're the dad, they want to teach their kids a little bit of programming. I want to give a shout out to CodeCombat. - -00:06:38 Com. +00:06:31 If people are out there and they're the dad, they want to teach their kids a little bit of programming. I want to give a shout out to CodeCombat.com 00:06:38 Such a cool place. @@ -182,13 +180,13 @@ 00:07:40 Yeah. And this is at CERN or your University or where is this? -00:07:44 So now I'm at Princeton University, and I'm part of a local group of RSEs research software engineers, and I'm also part of Irish, which will talk about a little bit, but that's sort of a very spread out group. +00:07:44 So now I'm at Princeton University, and I'm part of a local group of RSEs Research Software Engineers, and I'm also part of Irish, which will talk about a little bit, but that's sort of a very spread out group. 00:08:00 Some of us are at CERN, a few or in some other places. 00:08:05 Fermi lab and physicists are just used to working remote. -00:08:09 The dynamic wasn't that big of a change for us. We were already doing all our meetings remote. We just eventually changed from video to Zoom. But other than that. +00:08:09 The pandemic wasn't that big of a change for us. We were already doing all our meetings remote. We just eventually changed from video to Zoom. But other than that. 00:08:17 Exactly, it was real similar for me as well. That's interesting. @@ -210,7 +208,7 @@ 00:08:41 And it's a really neat place. -00:08:43 It is. Cern is a neat place, too. +00:08:43 It is. CERN is a neat place, too. 00:08:44 I would love to tour CERN, but it wasn't 20 minutes down the street from where I happen to be. @@ -220,13 +218,13 @@ 00:08:54 All right. -00:08:54 Well, let's talk about sort of the psychic Hep side of things and how you got into maintaining all of these packages. +00:08:54 Well, let's talk about sort of the scikit-HEP side of things and how you got into maintaining all of these packages. 00:09:05 So you found yourself in this place where you're working on tools that help other people build packages for the physicists and data scientists, and so on. Right. 00:09:14 So Where'd that all start? -00:09:16 So with maintenance itself, the first thing I started maintaining was a package called Plundham back in 2015, and at that point, I was starting to submit some PRS, and the author came to me and said I would like to have somebody do the releases. +00:09:16 So with maintenance itself, the first thing I started maintaining was a package called 'Plumbum' back in 2015, and at that point, I was starting to submit some PRS, and the author came to me and said I would like to have somebody do the releases. 00:09:33 I need a release manager. @@ -236,7 +234,7 @@ 00:09:53 And then I was working on physics tool called Goofy when I became a postdoc, and I worked on sort of really renovating that. It started out as a code written by physicists, and I worked on making it actually installable and packaged nicely and worked with a student to add Python bindings to it, things like that. -00:10:15 And as part of that, I wrote a C Plus Plus package, CLI Eleven first package I actually wrote and then maintained and section C, and it was written for good fit, but now it's fairly. +00:10:15 And as part of that, I wrote a C++ package, CLI 11 first package I actually wrote and then maintained and section C++, and it was written for good fit, but now it's fairly. 00:10:28 I think it's done pretty well on its own. @@ -272,13 +270,13 @@ 00:11:31 I definitely think it's included now, which is great. -00:11:34 So CLI Eleven, this is a C Plus Plus Eleven command line parser. +00:11:34 So CLI 11, this is a C++ 11 command line parser. 00:11:39 Right. -00:11:39 Like click or Arg pars or something like that. But for C Plus Plus, right? +00:11:39 Like click or Arg pars or something like that. But for C++, right? -00:11:43 Yes. It was designed off of the Pumbum command line Parcer on sort of a toolkit that has several different things. +00:11:43 Yes. It was designed off of the Plumbum command line Parser on sort of a toolkit that has several different things. 00:11:49 I wish those things had been pulled out because I think on their own they might have maybe even been popular on their own. @@ -294,13 +292,13 @@ 00:12:22 It's a bit challenging. I think in Python, there's a really strong emphasis to having the individual separate pieces and packages, especially in Python, partially because it has a really good packaging system and being able to take things have just pieces and be able to swap out one that you don't like is really nice. -00:12:41 And that's one of the things we'll talk about the pipe as well. And that's one of the things that they focus on is small individual packages that each do a job versus all in one poetry. +00:12:41 And that's one of the things we'll talk about the PyPI as well. And that's one of the things that they focus on is small individual packages that each do a job versus all in one poetry. 00:12:51 Yeah. 00:12:52 Well, you'll have to do some checking or some fact checking. Balancing modernizing. -00:12:58 For me. I did professional C Plus Plus development for a couple of years, and I really enjoyed it until there were better options. +00:12:58 For me. I did professional C++ development for a couple of years, and I really enjoyed it until there were better options. 00:13:05 And then I was like, Why am I still doing this? @@ -308,7 +306,7 @@ 00:13:09 But one of the things that struck me as a big difference to that world is basically the number of libraries you use. -00:13:17 The granularity of the libraries you use, the relative acceptance of things like Pip and the ease of using another library, right in C. +00:13:17 The granularity of the libraries you use, the relative acceptance of things like Pip and the ease of using another library, right in C++. 00:13:27 You've got the header and you've got the Linked file and you've got the DLL. @@ -316,17 +314,17 @@ 00:13:40 Is that still true? I feel like that difference is one of the things that allows for people to make these smaller composable pieces in Python. -00:13:47 I think that has a lot to do with it. What has happened in C Plus Plus is there's sort of a rise of a lot of header only libraries, and these libraries are a lot easier to just drop into your project because all you do is you put in the headers and you don't have to deal with a lot of the original issues. So a lot of these small standalone libraries are header only. +00:13:47 I think that has a lot to do with it. What has happened in C++ is there's sort of a rise of a lot of header only libraries, and these libraries are a lot easier to just drop into your project because all you do is you put in the headers and you don't have to deal with a lot of the original issues. So a lot of these small standalone libraries are header only. -00:14:10 And one of the next things that I picked up as a maintainer was Pi Bind Eleven, and I've sort of been in that space between C Plus Plus and Python for quite a bit. +00:14:10 And one of the next things that I picked up as a maintainer was Pybind11, and I've sort of been in that space between C++ and Python for quite a bit. 00:14:22 Kind of like being in that area. 00:14:25 Joining the two from listening to the things that you've worked on previously and things like this that you're interested in connecting and enabling piecing together. -00:14:34 Like here's my script that's going to pull together the compute on this cluster or here's this library that pulls together Python and C Plus and so on. +00:14:34 Like here's my script that's going to pull together the compute on this cluster or here's this library that pulls together Python and C++ and so on. -00:14:42 Yes, making different things work together and combining things like Sequel Flows and Python, or combining different packages in Python and piecing together a solution. +00:14:42 Yes, making different things work together and combining things like C++ and Python, or combining different packages in Python and piecing together a solution. 00:14:50 I think that's one of Python strengths versus something like MATLAB. It's been quite a bit of time in MATLAB early on and got to move a lot of stuff over to Python. @@ -336,43 +334,43 @@ 00:15:02 I know it's so expensive, and then you get the what are they called toolkits, the Add on tool kits, and they're like, each tool kit is the price of another $1,000 a year or $2,000 a year. It's ridiculous. -00:15:14 So I know of CFI, which is a way for Python and C to get clicked together in a simple way. +00:15:14 So I know of CFFI, which is a way for Python and C to get clicked together in a simple way. -00:15:25 How's high Bind Eleven fit into that? This is seamless interoperability between C Plus Plus Eleven and Python. +00:15:25 How's Pybind11 fit into that? This is seamless interoperability between C++11 and Python. -00:15:31 How are they different CFI? +00:15:31 How are they different CFFI? -00:15:34 I teach, like a little short course where I can go through the different sort of different binding tools, and it usually ends with me saying Pi by Eleven is my favorite. +00:15:34 I teach, like a little short course where I can go through the different sort of different binding tools, and it usually ends with me saying Pybind11 is my favorite. 00:15:42 Yeah. Cool. -00:15:43 Give us an overview of what the options are and stuff is closer to C types. It's more of it's focused on C versus C Close Plus, and it's actually the one I've used the least. +00:15:43 Give us an overview of what the options are and stuff is closer to C types. It's more of it's focused on C versus C++, and it's actually the one I've used the least. -00:15:54 I was just helping just talking with the CFI developer, but I've used it the least of those, but I think it basically parses your C headers and then automates a lot of what you would have to manually do a C type, so you have to specify what symbol you want to call and what the arguments are and what the return type is. And if one of those things is wrong, you get a SEG fault and that sort of thing. +00:15:54 I was just helping just talking with the CFFI developer, but I've used it the least of those, but I think it basically parses your C headers and then automates a lot of what you would have to manually do a C type, so you have to specify what symbol you want to call and what the arguments are and what the return type is. And if one of those things is wrong, you get a SEG fault and that sort of thing. -00:16:17 Whereas Pi by Eleven. +00:16:17 Whereas Pybind11. 00:16:19 This is about building modules, extension modules. -00:16:22 And the interesting thing about this is that it's written in Piercing Plus plus the other tools out there so Python can do this. It's not what it was designed for, but immediately became popular for doing this because Python turned code. +00:16:22 And the interesting thing about this is that it's written in pure C++ the other tools out there so Cython can do this. It's not what it was designed for, but immediately became popular for doing this because Cython turned code. -00:16:37 Python like code is a new language into transported into C or C at a toggle. You could change as a toggle. You can change. +00:16:37 Python like code is a new language into transported into C or C++ at a toggle. You could change as a toggle. You can change. -00:16:46 And then when you're there, you can now call C or C Plus Plus, but it's extremely verbose, and you repeat yourself and you have to learn another language. +00:16:46 And then when you're there, you can now call C or C++, but it's extremely verbose, and you repeat yourself and you have to learn another language. -00:16:54 This weird combined Python thing and just thinking in Python is difficult because you have to think about, well, am I in Python or am I in Python that can that's going to be bound to Python or am I in Python? That's just going straight to C or am I just in C or C? +00:16:54 This weird combined Python thing and just thinking in Cython is difficult because you have to think about, well, am I in Python or am I in Cython that can that's going to be bound to Python or am I in Cython? That's just going straight to C or am I just in C or C++? 00:17:10 But I've actually used it's a lot of layers there. 00:17:12 Yeah. -00:17:12 But Python loan is just C Plus Plus, and it's basically the C API for Python, but C Plus plus API, it's quite natural, and you don't have to learn a new language. It uses a fairly advanced C Plus Plus, but that's it. You're learning something useful anyway. +00:17:12 But Python is just C++, and it's basically the C API for Python, but C++ API, it's quite natural, and you don't have to learn a new language. It uses a fairly advanced C ++, but that's it. You're learning something useful anyway. 00:17:29 Right. 00:17:30 So do you do some sort of like template type thing and then say, I'm going to expose this class to Python or something like that, and then it figures out, does it write the Python code or what is it it's writing the build like so files or what do you do here? -00:17:45 It compiles into the C API calls, and then that would compile into a dot. So there's no such Python or Swig or these other tools because it's just cuppa like you do any other C, but it's actually internally using the C Python API or Pipe's wrapper for it. +00:17:45 It compiles into the C API calls, and then that would compile into a data cell. So there's no such Python or Swig or these other tools because it's just C++ like you do any other C++, but it's actually internally using the C Python API or PyPI's wrapper for it. 00:18:04 And the language looks a lot like Python. The names are similar. @@ -388,23 +386,23 @@ 00:18:26 And for people who haven't used those kind of outputs, basically, it's just import module name, whether it's a PY file or it's a. -00:18:37 PyTorch. If you've used used one of those things, you've been importing some Pi bind Eleven code. +00:18:37 PyTorch. If you've used CyPy used one of those things, you've been importing some Pybind11 code. -00:18:45 So let's talk a little bit about psychedelic. +00:18:45 So let's talk a little bit about Scikit-HEP. -00:18:48 This is one of the projects that has a lot of these packages inside of it. And your library is one of the things that is used to maintain and build all those packages, because I'm sure they have a lot of interesting and oddball dependencies. +00:18:48 This is one of the projects that has a lot of these packages inside of it. And your library Cibuild wheel is one of the things that is used to maintain and build all those packages, because I'm sure they have a lot of interesting and oddball dependencies. 00:19:08 Right. -00:19:09 I mean, C Plus Plus is kind of standard, but there's probably others as well. Right. +00:19:09 I mean, C++ is kind of standard, but there's probably others as well. Right. 00:19:13 It is. -00:19:14 One thing that is kind of somewhat unique to help is that we are very heavily invested in C. +00:19:14 One thing that is kind of somewhat unique to help is that we are very heavily invested in C++. -00:19:20 It's usually either you're going to see Python or you're going to see some sort of C Plus Plus package of some sort. +00:19:20 It's usually either you're going to see Python or you're going to see some sort of C++ package of some sort. -00:19:26 It could be varies in size there, but it's mostly C Plus Plus or Python. We really haven't used other languages as much for the past 39 years. +00:19:26 It could be varies in size there, but it's mostly C ++ or Python. We really haven't used other languages as much for the past 39 years. 00:19:36 Is that inertia or is that by choice? @@ -416,19 +414,19 @@ 00:19:56 And we have been fairly good about sort of meeting together and sort of deciding the direction that we want to go in and sort of sticking to that. -00:20:04 So for C Plus Plus, it was heavily root, which is a giant C Plus framework, and it's got everything in it. +00:20:04 So for C++, it was heavily root, which is a giant C ++ framework, and it's got everything in it. -00:20:13 And that was C Plus Plus, and that's what everybody used. +00:20:13 And that was C++, and that's what everybody used. 00:20:16 Right. 00:20:18 I was going to write code that would run and interact with, like the grid computing or the data access and all that kind of stuff at LHC. -00:20:27 I would use this route library if I was doing that C Plus Plus, right. +00:20:27 I would use this route library if I was doing that C++, right. 00:20:30 Yes. -00:20:31 You might be using interpreted C Plus Plus, which is something we invented. +00:20:31 You might be using interpreted C++, which is something we invented. 00:20:34 Oh, okay. @@ -438,11 +436,11 @@ 00:20:39 Yes. -00:20:39 We actually was the original interpreter, and then it got replaced by Kleen, which is built on the LLVM. +00:20:39 We actually CINT was the original interpreter, and then it got replaced by Kleen, which is built on the LLVM. -00:20:47 And I think recently it was merged to mainline LLVM as Clang Reply. I think it's called, but sort of a lightweight version. +00:20:47 And I think recently it was merged to mainline LLVM as Clang Ripple. I think it's called, but sort of a lightweight version. -00:20:57 It's a C Plus interpreter. You can actually get Zeus Cling, which I think quantack, but they package it as well. I think it's just Zeus clean. +00:20:57 It's a C++ interpreter. You can actually get Zeus Cling, which I think quanstack, but they package it as well. I think it's just Zeus clean. 00:21:08 Okay. @@ -450,9 +448,9 @@ 00:21:09 Very interesting. -00:21:10 Sequels plus really wasn't designed for a notebook, though. +00:21:10 C++ really wasn't designed for a notebook, though. -00:21:13 It does work, but you can't rerun a cell often because you can't redefine things. Python is just really natural in a notebook. And C Plus Plus is not. +00:21:13 It does work, but you can't rerun a cell often because you can't redefine things. Python is just really natural in a notebook. And C++ is not. 00:21:22 Yes. @@ -476,9 +474,9 @@ 00:21:59 They definitely have sort of agreed that Python should be a first class language. -00:22:05 And join C Plus Plus. That was decided a few years ago, and I think that's been a great step in the right direction, because what was happening, people are coming in with Python knowledge. They wanted to use Pandas, and I came in that way as well. +00:22:05 And join C++. That was decided a few years ago, and I think that's been a great step in the right direction, because what was happening, people are coming in with Python knowledge. They wanted to use Pandas, and I came in that way as well. -00:22:18 Pandas and number. And all these tools were really nice, and we were basically just having to write them all ourselves in people's website. +00:22:18 Pandas and number. And all these tools were really nice, and we were basically just having to write them all ourselves in C++. 00:22:26 It has a data frame. But why not just use why not just use Python, which is what people know. @@ -486,33 +484,33 @@ 00:22:33 There's a ton of people already doing the work maintaining it for us. -00:22:37 Root literally has a string class, literally. +00:22:37 Literally has a string class, literally. 00:22:41 They do everything the idea. -00:22:44 And that's the idea behind psychic. Hep was to build this collection of packages that would just fill in the missing pieces, the things that energy physicists were used to and needed, and some of them are general. +00:22:44 And that's the idea behind Scikit-Hep was to build this collection of packages that would just fill in the missing pieces, the things that energy physicists were used to and needed, and some of them are general. 00:22:58 And we're just gaps in the data science ecosystem. And some things are very specific. -00:23:02 High energy physics psychotic actually sort of originated as a single package. +00:23:02 High energy physics Scikit-HEP actually sort of originated as a single package. 00:23:09 It sort of looked like root right at first. -00:23:13 And it was invented by someone called it Bartovic Reeves, who is actually in my office at CERN and her office mates. +00:23:13 And it was invented by someone called it Bartovic Rodriguez, who is actually in my office at CERN and her office mates. -00:23:20 He did something I think really brilliant when he did this. And that is, he created an organization called Psychedhap around it. +00:23:20 He did something I think really brilliant when he did this. And that is, he created an organization called Scikit-HEP around it. -00:23:26 And then he went out and spoke with people and got some of the other Python packages that existed at the time to join Psychiate. Hep moved them over and started building a collection of some of the most popular Python packages at the time. +00:23:26 And then he went out and spoke with people and got some of the other Python packages that existed at the time to join Scikit-HEP moved them over and started building a collection of some of the most popular Python packages at the time. 00:23:38 And I thought that was great. -00:23:40 And I really wanted secondhead to become a collection of separate tools. And for the second half package to just be sort of a meta package that just grabbed all the rest. +00:23:40 And I really wanted Scikit-HEP to become a collection of separate tools. And for the second half package to just be sort of a meta package that just grabbed all the rest. 00:23:50 And that's actually kind of where it is now. 00:23:51 Right. -00:23:52 I can Pip install psychedelic. Is that right? +00:23:52 I can Pip install Scikit-HEP. Is that right? 00:23:54 You can. And mostly other than a few little things that are still in there that never got pulled out that will mostly just install our most popular maybe 15 or so packages. @@ -524,7 +522,7 @@ 00:24:16 It's a really cool idea. And I like it. So maybe one of the things I thought would be fun is to go through some of the packages there to give people a sense of what's in here. -00:24:26 Some of these are pretty particular, and I don't think would find broad use outside of CERN, for example, Honda Forge Root. +00:24:26 Some of these are pretty particular, and I don't think would find broad use outside of CERN, for example, Conda Forge Root. 00:24:34 It sounds like that's about building route, so I can install it as a dependency or something like that. Right? @@ -532,11 +530,11 @@ 00:24:39 Building root is horrible. -00:24:40 And you actually now can get it as part of a condo package, which is just way better than anything that was available for attaching to a specific version of Python because it has to compile against a very specific version of Python, but that's what it does. So unless you want something in root, then that's very hip specific. +00:24:40 And you actually now can get it as part of a condo package, which is just way better than anything that was available for attaching to a specific version of Python because it has to compile against a very specific version of Python, but that's what it does. So unless you want something in root, then that's very HEP specific. 00:25:01 Some of the more general ones probably briefly mentioned our very first package that I think was really popular among energy physicists that we actually produced was uproot, which was just a pure Python package. So you didn't have to install it. -00:25:18 That red route files again, very specific for somebody who was in high energy physics, but you could actually read a root file and get your data without installing root. And that was a game changer. +00:25:18 That red root files again, very specific for somebody who was in high energy physics, but you could actually read a root file and get your data without installing root. And that was a game changer. 00:25:31 And now you can actually install root slightly easier, but normally it's a multi hour compile, and it's gotten better. But it's still a bit of a beast to compile, especially for Python. @@ -584,11 +582,11 @@ 00:27:18 And to store that compactly the same way you'd use NumPy was one thing, but you can use arrow, and there's some other things that do this. -00:27:28 But Akrida also gives you NumPy, like indexing and data manipulation. +00:27:28 But Awkward Array also gives you NumPy, like indexing and data manipulation. 00:27:34 And that was the sort of breakthrough thing here the original one was built on top of NumPy. -00:27:40 The new one actually has some 5911 compiled bits and pieces, but it makes working with that really well. In fact, Jim Pavarsky got a grant to expand this to. +00:27:40 The new one actually has some Pybind11 compiled bits and pieces, but it makes working with that really well. In fact, Jim Pavarsky got a grant to expand this to. 00:27:52 I don't remember the number of different disciplines that he's working with, but lots of different areas genomics and things like that have all use cases, and he's adding things like complex numbers and things that weren't originally needed by heavy physicist, but make it widely. @@ -614,9 +612,9 @@ 00:28:47 22 gigs. So yeah, that's a pretty appealing value proposition there. -00:28:51 Yeah. And it supports number. +00:28:51 Yeah. And it supports Numba. -00:28:53 Jim works very closely with the number teams and really is one of the experts on the number internals. +00:28:53 Jim works very closely with the Numba teams and really is one of the experts on the Numba internals. 00:28:59 So it has full number support now, and he's working on adding Dask. @@ -624,7 +622,7 @@ 00:29:10 Very cool. -00:29:11 Maybe not everyone out there knows what number is. Maybe give us a quick elevator pitch on number. +00:29:11 Maybe not everyone out there knows what Numba is. Maybe give us a quick elevator pitch on number. 00:29:16 I hear it makes Python code fast, right? @@ -636,9 +634,9 @@ 00:29:23 It takes Python. It actually takes the Byte code, and then it basically takes that back to something or it pushes the bytes code and turns it into LLVM. -00:29:34 It works a lot like Julia, except instead of a new language, it's actually reading Python bytecode, which is challenging because the Python bytecode is not something that stays static. We're supposed to be a public detail. +00:29:34 It works a lot like Julia, except instead of a new language, it's actually reading Python bytecode, which is challenging because the Python bytecode is not something that stays static or supposed to be a public detail. -00:29:48 There's no public promises about consistency of bytecode across versions, because they play with that all the time to try to speed up things and they add by codes and they try to do little optimizations. +00:29:48 There's no public promises about consistency of bytecode across versions, because they play with that all the time to try to speed up things and they add byte codes and they try to do little optimizations. 00:30:00 Yes, every Python release breaks number, so they just know the next Python release will not support number, and it usually takes a month or two. @@ -676,37 +674,37 @@ 00:31:32 Well, this operation on this data structure, this is what it should do. -00:31:36 And then that turned into LVM, and maybe it can get Victorized or things like that for you. +00:31:36 And then that turned into LlVM, and maybe it can get Vectorized or things like that for you. 00:31:41 Yeah, that's super neat. 00:31:43 Another package in the list that I got to talk about, because just the name and the graphic is fantastic is a gas. -00:31:50 What is a gas? It's got like the scream. +00:31:50 What is aghast? It's got like the scream. 00:31:53 I forgot who was the artist of that. But the scream sort of look as part of the logo is good. 00:31:59 About half of the logos come from Jim, and he did about half and then use other around or from the individual package office. -00:32:09 This is sort of part of the histogramming area, which is sort of the area I work in. +00:32:09 This is sort of part of the histogramming area, which is sort of the area I work in. So I can help -00:32:14 But Jim actually wrote a gap, and the idea was that it would convert between histogram representations. +00:32:14 But Jim actually wrote aghast, and the idea was that it would convert between histogram representations. 00:32:19 I think it came up because Jim got tired of writing histogram libraries. I think he's written at least five. 00:32:24 Yeah. -00:32:25 One of the things I got the sense of by looking through all the psychic Hep stuff. +00:32:25 One of the things I got the sense of by looking through all the Scikit-Hep stuff. 00:32:29 There's a lot of histogram stuff happening over there. 00:32:32 Yes, histogram is sort of the area that I was in, and it ended up coming in several pieces. -00:32:38 But I think one of the important things was actually. And I think a gas may not really matter. +00:32:38 But I think one of the important things was actually. And I think aghast may not really matter. -00:32:42 They get archived at some point because instead of sort of translating between different representations of histograms in memory, what you can do is define a static typing protocol, and it can be checked by my Pi that describes what an object needs to be called a histogram. +00:32:42 They get archived at some point because instead of sort of translating between different representations of histograms in memory, what you can do is define a static typing protocol, and it can be checked by MyPy that describes what an object needs to be called a histogram. -00:33:02 And so I've defined that as a package called, UHI, universal histogram interface and anything that implements, UHI, it can be fully checked by WiFi will then be able to take any object from any library that implements, UHI. +00:33:02 And so I've defined that as a package called, UHI, universal histogram interface and anything that implements, UHI, it can be fully checked by MyPy will then be able to take any object from any library that implements, UHI. 00:33:17 And so all the libraries we have that produce histogram so uproot when it reads a root histogram or Hist and boost histogram when they produce histograms. @@ -714,13 +712,13 @@ 00:33:33 And then they can be plotted in NPL. -00:33:36 Hep or they can be printed to the terminal with just a print. +00:33:36 Hep or they can be printed to the terminal with just a histo-print. 00:33:40 And there's no dependencies there. 00:33:42 One doesn't need the other. -00:33:43 And that's sort of making a gas somewhat unneeded, because now it really doesn't matter. You don't have to convert between two because they both just work. +00:33:43 And that's sort of making aghast somewhat unneeded, because now it really doesn't matter. You don't have to convert between two because they both just work. 00:33:52 They work on the same underlying structure. Basically. @@ -728,7 +726,7 @@ 00:33:55 They work through the same interface, right? -00:33:58 Yeah. So a gas is a way to work with different histogramming libraries. +00:33:58 Yeah. So aghast is a way to work with different histogramming libraries. 00:34:03 That kind of is the intermediary of that abstraction layer on that. Okay. @@ -736,13 +734,13 @@ 00:34:11 What are some other ones we should kind of give a shout out to. We talked about Goofit, which is an affiliated package. -00:34:17 It's not part of psyched habit, but we developed this idea of an affiliated package for sure, things that didn't need to be moved in, but had at least one site could have developer working or working with them. At least that's my definition. I was never able to actually get the rest to agree to exactly that definition. But that's my working definition. +00:34:17 It's not part of Scikit-HEP, but we developed this idea of an affiliated package for sure, things that didn't need to be moved in, but had at least one Scikit-HEP developer working or working with them. At least that's my definition. I was never able to actually get the rest to agree to exactly that definition. But that's my working definition. -00:34:37 So that's why Pi Band Eleven gets listed there. +00:34:37 So that's why Pybind 11 gets listed there. -00:34:39 It's an affiliated package because we share a developer with the Pi Eleven library, and we sort of have a say in that and how that is developed. +00:34:39 It's an affiliated package because we share a developer with the Pybind11 library, and we sort of have a say in that and how that is developed. -00:34:50 And most importantly, if we have somebody come into psychic Hep, we want them to use 511 over the other tools because that one we have a lot of experience with. +00:34:50 And most importantly, if we have somebody come into Scikit-HEP, we want them to Pybind11 over the other tools because that one we have a lot of experience with. 00:34:59 Very cool. @@ -764,7 +762,7 @@ 00:35:33 It's a very tiny little library. -00:35:35 It was the first one to be fully statically Typed because it was tiny, easy to do because my pie and first constants, there was like two functions or something. And then it was done. +00:35:35 It was the first one to be fully statically Typed because it was tiny, easy to do because MyPy and first constants, there was like two functions or something. And then it was done. 00:35:45 Yeah. @@ -784,7 +782,7 @@ 00:36:09 I love pint. -00:36:11 Actually, it takes the types through, and I use Pintsum, but it actually gives you a quantity out or a NumPy quantity whereas the happiness just stays out of the way. And it's a way to be more clear in your code, but it's not enforced. Pint is enforced, which I like enforcing, but it also can slow down. You can't. +00:36:11 Actually, it takes the types through, and I use Pintum, but it actually gives you a quantity out or a NumPy quantity whereas the happiness just stays out of the way. And it's a way to be more clear in your code, but it's not enforced. Pint is enforced, which I like enforcing, but it also can slow down. You can't. 00:36:32 These are not actual real numbers anymore. So you pay. @@ -808,7 +806,7 @@ 00:37:00 What else do people maybe pay attention to that? They can generally find useful over here. -00:37:04 Mission vector. It's a little bit newer, but certainly for general physics. +00:37:04 Convention vector. It's a little bit newer, but certainly for general physics. 00:37:09 I think it's useful because it's a library for 2D 3D and relativistic vectors, and it's a very common sort of learning example, you see, but there aren't really very many libraries that do this that actually have. @@ -816,9 +814,9 @@ 00:37:31 And vector is supported by awkward. -00:37:34 It has an awkward back end. It has a number back end, NumPy back end and then plain object back end. +00:37:34 It has an awkward back end. It has a numba back end, NumPy back end and then plain object back end. -00:37:39 Eventually we might work on more and it even has a number awkward. So you can use a vector inside an awkward array inside a number jet compiled loop and still take magnitudes and do stuff like that. +00:37:39 Eventually we might work on more and it even has a numba awkward. So you can use a vector inside an awkward array inside a number jit compiled loop and still take magnitudes and do stuff like that. 00:37:51 That's really cool, because we have a lot of those statistics. @@ -836,31 +834,31 @@ 00:38:26 And then maybe just mentioned we mentioned the histogramming stuff and that's the area that's the one that I really work on. -00:38:32 The ones I specifically work on that are general purpose boost. Histogram is a wrapper for the C Plus post boost. Histogram library boost is sort of the big C plus library just one step below the standard library. +00:38:32 The ones I specifically work on that are general purpose boost. Histogram is a wrapper for the C++ post boost. Histogram library boost is sort of the big C++ library just one step below the standard library. 00:38:45 And right at the time I was starting at Princeton, I met the author of Boostrami, who's from Physics, and he was in the process, I believe, of getting this accepted into Boost and I got accepted after that. -00:38:58 But one of the things that he decided to do is pull out his initial Python bindings that were written in Boost Python, which is actually very similar to Python Eleven but requires boost instead of not requiring anything. +00:38:58 But one of the things that he decided to do is pull out his initial Python bindings that were written in Boost Python, which is actually very similar to Pybind 11 but requires boost instead of not requiring anything. 00:39:11 But the design is intentionally very similar. -00:39:14 And so I proposed I would work on Boost Histogram and write these Python bindings for it inside Psychic habit, and that would be sort of the main project I started on when I started in Princeton, and that's what I did this. Histogram is an extremely powerful histogramming library. +00:39:14 And so I proposed I would work on Boost Histogram and write these Python bindings for it inside Scikit-HEP, and that would be sort of the main project I started on when I started in Princeton, and that's what I did this. Histogram is an extremely powerful histogramming library. 00:39:32 So it's a histogram as an object rather than like a NumPy. 00:39:36 There's a histogram function and you give it an array and then it spits a couple of arrays back out at you. -00:39:40 But you now have to manage these. They don't have any special meaning, whereas the historians really are much more natural than object. Just like a data frame is more natural as an object where you tie that information together. +00:39:40 But you now have to manage these. They don't have any special meaning, whereas the histogram really are much more natural than object. Just like a data frame is more natural as an object where you tie that information together. 00:39:53 Histograms really natural that way, where you still have the information about what the data actually was on the axes. -00:39:58 If you have labels, you want to keep those attached to that data and you may need to fill again, which is one of the main things that your physicist really wanted, because we tend to fill histograms and then keep filling them or rebinning them or doing operations on them. +00:39:58 If you have labels, you want to keep those attached to that data and you may need to fill again, which is one of the main things that your physicist really wanted, because we tend to fill histograms and then keep filling them or rebinding them or doing operations on them. -00:40:13 And you can do all those very naturally and boost Histograms Plus wrapper in Pibind Eleven. +00:40:13 And you can do all those very naturally and boost Histograms C++ wrapper in Pybind 11. -00:40:19 And I actually got involved in Ciao because of Boost Instagram because one of the things I wanted to to make sure it worked everywhere, and it obviously requires C Plus Plus compilation. +00:40:19 And I actually got involved in Cibuildable because of Boost histogram because one of the things I wanted to to make sure it worked everywhere, and it obviously requires C++ compilation. -00:40:30 And then His is a nice wrapper on top of that. That just makes it a lot more friendly to use, because the original Boost scrambled here wants to keep this. Hanstobinsky wants to keep this quite pure and clean. +00:40:30 And then Hist is a nice wrapper on top of that. That just makes it a lot more friendly to use, because the original Boost scrambled here wants to keep this. Hanstobinsky wants to keep this quite pure and clean. 00:40:41 So Hist is the more natural. And even if you're not in Hep, I think that's still the more natural one to use. @@ -874,7 +872,7 @@ 00:41:01 Let's talk a bit about that. And I mean, maybe the place to start here is you want our wheels, right. -00:41:06 The first sentence described as Python wheels are great building them across Maclinics windows and other multiple versions of Python. +00:41:06 The first sentence described as Python wheels are great building them across Mac Linux windows and other multiple versions of Python. 00:41:13 Not so much. @@ -916,7 +914,7 @@ 00:42:30 It just grabs the contents inside and dumps them, following a specific set of rules into places into your site packages. -00:42:39 You have something installed, there's no setup PY in your wheel, there's no pipedroject Como. +00:42:39 You have something installed, there's no setup PY in your wheel, there's no pyproject.main 00:42:45 Those sorts of things are not in the wheel. The wheel is already there. @@ -942,7 +940,7 @@ 00:43:24 Why don't we just run it once and then go here and then also, that saves you a ton of time. -00:43:28 Right. Like I just installed Microscope, and it took 30 seconds, 45 seconds to install because it didn't have a wheel. So it's up there and it just grinded away compiling it. +00:43:28 Right. Like I just installed Microwhiskey, and it took 30 seconds, 45 seconds to install because it didn't have a wheel. So it's up there and it just grinded away compiling it. 00:43:39 Yeah. So there's two possibilities. @@ -952,7 +950,7 @@ 00:43:59 If it doesn't pass through the wheel stage anyway. -00:44:01 And then every time you open the file, then it's going to the first time. It's going to have to make that by cut. So it'll be a little slower the first time you open it. +00:44:01 And then every time you open the file, then it's going to the first time. It's going to have to make that byte code. So it'll be a little slower the first time you open it. 00:44:08 There's a variety of reasons I think it's Python wheels. @@ -966,7 +964,7 @@ 00:44:37 But Future doesn't, for example, and so on. -00:44:39 Futures been there for a long time. +00:44:39 Features been there for a long time. 00:44:43 So wheels are really good, and they actually replaced an older mechanism that was trying to do something somewhat similar called Eggs. But I avoid talking about this understanding. @@ -978,7 +976,7 @@ 00:45:09 If you have the example. -00:45:11 It'S like it was doing GCC. +00:45:11 It's like it was doing GCC. 00:45:13 You don't have a compiler. It won't even work. @@ -988,7 +986,7 @@ 00:45:20 Right. -00:45:20 How many Windows users have seen cannot find Vcvars bat. +00:45:20 How many Windows users have seen cannot find vcvars.bat. 00:45:24 Right. @@ -1004,7 +1002,7 @@ 00:45:44 And they can tell you what Python version they are good for, what platform they are supported on. -00:45:52 They have a build number, and then they have the Python is actually in two pieces. There's the Abi and the interface. +00:45:52 They have a build number, and then they have the Python is actually in two pieces. There's the Api and the interface. 00:46:00 Python. @@ -1032,7 +1030,7 @@ 00:46:29 It knows how to pick out it looks for the right one. If it finds a binary, it will just download it, depending slightly on the system and how new your Pip is. -00:46:37 Right. And this is one of the main innovation, ideas and philosophies behind Honda and Anaconda. +00:46:37 Right. And this is one of the main innovation, ideas and philosophies behind Conda and Anaconda. 00:46:44 Let's just take that and make sure that we build all of these things in a really clear way and then sort of package up the testing and compilation and distributing all that together. @@ -1042,13 +1040,13 @@ 00:47:05 Building a wheel was challenging. -00:47:09 Build wheel has really changed that. +00:47:09 CiBuild wheel has really changed that. 00:47:10 If you want a pure Python, it's really easy today you should be using the build tool, which I'm a maintainer of that as well. 00:47:17 But build just builds an estimate for you or it builds a wheel. -00:47:22 So you say something like Python set up UI Bdist or something like that and then boom. +00:47:22 So you say something like Python setup.py Bdist or something like that and then boom. 00:47:28 You shouldn't be doing that anymore. Please don't. @@ -1060,7 +1058,7 @@ 00:47:35 Well, you could do Python or Pip install build and then Python build, and that will build both an estist and a wheel, and it'll build the wheel from the estist. -00:47:46 If you use Pipex, which I would recommend, then you can just say pipe run build and you don't have to do anything that'll download build into a virtual environment for you. +00:47:46 If you use Pipex, which I would recommend, then you can just say pipex run build and you don't have to do anything that'll download build into a virtual environment for you. 00:47:54 It'll do it, and then eventually it will throw away the original after a week. Interesting. @@ -1068,25 +1066,25 @@ 00:48:00 We should be using the build. -00:48:01 You should be using the build tool for SS. There's a big benefit to this, and that is it will use yourpipeproject Tobal, and if you say you require NumPy, then it will go like you're using the NumPy headers, the C headers, then it will go. +00:48:01 You should be using the build tool for estest. There's a big benefit to this, and that is it will use yourpyproject.tobal, and if you say you require NumPy, then it will go like you're using the NumPy headers, the C headers, then it will go. 00:48:19 When it's building Estus, it will make the Pep 517 virtual environment. -00:48:25 It will install Numba anything that's in your requirements in your Piproject Como, and then it will run the setup PY inside that environment. So you can now import NumPy directly in there and it'll work even when you're building a nest. +00:48:25 It will install Numba anything that's in your requirements in your Pyproject.mo, and then it will run the setup PY inside that environment. So you can now import NumPy directly in there and it'll work even when you're building a estest. 00:48:40 If you do Python Eston set up PY stuff, you can't do that because you're literally running Python giving it set up PY import NumPy. Now it's broken, right? -00:48:52 Nothing triggers that call to the piproject. +00:48:52 Nothing triggers that call to the pyproject. 00:48:56 Com to see what you need for a wheel. 00:49:00 The best way to do it is with Pip or the original way to do it was with Pip wheel, because Pip has to be able to build wheels in order to install things that got added to Pip before build existed. -00:49:13 But now the best way to do it would be with build wheel and that's actually doing the right thing. It's actually trying to build the wheel you want, whereas Pip wheel is actually just building a wheelhouse. So if you depend on NumPy and Numb, I don't have wheels. +00:49:13 But now the best way to do it would be with build wheel and that's actually doing the right thing. It's actually trying to build the wheel you want, whereas Pip wheel is actually just building a wheelhouse. So if you depend on NumPy and Numpy, does'nt have wheels. 00:49:27 They did better with Python 310, so I'm not going to complain about them for Python 310, but for three nine, they didn't have wheels for a while. -00:49:34 So it'll build the wheels there and it'll build your wheels and it'll dump them all in the wheelhouse, whatever the output is. So you'll be building Numbi wheels, which you definitely don't want to try to upload. +00:49:34 So it'll build the wheels there and it'll build your wheels and it'll dump them all in the wheelhouse, whatever the output is. So you'll be building Numpy wheels, which you definitely don't want to try to upload. 00:49:43 Yeah, definitely not. @@ -1112,7 +1110,7 @@ 00:50:17 It's built just for our platform. -00:50:19 The challenge is when you become a maintainer, you got to solve this matrix of different Python versions that are supported and different platforms. Like, for example, there's macOS intel, there's macOS M, one Apple Silicon. +00:50:19 The challenge is when you become a maintainer, you got to solve this matrix of different Python versions that are supported and different platforms. Like, for example, there's macOS intel, there's macOS M1 Apple Silicon. 00:50:33 There's multiple versions of Windows. @@ -1130,7 +1128,7 @@ 00:51:04 It'll only work on the version you compiled it on and not anything older. -00:51:08 And you don't even see people trying to compile on Mac OS 1014 because they want their wheels to work as in many places as you want. +00:51:08 And you don't even see people trying to compile on Mac OS 10.14 because they want their wheels to work as in many places as you want. 00:51:19 Exactly. @@ -1164,13 +1162,13 @@ 00:52:19 If you do that, it works really well. -00:52:21 So what actually started with Encyclope, I was building boost histogram, which needed to be able to run anywhere. That was something I absolutely wanted. It had to be Pip install this histogram and it just worked no matter what. +00:52:21 So what actually started with Scikit-HEP, I was building boost histogram, which needed to be able to run anywhere. That was something I absolutely wanted. It had to be Pip install this histogram and it just worked no matter what. 00:52:33 And also we had several other compiled packages at the time. Several we had inherited and was compiled and that was quite popular. 00:52:41 We had a couple of specific ones and we had a couple more that ended up being becoming interested in that. In fact, during this sort of period is when Awkward started compiling pieces. -00:52:52 When I started with was building my own system to do this, it was called Azure Wheel Helpers, which was you can guess by the name. Azure was basically set up scripts. +00:52:52 When I started with was building my own system to do this, it was called Azure Wheel Helpers, which was you can guess by the name. Azure was basically set up dev ops scripts. 00:53:04 It was right after Azure had come out and I wrote a series of blog posts on this and described the exact process and sort of the things I found out about how you build a compatible wheel on macOS. You have to make sure you get the most compatible C Python from Python. @@ -1190,7 +1188,7 @@ 00:54:23 And I was watching CI build Wheel and it was this package. It was a Python package that would do this and it didn't matter what CI system you were on because it was written in Python and it followed nice Python principles for good package design and had unit tests and all that sort of stuff. So it looked really good. There were a couple of things that was missing. I came in, I added, I made PRS for the things that I come up with that it didn't have and they got accepted. -00:54:49 And there was a shared maintainer between Pi Band Eleven and CI Build Wheel as well. I think that's one of the reasons that I heard about it was really watching it and I finally decided just to make the switch and I did. At some point a little later I actually became a maintainer of CI bill, but I think I started doing the switch before it made it really easy. Once I was a maintainer to say this is a package that we have some control over it's. Okay. Let's just take a choice to depend upon this because we have a say it just took out all that maintenance and now depend about does all the maintenance for us. +00:54:49 And there was a shared maintainer between Pi Bind11 and CI Build Wheel as well. I think that's one of the reasons that I heard about it was really watching it and I finally decided just to make the switch and I did. At some point a little later I actually became a maintainer of CI built, but I think I started doing the switch before it made it really easy. Once I was a maintainer to say this is a package that we have some control over it's. Okay. Let's just take a choice to depend upon this because we have a say it just took out all that maintenance and now depend about does all the maintenance for us. 00:55:20 Does the pin moves forward to pin and see a build wheel? That's it nice. @@ -1224,7 +1222,7 @@ 00:56:55 And the simplest test is just Echo, and that will just make sure it installs because I won't try to install your wheel unless there's something in that test command. -00:57:03 Even that's useful, sometimes even that's broken, sometimes because of Numbing not supporting one of those things in that matrix. +00:57:03 Even that's useful, sometimes even that's broken, sometimes because of Numpy not supporting one of those things in that matrix. 00:57:09 Yeah, it can install the dependency. @@ -1260,7 +1258,7 @@ 00:58:15 It does install to a standard location C colonback, CI buildwheel, but other than that, it's safe to run out there and Mac OS it will install to your macOS system. -00:58:25 It's all system versions of Python, so that's something we haven't sold yet might be able to do some day. +00:58:25 It's all system versions of Python, so that's something we haven't solved yet might be able to do some day. 00:58:30 It's not a good idea unless you really are okay with installing every version of Python that ever existed into your system. @@ -1280,11 +1278,11 @@ 00:59:05 There's lots of rules to exactly what can be in the environment and things like that. -00:59:10 And PIPA maintains that one thing that also helps is that we have the Mini Linux maintainer is also a CI build wheel. Maintainer. That's one reason that those things tend they fit well together. +00:59:10 And PyPI maintains that one thing that also helps is that we have the Mini Linux maintainer is also a CI build wheel. Maintainer. That's one reason that those things tend they fit well together. -00:59:22 Features tend to match and come out at the speed like munics, which is a big thing recently. +00:59:22 Features tend to match and come out at the speed like mutual linux, which is a big thing recently. -00:59:27 It's not actually in a released version of CIA build yet. +00:59:27 It's not actually in a released version of CI build yet. 00:59:30 What is mutual Linux so normal? Linux is based on G libc and that's actually what controls it's. One of two things that controls Mini Linux. @@ -1292,15 +1290,15 @@ 00:59:49 That was a mess, so they eventually switched to a standard numbering system that is your Glibc number. -00:59:55 And now Pip the current Pip will be able to install a future mini length as long as your system. +00:59:55 And now Pip the current Pip will be able to install a future mini linux as long as your system. 01:00:00 But that was a big problem. So Pip nine can only install Mini Linux one. It can't install many Linux even if your Glibc is fine for it. -01:00:09 The other thing is the Glibc version and Mini Linux one was based on Synths five -2010 was send to S six. +01:00:09 The other thing is the Glibc version and Mini Linux one was based on Centos5 -2010 was send to 6. 01:00:19 Mini Linux 2014 was sent to S seven and then now they switched to DBN because of the send to us sort of switching to the Stream model. -01:00:28 So Mini Linux 224 is G Lipsy two point 24. And that's DBN eight or something like that. +01:00:28 So Mini Linux 224 is G Lipsy 2.24. And that's DBN eight or something like that. 01:00:37 But that's Glibc based. @@ -1316,7 +1314,7 @@ 01:01:15 Yeah. You said I can run this locally as well. -01:01:18 I know I would use it in CI because I've got that matrix of all the versions of C, Python and Pipi and then all the platforms. And I want to check as many of those boxes as possible to put wheels in it, right? +01:01:18 I know I would use it in CI because I've got that matrix of all the versions of C, Python and PyPI and then all the platforms. And I want to check as many of those boxes as possible to put wheels in it, right? 01:01:31 Yeah. @@ -1342,7 +1340,7 @@ 01:02:22 So the Mini Linux maintainer used his Linux machine and he had Q emulation on it, and he built the Emulated images a few hours, but he just built locally and then uploaded filled in the missing wheels. -01:02:37 So if I'm maintaining a package, I got some package I'm putting on Pipi and I want to test it. +01:02:37 So if I'm maintaining a package, I got some package I'm putting on PyPI and I want to test it. 01:02:44 Does it make sense to do it locally or does it just make sense to put it on some CI system? @@ -1350,7 +1348,7 @@ 01:02:51 Usually I do some local testing, but I'm also developing same builder, but usually it's probably fine to do this just in your CI and usually don't want to run the full thing every time. Usually you have your regular unit tests. -01:03:03 The civility is going to be a lot slower because it's going through and it's making each set of wheels launching Docker images and things like that. +01:03:03 The CI build is going to be a lot slower because it's going through and it's making each set of wheels launching Docker images and things like that. 01:03:09 And it's installing Python each time for Mac OS and Windows. @@ -1380,7 +1378,7 @@ 01:04:11 But Circle, I think Azure and GitHub auctions. I think they do all three. -01:04:17 And you can always flip things up, Travis, for the Linux and then appear for Windows. +01:04:17 And you can always flip things up, Travis, for the Linux and then appfair for Windows. 01:04:23 You can do it that way. @@ -1400,11 +1398,11 @@ 01:05:07 The example script is just a few lines. It does not take much to do this comparing to take. -01:05:14 Yeah, sure. And I didn't even scroll down here. You've got a nice grid on GitHub comiccibuildwheel that shows on GitHub Actions, which is supported on Azure pipelines. +01:05:14 Yeah, sure. And I didn't even scroll down here. You've got a nice grid on GitHub.com/cibuildwheel that shows on GitHub Actions, which is supported on Azure pipelines. 01:05:24 What supported CI doesn't do this. -01:05:29 Out there, Travis, Azure and GitHub Dot. +01:05:29 Out there Appfair, Travis, Azure and GitHub Dot. 01:05:35 But we can't test it. @@ -1412,15 +1410,15 @@ 01:05:38 It got you. -01:05:39 And then I wonder about the M one, the Apple Silicon Arm versions versus the intel versions. +01:05:39 And then I wonder about the M1, the Apple Silicon Arm versions versus the intel versions. 01:05:46 I don't know how well that's permeated into the world yet, but the fact they have Mac at all is kind of impressive. -01:05:52 Nobody has an M one runner yet. +01:05:52 Nobody has an M1 runner yet. 01:05:54 There are a few places I think now that you can purchase time on one, but no runners. -01:06:00 Last I checked GitHub Actions, you couldn't even run it yourself on them. One that may have changed. +01:06:00 Last I checked GitHub Actions, you couldn't even run it yourself on m1. One that may have changed. 01:06:05 I don't know. @@ -1440,13 +1438,13 @@ 01:06:45 Ideally, what you could do is you just need one mini, and then you set up a GitHub, actions hosted Runner, locally hosted Runner and other systems. -01:06:56 Get Labci was big on that. +01:06:56 Git Labci was big on that. 01:06:58 You can do anything on GitLab CI. We just haven't tested that because they don't have those publicly. But if you have your own, you can do that. 01:07:05 I know somebody who does this with basically has a Mac mini and runs the M one builds on that. -01:07:11 But you could do that. I have a Mac mini and the lead developer of Sierra Bills Wheel also has M one. +01:07:11 But you could do that. I have a Mac mini and the lead developer of Cibuildwheel also has M one. 01:07:19 He has an M one or something. I don't know. @@ -1484,39 +1482,39 @@ 01:09:03 So you basically just basically write a CMake file and this could wrap an existing package. -01:09:09 Or maybe you need some of the other things that seem like has and this will then let you build that as a regular Python package. +01:09:09 Or maybe you need some of the other things that Cmake has and this will then let you build that as a regular Python package. 01:09:16 In fact, recently somebody sort of put together CI build wheel psychic build and C make example and built LLVM and pulled out just the claim format tool and made wheels out of that. 01:09:28 And now you can just do Pip and so claim format. It's one to two megabytes. It works on all systems including Apple Silicon and things. I just tried it on Apple Silicon yesterday and it's a Pip install. -01:09:37 Now you can claim format C plus post code and that's just mindblowing added to pre commit the precommit CI it runs in two. I mean, I've been fighting for about a week to reduce the size of the claim format recipe from 600 megabytes to just under the 250. That was the maximum for freaking at CI. +01:09:37 Now you can claim format C++code and that's just mindblowing added to pre commit the precommit CI it runs in two. I mean, I've been fighting for about a week to reduce the size of the claim format recipe from 600 megabytes to just under the 250. That was the maximum for freaking at CI. -01:09:54 And then you can now Pip and sell it under about a megabyte for Linux that sort of thing. And I think that would be a really great thing to work on. It's been around since 2014, but it needs some serious work. +01:09:54 And then you can now Pip install under about a megabyte for Linux that sort of thing. And I think that would be a really great thing to work on. It's been around since 2014, but it needs some serious work. -01:10:09 And so I'm currently actually working on writing a grant to try to get funded to just work on basically the second build system and looking for interesting science use cases that would be interested in adapting or switching an existing build system over or adapting to it or taking something that has never been available from Python and making it available. +01:10:09 And so I'm currently actually working on writing a grant to try to get funded to just work on basically the scikit build system and looking for interesting science use cases that would be interested in adapting or switching an existing build system over or adapting to it or taking something that has never been available from Python and making it available. 01:10:30 And yes, root route might be one. -01:10:33 I'm looking for a wide variety of scientific build package is fundamentally just the glue between set of tools, Python module and CMake. +01:10:33 I'm looking for a wide variety of scikit build package is fundamentally just the glue between set of tools, Python module and CMake. 01:10:41 Yeah. So it's a real way to take some of these things based on CMake and sort of expose them to Python. -01:10:46 Yeah. So you can just have a CMA package that does all the C make things well, like finding different libraries and that I'm a big Cemic person. +01:10:46 Yeah. So you can just have a Cmake package that does all the C make things well, like finding different libraries and that I'm a big Cemic person. 01:10:56 How do you use it very heavily. -01:10:57 Most Cbus does. It's about 60%. I think of all build systems or CMAQ based now get from get Wear's numbers, but they may seem like, but I think it's very powerful. It can be used for things like that. +01:10:57 Most C++ does. It's about 60%. I think of all build systems or CMake based now get from gateways numbers, but they may seem like, but I think it's very powerful. It can be used for things like that. -01:11:11 And we'll really open up a much easier C Plus plus more natural in C and Fortran and things like that in CUDA then is currently available. Set up tools just utilize is going away in Python 312. +01:11:11 And we'll really open up a much easier C++ more natural in C++ and Fortran and things like that in CUDA then is currently available. Set up tools just utilize is going away in Python 3.12. -01:11:24 Setup tools is not really designed to build people's plus packages or packages. It was really just a hack on top of distributors which happened to be build. Just Python itself. +01:11:24 Setup tools is not really designed to build C++ packages or packages. It was really just a hack on top of distributors which happened to be build. Just Python itself. -01:11:35 Well, Psychiatrist sounds like the perfect tool to apply to the science space because there's so many of these weird compiled things that are challenging to install and deploy and share and so on. So making that easier sounds good. +01:11:35 Well, Scikit sounds like the perfect tool to apply to the science space because there's so many of these weird compiled things that are challenging to install and deploy and share and so on. So making that easier sounds good. 01:11:49 All right. -01:11:50 Well, I think we're probably going to need to leave it there just for the sake of time, but it's been awesome to talk about all the internals of supporting psychedelic, and people should check out CI Build wheel. +01:11:50 Well, I think we're probably going to need to leave it there just for the sake of time, but it's been awesome to talk about all the internals of supporting Scikit-HEP, and people should check out CI Build wheel. 01:12:02 It looks like if you're maintaining a package either publicly or just for internal for your organization, it looks like it would be a big help if it's got binary. @@ -1552,7 +1550,7 @@ 01:12:46 But all with the VI mode, there are plugins added, of course. -01:12:50 And then notable Pi package. I mean, we probably talked about 20 already. +01:12:50 And then notable PyPI package. I mean, we probably talked about 20 already. 01:12:55 If you want to just give a shout out to one of those, that's fine. Or if you got a new idea. @@ -1562,7 +1560,7 @@ 01:13:15 And that is plot text. -01:13:17 Y-L-O-T-T-E-X-T. +01:13:17 P-L-O-T-T-E-X-T. 01:13:19 And I'm really excited about that because it makes these the actual plot. It makes a really, really nice. @@ -1574,7 +1572,7 @@ 01:13:45 Yes. -01:13:45 You could cruise around your files, use your root I O integration, pull these things up here and put the plot right on the screen. +01:13:45 You could cruise around your files, use your root IO integration, pull these things up here and put the plot right on the screen. 01:13:53 Right. But in the terminal. @@ -1606,35 +1604,35 @@ 01:14:26 All right, Henry, final call to action. People want to do more with wheels. -01:14:29 See, I build Wheel or maybe some of the other stuff we talked about. What do you tell them? +01:14:29 ci build Wheel or maybe some of the other stuff we talked about. What do you tell them? 01:14:33 Look through. -01:14:34 I think one of the best places to go is the Psychic Hep Developer pages. If you have no interest in psychic Hep tools or Hep at all and that sort of shows you how these things integrate together really well. +01:14:34 I think one of the best places to go is the Scikit Hep Developer pages. If you have no interest in Scikit Hep tools or Hep at all and that sort of shows you how these things integrate together really well. 01:14:43 And nice documentation. -01:14:47 Build Wheel itself is nice. And the PIPA a lot of the IPA projects have gotten good documentation as well as packaging Python. +01:14:47 Build Wheel itself is nice. And the PyPI a lot of the IPA projects have gotten good documentation as well as packaging Python. 01:14:54 Org. -01:14:55 We've updated that quite a bit look like to reflect some of these things, but I really like the Psychic Developer pages. I mean, I'm biased because I wrote most of them nice. +01:14:55 We've updated that quite a bit look like to reflect some of these things, but I really like the Scikit Developer pages. I mean, I'm biased because I wrote most of them nice. 01:15:06 Yeah, I'll link those together. -01:15:08 I'll try to link to pretty much everything else we spoke to as well, so people can check out the podcast player showing us to find all that stuff. I guess one final thing that we didn't call it that, I think is worth pointing out is CI build Wheel is under the PIPA, the Python Packaging Authority, so it gives it some officialness. +01:15:08 I'll try to link to pretty much everything else we spoke to as well, so people can check out the podcast player showing us to find all that stuff. I guess one final thing that we didn't call it that, I think is worth pointing out is CI build Wheel is under the PyPI, the Python Packaging Authority, so it gives it some officialness. 01:15:23 I guess you should say yes. -01:15:24 That happened after I joined one of the first things I wanted to do was I thought this should really be in the PIPA, and I was sort of pushing for that. And the other developers were fine with that. +01:15:24 That happened after I joined one of the first things I wanted to do was I thought this should really be in the PyPI, and I was sort of pushing for that. And the other developers were fine with that. -01:15:35 And so we brought it up and I actually joined the PIPA just before that by becoming a member of Build. +01:15:35 And so we brought it up and I actually joined the PyPI just before that by becoming a member of Build. 01:15:40 So I got to vote on Build oil coming in. But it was a very enthusiastic vote, even without my vote. -01:15:46 And Pipe joined right at the same time too. So those were fighting time. +01:15:46 And Pipex joined right at the same time too. So those were fighting time. -01:15:50 Pipe is a great library. +01:15:50 Pipex is a great library. 01:15:51 I really like the way Pipex work. It's a great tool. @@ -1642,21 +1640,21 @@ 01:15:57 Thanks for all the insight on all these internals around building and installing Python packages. -01:16:01 There's also a lot more on my blog, so I sign Empy GitLab IO that links to all those other things, obviously too. +01:16:01 There's also a lot more on my blog, so I sign numpy.GitLab.IO that links to all those other things, obviously too. 01:16:09 Thank you for being here. Yeah. 01:16:11 See you. Thanks for having me. -01:16:12 You bet this has been another episode of Talk Python to me. +01:16:12 You bet. This has been another episode of Talk Python to me. -01:16:16 Our guest on this episode was Henry Schreiner, and it's brought to you by us over at Talk Python training and the transcripts were brought to you by assembly AI. +01:16:16 Our guest on this episode was Henry Schreiner, and it's brought to you by us over at Talk Python training and the transcripts were brought to you by 'AssemblyAI'. 01:16:25 Do you need a great automatic speech to text API? 01:16:27 Get human level accuracy in just a few lines of code? -01:16:30 Visit Talk Python FM AssemblyAI want to level up your Python? +01:16:30 Visit 'talkpython.fm/AssemblyAI' want to level up your Python? 01:16:35 We have one of the largest catalogs of Python video courses over at Talk Python. @@ -1664,18 +1662,18 @@ 01:16:44 And best of all, there's not a subscription in site. -01:16:46 Check it out for yourself at Training Talkpython FM be sure to subscribe to the show. +01:16:46 Check it out for yourself at 'Training.Talkpython .FM be sure to subscribe to the show. 01:16:51 Open your favorite podcast app and search for Python. 01:16:54 We should be right at the top. -01:16:55 You can also find the itunes feed at itunes, the Google Play feed at Slashplay and the Direct RSS feed at Slash. Rsson Talk Python FM We're live streaming most of our recordings these days. +01:16:55 You can also find the itunes feed at /itunes, the Google Play feed at /play and the Direct RSS feed at /RSS on Talk Python.FM We're live streaming most of our recordings these days. -01:17:08 If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at Talk Python FM. +01:17:08 If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at 'TalkPython.FM/youtube'. -01:17:15 Youtube this is your host, Michael Kennedy. +01:17:15 This is your host, Michael Kennedy. 01:17:18 Thanks so much for listening. I really appreciate it. Now get out there and write some Python code. -01:17:43 Python. \ No newline at end of file +01:17:43 Python.