025_effective_python.vtt

WEBVTT

00:00:00.001 --> 00:00:05.960
What if you could bottle up all the wisdom and hard-fought experience of many expert Python developers and power up your skills?

00:00:05.960 --> 00:00:10.240
That's what Brett Slatkin did, and he put it in his new book, Effective Python.

00:00:10.240 --> 00:00:16.720
This is episode number 25 of Talk Python to Me, recorded Wednesday, August 12, 2015.

00:00:16.720 --> 00:00:46.700
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the library's,

00:00:46.720 --> 00:00:48.280
the ecosystem, and the personalities.

00:00:48.280 --> 00:00:50.100
This is your host, Michael Kennedy.

00:00:50.100 --> 00:00:52.340
Follow me on Twitter, where I'm @mkennedy.

00:00:52.340 --> 00:00:56.220
Keep up with the show and listen to past episodes at talkpython.fm.

00:00:56.220 --> 00:00:59.260
And follow the show on Twitter via at Talk Python.

00:00:59.260 --> 00:01:02.560
This episode is brought to you by Hired and CodeChip.

00:01:02.560 --> 00:01:07.760
Thank them for supporting the show on Twitter via at Hired underscore HQ and at CodeChip.

00:01:07.760 --> 00:01:10.880
It's another week with a book giveaway.

00:01:10.880 --> 00:01:14.360
And just like last week's, this one is a really excellent one.

00:01:14.880 --> 00:01:20.220
Make sure you're a friend of the show and you'll be in the running to win the free e-book version of Effective Python.

00:01:20.220 --> 00:01:25.380
Just visit talkpython.fm, click on Friends of the Show, and boom, winning happens.

00:01:25.380 --> 00:01:27.300
Well, sometimes, if you're lucky.

00:01:27.300 --> 00:01:30.620
Now let me introduce Brett so we can get right to the show.

00:01:30.620 --> 00:01:33.700
Brett Slatkin is the author of Effective Python.

00:01:33.700 --> 00:01:37.880
He's the engineering lead and co-founder of Google Consumer Surveys.

00:01:37.900 --> 00:01:44.180
He formerly worked on Google App Engine, the PubSub Hubbub protocol, and managing Google's fleet of servers.

00:01:44.180 --> 00:01:50.920
Outside his day job, he works on open source tools and writes about software, bicycles, and other topics on his personal website.

00:01:50.920 --> 00:01:53.040
And today, he lives in San Francisco.

00:01:53.040 --> 00:01:55.200
Brett, welcome to the show.

00:01:55.200 --> 00:01:57.000
Thanks for having me, Michael.

00:01:57.500 --> 00:02:01.720
Yeah, I'm really excited to talk to you about your book and the other stuff you have going on with Python.

00:02:01.720 --> 00:02:06.600
We're going to talk about this book that you recently released called Effective Python.

00:02:06.600 --> 00:02:11.380
And I was a big fan of the Effective series back in the C++ days.

00:02:11.380 --> 00:02:14.340
And so when I saw your book title, I was like, oh, this is going to be awesome.

00:02:14.340 --> 00:02:16.400
So it's going to be a fun conversation.

00:02:17.140 --> 00:02:17.560
Yeah, definitely.

00:02:17.560 --> 00:02:23.820
And I similarly, Effective C++ was the best programming book that I ever had.

00:02:23.820 --> 00:02:27.060
And so I totally agree with that.

00:02:27.060 --> 00:02:31.040
So that was 1997 or 1998 when that came out when I first read it.

00:02:31.040 --> 00:02:38.300
And being able to make the Python version of that was really a great honor for me to be able to write.

00:02:38.300 --> 00:02:39.700
Yeah, I bet it was.

00:02:39.700 --> 00:02:40.380
That's great.

00:02:40.880 --> 00:02:47.540
Before we get into the details of your book, though, maybe we could just talk a bit about how you got to where you are.

00:02:47.540 --> 00:02:49.340
How did you get into programming in Python?

00:02:49.340 --> 00:02:55.760
Yeah, so I won't bore you with the super long details of my history of writing code.

00:02:55.760 --> 00:02:58.740
But I've been writing code since I was about 10 or 11 years old.

00:02:58.740 --> 00:03:02.700
My first job out of college was at Google.

00:03:02.700 --> 00:03:06.400
And I showed up for my first day of work.

00:03:06.400 --> 00:03:10.140
And they said, OK, here's this Python code base.

00:03:10.220 --> 00:03:11.200
You need to make it better.

00:03:11.200 --> 00:03:12.580
Here's a book.

00:03:12.580 --> 00:03:15.240
And Alex Martelli sits right there.

00:03:15.240 --> 00:03:16.120
Figure it out.

00:03:16.120 --> 00:03:18.740
So I had seen Python.

00:03:18.740 --> 00:03:20.580
I hadn't written any Python at all.

00:03:20.580 --> 00:03:28.220
And I went from basically knowing nothing about Python and then actually writing a book about it in about 10 years, which is pretty funny.

00:03:28.220 --> 00:03:29.180
Yeah, that's great.

00:03:29.180 --> 00:03:32.080
The student becomes the master sort of thing, right?

00:03:32.080 --> 00:03:32.260
Yeah.

00:03:32.260 --> 00:03:33.480
I hope so.

00:03:33.480 --> 00:03:34.680
I hope it's worthwhile.

00:03:34.680 --> 00:03:40.360
Well, I think what was funny is that my first impression of Python was from the BitTorrent code base.

00:03:40.360 --> 00:03:44.520
Because Bram Cohen had had this post all about Python and how it makes him more efficient.

00:03:44.520 --> 00:03:46.100
And that's why BitTorrent's so awesome.

00:03:47.740 --> 00:03:50.520
I don't think people realize that BitTorrent was written by Python.

00:03:50.520 --> 00:03:58.820
But then like Zope and Plone, I thought were just really, they were code bases that I was very unhappy with using when I tried Python for the first time.

00:03:58.820 --> 00:03:59.800
And that kind of turned me off.

00:03:59.800 --> 00:04:04.900
So then when I showed up at Google and they said, OK, you're using Python, I was cautiously optimistic.

00:04:06.180 --> 00:04:09.280
But it ended up turning out real well.

00:04:09.280 --> 00:04:15.020
And so I can just go real quick, just the kind of projects that I used Python for at Google.

00:04:15.020 --> 00:04:16.900
Yeah, people would love to hear that.

00:04:16.900 --> 00:04:23.040
My first job there was kind of doing janitorial services for data centers.

00:04:23.040 --> 00:04:26.260
Google has a bunch of machines in their data centers.

00:04:26.260 --> 00:04:28.000
Those machines have a life cycle.

00:04:28.000 --> 00:04:28.740
They're built.

00:04:28.740 --> 00:04:29.780
They're repaired.

00:04:29.780 --> 00:04:31.420
They break.

00:04:31.860 --> 00:04:37.260
And I worked on a bunch of the tools to help that life cycle, which are primarily written in Python.

00:04:37.260 --> 00:04:50.820
And I got to scale that out into Google's kind of worldwide network of data centers, a lot of system level code, a lot of networking, kind of database accessing, all kinds of stuff like that, workflows.

00:04:50.820 --> 00:04:53.500
So that was my first kind of big role.

00:04:53.500 --> 00:05:00.560
I did some other things similar to that around security and securing the machines in a data center.

00:05:01.620 --> 00:05:04.760
And then shortly after that, that was a couple of years.

00:05:04.760 --> 00:05:09.980
And then shortly after that, I heard of this really cool project that was being created called App Engine that had just started.

00:05:09.980 --> 00:05:14.400
And I thought it really had a lot of potential.

00:05:14.400 --> 00:05:16.340
And so I went and talked to that team.

00:05:16.340 --> 00:05:19.820
I think it was four people at that point, the original founding team.

00:05:19.820 --> 00:05:22.160
And I was like, what can I do to get on this team?

00:05:22.160 --> 00:05:23.920
How can I work on this?

00:05:23.920 --> 00:05:36.560
And in my 20% time, I built the App Engine Dev App Server, which is, I don't know if you've ever used App Engine, but you spend a lot of your time using this development tool to actually write your apps.

00:05:37.460 --> 00:05:38.580
And so, yes, I built that.

00:05:38.580 --> 00:05:39.760
And they're like, oh, this is pretty good.

00:05:39.760 --> 00:05:41.560
And then they invited me to join the team.

00:05:41.560 --> 00:05:49.740
And then I helped build App Engine out and launch it, which was, so it's, you know, platform as a service, Google's cloud kind of system.

00:05:49.740 --> 00:05:51.860
Yeah, that's, it's really cool.

00:05:51.960 --> 00:05:55.660
And was Python the very first language or was it Python and Go?

00:05:55.660 --> 00:05:56.320
What was it?

00:05:56.320 --> 00:05:58.500
It launched with just a restricted set, right?

00:05:58.500 --> 00:05:58.760
Yeah.

00:05:58.760 --> 00:06:00.300
So Python was the first language.

00:06:00.300 --> 00:06:02.280
Java came like a year and a half or two years later.

00:06:02.280 --> 00:06:09.160
And then, and then Go came a little bit after, like maybe two, two or three years after that.

00:06:09.160 --> 00:06:16.240
I think the first language is actually supposed to be JavaScript, server-side JavaScript in the old Netscape, LiveScript scheme of things.

00:06:17.440 --> 00:06:25.980
But in 2008, that seemed like a completely ridiculous proposition, which is pretty funny looking at all the Node.js stuff that's happened in the last few years with V8.

00:06:25.980 --> 00:06:27.140
This is way before V8.

00:06:27.140 --> 00:06:29.300
JavaScript is still really slow.

00:06:29.300 --> 00:06:35.740
And so anyway, so they went with Python because it was one of the main languages that Google had used.

00:06:35.740 --> 00:06:37.660
And Google had a lot of expertise in Python in general.

00:06:37.660 --> 00:06:42.560
So on the App Engine team, I spent a lot of time working on Python infrastructure, Python APIs,

00:06:43.280 --> 00:06:50.240
just kind of the feel of what it was like to write apps on top of App Engine using Python and fix a lot of bugs,

00:06:50.240 --> 00:07:01.320
vetted new APIs to go out to make sure they felt good, built infrastructure around task queues and MapReduce and offline processing and all kinds of different things.

00:07:01.320 --> 00:07:03.360
Did that for a few years.

00:07:04.620 --> 00:07:14.140
And then also started this project called PubSubHubbub in there, which was a real-time RSS project for making RSS feeds real-time,

00:07:14.140 --> 00:07:22.700
which is kind of hilarious also in hindsight now since that's pretty – it's kind of like saying I made something to design pogs or something like that.

00:07:22.700 --> 00:07:23.400
It's kind of old.

00:07:23.400 --> 00:07:25.000
Yeah, that's pretty funny.

00:07:25.000 --> 00:07:27.840
But that was the Google Reader days and all that kind of stuff, right?

00:07:27.840 --> 00:07:29.140
Yes, the good old days.

00:07:29.140 --> 00:07:29.640
Yeah.

00:07:29.740 --> 00:07:30.880
I kind of miss that thing still.

00:07:30.880 --> 00:07:33.180
I do too, all the time.

00:07:33.180 --> 00:07:38.160
I've been using News Blur recently, which I believe is also a Python app written in Django.

00:07:38.160 --> 00:07:40.940
But, you know, anyway, I miss Reader.

00:07:40.940 --> 00:07:42.180
Yeah, definitely.

00:07:42.180 --> 00:07:44.900
So that is a bunch of really cool stuff.

00:07:44.900 --> 00:07:49.080
And we might have to come back and talk about that at some later date in more detail.

00:07:49.080 --> 00:07:49.840
That'd be really fun.

00:07:49.840 --> 00:07:51.900
But let's talk about your book today.

00:07:51.900 --> 00:07:53.740
It's called Effective Python.

00:07:53.740 --> 00:07:54.720
What's the subtitle?

00:07:55.640 --> 00:07:58.600
It's 59 Specific Ways to Write Better Python.

00:07:58.600 --> 00:08:04.660
It's kind of based in this – the heritage of the Effective series, which, as we kind of hinted at the beginning,

00:08:04.660 --> 00:08:08.820
I think that started out with Scott Myers and Effective C++, right?

00:08:08.820 --> 00:08:10.700
Yeah, he invented the format.

00:08:10.700 --> 00:08:17.200
And, you know, he's really the guy who came up with this way of educating people.

00:08:17.200 --> 00:08:19.460
And all the other books are in his style.

00:08:20.200 --> 00:08:20.780
Yeah, that's cool.

00:08:20.780 --> 00:08:25.700
And so I remember when I was learning C++, I could sort of do stuff with it.

00:08:25.700 --> 00:08:27.060
You know, I could write code and so on.

00:08:27.060 --> 00:08:34.000
But after I read his book, I felt like it really took my understanding and effectiveness to a new level.

00:08:34.000 --> 00:08:37.600
And so you're kind of trying to bring this to the Python developers, right?

00:08:37.600 --> 00:08:39.020
Definitely.

00:08:39.020 --> 00:08:41.300
Yeah, that's the goal of the book.

00:08:41.300 --> 00:08:43.620
It's a bit of an audacious goal, but yeah.

00:08:44.360 --> 00:08:49.260
Yeah, I mean, I think it's really simple is that, like, I read an introductory Python book.

00:08:49.260 --> 00:08:54.880
This Effective Python is the book I would have given to myself as a second book if I had had the chance.

00:08:54.880 --> 00:08:59.620
So there was a gap in my knowledge that took me years and years and years to fill.

00:08:59.620 --> 00:09:05.380
And this book is kind of a way to shortcut and get all that practical knowledge without having to actually pay the dues.

00:09:05.380 --> 00:09:09.260
So I wish I had had this book when I was starting out.

00:09:09.960 --> 00:09:14.420
You know, we talk a lot about the concept of Pythonic code and idiomatic Python and stuff.

00:09:14.420 --> 00:09:16.900
And there's a lot of that concept in there.

00:09:16.900 --> 00:09:26.580
You know, like, this is the way you should do things in Python, even if you already know if you could accomplish it using your old idioms from, say, Java or C# or something.

00:09:26.580 --> 00:09:35.120
But then there's another part, another angle to it that's just sort of broadening your horizon, I think.

00:09:35.120 --> 00:09:37.120
So it's not just about writing Pythonic code.

00:09:37.120 --> 00:09:47.380
And I'm thinking of things like how you would use, say, subprocess to manage child processes and parallelism from your Python app, right?

00:09:47.380 --> 00:09:51.560
That's not technically a Pythonic thing, but it definitely ups your game and what you can do.

00:09:51.560 --> 00:09:57.700
Yeah, I think, you know, a lot of the advice is pure language construct kind of stuff.

00:09:58.520 --> 00:10:01.160
And then other things are like, yeah, the subprocess is item.

00:10:01.160 --> 00:10:06.920
But, you know, it's important to remember that one thing that they say about Python is, you know, batteries are included, right?

00:10:06.920 --> 00:10:14.920
And so, you know, the difference between Python as a language and Python as an ecosystem or Python as a set of libraries that you can use.

00:10:15.400 --> 00:10:23.800
When people think of Python, they think of all of it together, both the syntax, the language features that it has, and then the libraries that you know you can rely on to always be there.

00:10:23.800 --> 00:10:34.580
So those are tools you should have in your toolbox that are part of being a Python programmer, just like, you know, the collections API if you're a Java programmer, because it's basically all you write all the time.

00:10:34.800 --> 00:10:35.760
Yeah, absolutely.

00:10:35.760 --> 00:10:43.140
I have many friends that work in other languages, and they often want to compare their language to Python.

00:10:43.140 --> 00:10:54.100
And like you said, I think that's not even a conversation you can begin to have if you don't think about the entire standard library and the 60,000 PyPI packages.

00:10:54.100 --> 00:10:56.040
And, you know, you've got to take it as a whole, right?

00:10:56.040 --> 00:10:58.720
It's not just, here's how you do properties here and properties there.

00:10:58.720 --> 00:11:01.280
So this is either better or less good, right?

00:11:02.000 --> 00:11:02.620
Yeah, totally.

00:11:02.620 --> 00:11:10.260
I mean, a lot of these things, you have so much, I like to use the word leverage, you know, to describe, you get to stand on the shoulders of giants.

00:11:10.260 --> 00:11:11.400
There's a lot of different ways to say it.

00:11:11.400 --> 00:11:17.360
But yeah, things like NumPy, as an example of a library, it's a whole ecosystem unto itself.

00:11:17.360 --> 00:11:22.700
And trying to do the equivalent in Java or C++ is extremely difficult.

00:11:22.700 --> 00:11:24.040
Yeah, that's for sure.

00:11:24.600 --> 00:11:36.300
So I thought one way that would be fun to have a conversation about your book is kind of go from section to section or chapter to chapter and pick out a few interesting pieces of guidance from each one of them.

00:11:36.300 --> 00:11:37.340
Yeah, definitely.

00:11:37.340 --> 00:11:38.000
Yeah, cool.

00:11:38.000 --> 00:11:42.600
So you broke your book into six parts, and we'll just take them one at a time.

00:11:42.600 --> 00:11:47.980
So the first one is this concept of sort of the libraries and Pythonic thinking, right?

00:11:48.340 --> 00:11:49.440
Yeah, that's the first part.

00:11:49.440 --> 00:11:53.800
It's actually eight separate chapters is how many there are, because it keeps going.

00:11:53.800 --> 00:11:58.020
But yeah, the first one is Pythonic thinking.

00:11:58.020 --> 00:11:59.080
Excellent.

00:11:59.080 --> 00:12:00.560
So why'd you start with that?

00:12:01.860 --> 00:12:07.300
Yeah, I think that it's kind of like the strange universe of Python syntax.

00:12:07.300 --> 00:12:09.020
You have to get used to doing things.

00:12:09.020 --> 00:12:19.020
Even simple expressions like, I remember when I first started writing Python, and the not operator, as opposed to using the bang or exclamation point, getting used to where that not goes.

00:12:19.020 --> 00:12:25.180
And things like is not, as opposed to, you could do A is not B, or you could do not A is B.

00:12:26.220 --> 00:12:30.920
And, you know, understanding that the Pythonic way is A is not B. That's how you'd say it.

00:12:30.920 --> 00:12:37.400
So you really need this base to start with, so that all the other things make sense or are in context.

00:12:37.400 --> 00:12:45.500
Right. I also think it's probably a good way to kind of shock the system a little bit to prepare them for, like, this is a different world, probably, than you're coming from.

00:12:45.500 --> 00:12:46.000
Yeah.

00:12:46.320 --> 00:12:48.600
And so get prepared to think differently.

00:12:48.600 --> 00:12:56.940
Yeah, it's to open, I think it's a good way to say it, because I think it's, if you try to fit it into the way you're writing other languages, you're going to have a bad time.

00:12:56.940 --> 00:13:00.100
And you really need to embrace the Pythonic way of doing it.

00:13:00.100 --> 00:13:04.740
I had one person told me once that my C++ code looks like Python.

00:13:04.740 --> 00:13:06.900
I thought that was a really great compliment, I guess.

00:13:08.100 --> 00:13:10.600
But you don't really want your Python to look like C++, right?

00:13:10.600 --> 00:13:11.060
Yeah.

00:13:11.060 --> 00:13:12.160
Absolutely.

00:13:12.160 --> 00:13:13.680
Cool.

00:13:13.680 --> 00:13:21.920
So the very first thing that you talk about, literally the first piece of guidance, is to know what version of Python you're using.

00:13:21.920 --> 00:13:25.580
And I do a lot of training in Python, and it's always a problem.

00:13:25.580 --> 00:13:33.040
Someone's always running the wrong version of Python, or even worse, they're trying to use a package and they install it into the wrong version of Python,

00:13:33.040 --> 00:13:35.120
and then they're running the right one or something like that, right?

00:13:35.740 --> 00:13:39.120
Yeah, that's a big problem, especially with the Python 3 move that's going on.

00:13:39.120 --> 00:13:41.380
You know, you can get confused really easily.

00:13:41.380 --> 00:13:46.320
And then a lot of the features that people like to use, like even with statements, are actually more recent developments.

00:13:46.320 --> 00:13:52.280
So if you're on some old version, some versions of macOS are like running Python 2.5, and you don't know it,

00:13:52.280 --> 00:13:54.920
and you try to do simple things like a with statement, it doesn't work.

00:13:54.920 --> 00:13:57.060
And so that can be very surprising.

00:13:57.060 --> 00:13:58.260
So it's always good to check your assumptions.

00:13:58.260 --> 00:13:59.440
Yeah, absolutely.

00:13:59.440 --> 00:14:02.340
So sort of level set, all right, we all know what we're talking about.

00:14:02.340 --> 00:14:03.500
Now we can move forward, right?

00:14:03.500 --> 00:14:05.040
Yeah, exactly.

00:14:05.380 --> 00:14:12.020
So what I think probably a lot of people are familiar with, but some are not, is sort of working with sequences.

00:14:12.020 --> 00:14:18.860
And you say you should prefer using list comprehensions over like chaining a map and a filter call together.

00:14:18.860 --> 00:14:20.180
Yeah.

00:14:21.000 --> 00:14:31.960
Yeah, so I think if you, yeah, I think the story, so it's interesting, because if you look at the history of Python, people will talk about how, oh, I like Python, because it's kind of functional, but it's kind of object oriented, and it's kind of scripty.

00:14:33.240 --> 00:14:36.600
And so you'll still see examples or guides today.

00:14:36.600 --> 00:14:40.880
You'll be searching around the internet, and you'll find some old guide on active state from like 2006.

00:14:40.880 --> 00:14:42.960
And it's like, here's how you should use map and filter.

00:14:42.960 --> 00:14:47.280
And, or even 2003, like, you know, a long time ago.

00:14:47.520 --> 00:14:53.880
And those are useful tools, sometimes they're useful, but the list comprehension syntax is just so much more pithy.

00:14:54.780 --> 00:15:07.600
I think that Python over the years has added more and more tools to maximize the readability of code, and to minimize the visual noise, the extra parentheses and brackets and various symbols that you need to express something.

00:15:07.600 --> 00:15:18.200
And so I think that map and filter are kind of just antiquated tools that are in the language, because they're hard to take out, because they're, you know, kind of keywords, built in functions.

00:15:18.200 --> 00:15:22.900
If I had my way, I would just take them out, because I think list comprehensions are better.

00:15:22.900 --> 00:15:36.360
This episode is brought to you by Hired.

00:15:36.360 --> 00:15:42.820
Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.

00:15:42.820 --> 00:15:51.960
Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company.

00:15:52.420 --> 00:15:58.340
Typically, candidates receive five or more offers in just the first week, and there are no obligations, ever.

00:15:58.340 --> 00:16:00.420
Sounds pretty awesome, doesn't it?

00:16:00.420 --> 00:16:02.460
Well, did I mention there's a signing bonus?

00:16:02.460 --> 00:16:06.580
Everyone who accepts a job from Hired gets a $2,000 signing bonus.

00:16:06.580 --> 00:16:10.920
And as Talk Python listeners, it gets way sweeter.

00:16:10.920 --> 00:16:18.480
Use the link Hired.com slash Talk Python To Me, and Hired will double the signing bonus to $4,000.

00:16:19.380 --> 00:16:20.200
Opportunity's knocking.

00:16:20.200 --> 00:16:23.800
Visit Hired.com slash Talk Python To Me and answer the call.

00:16:34.020 --> 00:16:45.120
But yeah, the main thing is that, you know, if you have two ways of doing something, I think the Pythonic way of doing it is always the more clear and more explicit and obvious way of doing it.

00:16:45.120 --> 00:16:48.420
And so yeah, so that's why I think list comprehensions are definitely the way to go.

00:16:48.420 --> 00:16:50.040
Yeah, that's cool.

00:16:50.040 --> 00:16:50.780
I totally agree.

00:16:51.580 --> 00:16:56.560
It's not exactly a more declarative way of programming, but it's closer, right?

00:16:56.560 --> 00:16:57.200
That's nice.

00:16:57.200 --> 00:16:58.520
Yeah, it's more obvious.

00:16:58.520 --> 00:17:05.360
Sort of related to that, one of your other pieces of guidance was that you should use generators for large list comprehensions.

00:17:05.360 --> 00:17:06.420
Yeah.

00:17:06.420 --> 00:17:09.580
It's so easy to switch from one to the other, but when should I use one?

00:17:09.580 --> 00:17:10.280
When shouldn't I?

00:17:10.280 --> 00:17:11.260
What's the story of that?

00:17:11.920 --> 00:17:16.240
Yeah, so I think that, you know, Python can use a lot of memory is one big thing.

00:17:16.240 --> 00:17:24.380
And depending on what you're trying to do, if you're dealing with a small data set that entirely fits within memory, then you don't have to really think about this.

00:17:24.380 --> 00:17:27.980
You can just read everything as a list and just work on lists.

00:17:27.980 --> 00:17:35.000
So if you're reading a file, like a CSV file, and you want to deal with it line by line or something like that, then reading the whole thing into memory is fine.

00:17:35.000 --> 00:17:44.880
But what always happens is you end up wanting to do something a little bit bigger than you expected, especially with a lot of the data processing stuff that people are doing in Python these days.

00:17:44.880 --> 00:17:52.660
And so I'm trying to change the culture a little bit to say, hey, why not use a generator if you can?

00:17:52.660 --> 00:17:56.360
Because if you use a generator, then all of these memory problems go away.

00:17:57.740 --> 00:17:59.520
A generator returns one item at a time.

00:17:59.520 --> 00:18:01.760
It doesn't fully materialize the list.

00:18:01.760 --> 00:18:07.420
So the total memory space that's occupied is just the last thing you return from the generator.

00:18:07.420 --> 00:18:10.140
The rest of it gets cleaned up, garbage collected.

00:18:10.140 --> 00:18:11.500
Yeah, that's fantastic.

00:18:11.500 --> 00:18:17.080
Especially if you're like chaining one to another to another, and they're kind of building up.

00:18:17.080 --> 00:18:19.120
It's really efficient.

00:18:19.120 --> 00:18:27.560
Yeah, and to that point, I mean, I think that if you start by at the base level, like if you want to switch to generator code, it's difficult because you have to, you have to,

00:18:27.560 --> 00:18:30.560
put generators all the way down to the leaves of your call stacks.

00:18:30.560 --> 00:18:35.660
But if you start with generators at the leaves, then it's very easy to start saying, hey, you know, I'm going to turn this into a streaming generator.

00:18:35.660 --> 00:18:38.760
And so creating those cascades of generators becomes very straightforward.

00:18:39.260 --> 00:18:48.520
And so you can get quick, you know, very fast executing code that's easy to follow and uses a low amount of memory very, you know, very cheaply.

00:18:48.520 --> 00:18:50.060
And it's very readable.

00:18:50.060 --> 00:18:52.920
So that's why I suggest that.

00:18:53.040 --> 00:19:04.280
So the final language one that I wanted to sort of point out is something that I always thought was weird is the whole concept of the else statement on a loop.

00:19:04.280 --> 00:19:04.900
Yeah.

00:19:05.760 --> 00:19:12.120
And something I recently learned is that try except finally also supports else.

00:19:12.120 --> 00:19:13.200
It does.

00:19:13.200 --> 00:19:13.540
Yeah.

00:19:13.540 --> 00:19:14.000
Yeah.

00:19:14.000 --> 00:19:15.420
And you have really interesting guidance.

00:19:15.420 --> 00:19:20.940
You say, you know what, these, this else statement on loops, maybe not so much, but on exception handling, it's very cool.

00:19:20.940 --> 00:19:22.380
It's like an alternative to the catch.

00:19:22.760 --> 00:19:23.720
Yeah, definitely.

00:19:23.720 --> 00:19:24.200
Yeah.

00:19:24.200 --> 00:19:31.840
So I think else, else makes a lot of sense for try except finally, because you're basically saying, you know, try to do this.

00:19:31.840 --> 00:19:33.720
If an exception happens, do this.

00:19:33.720 --> 00:19:36.660
And if no exception happens, then do this other thing.

00:19:36.660 --> 00:19:40.140
And that's what the else block is on a, on an exception, kind of try except.

00:19:40.140 --> 00:19:45.720
Uh, and so that makes things really clear, clearly delineated what will and will not happen inside your exception handling.

00:19:45.720 --> 00:19:52.300
Um, and when you're doing exception handling, you want the, the try block to be as small as possible to narrow what you're catching.

00:19:52.300 --> 00:19:56.100
So that, that exceptions you didn't plan to catch are raised back up.

00:19:56.100 --> 00:19:57.460
That's, that's like a really important thing.

00:19:57.460 --> 00:19:58.640
Yeah, that makes sense.

00:19:58.640 --> 00:20:11.360
And the other thing is, you know, you might think, well, just the last bit of code in your try bit, your try try block could be what would go in the else statement, but that doesn't account for like early returns.

00:20:11.360 --> 00:20:22.280
And so, you know, if you were to do, try to do some stuff, if this, oh, just return, it would still run that else, but then you'd have to be really careful, you know, trying to put that cleanup all over the try.

00:20:22.280 --> 00:20:23.920
So yeah, it's really nice.

00:20:23.920 --> 00:20:24.560
Yeah, it's nice.

00:20:24.640 --> 00:20:28.700
And yeah, you can put like final cleanup stuff in the finally, and that'll always run if you have an early return.

00:20:29.000 --> 00:20:32.620
And if you have an else with an early return, that'll also still run the finally.

00:20:32.620 --> 00:20:33.920
So it's, it's super nice.

00:20:34.720 --> 00:20:36.520
Going to that back to that for loop part of it.

00:20:36.520 --> 00:20:37.960
Yeah, I think that's one of the big things.

00:20:37.960 --> 00:20:40.980
Like when people first learn about that in Python, they're like, oh, this is so great.

00:20:40.980 --> 00:20:43.900
I can, I can use it for doing this and that.

00:20:43.900 --> 00:20:47.680
And it's, there's really, yeah, I think it's something to avoid.

00:20:47.680 --> 00:20:56.080
It's kind of like, it's a shiny new toy and you want to use all the different parts of it for you, you know, and you have to like stab yourself in the eye before you realize that you're getting, that wasn't a good idea.

00:20:57.360 --> 00:20:58.020
That thing hurts.

00:20:58.020 --> 00:20:58.680
Yeah, it hurts.

00:20:58.680 --> 00:20:59.320
Like, oh, it's short.

00:20:59.320 --> 00:21:00.000
It's pointy.

00:21:00.000 --> 00:21:00.840
I shouldn't touch that.

00:21:00.840 --> 00:21:10.940
And I was talking to, I was talking to Guido about this one in particular because he said, you know, cause I was asking him what he thinks about this one.

00:21:10.940 --> 00:21:12.280
Cause it's kind of, it's kind of tough.

00:21:12.280 --> 00:21:17.180
I had worked with Guido on App Engine for a few years and, so, and I see him at PyCon from time to time.

00:21:17.180 --> 00:21:20.960
And so, you know, I felt bad by saying like, hey, you shouldn't use this part of the language.

00:21:20.960 --> 00:21:23.340
I just, you know, wanted to make sure it wasn't ridiculous to him.

00:21:23.980 --> 00:21:29.560
And, you know, he said that it's actually an implementation detail of how the Python, the CPython runtime is implemented.

00:21:29.560 --> 00:21:37.600
Um, the way that the for else block, the else with loops works has to do with the way these go to statements actually work in the Python interpreter.

00:21:37.600 --> 00:21:47.060
Uh, and so if you, if you, if you wrote that code, then it makes perfect sense because the else has to do with the way the certain switch or go to statement works.

00:21:47.060 --> 00:21:48.440
Um, it's switch statements.

00:21:48.440 --> 00:21:48.600
Sorry.

00:21:48.600 --> 00:21:49.260
It's not a go to thing.

00:21:49.260 --> 00:21:49.680
It's a switch.

00:21:49.680 --> 00:21:53.960
And, but if you don't have that mental model, then it makes no sense.

00:21:53.960 --> 00:22:04.940
And, you know, I think that, so for, for Guido and maybe other people who've hacked on the core, they just get it and it makes sense to them, but everyone else they have, it does the opposite of what you would expect.

00:22:04.940 --> 00:22:08.340
And so just because of that, I think it's too sharp and I think it's something you should avoid.

00:22:08.340 --> 00:22:13.020
Talk Python To Me is partially supported by our training courses.

00:22:13.020 --> 00:22:19.220
Do you want to learn Python, but you can't bear to subscribe to yet another service at Talk Python Training?

00:22:19.580 --> 00:22:20.560
We hate subscriptions too.

00:22:20.560 --> 00:22:21.040
We hate subscriptions too.

00:22:21.040 --> 00:22:26.580
That's why our course bundle gives you full access to the entire library of courses for one fair price.

00:22:26.580 --> 00:22:27.700
That's right.

00:22:27.700 --> 00:22:33.640
With the course bundle, you save 70% off the full price of our courses and you own them all forever.

00:22:33.640 --> 00:22:39.760
That includes courses published at the time of the purchase, as well as courses released within about a year of the bundle.

00:22:40.000 --> 00:22:45.260
So stop subscribing and start learning at talkpython.fm/everything.

00:22:46.700 --> 00:22:47.260
All right.

00:22:47.260 --> 00:22:50.420
So the next chapter was functions and there's a lot of good stuff in there.

00:22:50.420 --> 00:22:57.760
One of them that I'm a fan of is the whole sort of concept of it's easier to ask for forgiveness than permission.

00:22:57.760 --> 00:22:58.860
Okay.

00:22:59.260 --> 00:23:05.560
And, you know, that sort of manifests in your recommendation of saying prefer exceptions rather than returning none from functions.

00:23:05.560 --> 00:23:06.000
Yeah.

00:23:06.000 --> 00:23:06.900
Yeah, definitely.

00:23:06.900 --> 00:23:13.440
Yeah, this one I think that people, they're always looking, I don't know if you've ever heard of like a tribool.

00:23:13.440 --> 00:23:15.140
There's always this joke about a tribool.

00:23:15.140 --> 00:23:18.260
It's a Boolean value of three, you know, values, tribool.

00:23:18.260 --> 00:23:23.700
People are always dying for a tribool because it's, you know, it's true or it's false or it's some exceptional case that's neither true or false.

00:23:24.420 --> 00:23:28.040
And none in Python ends up playing that role a lot of the time.

00:23:28.040 --> 00:23:37.040
And that's a problem because in Python, both false and none evaluate false.

00:23:37.040 --> 00:23:38.080
They're falsy values.

00:23:38.080 --> 00:23:45.700
And so what ends up happening is it introduces a lot of bugs into your code because you say, hey, you know, x equals this function call.

00:23:45.700 --> 00:23:48.020
And then if x, then do this other thing.

00:23:48.020 --> 00:23:51.360
Or if not x, do this, then there's an error.

00:23:51.880 --> 00:23:59.240
But if you're returning none, then that if not x kind of statement is going to run no matter what.

00:23:59.240 --> 00:24:03.980
And so my whole point, I've been hit by this bug in production a whole bunch of times.

00:24:03.980 --> 00:24:07.380
I had to learn this one the hard way over the course of many years.

00:24:07.380 --> 00:24:12.080
And I'm still managing a code base that has a bunch of this in it and trying to take it out.

00:24:13.280 --> 00:24:16.980
And so my advice here is if you have anything exceptional, just always raise an exception.

00:24:16.980 --> 00:24:18.620
That's what exceptions are for.

00:24:18.620 --> 00:24:28.420
I think people coming from other languages are used to exceptions costing a lot of CPU time or somehow, you know, cause a lot of cache problems.

00:24:28.420 --> 00:24:30.700
And that's true for C++ and other languages.

00:24:31.380 --> 00:24:36.380
But in Python, raising exception is a little bit of cost to it, but it's not enough to matter.

00:24:36.380 --> 00:24:38.420
And the clarity is worth it.

00:24:38.420 --> 00:24:41.560
Not enough to introduce bugs into your code, right?

00:24:41.560 --> 00:24:42.200
Yeah, exactly.

00:24:42.200 --> 00:24:44.280
It's like, okay, well, it's fast, but it doesn't work correctly.

00:24:44.280 --> 00:24:46.620
It's like, all right, well, if that's what you want, then that's fine.

00:24:46.620 --> 00:24:49.180
Yeah, that's not a trade-off you want to make, right?

00:24:49.180 --> 00:24:50.480
Yeah, not the right trade-off.

00:24:50.480 --> 00:24:51.100
Sure.

00:24:51.240 --> 00:24:59.460
So another one you had around functions that I thought was good advice was if you're going to return a lot of data, just make that function a generator rather than returning a list.

00:24:59.460 --> 00:25:08.120
Yeah, and this kind of goes hand in hand with, you know, generator statements that I was talking about before.

00:25:08.120 --> 00:25:10.820
I think I kind of covered both of those when I was talking before.

00:25:10.820 --> 00:25:17.380
So, you know, generator expressions, there's list comprehensions, which, you know, are a fastest way of doing map and filter on lists.

00:25:17.720 --> 00:25:22.620
Then there's generator expressions, which is a way of doing exactly the same thing, but the data is generated one at a time.

00:25:22.620 --> 00:25:28.780
And then a generator function is just a, it's not just a one-liner, it's like a full function that does this.

00:25:28.780 --> 00:25:32.500
And that's, so what I was talking about before, I think I kind of conflated both of those together.

00:25:32.500 --> 00:25:39.600
Yeah, I think just having functions that return, that are generators, like I was saying, lets you kind of start with the leaves and move up to make things streaming.

00:25:39.600 --> 00:25:46.960
Yeah, and it's so easy to just throw in a yield, you know, instead of list.append or whatever, right?

00:25:47.060 --> 00:25:47.620
Yeah, exactly.

00:25:47.620 --> 00:25:48.900
It's very clear.

00:25:48.900 --> 00:25:58.300
And what's interesting is that, you know, JavaScript and other languages are adopting the yield semantics because it's such a powerful tool that people love to use, which is really great.

00:25:58.300 --> 00:25:59.900
And I'm sure Python stole it from something else.

00:25:59.900 --> 00:26:03.880
I actually don't know exactly where it yields from originally, but it's a really great tool.

00:26:03.880 --> 00:26:05.700
Yeah, it's really great.

00:26:05.700 --> 00:26:12.780
And it's one of those that I find that a lot of people, they haven't taken the time to really learn it and appreciate it.

00:26:12.780 --> 00:26:15.180
They know it's there and then they just kind of ignore it, right?

00:26:15.180 --> 00:26:18.280
But knowing, sort of raising the awareness is great.

00:26:18.280 --> 00:26:19.480
Yeah, definitely.

00:26:19.760 --> 00:26:21.040
And it should be the default.

00:26:21.040 --> 00:26:24.080
I think you should just always, you should like plan to yield.

00:26:24.080 --> 00:26:29.560
Plan, like if you're returning and it's not, if you're returning a sequence, like question what you're doing.

00:26:29.560 --> 00:26:31.320
Yeah, exactly.

00:26:31.320 --> 00:26:32.860
The thing better be pretty short.

00:26:33.320 --> 00:26:47.840
So the last one I want to talk about in functions, which I haven't necessarily decided my feelings on it yet, but is if you use keyword arguments, that can help in the expressivity and clarity of functions.

00:26:47.840 --> 00:26:48.980
Yes.

00:26:48.980 --> 00:26:55.080
So I think the actual recommendation was to use just keyword arguments if you get a chance sometimes, right?

00:26:55.080 --> 00:26:56.040
Go ahead.

00:26:56.040 --> 00:26:56.380
Yeah.

00:26:56.380 --> 00:26:57.220
Maybe speak to that a bit.

00:26:57.220 --> 00:26:57.580
Sure.

00:26:57.580 --> 00:27:01.880
So, well, I was curious what part you don't necessarily agree with because I'm curious.

00:27:02.540 --> 00:27:08.300
So, yeah, I mean, because I have a few different parts to my advice, but yeah, keyword arguments are extremely powerful.

00:27:08.300 --> 00:27:13.460
They let you be explicit about your intention.

00:27:13.460 --> 00:27:21.140
And yeah, I think that using them in general for optional parameters is really important.

00:27:21.140 --> 00:27:26.700
So if you have a function that takes an optional flag, then you should always use, you know, keyword argument is a great way to do that.

00:27:26.700 --> 00:27:28.820
No, I was going to say that I totally agree with.

00:27:28.820 --> 00:27:36.180
I think where there's an optional thing, it definitely helps because it's not part of what you expect to pass all the time.

00:27:36.180 --> 00:27:36.460
Right.

00:27:36.460 --> 00:27:39.340
Yeah, because Python doesn't have polymorphism like other languages.

00:27:39.340 --> 00:27:43.080
So in Java, you can define the same function name that takes three different sets of parameters.

00:27:43.080 --> 00:27:44.560
You know, C++ can do that too.

00:27:44.560 --> 00:27:46.060
And in Python, you can't do that.

00:27:46.740 --> 00:27:57.720
So the only way to deal with this is to basically add additional optional parameters to one function and have it kind of deal with the various types of input or define a totally different function of the different name, which can get verbose.

00:27:57.720 --> 00:27:59.600
Right.

00:27:59.920 --> 00:28:01.260
Yeah, I'm not sure I disagree with you.

00:28:01.260 --> 00:28:03.440
I just haven't fully internalized that one yet.

00:28:03.440 --> 00:28:03.600
Yeah.

00:28:03.600 --> 00:28:20.700
So the use case that I was thinking of is if you have some kind of rich IDE, something like PyCharm or Python tools for Visual Studio or something, and you hit open parenthesis and there's like a nice listing of all the parameters without even looking at the docs, which for optionals, I think that still does show up.

00:28:20.700 --> 00:28:23.280
But if you just do like the star star KWRs.

00:28:23.280 --> 00:28:23.760
Oh, yeah.

00:28:23.760 --> 00:28:24.720
Then you got a problem.

00:28:24.720 --> 00:28:28.660
Then it's really like, okay, what the heck can I pass?

00:28:28.660 --> 00:28:29.840
Yeah, no, that's totally true.

00:28:29.840 --> 00:28:33.280
So star star KWRs, which basically is a catch all.

00:28:33.280 --> 00:28:36.220
It's you catch any keyword arguments no matter what they are, including garbage.

00:28:36.220 --> 00:28:37.740
It can be a problem.

00:28:37.740 --> 00:28:46.340
And this is one of the things that I've tried really hard in the book to do, which is I provide advice that applies both to Python 2 and 3.

00:28:46.340 --> 00:28:48.260
So it's not a Python 2 book.

00:28:48.260 --> 00:28:49.300
It's not a Python 3 book.

00:28:49.380 --> 00:28:52.520
It's like the most overlapping subset that I could find of both.

00:28:52.520 --> 00:28:55.380
So it should be relevant to everyone.

00:28:55.380 --> 00:28:56.800
And this is a great example.

00:28:56.800 --> 00:29:04.660
Python 3 has specific language features that would make it so your IDE will continue to work properly, whereas Python 2 does not.

00:29:04.660 --> 00:29:10.220
So in Python 2, you can't say these arguments can only be passed by keyword.

00:29:10.220 --> 00:29:16.480
And you want to do that in situations where you're passing two integer values and you don't know which is which.

00:29:16.660 --> 00:29:22.620
And so you want to use the keyword part in the function call to make it really clear that, like, hey, this is the numerator and this is the denominator.

00:29:23.200 --> 00:29:24.580
And it's not the other way around.

00:29:24.580 --> 00:29:28.080
So here's a label in the function call.

00:29:28.080 --> 00:29:31.480
Numerator equals 5 and denominator equals 10 or something like that.

00:29:32.580 --> 00:29:37.280
And you want to enforce that behavior so that people can't call it by just passing 5 and 10 or 10 and 5 on accident.

00:29:37.280 --> 00:29:44.360
And so in Python 3, you can just add this star into the arguments list and then that's enforced by the compiler and everything's great.

00:29:46.040 --> 00:29:47.800
And in Python 2, you can't do that.

00:29:47.800 --> 00:29:50.100
So that star star kwrks is a way to deal with that.

00:29:50.100 --> 00:29:51.580
And you're right.

00:29:51.580 --> 00:29:52.220
It breaks your IDE.

00:29:52.220 --> 00:29:54.000
It makes it harder to understand the documentation.

00:29:54.000 --> 00:29:58.360
You have to make that tradeoff for you and say, hey, how error-prone is this function?

00:29:58.360 --> 00:30:03.200
Is it worth doing this, you know, having these optional parameters that are in this way?

00:30:04.080 --> 00:30:11.960
And, yeah, maybe there's a better design that you could have, maybe a helper class that would be a better way to approach it than having kwrks, star star kwrks, sitting there.

00:30:11.960 --> 00:30:12.600
Right.

00:30:12.600 --> 00:30:13.960
It's definitely a cool language feature.

00:30:13.960 --> 00:30:21.720
It's just I think sometimes it gets overused and you're like, okay, I've got to basically keep the documentation up for a while so I know what my possibilities are, you know?

00:30:21.720 --> 00:30:22.800
You're totally right.

00:30:22.800 --> 00:30:27.540
I would say don't use star star kwrks unless you're doing something like this.

00:30:27.540 --> 00:30:29.500
I think it's one of the few times you should use it.

00:30:30.080 --> 00:30:36.700
There's some other glue kind of infrastructure times that are good to use it if you're doing, like, generic wrappers of functions and stuff like that.

00:30:36.700 --> 00:30:38.880
But in general, I try to stay away from it.

00:30:38.880 --> 00:30:43.780
And I'm really happy that Python 3 has this extra forcing keyword arcs feature.

00:30:43.780 --> 00:30:44.560
Yeah, that's cool.

00:30:44.560 --> 00:30:46.380
Okay, then I definitely agree with you.

00:30:46.380 --> 00:30:47.380
Okay.

00:30:47.380 --> 00:30:47.880
Thanks.

00:30:47.880 --> 00:31:05.180
This episode is brought to you by Codeship.

00:31:05.180 --> 00:31:14.020
Codeship has launched organizations, create teams, set permissions for specific team members, and improve collaboration in your continuous delivery workflow.

00:31:14.620 --> 00:31:19.900
Maintain centralized control over your organization's projects and teams with Codeship's new organizations plan.

00:31:19.900 --> 00:31:25.440
And as Talk Python listeners, you can save 20% off any premium plan for the next three months.

00:31:25.440 --> 00:31:28.960
Just use the code TALKPYTHON, all caps, no spaces.

00:31:28.960 --> 00:31:34.700
Check them out at Codeship.com and tell them thanks for supporting the show on Twitter where they're at Codeship.

00:31:40.060 --> 00:31:41.760
So let's move on to classes.

00:31:41.760 --> 00:31:46.320
One thing that Python supports is multiple inheritance.

00:31:46.320 --> 00:31:56.920
And speaking about, oh, that's sharp, that hurts, is I did a lot of C++ and even COM with insane templating, multiple inheritance.

00:31:56.920 --> 00:32:02.960
And I just, I kind of prefer single inheritance and just keep it really simple.

00:32:03.140 --> 00:32:09.300
And so you were, you're saying that to some degree and you're saying, look, this multiple inheritance stuff should be used for mix-ins.

00:32:09.300 --> 00:32:10.580
Yeah, definitely.

00:32:10.580 --> 00:32:11.100
Yeah.

00:32:11.100 --> 00:32:11.320
So I.

00:32:11.320 --> 00:32:15.240
Can you tell people what you mean by mix-ins in this case and like how that would go?

00:32:15.240 --> 00:32:16.340
Yeah.

00:32:16.420 --> 00:32:22.360
So I think that a mix-in to me is, is a set of functionality that you can add to a class.

00:32:22.360 --> 00:32:23.980
You can add to any class.

00:32:23.980 --> 00:32:34.280
So helpers for serialization, helpers for logging, helpers for, you know, doing some kind of introspection on functions and stuff like that.

00:32:34.280 --> 00:32:37.060
Those are the mix-ins where you're like, hey, I have this class.

00:32:37.060 --> 00:32:40.120
Wouldn't it be nice if it automatically logged every function call or something like that?

00:32:40.120 --> 00:32:41.580
Or every attribute access.

00:32:41.580 --> 00:32:45.060
Let me, let me, let me inherit from this mix-in utility to do that for me.

00:32:45.600 --> 00:32:52.220
So I think that those are the times where you want to use multiple inheritance, where the class structure doesn't actually matter.

00:32:52.220 --> 00:32:59.980
It's more like aspect-oriented programming or something like that, where, which I'm not necessarily a huge fan of, but I'm just trying to say that it's nice to be able to kind of compose functionality.

00:32:59.980 --> 00:33:05.140
And so mix-ins are a way of doing composition with class inheritance.

00:33:05.140 --> 00:33:09.180
If there were another way to express that composition in Python, that'd be great.

00:33:09.180 --> 00:33:12.340
But the tool that we have is, is multiple inheritance.

00:33:12.340 --> 00:33:12.820
Yeah.

00:33:13.220 --> 00:33:18.420
And I have seen some really beautiful mixing code that, that when done right, right, is nice.

00:33:18.420 --> 00:33:19.420
Yeah.

00:33:19.420 --> 00:33:21.880
When it's done correctly, it's, it's really, really nice.

00:33:21.880 --> 00:33:24.720
And, and it doesn't, it's not brittle.

00:33:24.720 --> 00:33:29.720
I think a lot of the time, multiple inheritance is, is really brittle and it breaks in weird ways when you refactor it.

00:33:29.720 --> 00:33:32.240
And mix-ins are built not to be brittle.

00:33:32.240 --> 00:33:33.600
And so that's why it works out.

00:33:33.720 --> 00:33:36.300
So it's about composition, primarily.

00:33:36.300 --> 00:33:37.320
Nice.

00:33:37.320 --> 00:33:40.480
So let's move on to the next section, meta classes.

00:33:40.480 --> 00:33:41.120
Yeah.

00:33:41.600 --> 00:33:49.960
And in there, one thing that was sort of stood out to me was you're talking about that you can register class existence with the meta class.

00:33:49.960 --> 00:33:51.360
What's the story of that?

00:33:51.360 --> 00:33:52.340
Yeah.

00:33:52.340 --> 00:33:56.440
So, you know, meta classes are like the sharpest tool you can poke yourself with in Python.

00:33:57.320 --> 00:34:02.560
And it's kind of like, once you use it enough, Python enough, you're like, okay, I'm going to try out these meta classes thing.

00:34:02.560 --> 00:34:10.480
And maybe you read Eric Raymond's thing about meta classes and some email filtering thing he wrote with them and how awesome that was.

00:34:10.480 --> 00:34:14.600
And you realize that essay is from 12 years ago and it's worth to think about that again.

00:34:14.600 --> 00:34:18.440
But it's a really powerful tool.

00:34:18.440 --> 00:34:22.720
I have three different kind of main ways I think you should use it in Python.

00:34:22.720 --> 00:34:25.560
I don't think you should use it in other ways unless you really know what you're doing.

00:34:25.560 --> 00:34:28.820
And the registration is one of the big ones.

00:34:28.820 --> 00:34:38.060
So it's really nice to know, you know, if you're creating infrastructure and you're creating, you know, classes to represent database tables or sensors that self-register,

00:34:38.560 --> 00:34:45.740
a lot of that kind of like, you want the program to kind of sign itself up and initialize itself up so you don't have to write a lot of boilerplate to say,

00:34:45.740 --> 00:34:50.160
oh yeah, hey, remember, configure this sensor, configure that sensor, configure this database row.

00:34:50.160 --> 00:34:57.240
You know, you have a lot of programs end up having these main functions that are just like lists of registration calls for 100 lines of registration.

00:34:57.240 --> 00:34:58.700
And then it says go after that.

00:34:58.700 --> 00:35:07.200
And so meta classes give you a really nice way of saying, hey, anytime anyone creates a class,

00:35:07.360 --> 00:35:15.860
it's part of this hierarchy, register it in a central database of classes or central, you know, dictionary of classes so that when I start my program,

00:35:15.860 --> 00:35:18.860
I can go through and do all the housekeeping I need to do for initialization.

00:35:20.540 --> 00:35:23.300
So that's one thing, lets you do initialization really easily.

00:35:23.300 --> 00:35:25.780
And it also lets you do lookups really easily.

00:35:25.780 --> 00:35:29.020
So you say, hey, you know, is there a class that does this in the system?

00:35:29.100 --> 00:35:29.600
Yes, there is.

00:35:29.600 --> 00:35:30.520
I know exactly where it is.

00:35:30.520 --> 00:35:31.020
It's right here.

00:35:31.020 --> 00:35:31.940
I've already imported it.

00:35:31.940 --> 00:35:32.760
I've already looked at it.

00:35:32.760 --> 00:35:34.000
I know what it's capable of.

00:35:34.000 --> 00:35:41.920
So anytime you see yourself doing that pattern where you have one of something now and you're going to have 50 of them later and you need a registry of those things,

00:35:41.920 --> 00:35:45.240
there's no reason to explicitly register things.

00:35:45.240 --> 00:35:49.500
You should always just have them automatically register themselves so that it's less error prone.

00:35:49.700 --> 00:35:54.020
And that's the big thing that I'm trying to kind of advise you to do here.

00:35:54.020 --> 00:35:55.940
It's, you know, we're humans.

00:35:55.940 --> 00:35:58.160
We leave things out all the time.

00:35:58.160 --> 00:36:02.200
And so helping people make fewer mistakes is really important.

00:36:02.200 --> 00:36:05.840
And so metaclasses are a great way of helping people make fewer mistakes.

00:36:05.840 --> 00:36:09.760
And that's what that item is specifically about.

00:36:09.760 --> 00:36:10.860
Nice.

00:36:10.860 --> 00:36:15.100
The other sort of topic you talk a little bit about there is attributes.

00:36:15.280 --> 00:36:23.540
I think you talk about like private versus somewhat private, you know, the sort of underscore versus double underscore attributes in there.

00:36:23.540 --> 00:36:23.940
Is that right?

00:36:23.940 --> 00:36:29.940
Yeah, I think it's a common thing also that people have when they come from other languages that have like strict definitions of public and private.

00:36:29.940 --> 00:36:31.820
And, you know, Python doesn't have those things.

00:36:31.820 --> 00:36:35.320
It has kind of best equivalents of that.

00:36:35.320 --> 00:36:41.360
And the gist of it is, you know, you should just make everything public by default in almost all cases.

00:36:42.220 --> 00:36:46.420
Because people are going to go in there and reach in and make things that are private, public anyway.

00:36:46.420 --> 00:36:49.840
There's some nuance to it, but that's the gist of it.

00:36:49.840 --> 00:36:53.280
And Guido refers to this as, you know, we're all consenting adults.

00:36:53.280 --> 00:36:58.100
You know, you should let people reach into the class and use it however they want if they want to.

00:36:59.620 --> 00:37:04.300
As long as you know you're getting yourself into trouble, then, you know, if you're willing to take on that risk, then it's up to you.

00:37:04.300 --> 00:37:08.300
So, yeah, I think the whole idea here is that Python is a dynamic language.

00:37:08.300 --> 00:37:09.800
So things are mutable and more fluid.

00:37:09.800 --> 00:37:16.160
So you don't have to have this kind of draconian public-private enforcement that maybe Java programmers have.

00:37:16.160 --> 00:37:22.020
It's interesting that other languages are so focused on not letting people do things by default,

00:37:22.120 --> 00:37:26.240
where Python is focused on really letting anyone do anything with a class by default.

00:37:26.240 --> 00:37:30.960
Yeah, that's an interesting just language philosophy statement altogether, isn't it?

00:37:31.940 --> 00:37:32.600
Yeah, it is.

00:37:32.600 --> 00:37:35.980
And some people say that's why Python gets difficult for really large projects,

00:37:35.980 --> 00:37:39.580
because that's why defaulting to closed is good.

00:37:39.580 --> 00:37:42.540
It's because once the project's a million lines of code, then, you know, it's too late.

00:37:42.540 --> 00:37:44.280
I understand that.

00:37:44.280 --> 00:37:48.120
But I think, you know, a million line Python project is a very large program.

00:37:48.120 --> 00:37:50.840
A lot of people have never worked on something that size and probably never will.

00:37:50.840 --> 00:37:53.800
So it's kind of like one size does not fit all.

00:37:53.800 --> 00:37:55.220
Yeah, that's a good point.

00:37:55.220 --> 00:37:59.740
I mean, I see a lot of people that learn design patterns and other things, and they just want to apply them.

00:37:59.740 --> 00:38:00.480
Exactly.

00:38:00.780 --> 00:38:01.760
To whatever, right?

00:38:01.760 --> 00:38:06.040
But these design patterns maybe make sense on very large scales or certain circumstances, right?

00:38:06.040 --> 00:38:06.640
Absolutely.

00:38:06.640 --> 00:38:09.200
And that's why I think all the advice needs to be in context.

00:38:09.200 --> 00:38:10.940
You know, okay, you're giving me this advice.

00:38:10.940 --> 00:38:12.080
Who are you?

00:38:12.080 --> 00:38:12.660
What are you doing?

00:38:12.660 --> 00:38:13.640
How big is the code base?

00:38:13.640 --> 00:38:15.480
You know, like, how often do you have to do it?

00:38:15.480 --> 00:38:18.900
Like, I need to know those things because your design constraints are different than mine.

00:38:18.900 --> 00:38:19.320
So.

00:38:19.320 --> 00:38:20.260
Right.

00:38:20.260 --> 00:38:21.500
How similar are we?

00:38:21.500 --> 00:38:21.740
Yep.

00:38:21.740 --> 00:38:27.640
So one of the sections I really liked, and I'm looking forward to digging into it more,

00:38:27.640 --> 00:38:30.020
is the concurrency and parallelism section.

00:38:30.620 --> 00:38:30.940
Yeah.

00:38:31.680 --> 00:38:36.720
And you draw an interesting distinction between what you're calling concurrency and parallelism.

00:38:36.720 --> 00:38:38.080
Can you maybe speak to that first?

00:38:38.080 --> 00:38:42.580
And I think this goes back to, there's this great talk that Rob Pike, who's one of the

00:38:42.580 --> 00:38:47.560
creators of the Go programming language, a language who also works at Google, he did called concurrency

00:38:47.560 --> 00:38:48.520
is not parallelism.

00:38:48.520 --> 00:38:51.200
It's a great, great talk to check out.

00:38:51.740 --> 00:38:54.200
But yeah, it's basically, you know, concurrency is a programming pattern.

00:38:54.200 --> 00:38:59.480
It's a way of doing multiple things at the same time and having tools to let you do that at the programming level.

00:38:59.480 --> 00:39:00.880
So threads are one of those tools.

00:39:00.880 --> 00:39:02.400
Coroutines are one of those tools.

00:39:02.880 --> 00:39:07.860
And then parallelism is actually running two lines of execution on a processor at the same time.

00:39:07.860 --> 00:39:11.720
And that's not necessarily a programming tool.

00:39:11.720 --> 00:39:13.400
It's just the way that the program runs.

00:39:13.400 --> 00:39:17.440
And Python is great at doing concurrency style stuff.

00:39:18.120 --> 00:39:26.340
But Python can actually do, cannot do parallelism because of the global interpreter lock, which I'm sure people listening to this have probably heard of.

00:39:26.340 --> 00:39:29.820
But that's one of its big shortcomings as a language.

00:39:29.820 --> 00:39:37.500
Concurrently, you're thinking of maybe I'm going to make two web service calls at the same time and I can wait for one to come back.

00:39:37.500 --> 00:39:44.920
But parallelism is I'm trying to compute this financial algorithm and it's just a computational thing.

00:39:45.520 --> 00:39:50.160
And breaking that up into steps and trying to run that with threads is not really going to do much for you, right?

00:39:50.160 --> 00:39:51.180
Yeah.

00:39:51.180 --> 00:40:00.720
Or if you're trying to decode two frames from a video stream simultaneously on two different CPU cores, you can't do that in Python without actually going down to the C API.

00:40:00.720 --> 00:40:04.280
Whereas asynchronous things, you can do it in parallel.

00:40:04.280 --> 00:40:09.040
You can even run threads where that computation looks like it's happening in parallel, but it's actually not.

00:40:09.040 --> 00:40:10.880
It's happening concurrently.

00:40:11.100 --> 00:40:19.820
So you can't increase your throughput of computation with threads or any kind of concurrent infrastructure in Python.

00:40:19.820 --> 00:40:23.220
You can't get parallelism, cannot get parallelism that way.

00:40:23.220 --> 00:40:23.440
Right.

00:40:23.440 --> 00:40:29.480
So your first piece of guidance was to use the subprocess module to manage child processes.

00:40:29.740 --> 00:40:34.680
And those might be Python subprocesses or they might just be other executables, right?

00:40:34.680 --> 00:40:35.860
Yeah.

00:40:35.860 --> 00:40:36.380
Yeah.

00:40:36.380 --> 00:40:43.160
And that's a cheap way to get some parallelism because, you know, at the system level, multiple processes can be doing multiple things on multiple cores.

00:40:43.160 --> 00:40:45.680
So you can really take advantage of your computer that way.

00:40:45.680 --> 00:40:46.040
Yeah.

00:40:46.040 --> 00:40:46.280
Nice.

00:40:46.280 --> 00:40:52.500
Another one that you had was to use threads for blocking IO, but to avoid threads for CPU parallelism.

00:40:52.960 --> 00:40:55.180
And that kind of is a little bit what we were talking about, right?

00:40:55.180 --> 00:40:56.060
Yeah.

00:40:56.060 --> 00:40:57.960
Your web service calls exactly that.

00:40:57.960 --> 00:41:04.840
So you can do a web service call in one thread, another web service call in another thread, and they will happen concurrently.

00:41:04.840 --> 00:41:07.800
But they actually won't process in parallel.

00:41:07.800 --> 00:41:14.520
So it's great for kind of blocking tasks or IO bound tasks, threads are, but they don't actually help you go faster.

00:41:15.120 --> 00:41:15.220
Right.

00:41:15.220 --> 00:41:17.440
So verify my mental model here.

00:41:17.440 --> 00:41:28.220
I think I recall that if I have a thread in Python and it does some blocking IO, that thread will release the global lock, right?

00:41:28.220 --> 00:41:29.320
Exactly.

00:41:29.320 --> 00:41:29.680
Yeah.

00:41:29.680 --> 00:41:32.160
So you can use threads to avoid waiting.

00:41:32.160 --> 00:41:33.420
It's basically all you get.

00:41:33.420 --> 00:41:35.940
But you can't use them to.

00:41:35.940 --> 00:41:39.260
And so the gill, yeah, will unlock and then another thread will start running.

00:41:39.260 --> 00:41:41.260
And that's nice.

00:41:41.260 --> 00:41:42.940
And that's what threads are good for.

00:41:42.940 --> 00:41:44.680
But that's as far as it'll go.

00:41:45.000 --> 00:41:45.240
Right.

00:41:45.240 --> 00:41:47.400
Sort of the Node.js style of parallelism, maybe.

00:41:47.400 --> 00:41:48.920
Yeah.

00:41:48.920 --> 00:41:49.860
It's very similar, actually.

00:41:49.860 --> 00:41:51.680
Node has a lot of the same constraints and problems.

00:41:51.680 --> 00:41:54.320
It only has one executing thread at any time.

00:41:54.320 --> 00:41:55.140
Okay.

00:41:55.140 --> 00:42:00.900
So another one was to use or consider coroutines for running many functions concurrently.

00:42:00.900 --> 00:42:02.560
How's that look?

00:42:02.560 --> 00:42:02.980
Yeah.

00:42:02.980 --> 00:42:03.580
Yeah.

00:42:03.580 --> 00:42:05.940
This is my favorite item in the book, item 40.

00:42:05.940 --> 00:42:07.340
Yeah.

00:42:07.340 --> 00:42:09.720
I implement Conway's Game of Life as a set of coroutines.

00:42:09.720 --> 00:42:14.880
I would say if you understand this one, then you've understood the Xenopython pretty

00:42:14.880 --> 00:42:15.160
well.

00:42:15.160 --> 00:42:21.540
And it's basically just trying to say that you can express very complicated, high-level

00:42:21.540 --> 00:42:23.820
ideas of the way that things should interact.

00:42:23.820 --> 00:42:29.120
Whole system flows should work, workflows, using coroutines in this really abstract way that's

00:42:29.120 --> 00:42:32.080
extremely powerful, easy to test, easy to expand.

00:42:33.040 --> 00:42:36.860
It's really hard to go into detail without explaining the whole thing from the beginning.

00:42:36.860 --> 00:42:37.460
Sure.

00:42:37.460 --> 00:42:38.140
It's definitely worth checking out.

00:42:38.140 --> 00:42:39.640
It's actually online, fully published.

00:42:39.640 --> 00:42:44.100
If you search for it, I think you can actually find it on effectivepython.com because I'm so

00:42:44.100 --> 00:42:44.520
proud of that one.

00:42:44.520 --> 00:42:44.860
Yeah, nice.

00:42:44.860 --> 00:42:45.380
That's awesome.

00:42:45.380 --> 00:42:47.200
So you kind of pulled it out so everyone can get to it?

00:42:47.200 --> 00:42:47.860
Yeah, definitely.

00:42:47.860 --> 00:42:48.460
Cool.

00:42:48.560 --> 00:42:52.580
And then the final concurrency one that I thought was interesting was to consider concurrent

00:42:52.580 --> 00:42:54.920
dot futures for true CPU parallelism.

00:42:56.440 --> 00:42:59.820
Yeah, and this is the kind of hack that someone came up with.

00:42:59.820 --> 00:43:04.860
So there's this multiprocessing module that's built into Python now that will actually farm

00:43:04.860 --> 00:43:07.640
out work to sub-processes that are also Python.

00:43:08.720 --> 00:43:14.060
It's really ridiculous how it works, but for certain cases, it's extremely powerful, and

00:43:14.060 --> 00:43:17.480
it will actually speed up your program by the number of cores on your machine.

00:43:17.480 --> 00:43:19.440
It doesn't always work.

00:43:19.440 --> 00:43:25.620
It also breaks in a lot of weird ways, but when it does work, it's magical because you can go

00:43:25.620 --> 00:43:28.220
from something that was slow to something that's 10 times faster.

00:43:28.220 --> 00:43:30.280
So it's worth checking out, at least trying.

00:43:30.280 --> 00:43:31.300
I just wouldn't go too far.

00:43:31.300 --> 00:43:35.880
I think you just summed up multi-threaded parallel programming right there.

00:43:36.440 --> 00:43:41.120
It sometimes works, it sometimes breaks in weird ways, and when it works really well,

00:43:41.120 --> 00:43:42.020
it's magical, right?

00:43:42.020 --> 00:43:43.260
Yeah, it's true.

00:43:43.260 --> 00:43:44.380
Yeah, that's very true.

00:43:44.380 --> 00:43:45.380
I guess that's no exception.

00:43:45.380 --> 00:43:51.000
Yeah, but Python has its own peculiarities about how it breaks and how it works and whatever,

00:43:51.000 --> 00:43:51.300
right?

00:43:51.300 --> 00:43:52.800
That's a really good point.

00:43:52.800 --> 00:43:55.500
People forget to mention that, oh, Python only has one thread.

00:43:55.500 --> 00:43:56.400
It's like, that is a problem.

00:43:56.400 --> 00:43:57.160
It's a problem.

00:43:57.160 --> 00:44:01.500
But multiprocessing, multiprocess programming, multithreading in any other language was also

00:44:01.500 --> 00:44:04.020
really hard and error-prone, so let's not forget that.

00:44:04.020 --> 00:44:05.840
That's a great point.

00:44:05.840 --> 00:44:11.520
There was a group that was referring to the bugs that you get from multithreading as

00:44:11.520 --> 00:44:14.720
Heisen bugs, and I really like that way of thinking about it.

00:44:14.720 --> 00:44:15.480
Yeah, definitely.

00:44:15.480 --> 00:44:16.840
They're very hard to track down.

00:44:16.840 --> 00:44:22.220
So let's talk about just one more section in the book, and that was collaboration, like

00:44:22.220 --> 00:44:26.540
how you work well with other people and sort of like on teams and so on, right?

00:44:27.400 --> 00:44:28.240
Yeah, yeah.

00:44:28.240 --> 00:44:31.800
And I think Python is a huge community, like we were talking about before, and so working

00:44:31.800 --> 00:44:33.340
with others in Python is really important.

00:44:33.340 --> 00:44:34.540
Yeah, that's cool.

00:44:34.540 --> 00:44:39.280
And one of your pieces of guidance was to use packages rather than just patch around a bunch

00:44:39.280 --> 00:44:39.940
of files, right?

00:44:40.720 --> 00:44:41.000
Yeah.

00:44:41.000 --> 00:44:43.040
And this all depends on the size of your code base.

00:44:43.040 --> 00:44:44.420
You know, when you're starting out, it doesn't matter.

00:44:44.420 --> 00:44:48.020
Like until you have 10 files or something like that, it really doesn't matter.

00:44:48.020 --> 00:44:52.700
But you know, you should be willing to refactor your code into packages once you realize you

00:44:52.700 --> 00:44:57.180
have enough different working pieces, especially if you're working with different people.

00:44:57.860 --> 00:45:01.740
It's nice to split things a little earlier than you might have expected.

00:45:01.740 --> 00:45:02.980
So you give them room to grow.

00:45:02.980 --> 00:45:06.360
It's kind of like you'd buy, you know, if you have a kid, you buy them shoes a little, like

00:45:06.360 --> 00:45:08.520
one size a little bit bigger so that they can grow into them.

00:45:08.520 --> 00:45:09.340
Yeah.

00:45:09.340 --> 00:45:10.960
So packages are kind of like that.

00:45:10.960 --> 00:45:15.060
I think it's a good way to modularize your code base before it becomes too late.

00:45:15.060 --> 00:45:17.440
And I've been in code bases where it's been way too late.

00:45:17.440 --> 00:45:20.000
And we have files, six lines of Python code in them.

00:45:20.000 --> 00:45:20.820
And you're just like, why?

00:45:20.820 --> 00:45:22.140
How did this even happen?

00:45:22.140 --> 00:45:24.360
It's terrible.

00:45:24.360 --> 00:45:26.180
Yeah, I can imagine.

00:45:26.180 --> 00:45:29.800
Do you recommend having like an internal private package server?

00:45:29.800 --> 00:45:33.840
You know, it really depends on your company.

00:45:33.840 --> 00:45:37.420
I think if you're comfortable with PIP, then yeah, it's pretty good, especially if you're

00:45:37.420 --> 00:45:38.460
doing deployments off of it.

00:45:38.460 --> 00:45:39.320
It can be really useful.

00:45:39.320 --> 00:45:42.780
If you have a security team that needs to vet things, it can be really valuable.

00:45:42.780 --> 00:45:44.780
Right.

00:45:44.780 --> 00:45:48.960
Maybe one team that builds the API and another team that builds the web part that consumes

00:45:48.960 --> 00:45:50.100
the API or something like that.

00:45:50.100 --> 00:45:51.520
Yeah, you can do stuff like that.

00:45:51.520 --> 00:45:53.360
I mean, it also depends on your source code repo.

00:45:53.360 --> 00:45:56.060
Google, we have just one giant repo for the entire company.

00:45:56.060 --> 00:45:58.640
So for us, I don't need a package server because I have the code.

00:45:58.640 --> 00:46:04.220
But yeah, it's good hygiene to have people publishing APIs that are stable and to think

00:46:04.220 --> 00:46:05.200
about things in that way.

00:46:05.200 --> 00:46:08.800
It's kind of the Netflix or Amazon approach to software engineering.

00:46:08.800 --> 00:46:12.520
And it works really well, more like a service-oriented architecture, but at the API level.

00:46:12.520 --> 00:46:13.520
Yeah, that's cool.

00:46:13.520 --> 00:46:22.820
So another piece of advice you had for collaboration was to consider the Reaper method for debugging

00:46:22.820 --> 00:46:26.020
output and helping people understand the state of your program.

00:46:26.020 --> 00:46:26.480
Yeah.

00:46:26.480 --> 00:46:27.460
Yeah.

00:46:27.460 --> 00:46:32.020
So yeah, Reaper Reaper print representation.

00:46:32.020 --> 00:46:33.600
I don't know why they call it that.

00:46:33.660 --> 00:46:34.580
I think it's pretty silly.

00:46:34.580 --> 00:46:37.700
But yeah, this is the most valuable tool for me in Python.

00:46:37.700 --> 00:46:40.760
Like, print is the first most valuable tool for me in Python.

00:46:40.760 --> 00:46:43.500
But the second most valuable tool besides print is Reaper.

00:46:43.500 --> 00:46:47.580
Because you can get detailed information about an object.

00:46:48.860 --> 00:46:53.400
And you can basically make an object that'll self-describe itself so that when you're debugging, you can

00:46:53.400 --> 00:46:55.180
print it out and say, hey, what's in this thing?

00:46:55.180 --> 00:46:56.840
What were the parameters that created this object?

00:46:56.840 --> 00:46:57.780
How did I even get here?

00:46:57.780 --> 00:47:01.260
Reaper and under-under-repper is a really great way to do that.

00:47:01.260 --> 00:47:03.400
And Python has built-in support for it all over the place.

00:47:03.400 --> 00:47:05.220
So it's definitely worth checking out if you don't use it.

00:47:05.220 --> 00:47:05.780
Right.

00:47:05.780 --> 00:47:06.100
Absolutely.

00:47:06.100 --> 00:47:11.400
If you just take a list and print it, you'll get that representation for all the items it

00:47:11.400 --> 00:47:11.760
contains.

00:47:11.760 --> 00:47:12.720
That's right.

00:47:12.720 --> 00:47:16.460
And it basically evaluates back to the Python statement that it would have taken to create the

00:47:16.460 --> 00:47:17.000
same list.

00:47:17.000 --> 00:47:22.220
And so you usually can copy and paste reppers back into an interpreter to test them out.

00:47:22.220 --> 00:47:26.540
So it's a really nice way to do kind of an interactive debugging session with your own code.

00:47:26.540 --> 00:47:27.180
That's cool.

00:47:27.180 --> 00:47:32.460
And the final collaboration thing that I actually hadn't heard of was you can understand memory

00:47:32.460 --> 00:47:35.980
allocations and leaks with something called trace malloc.

00:47:35.980 --> 00:47:37.080
Yeah.

00:47:37.080 --> 00:47:39.860
And this is another Python 3 only thing.

00:47:39.860 --> 00:47:43.040
And it's wonderful.

00:47:44.660 --> 00:47:47.860
Because if you've ever had memory leaks in Python, for especially long-running processes,

00:47:47.860 --> 00:47:51.720
it's really hard to track down where your memory usage is going.

00:47:51.720 --> 00:47:56.060
Any project of size that I've been on, we run a lot of servers at Google, so we have this

00:47:56.060 --> 00:47:56.380
a lot.

00:47:56.380 --> 00:48:00.460
You end up having this problem where you look at your garbage collection or your heap, and

00:48:00.460 --> 00:48:03.220
it just keeps growing.

00:48:03.220 --> 00:48:05.160
You're like, where's the actual space going?

00:48:05.760 --> 00:48:07.640
And the tools to do that in Python are horrible.

00:48:07.640 --> 00:48:10.420
Python 2.7 and before, they're all bad.

00:48:10.420 --> 00:48:13.120
And trace malloc, they finally fixed this in Python 3.

00:48:13.120 --> 00:48:18.160
And yeah, it basically will tell you exactly like, this is the object you have too many of.

00:48:18.160 --> 00:48:19.720
This is exactly where it was allocated.

00:48:19.720 --> 00:48:21.580
Here are the parameters that were used at the time.

00:48:22.260 --> 00:48:26.980
And so now it's like, orders of magnitude easier to figure it out, which is really great.

00:48:26.980 --> 00:48:31.180
There is a port of that code to Python 2.7, I believe.

00:48:31.180 --> 00:48:33.320
But I don't know how well-maintained it is.

00:48:33.320 --> 00:48:38.120
And I don't think it's actually, it has to actually modify the CPython run, like binary

00:48:38.120 --> 00:48:38.640
itself.

00:48:38.640 --> 00:48:40.660
It can't be done as a C extension module.

00:48:40.840 --> 00:48:44.860
So I think it'll never be added to Python 2, which is unfortunate.

00:48:44.860 --> 00:48:45.380
Right.

00:48:45.380 --> 00:48:49.280
So you might have to like build your own version of the source code with that baked in and then

00:48:49.280 --> 00:48:50.740
run your code on that.

00:48:50.740 --> 00:48:53.360
Which is pretty scary, but it might be worth it.

00:48:53.360 --> 00:48:55.800
Yeah, maybe not for use in production too much.

00:48:55.800 --> 00:48:56.980
Yeah, I wouldn't use in production.

00:48:56.980 --> 00:48:58.200
For sure.

00:48:58.200 --> 00:48:59.380
Okay.

00:48:59.380 --> 00:49:04.760
So those are a bunch of really awesome things that I think will help, you know, pretty

00:49:04.760 --> 00:49:05.720
much people of all level.

00:49:05.720 --> 00:49:06.580
Thanks, man.

00:49:06.580 --> 00:49:07.360
Yeah.

00:49:07.360 --> 00:49:09.380
How long has the book been out?

00:49:09.720 --> 00:49:12.540
It's been out since April, basically.

00:49:12.540 --> 00:49:15.520
And it's doing pretty well.

00:49:15.520 --> 00:49:20.680
And this week we also have a video that's coming out, which is a lot of the same content, but

00:49:20.680 --> 00:49:23.380
in me just typing out the actual code format.

00:49:23.380 --> 00:49:29.200
So some people who learn better from seeing demonstrations, it might be better for them.

00:49:29.200 --> 00:49:34.520
But yeah, it's had a lot of good feedback, a lot of good reviews on Amazon and other places.

00:49:34.520 --> 00:49:38.160
A lot of bugs reported of things I did wrong, but not too many bugs.

00:49:38.700 --> 00:49:43.260
There's only one bug so far that I'm really embarrassed about where I did a multiply instead

00:49:43.260 --> 00:49:46.220
of divide, but the rest of them are all pretty good.

00:49:46.220 --> 00:49:50.460
Well, the multiply, divide, what's the difference?

00:49:50.460 --> 00:49:50.740
Yeah.

00:49:50.740 --> 00:49:52.920
You know, I just, I got confused.

00:49:52.920 --> 00:49:56.460
Math is hard.

00:49:57.220 --> 00:50:03.260
So people can go get it on Amazon and I'll definitely link to the book, but you have

00:50:03.260 --> 00:50:07.300
a book code that we can give to listeners so they can get a bit of a discount, right?

00:50:07.300 --> 00:50:07.540
Yeah.

00:50:07.540 --> 00:50:12.260
And it's, it's the, the, the code is, FPI, E F F P Y. And the simplest way to use

00:50:12.260 --> 00:50:20.660
it is you go to inform it.com like I N F O R M I T.com slash F PI, all caps E F F P Y.

00:50:21.140 --> 00:50:23.700
And, that, that lets you buy the book directly from the publisher.

00:50:23.700 --> 00:50:28.220
And for some places, that are not in the United States also, it can be cheaper

00:50:28.220 --> 00:50:29.260
this way than going through Amazon.

00:50:29.260 --> 00:50:35.140
Uh, and it's a, it gives a discount off the ebook version, the physical book, and then also

00:50:35.140 --> 00:50:36.860
this new video that just came out this week.

00:50:36.860 --> 00:50:38.100
Uh, you can get them all there.

00:50:38.100 --> 00:50:38.380
Yeah.

00:50:38.380 --> 00:50:39.280
You just released that.

00:50:39.280 --> 00:50:43.100
You, released this, this nice new video training version of the book.

00:50:43.200 --> 00:50:45.380
So, I'll be sure to link to that as well.

00:50:45.380 --> 00:50:45.960
That'd be cool.

00:50:45.960 --> 00:50:46.300
Yeah.

00:50:46.300 --> 00:50:46.880
Thank you.

00:50:46.880 --> 00:50:47.740
Yeah.

00:50:47.740 --> 00:50:48.200
You bet.

00:50:48.200 --> 00:50:52.260
So Brett, anything else you want to give a shout out to or let people know about that

00:50:52.260 --> 00:50:53.420
we somehow didn't touch on?

00:50:53.420 --> 00:50:58.220
Um, the only other thing is, yeah, like the, the book websites, effective python.com and,

00:50:58.220 --> 00:51:03.400
uh, everything you want to know about me, you can find there, and, please report

00:51:03.400 --> 00:51:04.100
any bugs you have.

00:51:04.100 --> 00:51:06.060
Those are on, on GitHub with all the example code.

00:51:06.060 --> 00:51:10.740
And I'm trying to make sure that, anything that's wrong, I, I, I address in there.

00:51:10.740 --> 00:51:13.180
Um, but yeah, if you have questions, there's also a link to

00:51:13.180 --> 00:51:13.900
email me directly.

00:51:13.900 --> 00:51:16.600
So if you, if you hit an issue, I'd love to hear from you.

00:51:16.600 --> 00:51:18.320
I'm happy to answer your question.

00:51:18.320 --> 00:51:19.000
Excellent.

00:51:19.000 --> 00:51:21.500
So two final questions before I let you go.

00:51:21.500 --> 00:51:24.100
What is your current favorite code editor?

00:51:24.100 --> 00:51:26.920
If you're going to sit down and write some Python, what do you, what do you open up?

00:51:26.920 --> 00:51:28.680
I, I am a sublime fan.

00:51:28.680 --> 00:51:29.940
I use text mate for a long time.

00:51:29.940 --> 00:51:32.740
Uh, I still use Emacs, when I'm on a, on a terminal.

00:51:32.740 --> 00:51:34.480
So I use Emacs every day also.

00:51:34.480 --> 00:51:39.480
Um, I don't use VI, but I can use less so I can, I can do the read part of it, I guess.

00:51:39.480 --> 00:51:43.160
Uh, and, but yeah, sublime is my favorite, editor.

00:51:43.160 --> 00:51:44.160
Nice.

00:51:44.160 --> 00:51:52.140
And then of all the 60,000 plus, whatever it is, PyPI packages, what, what ones do you

00:51:52.140 --> 00:51:53.060
think people should know about that?

00:51:53.060 --> 00:51:53.820
Maybe they don't know.

00:51:53.820 --> 00:51:57.240
NumPy, obviously, I guess that's, everyone knows about that one.

00:51:58.040 --> 00:52:04.340
SQLAlchemy is an amazing tool, especially the relational package, for doing, database

00:52:04.340 --> 00:52:04.800
lookups.

00:52:04.800 --> 00:52:07.880
Uh, flask is a great web framework that I like a lot.

00:52:07.880 --> 00:52:13.020
Uh, and then there's a bunch of stuff around, speeding up programs that I'm, I'm trying

00:52:13.020 --> 00:52:13.600
to get more into.

00:52:13.600 --> 00:52:16.220
Um, Theano is one and Numba is another.

00:52:16.220 --> 00:52:20.540
These are things that let you use your GPU through Python, which is super cool.

00:52:20.540 --> 00:52:23.640
And you can use that for things like neural network training and machine learning and stuff

00:52:23.640 --> 00:52:24.860
I haven't really spent enough time doing.

00:52:24.860 --> 00:52:28.900
So I think I'm really excited to try out some more of these computational tools and Python

00:52:28.900 --> 00:52:29.240
packages.

00:52:29.600 --> 00:52:30.400
Yeah, that's really awesome.

00:52:30.400 --> 00:52:35.080
If you, you know, people who've looked at it, they know, but if you've not looked at

00:52:35.080 --> 00:52:39.420
the parallel capabilities of GPUs, it's like, it's astounding, right?

00:52:39.420 --> 00:52:40.120
It's crazy.

00:52:40.120 --> 00:52:43.120
And the fact that you can like use that in three lines of Python is crazy.

00:52:43.120 --> 00:52:48.240
So I think it's just speaks to the community that Python has of scientists and, and, you

00:52:48.240 --> 00:52:49.980
know, it's just such a, such a diverse community.

00:52:49.980 --> 00:52:54.900
So it's really great to see these packages, blossoming in Python before many other languages.

00:52:54.900 --> 00:52:55.900
Yeah, absolutely.

00:52:55.900 --> 00:52:58.340
Brett, it's been a very interesting conversation.

00:52:58.560 --> 00:52:59.560
I appreciate it.

00:52:59.560 --> 00:53:00.120
Yeah.

00:53:00.120 --> 00:53:02.520
Thanks a lot for having me on and asking all those great questions.

00:53:02.520 --> 00:53:02.980
Yeah.

00:53:02.980 --> 00:53:07.480
And I love the book and I encourage the readers to go check it out and they'll enjoy it.

00:53:07.480 --> 00:53:08.500
Thanks very much.

00:53:08.500 --> 00:53:08.880
Yeah.

00:53:08.880 --> 00:53:09.400
Talk to you later.

00:53:09.400 --> 00:53:10.080
All right.

00:53:10.080 --> 00:53:10.360
See ya.

00:53:10.360 --> 00:53:14.380
This has been another episode of talk Python to me.

00:53:14.380 --> 00:53:16.140
Today's guest was Brett Slacken.

00:53:16.140 --> 00:53:19.060
And this episode has been sponsored by hired and code chip.

00:53:19.060 --> 00:53:21.100
Thank you guys for supporting the show.

00:53:21.100 --> 00:53:23.980
Hired wants to help you find your next big thing.

00:53:23.980 --> 00:53:28.380
Visit hired.com slash talk Python to me to get five or more offers with salary.

00:53:28.520 --> 00:53:32.800
And equity presented right up front and a special listener signing bonus of $4,000.

00:53:32.800 --> 00:53:36.180
Code chip wants you to always keep shipping.

00:53:36.180 --> 00:53:40.380
Check them out at code chip.com and thank them on Twitter via at code chip.

00:53:40.380 --> 00:53:42.840
Don't forget about the discount code for listeners.

00:53:42.840 --> 00:53:43.500
It's easy.

00:53:43.500 --> 00:53:45.580
Talk Python, all caps, no spaces.

00:53:46.400 --> 00:53:52.560
You can find the links from the show on talkpython.fm/episodes slash show slash 25.

00:53:52.560 --> 00:53:55.300
While you're there, be sure to subscribe to the show.

00:53:55.300 --> 00:53:57.680
Open your favorite podcatcher and search for Python.

00:53:57.680 --> 00:53:58.940
We should be right at the top.

00:53:58.940 --> 00:54:03.020
You can also find the iTunes and direct RSS feeds in the footer of the website.

00:54:03.680 --> 00:54:08.240
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

00:54:08.240 --> 00:54:12.340
You can hear the entire song at talkpython.fm/home slash music.

00:54:12.340 --> 00:54:15.040
This is your host, Michael Kennedy.

00:54:15.040 --> 00:54:16.180
Thanks for listening.

00:54:16.180 --> 00:54:18.440
Smix, take us out of here.

00:54:21.960 --> 00:54:23.680
Chilled in, haven't been sleeping.

00:54:23.680 --> 00:54:25.320
I've been using lots of rest.

00:54:25.320 --> 00:54:28.180
I'll pass the mic back to who rocked it best.

00:54:28.180 --> 00:54:31.240
Developers, developers, developers.

00:54:31.240 --> 00:54:38.240
Developers, developers, developers.

00:54:38.240 --> 00:54:38.960
developers.

00:54:38.960 --> 00:54:39.460
you

00:54:39.460 --> 00:54:39.960
you