RFC: Hack The Kernel

[spamegg1]

Problem:
Hack The Kernel in Core Systems is poorly designed and supported, and is extremely difficult for its place in the curriculum.

Duration:
April 5, 2020 - May 5, 2020

Current State:
Awaiting feedback (updated Proposal and Alternatives with suggestions)

Proposer:
spamegg1

Proposal:
Replace Hack The Kernel with the textbook OSTEP (Operating Systems: Three Easy Pieces) and its homework assignments, supplemented by material from Wisconsin's CS 537 and its video lectures which closely follows the textbook.

Alternatives:
Replace Hack The Kernel with Udacity's UD923 Introduction to Operating Systems

Details (very, very long):
I took Hack The Kernel. It was BY FAR the most difficult thing ever. It took me 6 months.

This course is poorly designed and supported.

They recorded the university lectures with a camera in the back of the classroom and put them on the web. In most lecture videos you cannot see the projection screen because of light/glare, so you have to read the slides separately. Moreover the audio quality is low, so you cannot hear the questions the students are asking.

I found the lectures to be very vague, unhelpful and underwhelming. The instructor has a tendency to give purely verbal explanations while walking back and forth in front of the classroom, instead of writing anything on the board or typing some code on the screen. There is no indication in the lectures how I am supposed to convert the high level abstract concepts in the lectures to the C programming API (the OSTEP book takes care of this with dedicated chapters). Also the lecture videos are very long and boring. (CS50 did not suffer from this, despite having even longer lecture videos.)

Recitation videos are a little bit more helpful but not by much. They had low audio quality and audio stutters. The TAs are trying VERY HARD "not to give away any answers" they end up talking in the same circles as the instructor; and also they deliberately avoid typing any code at all, instead giving vague hints. A few TAs gave some very helpful hints but again there is no info on how to convert them to C. Moreover the recitation videos are a collection of a bunch of videos, all from different years, where back in the past the assignments were somewhat different. No offence to the TAs, but some of them have very strong foreign accents and were difficult/impossible to understand with the low audio quality (I was a foreign TA in the US so I can sympathize).

The only helpful videos were the ones where the lecturer/TAs had "screencasts" of themselves typing stuff into the terminal, explaining how to setup the code system, how to browse the gigantic code base, using grep on the command line etc. But even these videos were hard to follow due to tiny font sizes and high resolution.

The slides for the course are also not helpful and very vague. The explanations of the concepts in slides don't really stick unless you read a textbook. The slides look like you took pages from a textbook and removed 90% of the text. I could not understand anything from them.

It looks like the course was not updated since 2017. The discussion forums were dead (last posts from 2017) so there is no way to get help. (This is not a unique problem, a lot of Udacity courses are also like this: outdated and "abandoned".) The assignment codes only work with a very outdated, insecure version of Ubuntu (Xenial 16.04 from 2016). The code had some bugs in it, which broke some of the assignment tests, with students reporting them all the way back to then. I was taking it in 2019.

There is no clear indication on the course website after which lecture/slide/recitation I should start each assignment. Also I did not know when I should watch each recitation video, because they were only listed under the pages of each assignment, rather than being incorporated into "the course flow". Students who physically attended this class at Harvard probably had no problem because they are on a schedule, but the creators did not translate this to the web version. Sometimes I ended up going too far in the course material/subjects and started the assignments too late. Sometimes I started the assignments too early without learning necessary concepts.

So at some point (way too late) I decided I should start ASST1. After watching all the related videos and reading all the related text, I had to stop taking the course because I could not get anywhere. Instead I dedicated one month to reading OSTEP (Operating Systems: Three Easy Pieces) cover to cover and doing all of its homework. All of a sudden EVERYTHING MADE SENSE. This book is so good I cannot recommend it enough. It's like going to your favourite professor's office and hanging out with him/talking to him. In fact the book opens and closes each chapter with a Student/Professor dialogue.

Not only OSTEP explained everything at least 10 times better than the lecture/slides/recitations, it had CODE SEGMENTS that implemented the concepts in C. A programming course that shows you code! How evil indeed! Moreover it had dedicated chapters explaining the C Thread API and all its functions. In fact without this code from the book I could not possibly do the assignments. On top of that each chapter has homework assignments with code you can get from the author's website. Most of these homeworks do not make you write code, but rather use author's code to step through multi-thread executions and predict their outcomes.

First I'd like to say that the instructors of the course created a very impressive kernel simulation software for the assignments. Super impressive indeed. It's like a miniature copy of a Linux OS. However the code base is 30000+ lines of code and it's INSANE. Impossible to understand. I had to trace files/code that depend on each other so much, I had to BUY A SECOND MONITOR just so I could look at it all and try to understand it. The code has so much required reading it's crazy. 95% of my time was spent reading insanely complicated low level kernel code (sometimes even assembly).

OK I can understand that... this is supposed to be "real world experience" because if I get a job I'll have to work with large systems and read others' code. The lecturer said that much. But this was definitely not Core CS level. It's even beyond Advanced CS level I think. It's at least Pro CS level (but even most pros would fail at this, since Linux kernel development is far more difficult than other "pro" work). I sincerely do not expect any one of the Advanced Applications or Specializations to be ONE TENTH of this course in difficulty. If I ever entertained any delusions about becoming a kernel developer, they are dead and done.

I would say that the first assignment was.... OK, fair and doable with help from OSTEP/Github (but definitely not with what I got from the lectures). It required not too many lines of code, maybe a bit over 200. I still had to get some hints from other people on Github who wrote their own solutions. Semaphores were already provided, I could use that code as example and just make some changes. The code I had to write was clearly indicated as "fill in the blanks here". I implemented other synchronization primitives (locks, condition variables, reader-writer locks), and solved some classic synchronization problems. In fact I had an "aha!" moment when I solved the Buffalo Intersection problem, it felt amazing. It just clicked, and I put a semaphore into the code, and, voila! Max concurrency achieved!

Now we get to the second and third assignments. Oh boy this was pure hell.

ASST2 asked us to design our own process/file handle/file descriptor/PID/system call system FROM SCRATCH. Holy hell. There was far less code provided. We also had to create a very very long design document. It took me 2 months to just write the design document. Just tracing one system call takes you through 10+ different files... I had absolutely no idea whatsoever where to start on designing any of that. I could read the actual, real-life Linux kernel code and base my design on that... but come on. I had to get some idea from others on Github. After a ton of struggle for a total of 3 months my 800+ lines of code could pass most of the tests with the exception of some tests that were reported possibly bugged. The instructors already admit that "at this point the code is so complicated, it's already becoming impossible to debug anything so don't even bother". Good luck trying to sort through the horrible text interface of GDB.

ASST3 was straight up INSANE. It asked us to design OUR OWN virtual memory system FROM SCRATCH. This is equivalent to 2200+ lines of code MINIMUM. No skeleton code provided whatsoever. There were far less instructions/help than ASST2. At least the lecturer is honest and he admitted that "this is supposed to drive you insane and make you tear out your hair." At this point I threw in the towel and just cheated. I looked at others' code on Github and tried to make sense of it (also realized a lot of repos stopped after ASST2). I wrote some code in no way I could call my own. Anyways I decided to consider it closed and move on to the next course in OSSU. I could go on for several more paragraphs into detail of this hell but this is already too long.

You may object by saying "you just suck at C" but in no way this was due to my lack of C knowledge or shortcomings. After all I finished ASST1 and 2 (admittedly with LOTS of help). No amount of C programming books/courses would help. Yes there was a function with triple pointers at some point, but I understand all that.

DESPITE ALL THAT I think the course might be worthwhile for some people out there, who are very hardcore and want to challenge themselves super hard, and want to go into low level programming. So it makes sense to keep it in Advanced Systems (where it would be an elective). Even in that case though, definitely read OSTEP anyways and check out others' code on Github.

[spamegg1]

Digression:
Anyways I decided to consider it closed and move on to the next course in OSSU. Otherwise I could have spent another year on HtK. It was becoming unhealthy. Then I started to read about other fellow OSSUians struggling with this course, not even making a dent into ASST1, having NO IDEA what to do. I wanted to spare them the pain and told them to move on. After all it's not useful if they are not planning on becoming kernel developers or do any low level coding in their future career.

Moreover I heard about the "notoriety" of this course in Harvard, how it "broke Mark Zuckerberg and made him drop out". I even watched the movie "The Social Network" where there is one scene of this class with Jesse Eisenberg answering a Virtual to Physical address translation question, then walking/dropping out. Then all my experiences in grad school came back to me: the uncooperative instructors deliberately withholding information and obscuring things, torturing students and calling it "tough", the gate keeping and elitism, shaming and intimidation labelled "Honour Codes".

This course completely changed my attitude towards these so-called "Honour Codes" and online learning in general. I realized that Honour Codes are EVIL, enemies of learning, ways for some instructors to protect their inflated fragile egos, and shaming tactics that Colleges and Online Learning Platforms alike try to keep charging you money for degrees/certificates and prevent answers from being distributed. Don't they always tell you "pay us money! it's a great motivator!" Especially in a subject such as CS, certificates and degrees are meaningless.

In my opinion all online classes MUST have COMPLETE solutions provided. LEARNING should come above and beyond stupid codes, money, and gate keeping. I could not have possibly done anything in this course without looking at others' solutions at Github. Call me a cheater if you want, but what is the point of getting stuck on something for 6 months, not getting any help, not learning, and then calling it "honour" and patting myself on the back for not looking at solutions?

Surely if I copy solutions without doing anything at all then I won't learn either; but in the case of free online learning with no help, this should be up to the learner's own free will and self restraint, not some degree/money/honour requirement. I was encouraged by the Remzi Arpaci (author of OSTEP) who makes his awesome book freely available on his website, and the words of one of the repository owners whose solutions I looked at. It was something like "please try very hard to do these assignments yourself first! Only look when you are absolutely stuck."

[aryzach]

Just wanted to say thanks so much for this review. I was going to get started on this class within the next month or so because it's the next on the curriculum. I was planning to use both this and Three Easy Pieces, because I know there was debate between that and HtK for what should be on OSSU. I'd say this is an easy decision for the curriculum.

And I totally agree on your stance on not torturing yourself when stuck. It took me a while to get here, and it still is sometimes painful to look at solutions prematurely, but it's definitely sped up my learning and probably improved it (from getting different view points on approaches). In reality, most people who take classes have other people (classmates, TA's, profs) to ask questions. Taking a class in isolation where you are both the smartest and dumbest one in the room is not a great / normal way to learn, but it's also taught me how and when to ask the right questions, and made me look at education differently, and see what parts are valuable and what parts are gatekeeping / elitist.

[spamegg1]

@aryzach Much appreciated, and I agree; this course certainly brought up feelings of inadequacy and made me want to quit OSSU altogether. I would not be surprised if this happened to other OSSUians. You should still read OSTEP and do its homeworks, they are very simple.

[waciumawanjohi]

I support replacing Hack the Kernel with the book/projects Three Easy Pieces (OSTEP).

1. I've had the pleasure of corresponding with Remzi H. Arpaci-Dusseau, one of the OSTEP authors. I found him engaging, curious about students using his work and open to the suggestion of developing more tools for online students. I am particularly happy that homeworks now have solutions available (I don't remember that being the case) and that there are intentions to port over the projects from the author's UWisc operating systems course. And I've always been tickled that I was included in a footnote in Chapter 15.

2. The authors are from U Wisconsin. The University's CS 537 follows the book closely. A syllabus with lecture notes is available.

3. Previous lectures for CS 537 have been recorded. One set is of Mike Swift in 2014. Another is of Adalbert Gerald Soosai Raj in 2017. Neither has been organized into a playlist, but both have hours of content. I enjoyed a number of the more recent lectures.

4. The RFC details that HtK is both hard and low quality. While the difficulty level might recommend putting the course in Advanced Systems, the information about course quality convinces me that Hack the Kernel should simply be removed from OSSU's curriculum.

[s7oev]

This course definitely doesn't sound like something that should be included in the main curriculum. Indeed, maybe put it as an elective somewhere, warning people about the general course experience.

As for replacing it in the curriculum, I believe that it would be great if there is also a video resource. I have found that Georgia Tech's Intro to OS course from their well-received online Master in CS is actually avalable for free on Udacity (bearing no credit for the Master, obviously). Additionally, even the follow-up course - Advanced OS, is available for free!

Here are the links:
https://www.udacity.com/course/introduction-to-operating-systems--ud923
https://www.udacity.com/course/advanced-operating-systems--ud189

Is there someone that took them and can share their opinion?

[s7oev]

Berkeley CS 162 is another resource that has public lecture recordings and assignments. It is also the recommended course by teachyourselfcs, yet, oddly, it is only named without further explanation.

[waciumawanjohi]

I took the Georgia Tech Intro to Operating Systems course as an OMSCS student. It's a good course, one of the highest rated in the curriculum. But the assignments for the course are not freely available online. Because of that, I would not recommend it for the OSSU curriculum. It is already listed in Extras/Courses.

[spamegg1]

The RFC details that HtK is both hard and low quality. While the difficulty level might recommend putting the course in Advanced Systems, the information about course quality convinces me that Hack the Kernel should simply be removed from OSSU's curriculum.

That's fine with me. Some people hold this class in high regard due to its notoriety and difficulty, and the assignments, if done correctly, certainly teach you A LOT and make you a bad ass programmer, so first I thought that making it an elective would give people the option to do it if they wanted. But now I realize you are right. Low quality should not be tolerated. I think their code is high quality but the lectures/videos/slides/assignments are low quality. I think their goal was to create a code base on which other schools/classes can base their own assignments.

It doesn't look like there are any alternatives for an Operating Systems class with programming assignments at such depth that can be submitted to a system and checked for correctness by an auto-grader. It's just way too difficult to build an auto-grader testing system for something like that.

@s7oev The Udacity UD923 course linked above has programming assignments, WITH SOLUTIONS provided (YES!), but without an auto-grader or some submission system. Also the assignments are a lot less involved/deep and not focused on Unix-like system calls and shell. But maybe that's OK. That course looks good to me. I'm gonna add it to Alternatives. It says:

Introduction to Operating Systems is a graduate-level introductory course in operating systems.

Yeah so there you go, Harvard treats this as a 100-level class (161), while another school treats it as a 900-level graduate course, and it's only introductory. HtK definitely should not be undergrad material, let alone Core CS. This shows you the very strange elitism going on at Harvard. I also had a similar opinion about CS50; it definitely did not feel like a 050 level class. Felt more sophomore/200-level.

I think HTK's code and the assignments are still valuable. Since their last commit was 3 years ago, they probably consider it "done". I think that maybe I (or someone else) should do something about this to salvage them. We could fork their kernel simulation code, but design/flesh out the 3 assignments by dividing them up into many more, smaller assignments and providing A LOT MORE skeleton code and hand-holding. (In fact there was something about this in their code saying something like "if you base your course on our material, here are your options..." and I vaguely remember on the web some other schools had 6 assignments based on this material.) But this would be a huge undertaking and would take a long time, so certainly not an option for right now.

[spamegg1]

@waciumawanjohi I will add the resources you listed to the proposal. Makes sense. I was fine without videos for OSTEP but others might find it helpful.

I took the Georgia Tech Intro to Operating Systems course as an OMSCS student. It's a good course, one of the highest rated in the curriculum. But the assignments for the course are not freely available online. Because of that, I would not recommend it for the OSSU curriculum.

The Udacity 923 course looks free, am I missing something? They are both Georgia Tech, are these two different courses? Udacity says "Offered at Georgia Tech as CS 8803" and your link says CS-6200. Confused.

[s7oev]

@s7oev The Udacity UD923 course linked above has programming assignments, WITH SOLUTIONS provided (YES!), but without an auto-grader or some submission system. Also the assignments are a lot less involved/deep and not focused on Unix-like system calls and shell.

@spamegg1 I'm guessing you signed up for the course, based on this paragraph! I haven't done so, but if you did and you could see there were assignments then may be the situation has changed since @waciumawanjohi last researched.

Especially if the course itself is well-rated, and assuming the assignments are available, I think it is a great option. Honestly, can't assess it in terms of depth (I'm not very far in my CS education), but having just an introductory course is okay, IMO.

Interested students can always continue - either with the Georgia Tech Advanced OS, and/or maybe even the HtK course for those that find it extremely interesting. But either way, those would be electives, as I do think that a general CS program does not need to go in great depth in OS (but, again, I'm in no way an authority on this).

[spamegg1]

@s7oev Yeah I signed up and looked at all of the UD923 course. It looks pretty good to me. Since I've already done HtK I am able to understand the level and depth of these assignments. Definitely much more introductory, which is much more appropriate for Core CS. The assignments are NOT graded, there is no auto-grader to submit your code but solutions are provided; so it's like self-study (just like how one would with OSTEP). Also after the first set of assignments it switches to a less technical, more traditional "college exam" type assignment. OSTEP has very similar homework assignments, like "explain why this is preferable to that" etc.

[xxylem]

Just wanted to chime in here to corroborate what @spamegg1 has said. I stopped the course after doing ASST1.

• The lectures: filmed in a room, slides are blurry, instructor rambles (spends a long time going over admin or berating the students in his class), doesn't explain things well (I only understood what he was talking about because I had already starting reading the textbook; but then, I may as well just read the textbook)
• The philosophy of the course: while at some point a teacher should withhold help so that the students can discover for themselves, that point is not day 1. The attitude of the instructor is that you should just be able to work everything out for yourself.
• The assignments: an old version of Ubuntu is required to get them running. I had to fix some bugs to get the code working before I could start on the assignment.

I have been reading the OSTEP book, but not doing the assignments (limited time at the moment, plan to do them in a break), so cannot comment on their quality, but the quality of the chapters is high and content is exposed/explained clearly and in a sensible order. The authors want you to understand, as opposed to HtK.

[spamegg1]

@xxylem Glad to hear I'm not the only one, but sad to hear that you had to go through it too. At least you had the good sense to stop much earlier than me. Yeah, the rambling lecturer, deliberate withholding of information, blurry videos with bad audio, and the attitude that expects me to do everything by myself really pissed me off.

[spamegg1]

Thanks everyone for the comments and suggestions. Closing this issue a bit early if that's OK, no comments have been made for a while (also I'll be offline for 1-2 months). Making a pull request with the Proposal changes.

[dipanzan]

I know this is an old thread, but would you still recommend HTK for someone who'd like to know about the internals of a kernel?

I'm an MSc student currently doing research with the Linux kernel itself, I'm barely scratching the surface, but I'm somewhat comfortable reading kernel code now (after months of struggling, I'll admit) and gaining a bit more proficiency in C. I really want to venture into making drivers or parallel computing someday, I figure I'd need to understand at a good level of how an operating system works and how it interacts with the software running above.

Thank you for the amazing review and posts!