My Recurse Center Syllabus
o o \ __/ \___ / \__ __/ \ / ____________________\/_________________ / _________________________________ \ | /__/ \__ _____________ \__/ \__\ | | | __ \____My_________ __ \| | | |\__/ \_________________ \__/ \__ | | | | __ \___Recurse_____ __ \| | | |\__/ \__ _____________ \__/ \__ | | | | __ \_______Center__ __ \| | | |\__/ \__ _____________ \__/ \__ | | | | __ \__Syllabus____ __ \| | | |\__/ \__ _____________ \__/ \__ | | | \________\________________________/ | | | | _ _ _ | | Ashley Blewer! (=) (=) (=) | \______________________________________/ "--" "--"
Creating a syllabus for myself
I will be attending the Recurse Center from April 2 until June 28. For the uninundated (non-inundated?), I have been describing RC succinctly as "like a writer's retreat for programmers." although maybe that's not quite sufficient. But in general, it is a place to focus intensely on my craft and be surrounded by and inspired by brilliant people.
I am verrrrrry excited!
Because the time you spend at RC and what you choose to work on is self-initiated and self-motivated, I am setting up this repository to create a sort of "syllabus" for myself and what I want to accomplish. BUT! I do not expect to follow it, just give myself some rough structure to make sure I hit my personal productivity goals while leaving enough room for unanticipated discovery.
This will also be a living syllabus,
updated with links to blog posts,
works I produced,
and things I used to tackle these problems
or build things for each week.
Some things will get bumped and other things will get shuffled around as part of the normal un-learning process.
Here is how I answered the application question "What would you like to work on at the Recurse Center?":
I really love video and I understand it (relatively) well, I think, but would really like to better understand compression algorithms and how things work. I think trying to make my own video codec would be a good way to do that, and RC seems like a great place to attempt a task that sounds difficult and academic. I think this kind of broad, open-ended task would allow me to have one major goal to accomplish, and would allow me to break down the problem and potentially tackle mini-problems along the way, accepting productive distraction (like committing to an open source video framework as a mini-goal towards the larger educational goal).
General suggestions from RC to help think about projects:
- What are the main 2-3 things I'd like to learn? (i.e. why am I doing this?)
- Which features could I build to learn them?
- Are there other things I can ignore because they won't help me achieve my learning goals?
- Does my project divide naturally into certain chunks?
- How will I know when my project is finished?
Thinking about how to think about this
I want to create goals for myself that are concrete enough to be able to tackle but open enough that I can move in different directions as I grow and learn more.
I also think it's important for me to think about this work as at most 80% of a week, leaving 20% to spread out and learn about or engage in other projects unrelated to this one. But I hope that for each week, I can formulate an idea of readings/tasks to complete, and ensure a solid balance of learning-time and active-programming time.
Week 0 (Now until April 2)
Create this syllabus.
Here are some things inspiring me (and related to this topic):
- Falsehoods Programmers Believe about Video
- Vittorio Giovara - Color Me Intrigued: A Jaunt Through Color Technology in Video
- Vittorio Giovara - Introduction to video reverse engineering
- Digital Video and HD: Algorithms and Interfaces by Charles Poynton
- Data compression paper
- Pirate TV
- Kieran O'Leary - Learning Through Fixity Errors
Popular algorithm names:
- Adaptive Huffman!
(Pre-RC Ideas Zone, this will move)
- Video container/codec review, how do significant properties relate to each other when established by each format?
- containers versus codecs, how do they work together (or not together?)
- Begin manipulation of raw video data
- "signal processing"
- Muxing, demuxing, compiling code
- ffmpeg filters
- Visualizing compression: Is there a way to extract only-the-compression from audio or video for visualization purposes?
- Build something in C -> I did build something in Rust with much help, so maybe this has been satisfied
- Codec manipulation
- Web Assembly, put C in the browser
- Electron, put JS on the desktop
- Thinking about open source contributions here
Week 1 (April 2-6): "Containers/codecs"
- Hmm, nothing that would merit a check-in...
- Making a Matroska Magic Hat.
- All the stuff between EBML and Matroska
- Exploring codecs and data streams
- Lenticular film, lenticular codec
- First Week at Recurse Center.
Exploring Matroska attachments
- file format polyglot-ing (put mkv file inside mkv file)
- This was very easy thanks to MKVToolNix!
Understand how codecs are made
- read up on codecs
Open or non-open codecs, standards:
- lenticular by reto kromer
- RTP Payload Format for H.264 Video
- Not open: h.264
- Not open: ProRes white paper
- Not open: ProRes RAW white paper
- FFmpeg has been coming up a lot in conversation, with a lot of people working on different things. Someone told me about this book which is like the best thing ever. I'm ready to start a graduate level program and teach only this book. So many example commands and a visual representation of what they do.
- I need to follow advice that Misty De Meo was given and passed along via Mastodon, which is "hacking around some stuff in FFmpeg's encoder to see what happens"
- Misty's Sega FILM demuxer is in ffmpeg now github and I'm #fangirling
Codec exploration: ./lenticular
- investigation of lenticular, a codec-in-progress for lenticular film
- Running lenticular partially successfully
- Ran into segmentation faults and I'd like to explore these more, but moving on for now
File formats, methods of implementation
- deep read: the secret life of nan. This blog post casually mentions standardization drama and I clicked on the link to read more SO fast. It's here
- deep read: google docs, not a file format I like these two quotes in a succinct blog post: "Proprietary formats are always risky. When the format isn’t a file format at all, you can’t even reverse-engineer it in principle." and "Don’t be a cheapskate. Pay for the storage you need."
- deep read: How can we preserve Google Documents?
gdb / debugging in C
brew install gdb
- extra stuff for macOS
- Is the problem: macOS, gcc, gdb, Homebrew, XCode, the code, or something else?
- "Reading symbols from lenticular...(no debugging symbols found)...done."
- "Undefined symbols for architecture x86_64" - sometimes this is macOS being weird, but sometimes this is just me not remembering to call an external lib in C
- "duplicate symbol _main in:"" - added -c to the two .o rules to tell gcc to stop at the compilation stage. But also noting that this could be main being defined twice in the program.
- My brain today: "How do you debug a debugger?"
brew install valgrind
- roundup of learning resources
- Docs v2
- playing in browser
- I like that the compiler tells you when something is unused
cargo buildto compile your crate
cargo runto run it
- "statements (executing code), ... expressions (returning a value)."
❣️Pairing with Casey on building a simple image manipulation script in Rust ❣️Data Compressor From Scratch
- Intro from Ernest
- pki, public key infrastructure
- "web of trust" / "strong set"
- expiration and revokation
- "security convenience balance"
- yubikey is closed source, nitrokey is open software/hardware
- Ring (end to end encrypted Skype) ring.cx
Cultural heritage and copyleft opinions
Lots of talks and inspiration from batchmate Marianne:
- http://ivory.idyll.org/blog/2017-digital-commons-funding.html paper about CC models
❣️https://framagit.org/mkcor/free-software-ethics Marianne's talk on free software ethics, so so so so good!
- "pragmatic idealism"
- deep read: Potential video cultural heritage paper to read (and maybe love?): https://hal-mines-paristech.archives-ouvertes.fr/hal-00975857v2/document
- "Intangible Cultural Heritage"
- TIL: "apt-get install anarchism" installs this
Preservation of complex media
- Re-Collection: Art, New Media, and Social Memory
❣️"variable media behaviors": contained, installed, performed, interactive, reproduced, interchangeable, encoded, networked
- wishing I could play this interactive fiction on laserdisk
- Something obvious to me when working in the context of the humanities is that we each have specific knowledge about a subject derived from our own lenses and cumulative individual experience (hello Black feminist critical theory), and the sooner I acknowledge this applies to technical subjects too, the better off I’ll be. RC is a great place for this because of the diverse experience and backgrounds collaborating in one space, but harder in a small monoculture like digital preservation.
- adding increasingly dangerous amounts of coffee, trying to reign it in
- first added a xml feed to my blog, next day I actually showcase that endpoint so people can use it: http://bits.ashleyblewer.com/feed.xml if you want to subscribe
- TIL zulip offers free hosting for OSS projects https://zulipchat.com/for/open-source/
- more zulip love: you can view source, you write in markdown
- Looking at http://www.vapoursynth.com/about/
- my review was requested on audiorecorder
- "are you upstate or something? I picture you in some sort of compound like the end of mad men"
- YCbCr: "But how do you make yellow?" Because colorspaces are cubes, and the absence of blue produces yellow, and the absence of red produces green.
- fell into the same Jekyll time zone trap as Marianne did this week: A few of my posts was originally written at something like "23:16:00 -0400" so when the time changed for daylight savings, it reverted to thinking it "should" be the next day, deleting the original page and replacing it with a date that was one day later. Tricky! My hack to resolve it was to not get so granular with time-of-posts and kept it at the day-level for these late-ish night additions.
- Lydia recommends: https://www.goodreads.com/book/show/44882.Code
Week 2 (April 9-13): R U S T
- menstruation rollercoaster (2D): ya my period data from the last decade is irregular.
- Internet Girlfriend Club Volume 7 goes live
- Considering Rust
- Considering Rust ...for Video
- Error Handling in Rust
- Stealing Snax using Rust
- Second Week at RC
Minimum Viable Codec
- What does this look like?
Rust vs multimedia
- Kostya's Boring Codec World
- but especially: NihAV — Concept and Principles
- The above blog post helps me break down my own problems-I-will-be-creating-for-myself with this statement: "I hope the domain for NihAV is clear: it will take ByteIO input, demux data using it (packets or elementary stream chunks—if you want them in packet format then use a parser), optionally fill timestamp information, decode frames, reorder them in display order if requested, similar approach for writing data."
- Huffman coding implementation in Rust
Rust in general
- "How to rewrite your project in Rust"
- Something I won't be doing soon PROBABLY, making a N64 emulator in rust
❣️Rust Cookbook ❣️rustlings: Small exercises to get you used to reading and writing Rust code. SUPER SMALL! Really excited about the simplicity here and it pushed me to really read the book, which is also full of examples.
- Luca Barbato: Rust-av: Rust and Multimedia
❣️Introducing rust-av, A pure-Rust approach to multimedia video from FOSDEM '18, also the "slides" used in this presentation are amazing ❣️rust-av github
- Chroma from Luma Intra Prediction for AV1
- Chroma from Luma lightning talk: intra-frame! Video from VDD '17
- CfL Design Doc for AV1 (draft)
- Data Compression Conference Proceedings
- Fast Averaging of High Color (16 bit) Pixels
- Videos about AV1
Video, in general
- "How VLC works": a helpful slide!
- "Multimedia Data"
- "data moshing" but actually doing it instead of just "enjoying" it
❣️Irish Film Archives great microservices collection
- What Is A LUT? Explained with Cats.
I think I may have to overcome my utter lack of maths knowledge soon...
- above is clearly everything I'm doing to avoid this topic
- Matrix coefficient?
- Essence of linear algebra (thanks Jess!)
- New fast.ai course: Computational Linear Algebra (thanks Jess!)
- Learning can feel painful and that's okay!
- How to make mistakes, acknowledge them, correct them, and move forward.
- The difficulties of committing (and wanting to commit) to 100% focus on something (this) but still having to manage winding down paused or ending freelance projects, maintaining pre-existing relationships, and trying not to worry about housing and employment after RC.
| hexdumpdoes exactly what you think it does
- destroyallsoftware: intro to computation series
- wondering why I didn't learn how to use Vim 3 years ago
🙄because it's great 😅
- bonus vimrc
- I made kombucha for the first time, it was rose flavored, and I drank it.
- Play all these at the same time: 1, 2, 3
- Rosa Menkman - A Vernacular of File Formats
Week 3 (April 16-20) "Minimum Viable Codec" (Encoding)
- avpres-training: video update
- ffmprovisr: open/close all recipe syntax and new recipe
- Tektronix - The Vectorscope -- It's All About Color! V 2.0
- Understanding Analog Video Signals
- Video format tables
- New York City in 1993 in HD - DTheater DVHS Demo Tape | theverge writup
- "cost of action calculator"
❣️World's Smallest h.264 Encoder
- Minimal video player example (h264 decoder!)
- The h.264 Sequence Parameter Set
- Common Intermediate Format
- Teaching in DC on Tues - Sad to miss even one day
- Analyze moshbot
| nc termbin.com 9999for quick code-sharing between computers
- Essential thoughtleadership theory
- Software Defined Radio hardware
xxdlooks better (imo) than
- h264bitstream command-line tool to read/write h264 bitstreams
- C Syntax cheatsheet
Week 4 (April 23-27) "Minimum Viable Codec" (Decoding)
Goals at the start of this week:
- Writing h264 decoder / mini-player
- Getting Rust to the browser with wasm - done!
- Writing encoder for video streams using huffman algo
- Not have a cold anymore - done!
- Uncompressed versus uncompressed packed video
- What does audiovisual compression look like?
- Rust Study Group Notes
- Week 4 of 12 at Recurse Center
❣️Introducing AV1 and Open Video Codecs
- four.cc: YUV Pixel Formats
- four.cc: YUV to RGB Conversion
- frei0r filters
ffplay -f lavfi mptestsrc=t=all
- Current Status of Codecs in Nov 2017 talk: Has me thinking about the massive costs associated with license-based codecs, really drives home how important the open collaborative work of AV1. Also thinking about how codecs are geographically-based in significance when these licensing/royalties costs to decode or for hardware-support are involved.
- "Same quality" -- how is quality assessed?
-psy <boolean> E..V..... Use psychovisual optimizations. (default auto)(sounds cooler than it is)
- Streaming Video and Audio with Low Delay
❣️Learn FFmpeg libav the Hard Way
- vaporwave player
- "transrating" changing the bitrate of a file
- deepfakes/faceswap: ostensibly more trustworthy source code
- df: for making corpus
- Exploring deepfakes
- The Era of Fake Video Begins
- Tensorflow tutorials
- Introduction to Deepfakes tutorial series
- "computer mediated mysticism"
- zodiacpress for the technical underpinning of how astrological natal charts are created and here's an example in case I want to rewrite
- "wholesome deepfakes"
- "What does it mean to raise a family or to grow up under constant surveillance?"
- Something to think about later: TLA+
- mosh_bot source code
- "side data" and "main data"
- "bit reservoir"
- "syntactic noise"
- how the internet works outside of home router
- "smart pointers"
- 3d snorlax
- Theorizing the Web 2018
YUV2RGB shader math
vec3 yuv = vec3(yColor.r - 16.0/256, uColor.r - 0.5, vColor.r - 0.5); oColor.x = dot(yuv, vec3(1.164, 0.000, 1.596)); oColor.y = dot(yuv, vec3(1.164, -0.391, -0.813)); oColor.z = dot(yuv, vec3(1.164, 2.018, 0.000));
Week 5 (April 30 - May 4) (Floating in space)
- The Story of Goldilocks and the Three WebAssembly Libraries
- Week 5 at Recurse Center
Calling C in Rust
A return to Rust+WASM+JS
- Parc3l: Combining Three.js, Rust, and WebAssembly!
Week 6 (May 7-11) (Planets in alignment)
Malaysian General Election 2018
- ffmpeg in the browser!!!!!!! videoconverter.js demo
- Building ffmpeg.js for Ubuntu
- Decode it like its 1999
- Engineers Making Movies (AKA Open Source Test Content)
- ashleyblewer.com finally went https!
- Resisting Surveillance: it's is not just about the metadata
- MENSTRUAPPS – How to turn your period into money (for others)
- Servo engine
- Aura reading
- American Mall Game
- Spending most of my time prioritizing being around people who are leaving this week
BONUS WEEK (May 14-18)
!!Con May 12-13 - I got a ticket!
This is when Never Graduate Week happens. SPRING BREAKKKKKKKK!
- I-Ching gets a save-state upgrade and makeover
Week 7 (May 21-25) (Reboot)
Back 2 video
Manipulating video in the browser
- the cryptopals crypto challenges
- ProofMode adds encrypted chunk of data to media
- Is This For Real? How InformaCam Improves Verification of Mobile Media Files
- tfw you get disheartened because you don't realize you're trying to solve an outstanding research problem
- "Here, we introduce a framework in which we quantify these perceptual distortions in terms of “just intolerable differences” (JIDs)."
Week 8 (May 28 - June 1)
- more-blue: another very tiny codec
- rav1e - av1 encoder in rust
- OBU = Open Bitstream Unit
- YUV4MPEG2 (.y4m) Encoder/Decoder Crate
- "Category:Undiscovered Video Codecs"
- closed standards are immoral
- RustFest Paris 2018 video playlist
- "Redox is a Unix-like Operating System written in Rust, aiming to bring the innovations of Rust to a modern microkernel and full set of applications."
- DEF CON 25 - Svea Eckert, Andreas Dewes - Dark Data
- DEF CON 25 - Matt Wixey - See no evil, hear no evil: Hacking invisibly & silently with light & sound
- Tactical Tech Collective
- Efficiency and Madness - Using Data and Technology to Solve Social, Environmental and Political Problems
- Ray Tracer #1: What is a ray tracer?
- conrod: "An easy-to-use, 2D GUI library written entirely in Rust."
YUV - RGB Conversion
R = Y + 1.4075 * (V - 128) G = Y - 0.3455 * (U - 128) - (0.7169 * (V - 128)) B = Y + 1.7790 * (U - 128) Y = R * .299000 + G * .587000 + B * .114000 U = R * -.168736 + G * -.331264 + B * .500000 + 128 V = R * .500000 + G * -.418688 + B * -.081312 + 128
Week 9 (June 4-8)
Real Life Hour
- Comparing video codecs and containers for archives
- Digital intermediate
- Sustainability of Digital Formats: Digital Moving-Picture Exchange (DPX), Version 2.0
- Sustainability of Digital Formats: Digital Video Encoding (DV, DVCAM, DVCPRO)
- Misinterpreting Mandelbrot
- Xiph.org test media
- A Framework For Purposeful Open Source
- Project Electron Update: Introducing Aurora 1.0
- Bit Manipulation
Week 10 (June 11-15)
🎊 🎂 🆎
Video Pixel Formats
- I feel like I should be thinking about pixel formats more.
- Microsoft: Pixel Formats for Uncompressed Video
- Apple: Core Video - Available Pixel Formats
- Localhost: Samantha Goldstein Crafting a Connected Home
- Critical .zip vulnerabilities?
- edX Introduction to Linux
- The Land Before Binary
- About Time: Sifting Through Analog Video Terminology
😵 😵 😵
Week 11 (June 18-22) ⿒
- getting my env un-fucked
- The key differences between Python 2.7.x and Python 3.x with examples
- returning to fish
- The Book of Shaders
- OIL OF EVERY PEARL'S UN-INSIDES
- x86 assembly doesn’t have to be scary (interactive)
- Explainer on Sigfried, FIDO, and DROID matching differences
- Resurrecting the first Dutch web index: NL-menu revisited
- How I use Wireshark
- Princeton Web Transparency & Accountability Project
- What's going on up the street?
Week 12 (June 25-28)
Back on my bullshit
scanlime video content
- Little Glitchy CRT
- Little live teardown of NTSC test pattern Tek
- TSG95 Color Bar Generator Teardown
- Moving from Codec to Camera, Tandberg teardown continued
- bitsquatting, rowhammer, aes encryption
- protocol analyzer tool, chip whisperer tool, shodan
- Grokking Algorithms: An illustrated guide for programmers and other curious people
- Lotte World
- ice cream!
There is no end! Never graduate!
Thanks for reading!