index.json

[
  {
    "title": "Maybe You Don't Need Kubernetes",
    "url": "https://endler.dev/2019/maybe-you-dont-need-kubernetes/",
    "body": "\n    \n    \n        A woman riding a scooter\n        \n        Illustration created by freepik, Nomad logo by HashiCorp.\n    \n\nKubernetes is the 800-pound gorilla of container orchestration.\nIt powers some of the biggest deployments worldwide, but it comes\nwith a price tag.\nEspecially for smaller teams, it can be time-consuming to maintain and has a\nsteep learning curve. For what our team of four wanted to achieve at trivago, it\nadded too much overhead. So we looked into alternatives — and fell in love with\nNomad.\nThe wishlist\nOur team runs a number of typical services for monitoring and performance\nanalysis: API endpoints for metrics written in Go, Prometheus exporters, log\nparsers like Logstash or Gollum, and databases like InfluxDB or Elasticsearch.\nEach of these services run in their own container. We needed a simple system to\nkeep those jobs running.\nWe started with a list of requirements for container orchestration:\n\nRun a fleet of services across many machines.\nProvide an overview of running services.\nAllow for communication between services.\nRestart them automatically when they die.\nBe manageable by a small team.\n\nOn top of that, the following things were nice to have but not strictly\nrequired:\n\nTag machines by their capabilities (e.g., label machines with fast disks for\nI/O heavy services.)\nBe able to run these services independently of any orchestrator (e.g. in\ndevelopment).\nHave a common place to share configurations and secrets.\nProvide an endpoint for metrics and logging.\n\nWhy Kubernetes was not a good fit for us\nWhen creating a prototype with Kubernetes, we noticed that we started adding\never-more complex layers of logic to operate our services. Logic on which we\nimplicitly relied on.\nAs an example, Kubernetes allows embedding service configurations using\nConfigMaps. Especially when merging multiple config files or\nadding more services to a pod, this can get quite confusing quickly.\nKubernetes - or helm, for that matter - allows injecting external configs\ndynamically to ensure separation of concerns. But this can\nlead to tight, implicit coupling between your project and Kubernetes.\nHelm and ConfigMaps are optional features so you don’t have to use them. You\nmight as well just copy the config into the Docker image. However, it’s tempting\nto go down that path and build unnecessary abstractions that can later bite you.\nOn top of that, the Kubernetes ecosystem is still rapidly evolving. It takes a\nfair amount of time and energy to stay up-to-date with the best practices and\nlatest tooling. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list\ngoes on and on. Not all tools are necessary to get started with Kubernetes, but\nit’s hard to know which ones are, so you have to be at least aware of them.\nBecause of that, the learning curve is quite steep.\nWhen to use Kubernetes\nAt trivago specifically, many teams use Kubernetes and are quite happy with it.\nThese instances are managed by Google or Amazon however, which have the capacity to do so.\nKubernetes comes with amazing\nfeatures,\nthat make container orchestration at scale more manageable:\n\nFine-grained rights management\nCustom controllers allow getting logic into the cluster. These are just\nprograms that talk to the Kubernetes API.\nAutoscaling! Kubernetes can scale your services up and down on demand. It\nuses service metrics to do this without manual intervention.\n\nThe question is if you really need all those features. You can't rely on these\nabstractions to just work; you'll have to learn what's going on under the\nhood.\nEspecially in our team, which runs most services on-premise (because of its\nclose connection to trivago's core infrastructure), we didn't want to afford\nrunning our own Kubernetes cluster. We wanted to ship services instead.\n\n\n    \n\n\nBatteries not included\nNomad is the 20% of service orchestration that gets you 80% of the way. All it\ndoes is manage deployments. It takes care of your rollouts and restarts your\ncontainers in case of errors, and that's about it.\nThe entire point of Nomad is that it does less: it doesn’t include\nfine-grained rights management or advanced network policies, and that’s by\ndesign. Those components are provided as enterprise services, by a third-party,\nor not at all.\nI think Nomad hit a sweet-spot between ease of use and expressiveness. It's good\nfor small, mostly independent services. If you need more control, you'll have to\nbuild it yourself or use a different approach. Nomad is just an orchestrator.\nThe best part about Nomad is that it's easy to replace. There is little to no\nvendor lock-in because the functionality it provides can easily be integrated\ninto any other system that manages services. It just runs as a plain old single\nbinary on every machine in your cluster; that's it!\nThe Nomad ecosystem of loosely coupled components\nThe real power of Nomad lies within its ecosystem. It integrates very well with\nother - completely optional - products like Consul (a key-value store) or\nVault (for secrets handling). Inside your Nomad file, you can have sections\nfor fetching data from those services:\n\ntemplate {\n  data = <<EOH\nLOG_LEVEL="{{key "service/geo-api/log-verbosity"}}"\nAPI_KEY="{{with secret "secret/geo-api-key"}}{{.Data.value}}{{end}}"\nEOH\n\n  destination = "secrets/file.env"\n  env         = true\n}\n\nThis will read the service/geo-api/log-verbosity key from Consul and expose it\nas a LOG_LEVEL environment variable inside your job. It's also exposing\nsecret/geo-api-key from Vault as API_KEY. Simple, but powerful!\nBecause it's so simple, Nomad can also be easily extended with other services\nthrough its API. For example, jobs can be tagged for service discovery. At\ntrivago, we tag all services, which expose metrics, with trv-metrics. This\nway, Prometheus finds the services via Consul and periodically scrapes the\n/metrics endpoint for new data. The same can be done for logs by integrating\nLoki for example.\nThere are many other examples for extensibility:\n\nTrigger a Jenkins job using a webhook and Consul watches to redeploy your\nNomad job on service config changes.\nUse Ceph to add a distributed file system to Nomad.\nUse fabio for load balancing.\n\nAll of this allowed us to grow our infrastructure organically without too much\nup-front commitment.\nFair warning\nNo system is perfect. I advise you not to use any fancy new features in\nproduction right now. There are bugs and missing features of course - but\nthat's also the case for\nKubernetes.\nCompared to Kubernetes, there is far less momentum behind Nomad. Kubernetes has\nseen around 75.000 commits and 2000 contributors so far, while Nomad sports about\n14.000 commits and 300 contributors. It will be hard for Nomad to keep up with\nthe velocity of Kubernetes, but maybe it doesn’t have to! The scope is much more\nnarrow and the smaller community could also mean that it'll be easier to get your \npull request accepted, in comparison to Kubernetes.\nSummary\nThe takeaway is: don't use Kubernetes just because everybody else does.\nCarefully evaluate your requirements and check which tool fits the bill.\nIf you're planning to deploy a fleet of homogenous services on large-scale\ninfrastructure, Kubernetes might be the way to go. Just be aware of the\nadditional complexity and operational costs. Some of these costs can be\navoided by using a managed Kubernetes environment like Google Kubernetes\nEngine or Amazon EKS.\nIf you're just looking for a reliable orchestrator that is easy to maintain and\nextendable, why not give Nomad a try? You might be surprised by how far it'll get you.\nIf Kubernetes were a car, Nomad would be a scooter. Sometimes you prefer one and\nsometimes the other. Both have their right to exist.\nCredits\nThanks to my awesome colleagues Esteban Barrios, Jorge-Luis Betancourt, Simon Brüggen, Arne Claus, Inga Feick, Wolfgang Gassler, Barnabas Kutassy, Perry Manuk, Patrick Pokatilo, and Jakub Sacha for reviewing drafts of this article.\n"
  },
  {
    "title": "What Is Rust Doing Behind the Curtains?",
    "url": "https://endler.dev/2018/cargo-inspect/",
    "body": "Rust allows for a lot of syntactic sugar, that makes it a pleasure to write. It is sometimes hard, however, to look behind the curtain and see what the compiler is really doing with our code.\n\nAt Rust Belt Rust 2018, I saw a talk by Tshepang Lekhonkhobe titled Syntax conveniences afforded by the compiler (Recording here).\nTo quote the abstract:\n\nThe Rust compiler provides a number of conveniences that make life easier for its users. It is good to know what these are, to avoid being mystified by what's going on under the hood... the less magical thinking we have of the world, the better.\n\nHe goes on to give a few examples of these conveniences:\n\nlifetime elisions\ntype inference\nsyntactic sugar\nimplicit dereferencing\ntype coercions\nhidden code (e.g. the prelude)\n\nIt was very educational and fun to see him compare code with and without these conveniences during the talk.\nComing home, I wanted to learn more. I wondered if there was a tool, which revealed what Rust was doing behind the curtains.\nOver on Reddit, I found a discussion about compiler flags to produce desugared output.\n(Note that I'm using rustup here to trigger the nightly compiler with the +nightly flag.)\n\nrustc +nightly -Zunpretty=hir example.rs\n\nHIR stands for high-level intermediate representation. This is basically an abstract syntax tree (AST) more suited for use by the compiler. It replaces syntactic sugar with basic building blocks that are easier to handle by the following compile steps.\nTo find out more, read this detailed write-up by Nico Matsakis.\n\nAnyway, the output looked surprisingly readable (see below). With some syntax highlighting and formatting, this could be quite a handy tool.\nI tried to use rustfmt on it, and it worked unreasonably well.\nMotivated by this quick win, I wrapped it up in a cargo subcommand and called it cargo-inspect.\nLet's try cargo-inspect on some real code!\nExample - Desugaring a range expression\nThe following examples can also be found in the project's examples folder.\nInput:\n\nfor n in 1..3 {\n    println!("{}", n);\n}\n\nOutput of cargo-inspect:\n\nThat's the neatly formatted terminal output. It sports line numbers and colors, thanks to prettyprint, which is a library on top of bat. Maybe you can't read that, so here's the gist of it:\n\nmatch ::std::iter::IntoIterator::into_iter(\n        ::std::ops::Range { start: 1, end: 3 })\n  mut iter => loop {\n    // ...\n  },\n};\n\nWe can see that 1..3 gets converted into std::ops::Range { start: 1, end: 3 }.\nTo the compiler backend, these are absolutely the same. So this holds:\n\nassert_eq!((1..3), std::ops::Range { start: 1, end: 3 });\nExample - File handling\nInput:\n\nuse std::fs::File;\nuse std::io::Error;\n\nfn main() -> Result<(), Error> {\n    let file = File::open("file.txt")?;\n    Ok(())\n}\n\nOutput:\n\n#[prelude_import]\nuse std::prelude::v1::*;\n#[macro_use]\nextern crate std;\nuse std::fs::File;\nuse std::io::Error;\n\nfn main() -> Result<(), Error> {\n  let file = match ::std::ops::Try::into_result(\n      <File>::open("file.txt")) {\n    ::std::result::Result::Err(err) =>\n    #[allow(unreachable_code)]\n    {\n      #[allow(unreachable_code)]\n      return ::std::ops::Try::from_error(\n          ::std::convert::From::from(err))\n    }\n    ::std::result::Result::Ok(val) =>\n    #[allow(unreachable_code)]\n    {\n      #[allow(unreachable_code)]\n      val\n    }\n  };\n  Ok(())\n}\n\nWe can see that the carrier operator ? gets desugared into a match on the Result of File::open. In case of an error, We apply std::convert::From::from to convert between error types. Otherwise, we simply return the Ok value.\nFuture work\nI'm not planning to rewrite the compiler here. rustc is doing a far greater job than I could. All this functionality already existed before; I'm merely trying to make the compiler more approachable for learners like me.\nRight now, the tool is quite fragile. It throws ugly error messages when things go wrong.\nIt mostly shines, when you run it on small, isolated example snippets. \nGet involved!\nOver on Github, I opened up a few issues for others to get involved.\nNamely, I wish there were options to:\n\n✅ Make it work with cargo projects.\n✅ Show the original code above the desugared code.\nShow only part of the full output\n...and much more.\n\nAlso, if you find a particularly exciting code example, don't be shy to contribute it to the examples folder.\n"
  },
  {
    "title": "The Unreasonable Effectiveness of Excel Macros",
    "url": "https://endler.dev/2018/excel/",
    "body": "I never was a big fan of internships, partially because all the exciting\ncompanies were far away from my little village in Bavaria and partially because\nI was too shy to apply.\nOnly once I applied for an internship in Ireland as part of a school program.\nOur teacher assigned the jobs and so my friend got one at Apple and I ended up\nat a medium-sized IT distributor — let's call them PcGo.\n\nJudging by the website, the company looked quite impressive, but in reality, it\nwas just a secluded, grey warehouse in the rainy industrial area of Cork. Upon\narrival, I was introduced to my colleague Evgeny, who was the main (and only)\nemployee responsible for assembling desktop computers. From what I can tell, he\nran the shop. He just spoke broken English, so he handed me an electric\nscrewdriver and a box of screws, and I got to work. Together we assembled a lot\nof computers in my first week, and we had a lot of fun. One day he drove me home\nfrom work because I missed my bus. It was a rainy day and while he was driving\nthrough the narrow streets of Cork we talked and laughed, but all of a sudden I\nheard a loud bang. I looked through the rear mirror only to find that there was\nno rear mirror anymore. Turns out he bumped into another car, and the thing went\noff. Evgeny didn't mind. In a thick Eastern-European accent he remarked "Lost\nthree mirrors before already," and kept driving.\nIn my second week, I had a visit from my boss. Apparently, I was done with the\nworkload that they planned for my three-week internship. I was used to\nassembling and installing computers, which explains why.\nTo keep me busy, they put together another task. On an old Windows 98 computer\nin the back, he pointed the browser to silverpages.ie, searched for "computer"\nand after a while we looked at an endless list of addresses of Irish companies\nhaving "something to do with computers." Each entry consisted of the expected\nfields: the company name, the address, the phone number, the website (if any)\nand a list of keywords.\nMy boss said that they needed an overview of all competing vendors. He carefully\nselected a field from an entry, copied it and pasted it into an Excel sheet. He\ndid the same for the remaining fields. "That's it!", he said with a fake smile.\nWe both knew that this would mean two boring weeks for me.\nThey wanted to keep me busy by letting me manually scrape the entirety of a web database.\nI could have taken that as an insult, but instead, I looked at it as a\nchallenge.\nI noticed that the page number on silverpages.ie could be controlled by a GET\nparameter.\n"Can I write a program that does the scraping?" My boss was noticeably puzzled.\n"Uhm... you can do whatever you want, but you're not allowed to install any\nadditional software!". With that, he was off.\nJudging from the installed programs, I wasn't left with many choices: Excel or\nMinesweeper. I knew that Excel's Visual Basic macros were quite powerful, but I\nwasn't sure if I could scrape a full website with it.\nAfter a while, I detected a feature to download a website into an Excel sheet\n(what a glorious functionality). This worked perfectly, so all I had to do was\nrecord a macro to create a temporary sheet for each page, copy all important\nfields into a "master slide" and then get rid of the temporary sheet. I recorded\nthe macro and looked at the code. The rest of the day was spent figuring out how\nto modify the URL in a loop and cleaning up the macro. I pressed the "run macro"\nbutton and then I sat there waiting. The computer was running at full speed. My\nbiggest fear was that the program would crash or that the computer would run out\nof memory. I refrained from playing minesweeper on it, so I mostly played pool\nor chatted with Evgeny.\nWhen I came to the office the next morning, my program was done. To my surprise, it scraped the entirety\nof SilverPages, and there were many thousands of entries in the list. I sent the\ndocument to my boss via E-Mail and then got back to playing minesweeper.\nAn hour later, three guys with suits were standing behind me. I had to show them\nthe list again. They couldn't believe I did that on my own, so I showed them the\ntool to scrape the data. For them, I had some sort of superpower.\nThey left without giving me another task; I was free to do whatever I wanted for\nthe remaining two weeks. I went on to write an inventory tool for them, which\nthey could also manage via Excel. It was just a glorious Excel form for a\nspreadsheet that they maintained manually. I spent two weeks of my summer\nvacation to finish that tool because they said they would pay me for that, which, of course, they didn't :).\nLessons learned\n\nNever underestimate the power of Excel macros.\nIf you have a boring task at hand, make it a challenge for yourself by adding\na handicap.\n\n"
  },
  {
    "title": "Switching from a German to a US Keyboard Layout - Is It Worth It?",
    "url": "https://endler.dev/2018/keyboard/",
    "body": "For the first three decades of my life, I've used a German keyboard layout.\nA few months ago, I switched to a US layout.\nThis post summarizes my thoughts around the topic.\nI was looking for a similar article before jumping the gun, but I couldn't find one — so I'll try to fill this gap.\nWhy switch?\nI was reasonably efficient when writing prose, but felt like \na lemur on a piano when programming: reaching the special keys ({, ;, or /)\nrequired lots of finger-stretching.\n\n    \n    \n        German Keyboard Layout\n        \n        Image by Wikipedia\n    \n\nHere's Wikipedia's polite\nexplanation why the\nGerman keyboard sucks for programming:\n\nLike many other non-American keyboards, German keyboards change the right Alt\nkey into an Alt Gr key to access a third level of key assignments. This is\nnecessary because the umlauts and some other special characters leave no room\nto have all the special symbols of ASCII, needed by programmers among others,\navailable on the first or second (shifted) levels without unduly increasing\nthe size of the keyboard.\n\nWhy switch now?\nAfter many years of using a rubber-dome Logitech Cordless Desktop\nWave, I\nhad to get a mechanical keyboard again. \nThose rubber domes just feel too mushy to me now. In addition to that, I enjoy the\nclicky sound of a mechanical keyboard and the noticeable tactile bump. (I'm using\nCherry MX Brown Keys with O-Ring dampeners to contain the anger of my coworkers.)\nMost mechanical keyboards come with an ANSI US layout only, so I figured, I'd\nfinally make the switch.\n\n    \n    \n        Picture of my lovely keyboard\n    \n\nHow long did it take to get accustomed to the new layout?\nWorking as a Software Engineer, my biggest fear was, that the switch would slow\ndown my daily work. This turned out not to be true. I was reasonably productive\nfrom day one and nobody even noticed any difference. (That's a good thing,\nright?)\nAt first, I didn't like the bar-shaped US-Return key. I preferred the European\nlayout with a vertical enter key. I was afraid that I would hit the key by\naccident. After a while, I find that the US return key to be even more convenient. \nI never hit it by accident.\nWithin two weeks, I was back to 100% typing speed.\nDid my programming speed improve noticeably?\nYup.\nEspecially when using special characters (/, ;, {, and so on) I'm much\nfaster now; partly because the key locations feel more intuitive, but mainly\nbecause my fingers stay at their dedicated positions now.\nSomehow the position of special characters feels right. I can now understand the\nreason why Vim is using / for search or why the pipe symbol is |: both are\neasy to reach! It all makes sense now!\n(For a fun time, try that on a German keyboard!)\nI now understand why Mircosoft chose \\ as a directory separator: it's easily\naccessible from a US keyboard. On the German layout, it's… just… awful\n(Alt Gr+ß on Windows, Shift + Option + 7 on Mac).\nThe opening curly brace on a German layout Mac is produced with Alt+8, which\nalways made me leave the home\nrow and break my typing\nflow. Now there are dedicated keys for parentheses. Such a relief!\nAm I slower when writing German texts now?\nIn the beginning, I was.\nSomehow my brain associated the German layout with German\ntexts. First, I used the macOS layout switcher.\nThis turned out to be cumbersome and take time.\nThen I found the "US with Umlauts via Option Key\nLayout". It works perfectly fine for\nme. It allows me to use a single Keyboard layout but insert German umlauts at will\n(e.g. ö is Option+o). There is probably a similar layout for other language combinations.\nIs switching between keyboards painful?\n\n    \n    \n        US keyboard layout\n        \n        Wikipedia\n    \n\nMy built-in MacBook Pro keyboard layout is still German. I was afraid, that switching between\nthe internal German and the external English keyboard would confuse me. This\nturned out not to be a problem. I rarely look at the print anyway.\nSummary\nIf you consider switching, just do it. I don't look back at all.\nThanks to Simon Brüggen for reviewing drafts of the article.\n"
  },
  {
    "title": "fastcat - A Faster `cat` Implementation Using Splice",
    "url": "https://endler.dev/2018/fastcat/",
    "body": "\nLots of people asked me to write another piece about the internals of well-known\nUnix commands. Well, actually, nobody asked me, but it makes for a good\nintro. I'm sure you’ve read the previous parts about yes and\nls — they are epic.\nAnyway, today we talk about cat, which is used to concatenate files - or, more\ncommonly, abused to print a file's contents to the screen.\n\n# Concatenate files, the intended purpose\ncat input1.txt input2.txt input3.txt > output.txt\n\n# Print file to screen, the most common use case\ncat myfile\nImplementing cat\nHere's a naive cat in Ruby:\n\n#!/usr/bin/env ruby\n\ndef cat(args)\n  args.each do |arg|\n    IO.foreach(arg) do |line|\n      puts line\n    end\n  end\nend\n\ncat(ARGV)\n\nThis program goes through each file and prints its contents line by line.\nEasy peasy! But wait, how fast is this tool?\nI quickly created a random 2 GB file for the benchmark.\nLet's compare the speed of our naive implementation with the system one\nusing the awesome pv (Pipe Viewer) tool.\nAll tests are averaged over five runs on a warm cache (file in memory).\n\n# Ruby 2.5.1\n> ./rubycat myfile | pv -r > /dev/null\n[196MiB/s]\n\nNot bad, I guess? How does it compare with my system's cat?\n\ncat myfile | pv -r > /dev/null\n[1.90GiB/s]\n\nUh oh, GNU cat is ten times faster than our little Ruby cat. 🐌\nMaking our Ruby cat a little faster\nOur naive Ruby code can be tweaked a bit.\nTurns out line buffering hurts performance in the end1:\n\n#!/usr/bin/env ruby\n\ndef cat(args)\n  args.each do |arg|\n    IO.copy_stream(arg, STDOUT)\n  end\nend\n\ncat(ARGV)\n\nrubycat myfile | pv -r > /dev/null\n[1.81GiB/s]\n\nWow... we didn't really try hard, and we're already approaching the speed of a\ntool that gets optimized since\n1971. 🎉\nBut before we celebrate too much, let's see if we can go even faster.\nSplice\nWhat initially motivated me to write about cat was this comment by user\nwahern on\nHackerNews:\n\nI'm surprised that neither GNU yes nor GNU cat uses splice(2).\n\nCould this splice thing make printing files even faster? — I was intrigued.\nSplice was first introduced to the Linux Kernel in 2006, and there is a nice\nsummary from Linus Torvalds himself,\nbut I prefer the description from the manpage:\n\nsplice() moves data between two file descriptors without copying\nbetween kernel address space and user address space.  It transfers up\nto len bytes of data from the file descriptor fd_in to the file\ndescriptor fd_out, where one of the file descriptors must refer to a\npipe.\n\nIf you really want to dig deeper, here's the corresponding source code from the\nLinux Kernel,\nbut we don't need to know all the nitty-gritty details for now.\nInstead, we can just inspect the header from the C implementation:\n\n#include <fcntl.h>\n\nssize_t splice (int fd_in, loff_t *off_in, int fd_out,\n                loff_t *off_out, size_t len,\n                unsigned int flags);\n\nTo break it down even more, here's how we would copy the entire src file to dst:\n\nconst ssize_t r = splice (src, NULL, dst, NULL, size, 0);\n\nThe cool thing about this is that all of it happens inside the Linux kernel, which means we won't copy a single byte to userspace (where our program runs).\nIdeally, splice works by remapping pages and does not actually copy\nany data, which may improve I/O performance\n(reference).\n\n    \n    \n        \n        File icon by Aleksandr Vector from the Noun Project. Terminal icon by useiconic.com from the Noun Project.\n    \n\nUsing splice from Rust\nI have to say I'm not a C programmer and I prefer Rust because it offers a safer\ninterface. Here's the same thing in Rust:\n\n#[cfg(any(target_os = "linux", target_os = "android"))]\npub fn splice(\n    fd_in: RawFd,\n    off_in: Option<&mut libc::loff_t>,\n    fd_out: RawFd,\n    off_out: Option<&mut libc::loff_t>,\n    len: usize,\n    flags: SpliceFFlags,\n) -> Result<usize>\n\nNow I didn't implement the Linux bindings myself. Instead, I just used a library called\nnix, which provides Rust friendly bindings to *nix APIs.\nThere is one caveat, though:\nWe cannot really copy the file directly to standard out, because splice\nrequires one file descriptor to be a pipe.\nThe way around that is to create a pipe, which consists of a reader and a\nwriter (rd and wr).\nWe pipe the file into the writer, and then we read from the pipe and push the data to stdout.\nYou can see that I use a relatively big buffer of 16384 bytes (214) to improve performance.\n\nextern crate nix;\n\nuse std::env;\nuse std::fs::File;\nuse std::io;\nuse std::os::unix::io::AsRawFd;\n\nuse nix::fcntl::{splice, SpliceFFlags};\nuse nix::unistd::pipe;\n\nconst BUF_SIZE: usize = 16384;\n\nfn main() {\n    for path in env::args().skip(1) {\n        let input = File::open(&path).expect(&format!("fcat: {}: No such file or directory", path));\n        let (rd, wr) = pipe().unwrap();\n        let stdout = io::stdout();\n        let _handle = stdout.lock();\n\n        loop {\n            let res = splice(\n                input.as_raw_fd(),\n                None,\n                wr,\n                None,\n                BUF_SIZE,\n                SpliceFFlags::empty(),\n            ).unwrap();\n\n            if res == 0 {\n                // We read 0 bytes from the input,\n                // which means we're done copying.\n                break;\n            }\n\n            let _res = splice(\n                rd,\n                None,\n                stdout.as_raw_fd(),\n                None,\n                BUF_SIZE,\n                SpliceFFlags::empty(),\n            ).unwrap();\n        }\n    }\n}\n\nSo, how fast is this?\n\nfcat myfile | pv -r > /dev/null\n[5.90GiB/s]\n\nHoly guacamole. That's over three times as fast as system cat.\nOperating System support\n\nLinux and Android are fully supported.\nOpenBSD\nalso has some sort of splice implementation called\nsosplice. I haven't tested that, though.\nOn macOS, the closest thing to splice is its bigger brother,\nsendfile, which can send a\nfile to a socket within the Kernel. Unfortunately, it does not support sending\nfrom file to file.2 There's also\ncopyfile,\nwhich has a similar interface, but unfortunately, it is not zero-copy. (I\nthought so in the beginning, but I was\nwrong.)\nWindows doesn't provide zero-copy file-to-file transfer\n(only file-to-socket transfer using the TransmitFile API).\n\nNevertheless, in a production-grade\nimplementation, the splice support could be activated on systems that support\nit, while using a generic implementation as a fallback.\nNice, but why on earth would I want that?\nI have no idea. Probably you don't, because your bottleneck is somewhere else.\nThat said, many people use cat for piping data into another process like\n\n# Count all lines in C files\ncat *.c | wc -l\n\nor\n\ncat kittens.txt | grep "dog"\n\nIn this case, if you notice that cat is the bottleneck try fcat (but first,\ntry to avoid cat altogether).\nWith some more work, fcat could also be used to directly route packets from one\nnetwork card to another, similar to netcat. \nLessons learned\n\nThe closer we get to bare metal, the more our hard-won abstractions fall\napart, and we are back to low-level systems programming.\nApart from a fast cat, there's also a use-case for a slow cat: old computers.\nFor that purpose, there's... well.. slowcat.\n\nThat said, I still have no idea why GNU cat does not use splice on Linux. 🤔\nThe source code for fcat is on Github.\nContributions welcome!\nThanks to Olaf Gladis for helping me run the benchmarks on his Linux machine and to Patrick Pokatilo and Simon Brüggen for reviewing drafts of the article.\nFootnotes\n1. Thanks to reader Freeky for making this code more idiomatic.↩\n2. Thanks to reader masklinn for the hint.↩  \n"
  },
  {
    "title": "That Octocat on the Wall",
    "url": "https://endler.dev/2018/github/",
    "body": "\nSo I'm in a bit of a sentimental mood lately.\nGithub got acquired by Microsoft.\nWhile I think the acquisition was well-deserved, I still wish it didn't happen.\nLet me explain.\nMy early days\nI joined Github on 3rd of January 2010.\nSince I was a bit late to the game, my usual handle (mre) was already taken.\nSo I naively sent a mail to Github, asking if I could bag the name as it seemed to be abandoned.\nTo my surprise, I got an answer.\nThe response came from a guy named Chris Wanstrath.\nAll he wrote was "it's yours." \nThat was the moment I fell in love with Github.\nI felt encouraged to collaborate on projects, that everybody could contribute something valuable.\nOnly later I found out that Chris was one of the founders and the CEO of the company.\nLiving on Github\nBefore Github, there was SourceForge, and I only went there to download binaries.\nGithub showed me, that there was an entire community of like-minded people\nout there who think alike and love to work on code in their free-time.\nTo me, Github is much more than a git interface; it's a social network.\nWhile other people browse Facebook or Instagram, I browse Github.\nI can still vividly remember getting my first star and my first issue on one of my projects coming from a real (!) person other than myself.\nAfter so many years, a pull-request still feels like the most personal gift anyone could give to me.\nGithub - the culture\nAfter a while, I started to admire some Github employees deeply:\n\nZach Holman (who is about my age) is a great writer, speaker, and one of the most creative developers I can think of.\nScott Chacon, who taught me a lot about git and whose presentation tool, showoff, I've used at University.\nTom Preston-Werner, who I admire for refusing an offer from Microsoft to pursue his dream and build Github, for establishing a super-nerdy company culture, and for Jekyll.\n\nAll three developers have since left the company.\nI can't help but notice that Github has changed.\nThe harassment accusations and letting Zach Holman go are only part of the story.\nIt has become a company like any other, maintaining a mature product.\nIt doesn't excite me anymore.\nAn alternative reality\nThere's still a bitter taste in my mouth when I think that Github has fallen prey to one of the tech giants. I loved Github while it was a small, friendly community of passionate developers.\nCould this have been sustainable?\nMaybe through paid features for project maintainers.\nYou see, if you do Open Source every day, it can be a lot of work.\nPeople start depending on your projects, and you feel responsible for keeping the lights on.\nTo ease the burden, I'd love to have deeper insights into my project usage: visitor statistics for longer than two weeks,\na front page where you could filter and search for events, a better way to handle discussions\n(which can get out of hand quickly), better CI integration à la Gitlab.\nThese features would be targeted at the top 10% of Github users, a group of 3 million people.\nWould this be enough to pay the bills? Probably. Would it be enough to grow? Probably not.\nSo what?\nI don't think the acquisition will kill the culture. Microsoft is a strong partner and Nat Friedman is one of us.\nOn the other side, I'm not as enthusiastic as I used to be.\nThere's room for competitors now and I'm beginning to wonder what will be the next Github.\nThat said, I will keep the Octocat on my office wall, in the hope that the excitement comes back.\n"
  },
  {
    "title": "Ten Years of Vim",
    "url": "https://endler.dev/2018/ten-years-of-vim/",
    "body": "\n\n\nWhen I opened Vim by accident for the first time, I thought it was broken. My\nkeystrokes changed the screen in unpredictable ways, and I wanted to undo things\nand quit. Needless to say, it was an unpleasant experience.  There was something\nabout it though, that kept me coming back and it became my main editor.\nFast forward ten years (!) and I still use Vim.\nAfter all the Textmates and Atoms and PhpStorms I tried, I still find myself at home in Vim.\nPeople keep asking me: Why is that?\nWhy Vim?\nBefore Vim, I had used many other editors like notepad or nano. They all behaved more or less as expected: you insert text, you move your cursor with the arrow keys or your mouse, and you save with Control + S or by using the menu bar. VI (and Vim, its spiritual successor) is different.\nEVERYTHING in Vim is different, and that's why it's so highly effective. Let me explain.\nThe Zen of Vim\nThe philosophy behind Vim takes a while to sink in: \nWhile other editors focus on writing as the central part of working with text, Vim thinks it's editing.\nYou see, most of the time I don't spend writing new text; instead, I edit existing text.\nI mold text, form it, turn it upside down.\nWriting text is craftsmanship and hard work. You have to shape your thoughts with your cold, bare hands until they somewhat form a coherent whole.\nThis painful process is what Vim tries to make at least bearable. It helps you keep control.\nIt does that, by providing you sharp, effective tools to modify text.\nThe core of Vim is a language for editing text.\nVim, the language\nThe Vim commands are not cryptic, you already know them.\n\nTo undo, type u.\nTo find the next t, type ft.\nTo delete a word, type daw.\nTo change a sentence, type cas.\n\nMore often than not, you can guess the correct command by thinking of an operation you want to execute and an object to execute it on.\nThen just take the first character of every word. Try it!\nIf anything goes wrong, you can always hit ESC and type u for undo.\nOperations: delete, find, change, back, insert, append,...\nObjects: word, sentence, parentheses, (html) tag,... (see :help text-objects)\nInserting text is just another editing operation, which can be triggered with i. \nThat's why, by default, you are in normal mode — also called command mode — where all those operations work. \nOnce you know this, Vim makes a lot more sense, and that's when you start to be productive.\nHow my workflow changed over the years\nWhen I was a beginner, I was very interested in how people with more Vim experience would use the editor.\nNow that I'm a long-time user, here's my answer: there's no secret sauce.\nI certainly feel less exhausted after editing text for a day, but 90% of the commands I use fit on a post-it note.\nThat said, throughout the years, my Vim habits changed.\nI went through several phases:\nYear 1: I'm happy if I can insert text and quit again.\nYear 2: That's cool, let's learn more shortcuts.\nYear 3-5: Let's add all the features!!!\nYear 6-10: My .vimrc is five lines long.\nYear three is when I started to learn the Vim ecosystem for real.\nI tried all sorts of flavors like MacVim and distributions like janus.\nFor a while, I even maintained my own Vim configuration\n, which was almost 400 lines long.\nAll of that certainly helped me learn what's out there, but I'm not sure if I would recommend that to a Vim beginner.\nAfter all, you don't really need all of that. Start with a vanilla Vim editor which works just fine!\nMy current Vim setup is pretty minimalistic. I don't use plugins anymore, mostly out of laziness and because built-in Vim commands or macros can replace them.\nHere are three concrete examples of how my workflow changed over the years:\n\n\nIn the beginning, I used a lot of "number powered movements". That is, if you have a command like b, which goes back one word in the text, you can also say 5b to go back five words. Nowadays I mostly use / to move to a matching word because it's quicker.\n\n\nI don't use arrow keys to move around in text anymore but forced myself to use h, j, k, l. Many people say that this is faster. After trying this for a few years, I don't think that is true (at least for me). I now just stick to it out of habit.\n\n\nOn my main working machine I use Vim for quick text editing and Visual Studio Code plus the awesome Vim plugin for projects. This way, I get the best of both worlds.\n\n\nWorkflow issues I still struggle with\nAfter all these years I'm still not a Vim master — far from it. \nAs every other Vim user will tell you, we're all still learning. \nHere are a few things I wish I could do better:\n\nJumping around in longer texts: I know the basics, like searching (/), jumping to a matching bracket (%) or jumping to specific lines (for line 10, type 10G), but I still could use symbols more often for navigation.\nUsing visual mode for moving text around: Sometimes it can be quite complicated to type the right combination of letters to cut (delete) the text I want to move around. That's where visual mode (v) shines. It highlights the selected text. I should use it more often.\nMultiple registers for copy and paste: Right now I only use one register (like a pastebin) for copying text, but Vim supports multiple registers. That's cool if you want to move around more than one thing at the same time. Let's use more of those!\nTabs: I know how tabs work, but all the typing feels clunky. That's why I never extensively used them. Instead, I mostly use multiple terminal tabs or an IDE with Vim bindings for bigger projects.\n\nWould I learn Vim again?\nThat's a tough question to answer.\nOn one side, I would say no. \nThere's a steep learning curve in Vim and seeing all those modern IDEs become better at understanding the user's intent, editing text became way easier and faster in general.\nOn the other side, Vim is the fastest way for me to write down my thoughts and code. As a bonus, it runs on every machine and might well be around in decades to come. In contrast, I don't know if the IntelliJ shortcuts will be relevant in ten years (note: if you read this in the future and ask yourself "What is IntelliJ?", the answer might be no).\nTakeaways\nIf I can give you one tip, don't learn Vim by memorizing commands. Instead, look at your current workflow and try to make it better, then see how Vim can make that easier. It helps to look at other people using Vim to get inspired (Youtube link with sound).\nYou will spend a lot of time writing text, so it's well worth the time investment to learn one editor really well — especially if you are a programmer.\nAfter ten years, Vim is somehow ingrained in my mind. I think Vim when I'm editing text. It has become yet another natural language to me. I'm looking forward to the next ten years.\n"
  },
  {
    "title": "Refactoring Go Code to Avoid File I/O in Unit Tests",
    "url": "https://endler.dev/2018/go-io-testing/",
    "body": "At work today, I refactored some simple Go code to make it more testable.\nThe idea was to avoid file handling in unit tests without mocking or using temporary files by separating data input/output and data manipulation.\n\n\n    \n    \n        A gopher reading a long computer printout\n        \n        Illustration by Marcus Olsson CC BY-NC-SA 4.0\n    \n\nI was surprised that I couldn't find a simple explanation on sites like StackOverflow,\nwhich is why I wrote down some notes myself so that others can refer to it in the future.\nOur example code\nThe initial version looked like this:\n\npackage main\n\nimport (\n\t"bufio"\n\t"io/ioutil"\n\t"os"\n)\n\nfunc main() {\n\tanalyze("test.txt")\n}\n\nfunc analyze(file string) error {\n\thandle, err := os.Open(file)\n\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer handle.Close()\n\n\tscanner := bufio.NewScanner(handle)\n\tfor scanner.Scan() {\n\t\t// Do something with line\n\t\t_ = scanner.Text()\n\t}\n\treturn nil\n}\n\nAs you can see, we take a filename as input, and we open that file inside the analyze function to do something with its contents.\nWriting our first test for the code\nA typical test harness for that code might look like this:\n\npackage main\n\nimport "testing"\n\nfunc Test_analyze(t *testing.T) {\n\tt.Run("Test something", func(t *testing.T) {\n\t\tif err := analyze("test.txt"); (err != nil) != false {\n\t\t\tt.Errorf("analyze() error = %v", err)\n\t\t}\n\t})\n}\n\nAll fine and good?\nProblems\nThis will work, but file I/O while running tests is not always the best idea.\nFor one, you could be running in a constrained environment, where you don't have access to the file.\nWe could use temporary files to avoid this.\nBut there might be problems with disk I/O, which makes for flaky tests and frustration.\nAnother process could also modify the file during the test.\nAll these issues have nothing to do with your code.\nFurthermore, it's not enough to just look at the test and see exactly what's going on. You also have to read the text file first.\nA lot of people suggest mocking instead. \nThere are quite a few powerful libraries like spf13/afero for this purpose.\nThese packages will create temporary files in the background and clean up afterward.\nIn my opinion, mocking should be the last resort when it comes to testing. Before you mock, check that you use the right abstractions in your code.\nMaybe implementing against an interface or using Dependency Injection helps decouple components?\nMore often than not, a clear separation of concerns is all you need.\nRefactoring to make testing easier\nIn my case above, we can easily avoid using mocks and temporary files by decoupling file I/O from the analysis.\nWe do so by refactoring our analyze function to call doSomething, which takes an io.Reader.\n(You could also use an array of strings for now.)\nOur main.go now looks like this:\n\npackage main\n\nimport (\n\t"bufio"\n\t"io"\n\t"os"\n)\n\nfunc main() {\n\tanalyze("test.txt")\n}\n\nfunc analyze(file string) error {\n\thandle, err := os.Open(file)\n\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer handle.Close()\n\treturn doSomething(handle)\n}\n\nfunc doSomething(handle io.Reader) error {\n\tscanner := bufio.NewScanner(handle)\n\tfor scanner.Scan() {\n\t\t// Do something with line\n\t\t_ = scanner.Text()\n\t}\n\treturn nil\n}\n\nNow we can test the actual analysis in isolation:\n\npackage main\n\nimport (\n\t"strings"\n\t"testing"\n)\n\nfunc Test_analyze(t *testing.T) {\n\tt.Run("Test something", func(t *testing.T) {\n\t\tif err := doSomething(strings.NewReader("This is a test string")); (err != nil) != false {\n\t\t\tt.Errorf("analyze() error = %v", err)\n\t\t}\n\t})\n}\n\nWe changed analyze("test.txt") to doSomething(strings.NewReader("This is a test string")).\n(Of course, we should also write a separate test for analyze(), but the focus is on decoupling the datasource-agnostic part here.)\nResult\nBy slightly refactoring our code, we gained the following advantages:\n\nSimple testability: No mocks or temporary files.\nSeparation of concerns: Each function does exactly one thing.\nEasier code re-use: The doSomething() function will work with any io.Reader and can be called from other places. We can even move it to its own library if we want.\n\nOn Reddit, user soapysops made an important remark:\n\nIn general, I prefer to not accept a file name in an API. A file name doesn't give users enough control. It doesn't let you use an unusual encoding, special file permissions, or a bytes.Buffer instead of an actual file, for example. Accepting a file name adds a huge dependency to the code: the file system, along with all of its associated OS specific stuff.\nSo I probably would have eliminated the file name based API and only exposed one based on io.Reader. That way, you have complete code coverage, fast tests, and far fewer edge cases to worry about.\n\nI totally agree with that sentiment.\nBut often times you can't simply change the user-facing API easily, because the API might be public and might already have users. \nThe refactoring above is just the first step towards better architecture. There is definitely a lot more you can do.\nIf that got you interested, also check out justforfunc #29: dependency injection in a code review, which talks about the same topic.\n"
  },
  {
    "title": "A Tiny `ls` Clone Written in Rust",
    "url": "https://endler.dev/2018/ls/",
    "body": "In my series of useless Unix tools rewritten in Rust, today I'm going to be covering one of my all-time favorites: ls.\nFirst off, let me say that you probably don't want to use this code as a replacement for ls on your local machine (although you could!).\nAs we will find out, ls is actually quite a powerful tool under the hood.\nI'm not going to come up with a full rewrite, but instead only cover the very basic output that you would expect from calling ls -l on your command line.\nWhat is this output? I'm glad you asked.\nExpected output\n\n> ls -l\ndrwxr-xr-x 2 mendler  staff    13468 Feb  4 11:19 Top Secret\n-rwxr--r-- 1 mendler  staff  6323935 Mar  8 21:56 Never Gonna Give You Up - Rick Astley.mp3\n-rw-r--r-- 1 mendler  staff        0 Feb 18 23:55 Thoughts on Chess Boxing.doc\n-rw-r--r-- 1 mendler  staff   380434 Dec 24 16:00 nobel-prize-speech.txt\n\nYour output may vary, but generally, there are a couple of notable things going on. From left to right, we've got the following fields:\n\nThe drwx things in the beginning are the file permissions (also called the file mode). If d is set, it's a directory. r means read, w means write and x execute.\nThis rwx pattern gets repeated three times for the current user, the group, and other computer users respectively.\nNext we got the hardlink count when referring to a file, or the number of contained directory entries when referring to a directory. (Reference)\nOwner name\nGroup name\nNumber of bytes in the file\nDate when the file was last modified\nFinally, the path name\n\nFor more in-depth information, I can recommend reading the manpage of ls from the GNU coreutils used in most Linux distributions and the one from Darwin (which powers MacOS). \nWhew, that's a lot of information for such a tiny tool.\nBut then again, it can't be so hard to port that to Rust, right? Let's get started!\nA very basic ls in Rust\nHere is the most bare-bones version of ls, which just prints all files in the current directory:\n\nuse std::fs;\nuse std::path::Path;\nuse std::error::Error;\nuse std::process;\n\nfn main() {\n\tif let Err(ref e) = run(Path::new(".")) {\n\t\tprintln!("{}", e);\n\t\tprocess::exit(1);\n\t}\n}\n\nfn run(dir: &Path) -> Result<(), Box<Error>> {\n\tif dir.is_dir() {\n\t\tfor entry in fs::read_dir(dir)? {\n\t\t\t\tlet entry = entry?;\n\t\t\t\tlet file_name = entry\n\t\t\t\t\t\t.file_name()\n\t\t\t\t\t\t.into_string()\n\t\t\t\t\t\t.or_else(|f| Err(format!("Invalid entry: {:?}", f)))?;\n\t\t\t\tprintln!("{}", file_name);\n\t\t}\n\t}\n\tOk(())\n}\n\n\nWe can copy that straight out of the documentation.\nWhen we run it, we get the expected output:\n\n> cargo run\nCargo.lock\nCargo.toml\nsrc\ntarget\n\nIt prints the files and exits. Simple enough.\nWe should stop for a moment and celebrate our success, knowing that we just wrote our first little Unix utility from scratch.\nPro Tip: You can install the binary with cargo install and call it like any other binary from now on.\nBut we have higher goals, so let's continue.\nAdding a parameter to specify the directory\nUsually, if we type ls mydir, we expect to get the file listing of no other directory than mydir. We should add the same functionality to our version.\nTo do this, we need to accept command line parameters.\nOne Rust crate that I love to use in this case is structopt. It makes argument parsing very easy.\nAdd it to your Cargo.toml. (You need cargo-edit for the following command).\n\ncargo add structopt\n\nNow we can import it and use it in our project:\n\n#[macro_use]\nextern crate structopt;\n\n// use std::...\nuse structopt::StructOpt;\n\n#[derive(StructOpt, Debug)]\nstruct Opt {\n\t/// Output file\n\t#[structopt(default_value = ".", parse(from_os_str))]\n\tpath: PathBuf,\n}\n\nfn main() {\n\tlet opt = Opt::from_args();\n\tif let Err(ref e) = run(&opt.path) {\n\t\t\tprintln!("{}", e);\n\t\t\tprocess::exit(1);\n\t}\n}\n\nfn run(dir: &PathBuf) -> Result<(), Box<Error>> {\n\t// Same as before\n}\n\nBy adding the Opt struct, we can define the command line flags, input parameters, and the help output super easily.\nThere are tons of configuration options, so it's worth checking out the project homepage.\nAlso note, that we changed the type of the path variable from Path to PathBuf. The difference is, that PathBuf owns the inner path string, while Path simply provides a reference to it. The relationship is similar to String and &str.\nReading the modification time\nNow let's deal with the metadata.\nFirst, we try to retrieve the modification time from the file.\nA quick look at the documentation shows us how to do it:\n\nuse std::fs;\n\nlet metadata = fs::metadata("foo.txt")?;\n\nif let Ok(time) = metadata.modified() {\n\tprintln!("{:?}", time);\n}\n\nThe output might not be what you expect: we receive a SystemTime object, which represents the measurement of the system clock. E.g. this code\n\nprintln!("{:?}", SystemTime::now());\n// Prints: SystemTime { tv_sec: 1520554933, tv_nsec: 610406401 }\n\nBut the format that we would like to have is something like this:\n\nMar  9 01:24\n\nThankfully, there is a library called chrono, which can read this format and convert it into any human readable output we like:\n\nlet current: DateTime<Local> = DateTime::from(SystemTime::now());\nprintln!("{}", current.format("%_d %b %H:%M").to_string());\n\nthis prints\n\n9 Mar 01:29\n\n(Yeah, I know it's getting late.)\nArmed with that knowledge, we can now read our file modification time.\n\ncargo add chrono\n\nuse chrono::{DateTime, Local};\n\nfn run(dir: &PathBuf) -> Result<(), Box<Error>> {\n\tif dir.is_dir() {\n\t\tfor entry in fs::read_dir(dir)? {\n\t\t\tlet entry = entry?;\n\t\t\tlet file_name = ...\n\n\t\t\tlet metadata = entry.metadata()?;\n\t\t\tlet size = metadata.len();\n\t\t\tlet modified: DateTime<Local> = DateTime::from(metadata.modified()?);\n\n\t\t\tprintln!(\n\t\t\t\t"{:>5} {} {}",\n\t\t\t\tsize,\n\t\t\t\tmodified.format("%_d %b %H:%M").to_string(),\n\t\t\t\tfile_name\n\t\t\t);\n\t\t}\n\t}\n\tOk(())\n}\n\nThis {:>5} might look weird. It's a formatting directive provided by std::fmt.\nIt means "right align this field with a space padding of 5" - just like our bigger brother ls -l is doing it.\nSimilarly, we retrieved the size in bytes with metadata.len().\nUnix file permissions are a zoo\nReading the file permissions is a bit more tricky.\nWhile the rwx notation is very common in Unix derivatives such as *BSD or GNU/Linux, many other operating systems ship their own permission management.\nThere are even differences between the Unix derivatives.\nWikipedia lists a few extensions to the file permissions that you might encounter:\n\n+ (plus) suffix indicates an access control list that can control additional permissions.\n. (dot) suffix indicates an SELinux context is present. Details may be listed with the command ls -Z.\n@ suffix indicates extended file attributes are present.\n\nThat just goes to show, that there are a lot of important details to be considered when implementing this in real life.\nImplementing very basic file mode\nFor now, we just stick to the basics and assume we are on a platform that supports the rwx file mode.\nBehind the r, the w and the x are in reality octal numbers. That's easier for computers to work with and many hardcore users even prefer to type the numbers over the symbols.\nThe ruleset behind those octals is as follows. I took that from the chmod manpage.\n\n\tModes may be absolute or symbolic. \n\tAn absolute mode is an octal number constructed \n\tfrom the sum of one or more of the following values\n\n\t 0400    Allow read by owner.\n\t 0200    Allow write by owner.\n\t 0100    For files, allow execution by owner.\n\t 0040    Allow read by group members.\n\t 0020    Allow write by group members.\n\t 0010    For files, allow execution by group members.\n\t 0004    Allow read by others.\n\t 0002    Allow write by others.\n\t 0001    For files, allow execution by others.\n\nFor example, to set the permissions for a file so that the owner can read, write and execute it and nobody else can do anything would be 700 (400 + 200 +100).\nGranted, those numbers are the same since the 70s and are not going to change soon, but it's still a bad idea to compare our file permissions directly with the values; if not for compatibility reasons, then for readability and to avoid magic numbers in our code.\nTherefore, we use the libc crate, which provides constants for those magic numbers.\nAs mentioned above, these file permissions are Unix specific, so we need to import a Unix-only library named std::os::unix::fs::PermissionsExt; for that.\n\nextern crate libc;\n\n// Examples:\n// * `S_IRGRP` stands for "read permission for group",\n// * `S_IXUSR` stands for "execution permission for user"\nuse libc::{S_IRGRP, S_IROTH, S_IRUSR, S_IWGRP, S_IWOTH, S_IWUSR, S_IXGRP, S_IXOTH, S_IXUSR};\nuse std::os::unix::fs::PermissionsExt;\n\nWe can now get the file permissions like so:\n\nlet metadata = entry.metadata()?;\nlet mode = metadata.permissions().mode();\nparse_permissions(mode as u16);\n\nparse_permissions() is a little helper function defined as follows:\n\nfn parse_permissions(mode: u16) -> String {\n\tlet user = triplet(mode, S_IRUSR, S_IWUSR, S_IXUSR);\n\tlet group = triplet(mode, S_IRGRP, S_IWGRP, S_IXGRP);\n\tlet other = triplet(mode, S_IROTH, S_IWOTH, S_IXOTH);\n\t[user, group, other].join("")\n}\n\nIt takes the file mode as a u16 (simply because the libc constants are u16)\nand calls triplet on it.\nFor each flag read, write, and execute, it runs a binary & operation on mode.\nThe output is matched exhaustively against all possible permission patterns.\n\nfn triplet(mode: u16, read: u16, write: u16, execute: u16) -> String {\n\tmatch (mode & read, mode & write, mode & execute) {\n\t\t(0, 0, 0) => "---",\n\t\t(_, 0, 0) => "r--",\n\t\t(0, _, 0) => "-w-",\n\t\t(0, 0, _) => "--x",\n\t\t(_, 0, _) => "r-x",\n\t\t(_, _, 0) => "rw-",\n\t\t(0, _, _) => "-wx",\n\t\t(_, _, _) => "rwx",\n\t}.to_string()\n}\nWrapping up\nThe final output looks like this. Close enough.\n\n> cargo run\nrw-r--r--     7  6 Mar 23:10 .gitignore\nrw-r--r-- 15618  8 Mar 00:41 Cargo.lock\nrw-r--r--   185  8 Mar 00:41 Cargo.toml\nrwxr-xr-x   102  5 Mar 21:31 src\nrwxr-xr-x   136  6 Mar 23:07 target\n\nThat's it! You can find the final version of our toy ls on Github.\nWe are still far away from a full-fledged ls replacement, but at least we learned a thing or two about its internals.\nIf you're looking for a proper ls replacement written in Rust, go check out exa.\nIf, instead, you want to read another blog post from the same series, check out A Little Story About the yes Unix Command.\n"
  },
  {
    "title": "Rust in 2018",
    "url": "https://endler.dev/2018/rust-2018/",
    "body": "I wrote about the future of Rust before and it seems like nobody stops me from doing it again! Quite the contrary: this time the Rust core team even asked for it.\nI'm a bit late to the party, but here are my 2 cents about the priorities for Rust in 2018.\n\nWho is this guy?\nThere's a depressingly high chance that we've never met before — which is a real shame.\nFor some context: I come from dynamically typed languages like Python and PHP.\nRust was the first language that allowed me to write real low-level code without feeling like arguing with a bouncer. \nTo me, Rust is not a fireflower, it's my own personal Megazord1.\nI want Rust to win, but for that, we need to tick a few points off the list.\nCompiler documentation for easier contribution\n\nWhen I was in Columbus, Ohio for Rust Belt Rust, I met Niko Matsakis, Ariel Ben-Yehuda, and Santiago Pastorino.\nThose fine gentlemen eagerly worked on non-lexical lifetimes during the impl-period.\nWatching them hack away on the compiler was deeply inspirational to me, and I started wondering if I could contribute, too.\nNeedless to say, the barrier to entry for hacking on the compiler can be quite high.\nI didn't contribute anything yet.\nOne thing I'd love to do is to spend short 30-60 minute chunks of time to fix a small thing in the compiler here and there. Could be as simple as renaming a variable, writing a test or adding some documentation.\nHence my first wish is, that contributing to the language will become easier.\nThat could be achieved by providing extensive mentorship, more entry-level tickets, and better compiler documentation.\nAll of that was already suggested by Niko.\nMore resources for intermediate programmers\nOn a related note, I'd like to see more talks/guidelines/books targeting intermediate Rust programmers.\nThis includes discussions on how to structure big projects in Rust and Rust-specific design patterns.\nI want to read more about professional Rust usage and see case-studies from various industries.\nFor example, there is a startup called snips, which builds an on-device voice-assistant using Rust.\nThey integrate with C and C++ libraries and I want to hear more about their journey.\nImprove the RFC process\nI try to follow the RFC process very closely, but my time is limited.\nMy wish is, that I can open any RFC and immediately get its status:\n\nA summary of the discussion with major pros and cons.\nA simple usage example, right at the beginning.\nThe next steps towards stabilization.\n\nFor example, if I look at this (not so) random issue, I don't even know where to start. What are the biggest blockers right now? Who is actively pushing this forward? How can I help out?\nGithub is great for code, but conversations about new features regularly get out of hand.\nThis is not a problem, that is limited to Rust, either. Just look at other big projects like Docker, Kubernetes, or Node.\nMaybe we need a new tool for that.\nThe usual suspects\nIf I could ask for two stable features in 2018, it would be ? in main \nand non-lexical lifetimes.\nThere's more I could mention of course, but I'm not gonna bore you with faster compile times, impl trait, generators, and the like.\nWe're on a good way here, see Nick Cameron's post instead.\nI'm convinced, that by improving documentation and mentorship, we can grow the number of contributors significantly\nand stabilize many highly-anticipated features this year.\n1. Disclaimer: I never watched a single episode of Power Rangers.↩  \n"
  },
  {
    "title": "Functional Programming for Mathematical Computing",
    "url": "https://endler.dev/2018/functional-mathematics/",
    "body": "Programming languages help us describe general solutions for problems; the result just happens to be executable by machines. Every programming language comes with a different set of strengths and weaknesses, one reason being that its syntax and semantics heavily influence the range of problems which can easily be tackled with it.\ntl;dr: I think that functional programming is better suited for mathematical computations than the more common imperative approach.\nUsing built-in abstractions for Mathematics\nThe ideas behind a language (the underlying programming paradigms) are distinctive for the community that builds around it. The developers create a unique ecosystem of ready-to-use libraries and frameworks around the language core. As a consequence, some languages are stronger in areas such as business applications (one could think of Cobol), others work great for systems programming (like C or Rust).\nWhen it comes to solving mathematical and numerical problems with computers, Fortran might come to mind. Although Fortran is a general-purpose language, it is mostly known for scientific computing. Of course, the language was created with that purpose in mind – hence the name, Formula Translation.\nOne reason for its popularity in this area is that it offers some built-in domain-specific keywords to express mathematical concepts, while keeping an eye on performance. For instance, it has a dedicated datatype for complex numbers – COMPLEX – and a keyword named DIMENSION which is quite similar to the mathematical term and can be used to create arrays and vectors.\nImperative vs functional style\nBuilt-in keywords can help expand the expressiveness of a language into a specific problem space, but this approach is severly limited. It’s not feasible to extend the language core ad infinitum; it would just be harder to maintain and take longer to learn. Therefore, most languages provide other ways of abstraction – like functions, subroutines, classes and objects – to split a routine into smaller, more manageable parts. These mechanisms might help to control the complexity of a program, but especially when dealing with mathematical problems, one has to be careful not to obfuscate the solution with boilerplate code.\nSpecimen I - Factorial\nAs an example, the stated problem might be to translate the following formula, which calculates the factorial of a positive number n, into program code:\n\nAn implementation of the above formula using imperative style Java might look like this:\n\npublic static long fact(final int n) {\n    if (n < 0) {\n        // Negative numbers not allowed\n        return 0;\n    }\n    long prod = 1;\n    for (int i = 1; i <= n; ++i) {\n        prod *= i;\n    }\n    return prod;\n}\n\nThis is quite a long solution for such a short problem definition.\n(Note that writing a version with an explicit loop from 1 to n was on purpose; a recursive function would be shorter, but uses a concept which was not introduced by the mathematical formula.)\nAlso, the program contains many language-specific keywords, such as public, static, and System.err.println(). On top of that, the programmer must explicitly provide all data types for the variables in use – a tiresome obligation.\nAll of this obfuscates the mathematical definition.\nCompare this with the following version written in a functional language, like Haskell.\n\nfact n = product [1..n]\n\nThis is an almost direct translation from the problem definition into code. It needs no explicit types, no temporary variables and no access modifiers (such as public).\nSpecimen II - Dot product\nOne could argue that the above Haskell program owes its brevity to the fact, that the language provides just the right abstractions (namely the product keyword and the [1..n] range syntax) for that specific task.\nTherfore let’s examine a simple function which is neither available in Haskell nor in Java: The dot product of two vectors. The mathematical definition is as follows:\n \nFor vectors with three dimensions, it can be written as\n\nFirst, a Haskell implementation:\n\ntype Scalar a = a\ndata Vector a = Vector a a a deriving (Show)\ndot :: (Num a) => Vector a -> Vector a -> Scalar a\n(Vector a1 a2 a3) `dot` (Vector b1 b2 b3) = a1*b1 + a2*b2 + a3*b3\n\nNote, that the mathematical types can be defined in one line each. Further note, that we define the dot function in infix notation, that is, we place the first argument of dot in front of the function name and the second argument behind it. This way, the code looks more like its mathematical equivalent.\nAn example call of the above function would be \n\n(Vector 1 2 3) ’dot’ (Vector 3 2 1)\n\nwhich is short, precise and readable.\nNow, a similar implementation in Java.\n\npublic static class Vector<T extends Number> {\n    private T x, y, z;\n\n    public Vector(T x, T y, T z) {\n        this.x = x;\n        this.y = y;\n        this.z = z;\n    }\n\n    public double dot(Vector<?> v) {\n        return (x.doubleValue() * v.x.doubleValue() +\n                y.doubleValue() * v.y.doubleValue() +\n                z.doubleValue() * v.z.doubleValue());\n        }\n    }\n\n    public static void main(String[] args) {\n        Vector<Integer> a = new Vector<Integer>(3, 2, 1);\n        Vector<Integer> b = new Vector<Integer>(1, 2, 3);\n        System.out.println(a.dot(b));\n    }\n}\n\nFor a proper textual representation of Vectors, the toString() Method would also need to be overwritten. In Haskell, one can simply derive from the Show typeclass as shown in the code.\nCreating new abstractions\nIf functions and types are not sufficient to write straightforward programs, Haskell also offers simple constructs to create new operators and keywords which extend the language core itself. This makes domain-specific-languages feasible and enables the developer to work more directly on the actual problem instead of working around peculiarities of the programming language itself (such as memory management or array iteration). Haskell embraces this concept; Java has no such functionality.\nConclusion\nI'm not trying to bash Java or worship Haskell here. Both languages have their place.\nI merely picked Java, because lots of programmers can read it.\nThe comparison is more between a functional and an imperative approach for numerical and symbolical programming; and for that, I prefer a functional approach every day. It removes clutter and yields elegant solutions. It provides convenient methods to work on a high level of abstraction and speak in mathematical terms and still, these strengths are disregarded by many programmers.\nAbraham H. Maslow’s observation in his 1966 book The Psychology of Science seems fitting:\n\n“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”\n\n"
  },
  {
    "title": "Rust for Rubyists",
    "url": "https://endler.dev/2017/rust-for-rubyists/",
    "body": "Recently I came across a delightful article on idiomatic Ruby.\nI'm not a good Ruby developer by any means, but I realized, that a lot of the patterns are also quite common in Rust.\nWhat follows is a side-by-side comparison of idiomatic code in both languages.\nThe Ruby code samples are from the original article.\nMap and Higher-Order Functions\nThe first example is a pretty basic iteration over elements of a container using map.\n\n\n\nuser_ids = users.map { |user| user.id }\n\nThe map concept is also pretty standard in Rust.\nCompared to Ruby, we need to be a little more explicit here:\nIf users is a vector of User objects, we first need to create an iterator from it:\n\n\n\nlet user_ids = users.iter().map(|user| user.id);\n\nYou might say that's quite verbose, but this additional abstraction allows us to express an important concept:\nwill the iterator take ownership of the vector, or will it not?\n\nWith iter(), you get a "read-only view" into the vector. After the iteration, it will be unchanged.\nWith into_iter(), you take ownership over the vector. After the iteration, the vector will be gone.\nIn Rust terminology, it will have moved.\nRead some more about the difference between iter() and into_iter() here.\n\nThe above Ruby code can be simplified like this:\n\n\n\nuser_ids = users.map(&:id)\n\nIn Ruby, higher-order functions (like map) take blocks or procs as an argument and the language provides a convenient shortcut for method invocation — &:id is the same as {|o| o.id()}.\nSomething similar could be done in Rust:\n\n\n\nlet id = |u: &User| u.id;\nlet user_ids = users.iter().map(id);\n\nThis is probably not the most idiomatic way to do it, though. What you will see more often is the use of Universal Function Call Syntax in this case:1\n\n\n\nlet user_ids = users.iter().map(User::id);\n\nIn Rust, higher-order functions take functions as an argument. Therefore users.iter().map(Users::id) is more or less equivalent to users.iter().map(|u| u.id()).2\nAlso, map() in Rust returns another iterator and not a collection.\nIf you want a collection, you would have to run collect() on that, as we'll see later.\nIteration with Each\nSpeaking of iteration, one pattern that I see a lot in Ruby code is this:\n\n\n\n["Ruby", "Rust", "Python", "Cobol"].each do |lang|\n  puts "Hello #{lang}!"\nend\n\nSince Rust 1.21, this is now also possible:\n\n\n\n["Ruby", "Rust", "Python", "Cobol"]\n    .iter()\n    .for_each(|lang| println!("Hello {lang}!", lang = lang));\n\nAlthough, more commonly one would write that as a normal for-loop in Rust:\n\n\n\nfor lang in ["Ruby", "Rust", "Python", "Cobol"].iter() {\n    println!("Hello {lang}!", lang = lang);\n}\n\nSelect and filter\nLet's say you want to extract only even numbers from a collection in Ruby.\n\n\n\neven_numbers = [1, 2, 3, 4, 5].map { |element| element if element.even? } # [ni, 2, nil, 4, nil]\neven_numbers = even_numbers.compact # [2, 4]\n\nIn this example, before calling compact, our even_numbers array had nil entries.\nWell, in Rust there is no concept of nil or Null. You don't need a compact.\nAlso, map doesn't take predicates. You would use filter for that:\n\n\n\nlet even_numbers = vec![1, 2, 3, 4, 5]\n    .iter()\n    .filter(|&element| element % 2 == 0);\n\nor, to make a vector out of the result\n\n\n\n// Result: [2, 4]\nlet even_numbers: Vec<i64> = vec![1, 2, 3, 4, 5]\n    .into_iter()\n    .filter(|element| element % 2 == 0).collect();\n\nSome hints:\n\nI'm using the type hint Vec<i64> here because, without it, Rust does not know what collection I want to build when calling collect.\nvec! is a macro for creating a vector.\nInstead of iter, I use into_iter. This way, I take ownership of the elements in the vector. With iter() I would get a Vec<&i64> instead.\n\nIn Rust, there is no even method on numbers, but that doesn't keep us from defining one!\n\n\n\nlet even = |x: &i64| x % 2 == 0;\nlet even_numbers = vec![1, 2, 3, 4, 5].into_iter().filter(even);\n\nIn a real-world scenario, you would probably use a third-party package (crate) like num for numerical mathematics:\n\n\n\nextern crate num;\nuse num::Integer;\n\nfn main() {\n    let even_numbers: Vec<i64> = vec![1, 2, 3, 4, 5]\n        .into_iter()\n        .filter(|x| x.is_even()).collect();\n}\n\nIn general, it's quite common to use crates in Rust for functionality that is not in the standard lib.\nPart of the reason why this is so well accepted is that cargo is such a rad package manager.\n(Maybe because it was built by no other than Yehuda Katz of Ruby fame. 😉)\nAs mentioned before, Rust does not have nil. However, there is still the concept of operations that can fail.\nThe canonical type to express that is called Result.\nLet's say you want to convert a vector of strings to integers.\n\n\n\nlet maybe_numbers = vec!["1", "2", "nah", "nope", "3"];\nlet numbers: Vec<_> = maybe_numbers\n    .into_iter()\n    .map(|i| i.parse::<u64>())\n    .collect();\n\nThat looks nice, but maybe the output is a little unexpected. numbers will also contain the parsing errors:\n\n\n\n[Ok(1), Ok(2), Err(ParseIntError { kind: InvalidDigit }), Err(ParseIntError { kind: InvalidDigit }), Ok(3)]\n\nSometimes you're just interested in the successful operations.\nAn easy way to filter out the errors is to use filter_map:\n\n\n\nlet maybe_numbers = vec!["1", "2", "nah", "nope", "3"];\nlet numbers: Vec<_> = maybe_numbers\n    .into_iter()\n    .filter_map(|i| i.parse::<u64>().ok())\n    .collect();\n\nI changed two things here:\n\nInstead of map, I'm now using filter_map.\nparse returns a Result, but filter_map expects an Option. We can convert a Result into an Option by calling ok() on it3.\n\nThe return value contains all successfully converted strings:\n\n\n\n[1, 2, 3]\n\nThe filter_map is similar to the select method in Ruby:\n\n\n\n[1, 2, 3, 4, 5].select { |element| element.even? }\n\nRandom numbers\nHere's how to get a random number from an array in Ruby:\n\n\n\n[1, 2, 3].sample\n\nThat's quite nice and idiomatic!\nCompare that to Rust:\n\n\n\nlet mut rng = thread_rng();\nrng.choose(&[1, 2, 3, 4, 5])\n\nFor the code to work, you need the rand crate. Click on the snippet for a running example.\nThere are some differences to Ruby. Namely, we need to be more explicit about what random number generator\nwe want exactly. We decide for a lazily-initialized thread-local random number generator, seeded by the system.\nIn this case, I'm using a slice instead of a vector. The main difference is that the slice has a fixed size while the vector does not.\nWithin the standard library, Rust doesn't have a sample or choose method on the slice itself. \nThat's a design decision: the core of the language is kept small to allow evolving the language in the future.\nThis doesn't mean that you cannot have a nicer implementation today.\nFor instance, you could define a Choose trait and implement it for [T].\n\n\n\nextern crate rand;\nuse rand::{thread_rng, Rng};\n\ntrait Choose<T> {\n    fn choose(&self) -> Option<&T>;\n}\n\nimpl<T> Choose<T> for [T] {\n    fn choose(&self) -> Option<&T> {\n        let mut rng = thread_rng();\n        rng.choose(&self)\n    }\n}\n\nThis boilerplate could be put into a crate to make it reusable for others.\nWith that, we arrive at a solution that rivals Ruby's elegance.\n\n\n\n[1, 2, 4, 8, 16, 32].choose()\n\nImplicit returns and expressions\nRuby methods automatically return the result of the last statement.\n\n\n\ndef get_user_ids(users)\n  users.map(&:id)\nend\n\nSame for Rust. Note the missing semicolon.\n\n\n\nfn get_user_ids(users: &[User]) -> Vec<u64> {\n    users.iter().map(|user| user.id).collect()\n}\n\nBut in Rust, this is just the beginning, because everything is an expression.\nThe following block splits a string into characters, removes the h, and returns the result as a HashSet.\nThis HashSet will be assigned to x.\n\n\n\nlet x: HashSet<_> = {\n    // Get unique chars of a word {'h', 'e', 'l', 'o'}\n    let unique = "hello".chars();\n    // filter out the 'h'\n    unique.filter(|&char| char != 'h').collect()\n};\n\nSame works for conditions:\n\n\n\nlet x = if 1 > 0 { "absolutely!" } else { "no seriously" };\n\nSince a match statement is also an expression, you can assign the result to a variable, too!\n\n\n\nenum Unit {\n    Meter,\n    Yard,\n    Angstroem,\n    Lightyear,\n}\n\nlet length_in_meters = match unit {\n    Unit::Meter => 1.0,\n    Unit::Yard => 0.91,\n    Unit::Angstroem => 0.0000000001,\n    Unit::Lightyear => 9.461e+15,\n};\n\nMultiple Assignments\nIn Ruby you can assign multiple values to variables in one step:\n\n\n\ndef values\n  [1, 2, 3]\nend\n\none, two, three = values\n\nIn Rust, you can only decompose tuples into tuples, but not a vector into a tuple for example.\nSo this will work:\n\n\n\nlet (one, two, three) = (1, 2, 3);\n\nBut this won't:\n\n\n\nlet (one, two, three) = [1, 2, 3];\n//    ^^^^^^^^^^^^^^^^^ expected array of 3 elements, found tuple\n\nNeither will this:\n\n\n\nlet (one, two, three) = [1, 2, 3].iter().collect();\n// a collection of type `(_, _, _)` cannot be built from an iterator over elements of type `&{integer}`\n\nBut with nightly Rust, you can now do this:\n\n\n\nlet [one, two, three] = [1, 2, 3];\n\nOn the other hand, there's a lot more you can do with destructuring apart from multiple assignments. You can write beautiful, ergonomic code using pattern syntax.\n\n\n\nlet x = 4;\nlet y = false;\n\nmatch x {\n    4 | 5 | 6 if y => println!("yes"),\n    _ => println!("no"),\n}\n\nTo quote The Book:\n\nThis prints no since the if condition applies to the whole pattern 4 | 5 | 6, not only to the last value 6.\n\nString interpolation\nRuby has extensive string interpolation support.\n\n\n\nprogramming_language = "Ruby"\n"#{programming_language} is a beautiful programming language"\n\nThis can be translated like so:\n\n\n\nlet programming_language = "Rust";\nformat!("{} is also a beautiful programming language", programming_language);\n\nNamed arguments are also possible, albeit much less common:\n\n\n\nprintln!("{language} is also a beautiful programming language", language="Rust");\n\nRust's println!() syntax is even more extensive than Ruby's. Check the docs if you're curious about what else you can do.\nThat’s it!\nRuby comes with syntactic sugar for many common usage patterns, which allows for very elegant code.\nLow-level programming and raw performance are no primary goals of the language.\nIf you do need that, Rust might be a good fit, because it provides fine-grained hardware control with comparable ergonomics.\nIf in doubt, Rust favors explicitness, though; it eschews magic.\nDid I whet your appetite for idiomatic Rust? Have a look at this Github project. I'd be thankful for contributions.\nFootnotes\n1. Thanks to Florian Gilcher for the hint.↩\n2. Thanks to masklin for pointing out multiple inaccuracies.↩\n3. In the first version, I sait that ok() would convert a Result into a boolean, which was wrong. Thanks to isaacg for the correction.↩\n"
  },
  {
    "title": "Making Myself Obsolete",
    "url": "https://endler.dev/2017/obsolete/",
    "body": "\n    \n    \n        The Stegosaurus had better days 150 million years ago.\n        \n        Paleontologists once thought it had a brain in its butt.\n    \n\nIn December 2015 I was looking for static analysis tools to integrate into trivago's CI process.\nThe idea was to detect typical programming mistakes automatically.\nThat's quite a common thing, and there are lots of helpful tools out there which fit the bill.\nSo I looked for a list of tools...\nTo my surprise, the only list I found was on Wikipedia — and it was outdated.\nThere was no such project on Github, where most modern static analysis tools were hosted.\nWithout overthinking it, I opened up my editor and wrote down a few tools I found through my initial research. After that, I pushed the list to Github.\nI called the project Awesome Static Analysis.\nFast forward two years and the list has grown quite a bit.\nSo far, it has 75 contributors, 277 forks and received over 2000 stars. (Thanks for all the support!)\n(Update May 2018: 91 contributors, 363 forks, over 3000 stars)\nAround 1000 unique visitors find the list every week. Not much by any means, but I feel obliged to keep it up-to-date\nbecause it has become an essential source of information for many people.\nIt now lists around 300 tools for static analysis. Everything from Ada to TypeScript is on there.\nWhat I find particularly motivating is, that now the authors themselves create pull requests to add their tools!\nThere was one problem though: The list of pull requests got longer and longer, as I was busy doing other things.\n\nAdding contributors\nI always try to make team members out of regular contributors. My friend and colleague Andy Grunwald as well as Ouroboros Chrysopoeia are both valuable collaborators. They help me weed out new PRs whenever they find the time.\nBut let's face it: checking the pull requests is a dull, manual task.\nWhat needs to be checked for each new tool can be summarized like this:\n\nFormatting rules are satisfied\nProject URL is reachable\nLicense annotation is correct\nTools of each section are alphabetically ordered\nDescription is not too long\n\nI guess it's obvious what we should do with that checklist: automate it!\nA linter for linting linters\nSo why not write an analysis tool, which checks our list of analysis tools!\nWhat sounds pretty meta, is actually pretty straightforward.\nWith every pull request, we trigger our bot, which checks the above rules and responds with a result.\nThe first step was to read the Github documentation about building a CI server.\nJust for fun, I wanted to create the bot in Rust.\nThe two most popular Github clients for Rust were github-rs and hubcaps.\nBoth looked pretty neat, but then I found afterparty, a "Github webhook server".\nThe example looked fabulous:\n\n#[macro_use]\nextern crate log;\nextern crate env_logger;\nextern crate afterparty;\nextern crate hyper;\n\nuse afterparty::{Delivery, Hub};\n\nuse hyper::Server;\n\npub fn main() {\n    env_logger::init().unwrap();\n    let addr = format!("0.0.0.0:{}", 4567);\n    let mut hub = Hub::new();\n    hub.handle("pull_request", |delivery: &Delivery| {\n        match delivery.payload {\n            Event::PullRequest { ref action, ref sender, .. } => {\n                // TODO: My code here!\n                println!("sender {} action {}", sender.login, action)\n            }\n            _ => (),\n        }\n    });\n    let srvc = Server::http(&addr[..])\n                   .unwrap()\n                   .handle(hub);\n    println!("listening on {}", addr);\n    srvc.unwrap();\n}\n\nThis allowed me to focus on the actual analysis code,\nwhich makes for a pretty boring read. It mechanically checks for the things mentioned above and could be written in any language.\nIf you want to have a look (or even contribute!), check out the repo.\nTalking to Github\nAfter the analysis code was done, I had a bot, running locally, waiting for incoming pull requests.\nBut how could I talk to Github?\nI found out, that I should use the Status API\nand send a POST request to /repos/mre/awesome-static-analysis/statuses/:sha\n(:sha is the commit ID that points to the HEAD of the pull request):\n\n{\n  "state": "success",\n  "description": "The build succeeded!"\n}\n\nI could have used one of the existing Rust Github clients, but I decided to write a simple function to update the pull request status code.\n\nfn set_status(status: Status, desc: String, repo: &str, sha: &str) -> Result<reqwest::Response> {\n    let token = env::var("GITHUB_TOKEN")?;\n    let client = reqwest::Client::new();\n    let mut params = HashMap::new();\n    params.insert("state", format!("{}", status));\n    params.insert("description", desc);\n    println!("Sending status: {:#?}", params);\n\n    let status_url = format!("https://api.github.com/repos/{}/statuses/{}", repo, sha);\n    println!("Status url: {}", status_url);\n    Ok(client\n        .request(\n            reqwest::Method::Post,\n            &format!(\n                "{}?access_token={}",\n                status_url,\n                token,\n            ),\n        )\n        .json(&params)\n        .send()?)\n}\n\nYou can see that I pass in a Github token from the environment and then I send the JSON payload as a post request using the reqwest library.\nThat turned out to become a problem in the end: while afterparty was using version 0.9 of hyper, reqwest was using 0.11. Unfortunately, these two versions depend on a different build of the openssl-sys bindings. That's a well-known problem and the only way to fix it, is to resolve the conflict.\nI was stuck for a while, but then I saw, that there was an open pull request to upgrade afterparty to hyper 0.10.\nSo inside my Cargo.toml, I locked the version of afterparty to the version of the pull request:\n\n[dependencies]\nafterparty = { git = "https://github.com/ms705/afterparty" }\n\nThis fixed the build, and I could finally move on.\nDeployment\nI needed a place to host the bot.\nPreferably for free, as it was a non-profit Open Source project.\nAlso, the provider would have to run binaries.\nFor quite some time, I was following a product named zeit.\nIt runs any Docker container using an intuitive command line interface called now.\n\n  \nYour browser does not support playing mp4 files.\n \nI fell in love the first time I saw their demo on the site, so I wanted to give it a try.\nSo I added a multi-stage Dockerfile to my project:\n\nFROM rust as builder\nCOPY . /usr/src/app \nWORKDIR /usr/src/app \nRUN cargo build --release\n\nFROM debian:stretch\nRUN apt update \\\n    && apt install -y libssl1.1 ca-certificates \\\n    && apt clean -y \\\n    && apt autoclean -y \\\n    && apt autoremove -y\nCOPY --from=builder target/release/check .\nEXPOSE 4567\nENTRYPOINT ["./check"]\nCMD ["--help"]\n\nThe first part would build a static binary, the second part would run it at container startup.\nWell, that didn't work, because zeit does not support multi-stage builds yet.\nThe workaround was to split up the Dockerfile into two and connect them both with a Makefile. Makefiles are pretty powerful, you know?\nWith that, I had all the parts for deployment together.\n\n# Build Rust binary for Linux\ndocker run --rm -v $(CURDIR):/usr/src/ci -w /usr/src/ci rust cargo build --release\n\n# Deploy Docker images built from the local Dockerfile\nnow deploy --force --public -e GITHUB_TOKEN=${GITHUB_TOKEN}\n\n# Set domain name of new build to `check.now.sh`\n# (The deployment URL was copied to the clipboard and is retrieved with pbpaste on macOS)\nnow alias `pbpaste` check.now.sh\n\nHere's the output of the deploy using now:\n\n> Deploying ~/Code/private/awesome-static-analysis-ci/deploy\n> Ready! https://deploy-sjbiykfvtx.now.sh (copied to clipboard) [2s]\n> Initializing…\n> Initializing…\n> Building\n> ▲ docker build\nSending build context to Docker daemon 2.048 kBkB\n> Step 1 : FROM mre0/ci:latest\n> latest: Pulling from mre0/ci\n> ...\n> Digest: sha256:5ad07c12184755b84ca1b587e91b97c30f7d547e76628645a2c23dc1d9d3fd4b\n> Status: Downloaded newer image for mre0/ci:latest\n>  ---> 8ee1b20de28b\n> Successfully built 8ee1b20de28b\n> ▲ Storing image\n> ▲ Deploying image\n> ▲ Container started\n> listening on 0.0.0.0:4567\n> Deployment complete!\n\nThe last step was to add check.now.sh as a webhook inside the awesome-static-analysis project settings.\nNow, whenever a new pull request is coming in, you see that little bot getting active!\n\nOutcome and future plans\nI am very pleased with my choice of tools: afterparty saved me from a lot of manual work, while zeit made deployment really easy.\nIt feels like Amazon Lambda on steroids.\nIf you look at the code and the commits for my bot, you can see all my little missteps, until I got everything just right. Turns out, parsing human-readable text is tedious.\nTherefore I was thinking about turning the list of analysis tools into a structured format like YAML. This would greatly simplify the parsing and have the added benefit of having a machine-readable list of tools that can be used for other projects.\nUpdate May 2018\nWhile attending the WeAreDevelopers conference in Vienna (can recommend that), I moved the CI pipeline from zeit.co to Travis CI.\nThe reason was, that I wanted the linting code next to the project, which greatly simplified things.\nFirst and foremost I don't need the web request handling code anymore, because travis takes care of that.\nIf you like, you can compare the old and the new version.\n"
  },
  {
    "title": "Modern Day Annoyances - Digital Clocks",
    "url": "https://endler.dev/2017/digitial-clocks/",
    "body": "This morning I woke up to the beeping noise of our oven's alarm clock.\nThe reason was that I tried to correct the oven's local time the day before — and I pushed the wrong buttons.\nAs a result I didn't set the correct time, instead, I set a cooking timer... and that's what woke me up today.\n\n\n    \n\nLet's add a clock to the microwave!\nOn occasions like these, I wonder why there's a digital clock on every single household device these days.\nThey're integrated into microwaves, fridges, ovens, dishwashers, dryers, mixers — and that's just the kitchen!\nThere is an inflation of digital clocks on modern-day devices.\nA lot of times I was wondering why that is the case. Here's my best guess:\nIt's easier to add a useless digital clock to the design than to leave it out.\nSay you are the engineer responsible for the control panel of a run-of-the-mill microwave.\nThe microwave chip comes with a digital timer, which is perfect for showing the remaining time until the food is warmed up.\nNow the question is, what will the timer show when you don't want to heat anything?\nWell, why not show the current time?\nIt's unobtrusive and adds value.\nExcept that these digital clocks can be quite annoying:\n\nThey run out of sync and show the wrong time.\nThey get reset when being plugged off or there's a power outage. (That's the dreaded, blinking 00:00 we all learned to love.)\nThey don't automatically switch between summer and winter time (hey Germany!).\n\nThat's why I constantly need to look after those clocks.\nLet me tell you a secret:\nWhen I'm not warming stuff in the oven, I don't want it to tell me the local time. I want the stove to be off.\nWhy I have trouble setting the clock on our oven\nOur oven has three buttons related to time: plus, minus and a clock symbol.\nTo set the time, you push the clock symbol. An arrow appears and the display changes to 00:00. You press time again and another arrow appears.\nPressing it two more times shows a blinking clock symbol. Then you can use the + and - buttons to adjust the time. After that, you wait to confirm.\nEasy!\nThe problem is, there is no immediate relationship between the controls and the result in the world.\nThe underlying concept is called mapping and is prevalent in interface design.\nTo add some functionality to a device you have two options:\n\nAdd more buttons.\nTeach an existing button a new trick.\n\nOption 1 might dilute your beautiful design, while option 2 might mean frustration for your users.\nNeither option is appealing.\nOur oven maps multiple functions to the same button.\nBut the most annoying thing is, that each device has a different mapping.\nLearning to set the time on my oven won't help me with the dishwasher, which sports an entirely different interface!\nTakeaways\nGood industrial designs are few and far between.\nA clock on your product will most likely not add any additional value.\nIn the best case it might be an annoyance, in the worst case it's harmfully misleading.\nWhen given a choice, I prefer home appliances without clocks.\nLooking at today's market, that's harder than it sounds.\nArguably, a device with a clock is cheaper than one without; just because the ones with timers get produced more often.\nNow I can understand why it took Steve Jobs two weeks to decide on a washing machine:\n\nWe spent some time in our family talking about what's the trade-off we want to make.\nWe spent about two weeks talking about this. Every night at the dinner table.\n\nHe chose a Miele Washing machine in the end - without a digital clock, I assume.\n"
  },
  {
    "title": "Learn Some Rust During Hacktoberfest",
    "url": "https://endler.dev/2017/hacktoberfest/",
    "body": "\n    \n    \n        Dirndl, Lederhose, Brezn, Beer, Rust\n        \n        Designed by Freepik\n    \n\nOctober is the perfect time to contribute to Open Source — at least according to Github and DigitalOcean.\nBecause that's when they organize Hacktoberfest, a global event where you get a free shirt and lots of street cred for creating pull requests. Read the official announcement here.\nSome people think they cannot contribute anything of value. Either because they lack the programming skills or because they don't know where to start.\nThis guide is trying to change that!\nLet me show you, how everybody can contribute code to Rust, a safe systems programming language.\nI was inspired to write this by a tweet from llogiq.\n1. Find a great Rust project to work on\nWe all want our work to be appreciated.\nTherefore I suggest to start contributing to medium-sized projects, because they gained some momentum but are still driven by a small number of maintainers, so help is always welcome. By contrast, tiny projects are mostly useful to the original author only, while large projects can be intimidating at first and have stricter guidelines.\nFor now, let's look at repositories with 5-100 stars, which were updated within this year.\nGithub supports advanced search options based on Lucene syntax. \n\nlanguage:Rust stars:5..100 pushed:>2017-01-01\n\nHere's a list of projects, which match this filter.\n2. Install the Rust toolchain\nTo start contributing, we need a working Rust compiler and the cargo package manager.\nFortunately, the installation should be straightforward.\nI recommend rustup for that.\nRun the following command in your terminal, then follow the onscreen instructions.\n\ncurl https://sh.rustup.rs -sSf | sh\n\nIf you're unsure, just accept the defaults.\nAfter the installation is done, we also need to get the nightly version of the compiler for later.\n\nrustup install nightly\n\nQuestions so far? Find more detailed installation instructions here.\n3. Fork the project and clone it to your computer\nFirst, click on the little fork button on the top right of the Github project page. Then clone your fork to your computer. \n\ngit clone git@github.com:yourusername/project.git\n\nFor more detailed instructions, go here.\n4. Does it build?\nBefore we start modifying the codebase, we should make sure that it is in a workable state.\nThe following commands should work right away from inside the project folder.\n\ncargo build\ncargo test\n\nIf not, you might want to consult the README for further instructions. (But feel free to choose another project.)\n5. The magic sauce\nHere's the trick: we use a linter called clippy to show us improvement areas in any Rust codebase.\nTo get clippy, install it like so:\n\ncargo +nightly install clippy\n\nAfterwards, run it from the project root as often as you like.\n\nrustup run nightly cargo clippy\n\nThis should give you actionable information on how to improve the codebase.\nHere's some sample output:\n\nwarning: useless use of `format!`\n   --> src/mach/header.rs:420:49\n    |\n420 |             let error = error::Error::Malformed(format!("bytes size is smaller than an Mach-o header"));\n    |                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    |\n    = note: #[warn(useless_format)] on by default\n    = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.165/index.html#useless_format\n\nwarning: this expression borrows a reference that is immediately dereferenced by the compiler\n   --> src/mach/header.rs:423:36\n    |\n423 |             let magic = mach::peek(&bytes, 0)?;\n    |                                    ^^^^^^ help: change this to: `bytes`\n    |\n    = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.165/index.html#needless_borrow\n\nJust try some of the suggestions and see if the project still compiles and the tests still pass.\nCheck out the links to the documentation in the help section to learn more.\nStart small to make your changes easier to review.\n6. Creating a Pull Request\nIf you're happy with your changes, now is the time to publish them!\nIt's best to create a new branch for your changes and then push it to your fork.\n\ngit checkout -b codestyle\ngit commit -am "Minor codestyle fixes"\ngit push --set-upstream origin codestyle\n\nAfterwards, go to the homepage of your fork on Github.\nThere should be a button titled Compare & pull request.\nPlease add a meaningful description and then submit the pull request.\nCongratulations! You've contributed to the Rust ecosystem. Thank you! 🎉\nTrophy case\n\nm4b/goblin\nfitzgen/cpp_demangle\nfdehau/tui-rs\nchristophertrml/rs-natural\n\nBonus!\nIf all of the manual fixing and checking sounds too dull, you can automate step number 5 using rustfix by Pascal Hertleif (@killercup):\n\nrustfix --yolo && cargo check\n"
  },
  {
    "title": "A Little Story About the `yes` Unix Command",
    "url": "https://endler.dev/2017/yes/",
    "body": "What's the simplest Unix command you know?\nThere's echo, which prints a string to stdout and true, which always terminates with an exit code of 0.\nAmong the rows of simple Unix commands, there's also yes.\nIf you run it without arguments, you get an infinite stream of y's, separated by a newline:\n\ny\ny\ny\ny\n(...you get the idea)\n\nWhat seems to be pointless in the beginning turns out to be pretty helpful  :\n\nyes | sh boring_installation.sh\n\nEver installed a program, which required you to type "y" and hit enter to keep going?\nyes to the rescue! It will carefully fulfill this duty, so you can keep watching Pootie Tang.\nWriting yes\nHere's a basic version in... uhm... BASIC.\n\n10 PRINT "y"\n20 GOTO 10\n\nAnd here's the same thing in Python:\n\nwhile True:\n    print("y")\n\nSimple, eh? Not so quick!\nTurns out, that program is quite slow. \n\npython yes.py | pv -r > /dev/null\n[4.17MiB/s]\n\nCompare that with the built-in version on my Mac:\n\nyes | pv -r > /dev/null\n[34.2MiB/s]\n\nSo I tried to write a quicker version in Rust. Here's my first attempt:\n\nuse std::env;\n\nfn main() {\n  let expletive = env::args().nth(1).unwrap_or("y".into());\n  loop {\n    println!("{}", expletive);\n  }\n}\n\nSome explanations:\n\nThe string we want to print in a loop is the first command line parameter and is named expletive. I learned this word from the yes manpage.\nI use unwrap_or to get the expletive from the parameters. In case the parameter is not set, we use "y" as a default.\nThe default parameter gets converted from a string slice (&str) into an owned string on the heap (String) using into().\n\nLet's test it.\n\ncargo run --release | pv -r > /dev/null\n   Compiling yes v0.1.0\n    Finished release [optimized] target(s) in 1.0 secs\n     Running `target/release/yes`\n[2.35MiB/s] \n\nWhoops, that doesn't look any better. It's even slower than the Python version!\nThat caught my attention, so I looked around for the source code of a C implementation.\nHere's the very first version of the program, released with Version 7 Unix and famously authored by Ken Thompson on Jan 10, 1979:\n\nmain(argc, argv)\nchar **argv;\n{\n  for (;;)\n    printf("%s\\n", argc>1? argv[1]: "y");\n}\n\nNo magic here.\nCompare that to the 128-line-version from the GNU coreutils, which is mirrored on Github. After 25 years, it is still under active development!\nThe last code change happened around a year ago.\nThat's quite fast:\n\n# brew install coreutils\ngyes | pv -r > /dev/null \n[854MiB/s]\n\nThe important part is at the end:\n\n/* Repeatedly output the buffer until there is a write error; then fail.  */\nwhile (full_write (STDOUT_FILENO, buf, bufused) == bufused)\n  continue;\n\nAha! So they simply use a buffer to make write operations faster.\nThe buffer size is defined by a constant named BUFSIZ, which gets chosen on each system so as to make I/O efficient (see here).\nOn my system, that was defined as 1024 bytes. I actually had better performance with 8192 bytes.\nI've extended my Rust program:\n\nuse std::env;\nuse std::io::{self, BufWriter, Write};\n\nconst BUFSIZE: usize = 8192;\n\nfn main() {\n    let expletive = env::args().nth(1).unwrap_or("y".into());\n    let mut writer = BufWriter::with_capacity(BUFSIZE, io::stdout());\n    loop {\n        writeln!(writer, "{}", expletive).unwrap();\n    }\n}\n\nThe important part is, that the buffer size is a multiple of four, to ensure memory alignment.\nRunning that gave me 51.3MiB/s.\nFaster than the version, which comes with my system, but still way slower than the results from this Reddit post that I found, where the author talks about 10.2GiB/s.\nUpdate\nOnce again, the Rust community did not disappoint.\nAs soon as this post hit the Rust subreddit, user nwydo pointed out a previous discussion on the same topic.\nHere's their optimized code, that breaks the 3GB/s mark on my machine:\n\nuse std::env;\nuse std::io::{self, Write};\nuse std::process;\nuse std::borrow::Cow;\n\nuse std::ffi::OsString;\npub const BUFFER_CAPACITY: usize = 64 * 1024;\n\npub fn to_bytes(os_str: OsString) -> Vec<u8> {\n  use std::os::unix::ffi::OsStringExt;\n  os_str.into_vec()\n}\n\nfn fill_up_buffer<'a>(buffer: &'a mut [u8], output: &'a [u8]) -> &'a [u8] {\n  if output.len() > buffer.len() / 2 {\n    return output;\n  }\n\n  let mut buffer_size = output.len();\n  buffer[..buffer_size].clone_from_slice(output);\n\n  while buffer_size < buffer.len() / 2 {\n    let (left, right) = buffer.split_at_mut(buffer_size);\n    right[..buffer_size].clone_from_slice(left);\n    buffer_size *= 2;\n  }\n\n  &buffer[..buffer_size]\n}\n\nfn write(output: &[u8]) {\n  let stdout = io::stdout();\n  let mut locked = stdout.lock();\n  let mut buffer = [0u8; BUFFER_CAPACITY];\n\n  let filled = fill_up_buffer(&mut buffer, output);\n  while locked.write_all(filled).is_ok() {}\n}\n\nfn main() {\n  write(&env::args_os().nth(1).map(to_bytes).map_or(\n    Cow::Borrowed(\n      &b"y\\n"[..],\n    ),\n    |mut arg| {\n      arg.push(b'\\n');\n      Cow::Owned(arg)\n    },\n  ));\n  process::exit(1);\n}\n\nNow that's a whole different ballgame!\n\nWe prepare a filled string buffer, which will be reused for each loop.\nStdout is protected by a lock. So, instead of constantly acquiring and releasing it, we keep it all the time.\nWe use a the platform-native std::ffi::OsString and std::borrow::Cow to avoid unnecessary allocations.\n\nThe only thing that I could contribute was removing an unnecessary mut. 😅\nLessons learned\nThe trivial program yes turns out not to be so trivial after all.\nIt uses output buffering and memory alignment to improve performance.\nRe-implementing Unix tools is fun and makes me appreciate the nifty tricks,\nwhich make our computers fast.\n"
  },
  {
    "title": "Lightning Fast Image Previews with Pure CSS and LQIP",
    "url": "https://endler.dev/2017/image-previews/",
    "body": "\n    \n    \n        \n        Adapted from Freepik\n    \n\nMy website is reasonably fast.\nI hope that every page load feels snappy, no matter your device or location.\nThat should not come as a surprise. After all, I'm just using plain HTML and CSS.\nJavaScript is avoided whenever possible.\nThere was one thing left, which really annoyed me: layout reflow after images got loaded.\nThe problem is, that the image dimensions are not known when the text is ready to be displayed.\nAs a result, the text will be pushed down on the screen as soon as an image is loaded above.\nAlso, while an image is loading, there is no preview, just blank space.\nHere's what that looks like on a slower connection:\n\nI could fix that, by hardcoding the image width and height, but that would be tedious and error-prone.\nAnd there would be no preview.\nSo I was wondering, what others were doing. 🤔\nTiny image thumbnails\nI vaguely remembered, that Facebook uses tiny preview thumbnails in their mobile app.\nThey extract the quantization table from the JPEG header to render the preview. This information \nis stored on the client, so it doesn't need to be downloaded every time.\nUnfortunately, this approach requires full control over the image encoder.\nIt works for apps, but hardly for websites.\nThe search continued.\nUntil my colleague Tobias Baldauf introduced me to LQIP (Low-Quality Image Placeholders).\nHere's the idea:\n\nLoad the page including inlined, low-quality image thumbnails.\nOnce the page is fully loaded (e.g. when the onload event is fired), lazy load full quality images.\n\nUnfortunately, this technique requires JavaScript.\nNevertheless, I liked the idea, so I started experimenting with different image sizes and formats. My goal was to create the smallest thumbnails using any standard image format.\nBenchmark\nHere are 15 pixel wide thumbnails encoded in different file formats:\n\nI used different tools to create the thumbnails.\nFor JPEG and PNG encoding, I used svgexport.\n\nsvgexport img.svg img.png "svg{background:white;}" 15: 1%\n\nFor webp, I used cwebp:\n\ncwebp img.png -o img.webp\n\nThe gif was converted using an online tool and optimized using gifsicle:\n\ngifsicle -O3 < img.gif > img_mini.gif\nComparison\nWebP is the smallest, but it's not supported by all browsers.\nGif was second, but when resizing the image and applying the blur filter, I was not happy with the result.\nIn the end, I settled for PNG, which provided an excellent tradeoff between size and quality.\nI optimized the images even further using oxipng, which supports zopfli compression.\nWith that, I end up with thumbnails of around 300-400 bytes in size.\nI integrated the thumbnail creation process into my build toolchain for the blog.\nThe actual code to create the images is rather boring.\nIf you really want to have a look, it's on Github.\nAvoiding JavaScript\nHere is the skeleton HTML for the image previews:\n\n<figure>\n  <div class="loader">\n    <object data="image.svg" type="image/svg+xml"></object>\n    <img class="frozen" src="data:image/png;base64,..." />\n  </div>\n</figure>\n\nThe trick is to wrap both the full-size image and the preview image into a loader div,\nwhich gets a width: auto CSS attribute:\n\n.loader {\n  position:relative;\n  overflow: hidden;\n  width: auto;\n}\n\nI wrap the SVG into an object tag instead of using an img element.\nThis has the benefit, that I can show a placeholder in case the SVG can't be loaded.\nI position the object at the top left of the loader div.\n\n.loader object {\n  position: absolute;\n}\n\n.loader img, .loader object {\n  display: block;\n  top: 0;\n  left: 0;\n  width: 100%;\n}\n\nHere's the placeholder hack including some references:\n\n// https://stackoverflow.com/a/29111371/270334\n// https://stackoverflow.com/a/32928240/270334\nobject {\n  position: relative;\n  float: left;\n  display: block;\n  \n  &::after {\n    position: absolute;\n    top: 0;\n    left: 0;\n    display: block;\n    width: 1000px;\n    height: 1000px;\n    content: '';\n    background: #efefef;\n  }\n}\n\nThe last part is the handling of the thumbnails.\nLike most other sites, I decided to apply a blur filter.\nIn a way, it looks like the image is frozen, so that's what I called the CSS selector.\nI also applied a scaling transformation to achieve sharp borders.\n\n.frozen {\n  -webkit-filter: blur(8px);\n  -moz-filter: blur(8px);\n  -o-filter: blur(8px);\n  -ms-filter: blur(8px);\n  filter: blur(8px);\n  transform: scale(1.04);\n  animation: 0.2s ease-in 0.4s 1 forwards fade;\n  width: 100%;\n}\n\n@keyframes fade {\n  0% {\n    opacity:1;\n  }\n  100% {\n    opacity:0;\n  }\n}\n\nI use CSS animations instead of JavaScript.\nThe duration of the animation is based on the 95% percentile load time of all visitors of the page. Although it's just an approximation, this should work for most readers.\nResult\n\nNo JavaScript needed\nWorks on all modern browsers\nSupports a fallback in case the main image can't be loaded\nTiny overhead\n\nResources\n\nIntroducing LQIP – Low Quality Image Placeholders\nHow Medium does progressive image loading\nSQIP, a new preview technique using pure SVG\n\n"
  },
  {
    "title": "Go vs Rust? Choose Go.",
    "url": "https://endler.dev/2017/go-vs-rust/",
    "body": "\n    \n    \n        \n        Gopher designed with Gopherize.me. Cogwheels designed by Freepik\n    \n\n"Rust or Go, which one should I choose?" is a question I get quite often.\nBoth languages seem to be competing for the same user base and they both seem to be\nsystems programming languages, so there must be a clear winner, right?\ntl;dr: It's not so easy. Both languages have a different scope. Golang shines for writing microservices and for typical "DevOps" tasks, but it is not a systems programming language. Rust is stronger for tasks where concurrency, safety and/or performance are important; but it has a steeper learning curve than Go.\nGo: practical, pragmatic, plain\n\nI don't think Go is an elegant language. Its biggest feature is simplicity.\nGo is not even a systems programming language. While it's great for writing microservices and tooling around backend infrastructure, I would not want to write a kernel or a memory allocator with it.\nBut with Go, you get things done — fast.\nGo is one of the most productive languages I've ever worked with.\nThe mantra is: solve real problems today. \nRust's strong guarantees come at a cost\n\nRust in comparison is hard. It took me many months to become somewhat productive.\nYou need to invest a serious amount of time to see any benefit.\nRust is already a powerful language and it gets stronger every day.\nIt feels much more like a pragmatic Haskell to me than a safer C.\nDon't get me wrong: I love Rust, and it helped me become a better programmer. It is certainly a nice language to learn. The big question is, if it is the right choice for your next major project.\nHere's the thing: if you choose Rust, usually you need the guarantees, that the language provides:\n\nSafety against Null pointers, race conditions and all sorts of low-level threats.\nPredictable runtime behavior (zero cost abstractions and no garbage collector).\n(Almost) total control over the hardware (memory layout, processor features).\nSeamless interoperability with other languages.\n\nIf you don't require any of these features, Rust might be a poor choice for your next project.\nThat's because these guarantees come with a cost: ramp-up time.\nYou'll need to unlearn bad habits and learn new concepts.\nChances are, you will fight with the borrow checker a lot when you start out.\nCase-study: Primality by trial division\nLet's say, you want to check if a number is prime.\nThe easiest way is to check if we can divide the number by any smaller natural number (without a remainder). If not, we found a prime number! This approach is called trial division.\nHere's how to do that in Golang (courtesy of Rosetta Code):\n\nfunc IsPrime(n int) bool {\n\tif n < 0 {\n\t\tn = -n\n\t}\n\tswitch {\n\tcase n < 2:\n\t\treturn false\n\tdefault:\n\t\tfor i := 2; i < n; i++ {\n\t\t\tif n%i == 0 {\n\t\t\t\treturn false\n\t\t\t}\n\t\t}\n\t}\n\treturn true\n}\n\nAnd here's the same thing in Rust:\n\npub fn is_prime(n: u64) -> bool {\n    match n {\n        0...1 => false,\n        _ => {\n            for d in 2..n {\n                if n % d == 0 {\n                    return false;\n                }\n            }\n            true\n        }\n    }\n}\n\nAt first sight, both solutions look pretty similar.\nBut if we look closer, we can spot some differences.\n\nIn Go, we use a simple switch-case statement. In Rust, we use a match statement, which is much more powerful.\nIn Go, we use a simple for-loop to iterate over the numbers 2 to n. In Rust, we use a range expression (2..n).\nIn Go, we use two return statements, in Rust we have one return expression. In general, most things in Rust are expressions, which can be returned and assigned to a variable. Read more about expressions here.\n\nIn many areas, Rust is more functional than Golang. You could rewrite the above code using the any method, which is implemented for Range.\n\nfn is_prime(n: u64) -> bool {\n    match n {\n        0...1 => false,\n        _ => !(2..n).any(|d| n % d == 0),\n    }\n}\n\nIt might seem a little alien at first, but it will become second-nature after a while.\nThis was just a quick example, of course. I suggest, you browse some code on Rosetta Code to get a better feeling for both languages.\nCase study: Finding duplicate words in text files\nIf you're more like a visual type, here is a video where I write a simple\nconcurrent program in Go and Rust to compare both languages:\n\n    \n    \n\nSome things I prefer in Go\n\nFast compile times\nPragmatic problem-solving approach\nNice ecosystem for typical DevOps tasks\nBatteries-included standard-library\nIDE support\nSimple error handling\nThe mascot 😉\n\nSome things I prefer in Rust\n\nSafety: No null pointers, no data races,...\nFine-grained system control\nIncredible runtime speed (comparable with C/C++)\nZero-cost abstractions\nAwesome, open-minded community\nSimple package management with cargo\nSupport for Generics in form of traits\nC interop and FFI\n\nConclusion\n99% of the time, Go is "good enough" and that 1% where it isn't, you'll know.\nAnd then take a look at Rust, because the two languages complement each other pretty well.\nAfter all is said and done, Rust and Go are not really competitors.\n"
  },
  {
    "title": "Afraid of Makefiles? Don't be!",
    "url": "https://endler.dev/2017/makefiles/",
    "body": "\n    \n    \n        What do clothes have to do with Makefiles? Find out in this post.\n        \n        Illustration by Anindyanfitri - Freepik.com\n    \n\nIn the last few years, I've had the pleasure to work with a lot of talented Software Engineers.\nOne thing that struck me is, that many of them did not have any working knowledge of Makefiles \nand why they are useful.\nWhen faced with the task to automate a build process, they often roll their own shell scripts.\nCommon culprits are called build.sh or run.sh or doall.sh etc.\nThey implement the same basic functionality over and over again:\n\nParsing input parameters and environment variables.\nManually managing dependencies between build steps.\nError handling... maybe.\n\nAlong the way, they keep making the same basic mistakes:\n\nIncorrectly handling input parameters and environment variables.\nMissing dependencies between build steps.\nForgetting to handle errors and — even worse — carrying on with the program execution.\n\nMakefiles are scary!\nIf you think that make is scary, you probably think of complicated build machinery for big software projects.\nIt doesn't need to be that way. Let's hear what the author of make, Stuart Feldman has to say:\n\nIt began with an elaborate idea of a dependency analyzer, boiled down to something much simpler, and turned into Make that weekend. Use of tools that were still wet was part of the culture. Makefiles were text files, not magically encoded binaries because that was the Unix ethos: printable, debuggable, understandable stuff.\n— The Art of Unix Programming (2003)\n\nMakefiles are simple!\nBefore I leave the house, I need to get dressed.\nI use the same simple routine every time:\nUnderpants, trousers, shirt, pullover, socks, shoes, jacket.\nMost likely you also have a routine, even though yours might be different.\nSome of these steps depend on each other.\nMake is useful for handling dependencies.\nLet's try to express my routine as a Makefile.\n\ndress: trousers shoes jacket\n\t@echo "All done. Let's go outside!"\n\njacket: pullover\n\t@echo "Putting on jacket."\n\npullover: shirt\n\t@echo "Putting on pullover."\n\nshirt:\n\t@echo "Putting on shirt."\n\ntrousers: underpants\n\t@echo "Putting on trousers."\n\nunderpants:\n\t@echo "Putting on underpants."\n\nshoes: socks\n\t@echo "Putting on shoes."\n\nsocks: pullover\n\t@echo "Putting on socks."\n\nIf we execute the Makefile, we get the following output:\n\n$ make dress\nPutting on underpants.\nPutting on trousers.\nPutting on shirt.\nPutting on pullover.\nPutting on socks.\nPutting on shoes.\nPutting on jacket.\nAll done. Let's go outside!\nWhat just happened?\nNoticed how the steps are in the correct order?\nBy plainly writing down the dependencies between the steps, make helps us to execute them correctly.\nEach build step has the following structure:\n\ntarget: [dependencies]\n\t<shell command to execute>\n\t<shell command to execute>\n\t...\n\n\n\nThe first target in a Makefile will be executed by default when we call make.\n\n\nThe order of the targets does not matter.\n\n\nShell commands must be indented with a tab.\n\n\nAdd an @ sign to suppress output of the command that is executed.\n\n\nIf target isn't a file you want to build, please add .PHONY <target> at the end of the build step.\nCommon phony targets are: clean, install, run,...\n\ninstall: \n\tnpm install\n.PHONY: install\n\nOtherwise, if somebody creates an install directory, make will silently fail, because the build target already exists.\n\n\nCongratulations! You've learned 90% of what you need to know about make.\nNext steps\nReal Makefiles can do much more! They will only build the files that have changed instead of doing a full rebuild.\nAnd they will do as much as possible in parallel.\n"
  },
  {
    "title": "Of Boxes and Trees - Smart Pointers in Rust",
    "url": "https://endler.dev/2017/boxes-and-trees/",
    "body": "Recently, I tried to implement a binary tree data structure in Rust.\nEach binary tree has a root value, a left, and a right subtree.\nI started from this Python implementation, which is quite straightforward.\n\n\nclass Tree:\n  def __init__(self, val, left=None, right=None):\n    self.val = val\n    self.left = left\n    self.right = right\n\nThis allows us to declare a fancy tree object like this:\n\nt = Tree(15,\n      Tree(12,\n           None,\n           Tree(13)),\n      Tree(22,\n           Tree(18),\n           Tree(100)))\n\nAnd the result can be visualized beautifully.\n(Yes I've drawn that myself.)\n\n    \n    \n        A binary search tree representing our data structure\n    \n\nPorting that code to Rust turned out to be a little... challenging.\nMy first attempt looked quite innocuous.\n\nstruct Tree {\n  root: i64,\n  left: Tree,\n  right: Tree,\n}\n\nThat's pretty much a one-to-one translation of the Python definition — but rustc says no.\n\nerror[E0072]: recursive type `Tree` has infinite size\n --> src/main.rs:1:1\n  |\n1 | struct Tree {\n  | ^^^^^^^^^^^ recursive type has infinite size\n  |\n  = help: insert indirection (e.g., a `Box`, `Rc`, or `&`) at some point to make `Tree` representable\n\nComing from memory-managed languages (like Python, PHP, or Ruby), I was confused by this.\nThe problem is easy to understand, though.\nComputers have a limited amount of memory.\nIt's the compiler's job to find out how much memory to allocate for each item.\nIn our case, it infers the following:\nA tree is a structure containing an i64, and two trees. Each of these trees is a structure containing an i64, and two trees. Each of these...\nYou get the idea.\n\nTree { i64, Tree, Tree }\nTree { i64, Tree { ... }, Tree { ... } }\n// The next expansion won't fit on the page anymore\n\nSince we don't know how many subtrees our tree will have, there is no way to tell how much memory we need to allocate up front. We'll only know at runtime!\nRust tells us how to fix that: by inserting an indirection like Box, Rc, or &.\nThese are different "pointer types" in Rust. They all point to places in memory. So, instead of knowing the total size of our tree structure, we just know the point in memory where the tree is located. But that's enough to define the tree structure.\nThese pointer types allow us to do that safely and without manual memory management.\nThey all offer different guarantees and you should choose the one that fits your requirements best.\n\n\n& is called a borrow in Rust speech. It's the most common of the three. It's a reference to some place in memory, but it does not own the data it points to. As such, the lifetime of the borrow depends on its owner.\nTherefore we would need to add lifetime parameters here. This can make it tedious to use.\n\nstruct Tree<'a> {\n  root: i64,\n  left: &'a Tree<'a>,\n  right: &'a Tree<'a>,\n}\n\n\nBox is a smart pointer with zero runtime overhead. It owns the data it points to.\nWe call it smart because when it goes out of scope, it will first drop the data it points to and then itself. No manual memory management required.\n\nstruct Tree {\n  root: i64,\n  left: Box<Tree>,\n  right: Box<Tree>,\n}\n\n\nRc is another smart pointer. It's short for "reference-counting". It keeps track of the number of references to a data structure. As soon as the number of references is down to zero, it cleans up after itself.\nChoose Rc if you need to have multiple owners of the same data in one thread.\nFor multithreading, there's also Arc (atomic reference count).\n\nstruct Tree {\n  root: i64,\n  left: Rc<Tree>,\n  right: Rc<Tree>,\n}\n\n\nPutting the tree into a box\nAll three options are totally valid. Which one you should choose, depends on your use-case.\nA rule of thumb is to keep it simple.\nIn my case, I chose to use a Box, because I did not need any special guarantees.\nMaking subtrees optional\nThe next problem I faced was that I could not instantiate a tree structure.\nThe left and right subtree have the type Box<Tree>, but at some\npoint I would need an empty subtree.\nIn the Python example, I used None to signal the end of my data structure.\nThanks to Rust's Option type we can do the same:\n\nstruct Tree {\n  root: i64,\n  left: Option<Box<Tree>>,\n  right: Option<Box<Tree>>,\n}\n\nAfter all of this, we can create our first tree:\n\nTree {\n    root: 15,\n    left: Some(Box::new(Tree {\n            root: 12,\n            left: None,\n            right: Some(Box::new(Tree {\n                    root: 13,\n                    left: None,\n                    right: None,\n            })),\n    })),\n    right: Some(Box::new(Tree {\n            root: 22,\n            left: Some(Box::new(Tree {\n                    root: 18,\n                    left: None,\n                    right: None,\n            })),\n            right: Some(Box::new(Tree {\n                    root: 100,\n                    left: None,\n                    right: None,\n            })),\n    })),\n};\n\nDepending on your point of view, you might say this is either verbose or explicit.\nCompared to the Python version, it looked a bit too cluttered.\nCan we do better?\nChris McDonald helped me to come up with the following representation:\n\nTree::new(15)\n  .left(\n    Tree::new(12)\n      .right(Tree::new(13))\n  )\n  .right(\n    Tree::new(22)\n      .left(Tree::new(18))\n      .right(Tree::new(100))\n  );\n\nTo me, this is much easier on the eye.\nHere's the full tree implementation that makes this possible:\n\n#[derive(Default)]\nstruct Tree {\n  root: i64,\n  left: Option<Box<Tree>>,\n  right: Option<Box<Tree>>,\n}\n\nimpl Tree {\n  fn new(root: i64) -> Tree {\n    Tree {\n      root: root,\n      ..Default::default()\n    }\n  }\n\n  fn left(mut self, leaf: Tree) -> Self {\n    self.left = Some(Box::new(leaf));\n    self\n  }\n\n  fn right(mut self, leaf: Tree) -> Self {\n    self.right = Some(Box::new(leaf));\n    self\n  }\n}\n\nUpdate: Danny Grein mentioned on Twitter, that\nwe can support the following syntax by implementing From<i64> for Tree:\n\nroot(15)\n  .left(root(12).right(13))\n  .right(root(22).left(18).right(100));\nWhy did it work in Python?\nNow you might be wondering, why our tree implementation worked so flawlessly in Python.\nThe reason is that Python dynamically allocates memory for the tree object at runtime.\nAlso, it wraps everything inside a PyObject, which is kind of similar to Rc from above\n— a reference counted smart pointer.\nRust is more explicit here. It gives us more flexibility to express our needs.\nThen again, we need to know about all the possible alternatives to make good use of them.\nIf you can, then stay away from smart pointers and stick to simple borrows.\nIf that's not possible, as seen above, choose the least invasive one for your\nuse-case. The Rust documentation is a good starting point here.\nAlso, read "Idiomatic tree and graph-like structures in Rust" for some clever use of allocators.\n"
  },
  {
    "title": "Why Type Systems Matter",
    "url": "https://endler.dev/2017/why-type-systems-matter/",
    "body": "I've written most of my code in dynamically typed languages such as Python or PHP. But ever since dabbling with Rust, I've developed a passion for static type systems.\nIt began to feel very natural to me; like a totally new way to express myself.\n\nTypes are here to help\nWith types, you communicate your guarantees and expectations. Both, to the machine and other developers. Types express intent.\nAs a programmer, you've probably gained some intuition about types.\n\nsentence = "hello world"\n\nYou might guess that sentence is a string. It's in quotes, after all. \nIt gets a little more tricky if the type gets inferred from some other location.\n\nsentence = x\n\nIs sentence still a string? Uhm... we don't know. It depends on the type of x. Maybe x is a number, and so sentence is also a number? Maybe x used to be a string but during refactoring it is now a byte array? Fun times had by all. 🎉\nWhat about this one?\n\nfilesize = "5000" # Size in bytes\n\nHere, we express a file size as a string.\nWhile this might work, it's an unsettling idea.\nEven simple calculations might lead to unexpected results:\n\nfile1 = "5000"\nfile2 = "3000"\ntotal = file1 + file2\nprint(total) # prints '50003000'\n\nHow can we fix that?\nWe can safely assume that a file size is always a number.\nTo be more precise, it must be a positive, natural number.\nThere can be no negative file size, and our smallest block of memory is one byte\n(on all but the most obscure systems).\nAnd since we're dealing with a discrete machine here, we know it can only be\na filesize the computer can handle.\nIf we only could express all of this in a precise way...?\nThis is where type systems enter the stage.\nIn Rust, you could define a File type with a field named size.\n\nstruct File {\n  name: String,\n  size: usize,\n}\n\nThe usize gives you the guarantee to be always big enough to hold any pointer into memory (on 64 bit computers usize = u64).\nNow there is no more ambiguity about the type of size.\nYou can't even create an invalid file object:\n\n// Error: `size` can't be a string.\nlet weird_file = File { name: 123, size: "hello" };\n\nThe type system will prevent invalid state. It will simply not allow you to\nbreak your own rules. It will hold you accountable for your design choices.\nDare I say it: it becomes an extension of your brain.\nAfter some time you start to rely on the type checker. "If it compiles, it runs"\nis a powerful mantra.\nTypes improve readability and provide context\nConsider the following Python snippet:\n\ndef filter_files(files):\n  matches = []\n  for file in files:\n    if file.status == 0:\n      matches.append(file)\n  return matches\n\nWhat does 0 represent?\nWe can't say. We lack the context!\nThe story gets a little clearer once we define an enum type like this:\n\nfrom enum import Enum\n\nclass FileStatus(Enum):\n  OPEN = 0\n  CLOSED = 1\n\nOur example from above becomes\n\ndef filter_files(files):\n  matches = []\n  for file in files:\n    if file.status == FileStatus.OPEN:\n      matches.append(file)\n  return matches\n\nIn a larger codebase, FileStatus.OPEN is much easier to search for than 0.\nNote: The native enum type was introduced very late in the history of Python. It serves as a nice\nexample of how enhancing the type system can help improve readability.\nWhen you combine different types, magic happens.\nAll pieces suddenly fall into place when you choose your types wisely. Out of nowhere, the compiler will start\nchecking your design decisions and if all your types work well together. It will point out flaws in your mental model.\nThis gives you a great amount of confidence during refactoring.\nFor example, let's think about sorting things.\nWhen I think of sorting, I first think about a list of numbers:\n\nsorted([1,5,4,3,2]) # [1,2,3,4,5]\n\nThat's the happy path. How about this one?\n\nsorted(1)\n\nOuch. This can't work because 1 is a single number and not a collection!\nIf we forget to check the type before we pass it to sorted, we get an error\nwhile the program runs.\n\nsorted([1, "fish"])\n\nIn Python 2, this would result in [1, 'fish'] (because strings will be compared by length)\nEdit: Reddit user jcdyer3 pointed out that the reason is that when incomparable types are compared, they sort by their type, so all ints will come before all strings. It's a CPython implementation detail).\n\n    \n    \n        1 < fish according to Python 2\n        \n        Illustration provided by Freepik\n    \n\nSince Python 3, this throws an Exception.\n\nTypeError: '<' not supported between instances of 'str' and 'int'\n\nMuch better! One less source of error. The problematic thing is though, that this happens at runtime.\nThat's because of Python's dynamic typing.\nWe could have avoided that with a statically typed language.\n\nfn sorted<T>(collection: &mut [T]) where T: PartialOrd {\n  // TODO: Sort the collection here.\n}\n\nLooks scary but it really isn't.\nWe define a function named sorted which takes one input parameter named\ncollection.\nThe type of collection consists of four parts:\n\nThe & means that we "borrow" the collection, we don't own it. After the function returns, it will still exist. It won't be cleaned up.\nThe mut means that the collection is mutable. We are allowed to modify it.\n[T] indicates that we expect a list/slice/vector as input. Everything else\nwill be rejected at compile time (before the program even runs).\nPartialOrd is\nthe magic sauce.  It is a trait, which is something like an interface. It means that all elements T in the collection must be partially ordered.\n\nAll of this information helps the compiler to prevent us from shooting ourselves in the foot.\nAnd we can understand the inputs and outputs of the function without looking elsewhere.\nTakeaways\n\nTypes force developers to do their homework and think about the guarantees and limitations of their code.\nDon't think of types as constraints, think of them as a safety net which will protect you from your own flawed mental models.\nAlways choose the type which most precisely expresses your intentions.\nIf there is no perfect type in the standard library, create your own from simpler types.\n\nFollowing these rules, I found that I was magically guided towards the most elegant representation of my ideas.\nMy code became much more idiomatic.\n"
  },
  {
    "title": "Being a Professional Programmer",
    "url": "https://endler.dev/2017/professional-programming/",
    "body": "When I was around 12, I set myself the goal to become a professional programmer.\nI can tell, because at this time I made the conscious decision to use my right hand to control the mouse — even though I'm left-handed.\nMy reasoning was, that if I ever had to help out a colleague with a computer problem I sure did not want to move her mouse to the other side before getting started.  That would be awkward. \n(Of course I did not foresee the advent of the wireless mouse... As a matter of fact, I still use the right hand out of habit.)\nOne thing I always wanted to know is how a typical workday of a programmer looked like.\nWas I wasting my time by pursuing this career?\nOnly later I found the answer — but I had to become a professional programmer myself.\nThis article aims to save you from a few years of uncertainty.\nBefore you dig into this, be sure to read the first part of this series titled "Why I love Programming".\nWhat's the difference between "professional" and "hobby" programming?\nIn one word: accountability.\nYou are expected to be responsible.\nProgramming in your free time is like throwing a party without having to clean up: pure fun!\nIf you get bored you're free to move on.\nNot so in professional programming, where you're expected to get the job done.\nEvery application requires constant bug fixing, refactoring and sometimes even monkey patching. Maintaining code is no amusement park; especially if it's not your own.\nBeing a Junior Developer\nFresh out of school you might think you're a pretty kick-ass programmer. Let me tell you: you're not.\nYou wouldn't guess what talented people can do with these blinking machines.\nYou'll have tons of things to learn in the first few years.\nProfessional software development is a lengthy process. Writing readable, well-tested, well-documented code is a substantial effort. You will need patience, lots of it. Both, with yourself and with others.\nAs a junior, you only think in black and white. You look at some code, and it's all wrong. Who in their right mind created this horrible monstrosity?!\nAs you become more experienced, you'll see the shades of grey.\nEventually, you'll understand that those neckbeards were not slower than you, but\nmore careful. You learn how to test your code, how to document it. You even begin to\nappreciate UML diagrams.\nBecoming obsolete\n"The world is moving too fast. What you learned today is obsolete tomorrow. Why bother?".\nI've heard that saying countless times throughout my career.\nIt's both, popular and wrong.\nIf a skill becomes obsolete, it's not a skill.\nThroughout your career you don't want to be known as "the Jenkins guy", you want to be the\nexpert in Software Quality. Hint: If you don't know what Jenkins is, that's the\nwhole point. You should not narrow down your scope too much.\nThe right skills never become obsolete.\nFrom time to time it happens, that due to some new company policy your beautiful creation will become obsolete.\nAs depressing as it sounds: it's a regular part of the software business.\nYou need to adapt.\nOne advice I can give you is not to take it too seriously.\nDrop the project, keep the wisdom.\nEmbrace change.\nWriting software in a non-perfect world\nA professional programmer has to deal with deficiencies all the time. The game is called "balancing constraints". Deadlines, budgets, and code quality are just a few competing constraints we have to consider.\nElegant designs fade away in the face of reality.\nIn the end you want to earn money with your software, so you have to ship it!\nThe best developers I know, keep the balance between pragmatism and elegance.\nThey know which parts matter and which don't. Those who don't will be replaced\nwhen there's a need.\nFor me, I was always leaning more towards elegance.\nThat's just a nicer way to say I was a perfectionist.\nI needed to learn the pragmatic part through hard work.\nMentoring less experienced Programmers\n\nThe better you become at programming, the less you code.\n\nInstead, you will spend more time thinking about Software Architecture,\nhigh-level designs and splitting up the work into smaller junks for other developers to consume.\nYou will start mentoring Junior Developers. Recruiting will require a lot of your\nattention. You will spend your time in Meetings, discussing project goals with\nbusiness people.\nOne might say, you take the role of a mediator. Others might call you a manager.\nOnce you know the ins and outs of the business, you are an essential asset for\nthe company. You might get asked to become a manager, or at least managing projects will slowly feel like a natural extension of your responsibilities.\nBut beware! This slow and gradual process is dangerous.\nMoving back to being a full-time programmer is not easy. \nDuring the time you were busy with project management, others were busy improving their\ncoding skills.\nYou can try to keep up-to-date in your free time but that's hard.\nI've seen excellent developers become great managers. At some point in your career\nit's a decision you need to make for yourself.\nHowever you decide, it pays off to invest some time into learning how to\ncommunicate. Empathy plays a prominent role in that.\nDeveloping software as a team is so complicated that a lot of time is spent on aligning goals and communicating problems. In fact, communication is what you get paid for. This includes documentation, tests and the code itself.\nTalk to others, listen to their problems. Read books about Software Project\nManagement, even though you don't want to be a manager yourself. It will help\nyou understand the role of your boss.\nA word about money\nThere are many good reasons to work in IT, but money is not one of them.\nWhile it can be tempting to base your career decisions on prospective salary,\ndon't do it. You will be very unhappy. You will spend eight hours or more each day sitting in front of a blinking cursor.\nThat's a lot of time, and time is much more valuable than money.\nDon't get me wrong. There's plenty of jobs that pay well. \nYou will most likely not get rich, though. If you want\nto make it big, I can't help you. Maybe look into Real Estate or so...\nThe only way to get rich as a developer is to work on something really hard, put in lots of hours and get\nlucky. Startups, basically. Keep in mind: One Bill Gates takes a thousand failed\nattempts.\nAnother way is to stop being a programmer and become a manager instead. \nI've already shared my opinion on that in the last section.\nFinal words\nWhile you should learn to read (and maybe write) code, working as a professional programmer is not for everyone.\nYou might ask: "Is it worth it?". \nFor me it was the right decision. Hopefully I could help you to make your own.\n"
  },
  {
    "title": "The Future of Rust",
    "url": "https://endler.dev/2017/future-of-rust/",
    "body": "Let me first point out the obvious: yes, the title is a little sensationalist. Also\nyou might be asking why I should be entitled to talk about the future of Rust. After\nall, I'm neither part of the Rust core team, nor a major contributor to the Rust\necosystem. To that I answer: why not? It's fun to think about the future of\nsystems programming in general and Rust in particular.\n\n    \n    \n        Ferris is the inofficial Rust mascot\n        \n        Illustration provided by zooenvato for FreePik.com\n    \n\nYou might have heard of the near-term goals that the core team has committed itself to. Faster compile times and a more gentle learning curve come to mind.\nThis post is not about that.\nInstead, I want to explore some more exotic areas where Rust could shine in\nfive to ten years from now. To make it big, we need both, roots and wings.\nData Science\nRight now, the most popular languages for Data Science are Python, Java, R, and C++.\n\n  \n  Programming language popularity for data science (Source).\n  \n\nWe've observed that while prototypes are mostly written in dynamically typed\nlanguages like Python and R, once an algorithm reaches production level quality\nit is often rewritten in faster languages such as C++ for scalability.\nIt is not unthinkable that Rust is going to be some healthy competition for C++ in the near future.\nThe benchmarks of leaf, a machine learning library written in Rust, are already nothing short of\nimpressive.\nBlockbuster games\nGames are another area where Rust might shine. \nIt's financially attractive for Game Studios to support multiple platforms without much\neffort. Cargo and rustup make cross-compiling easy.\nModern libraries slowly fill the tooling gaps for large-scale game development.\nRust's support for the Vulkan 3D graphics API might already be the best of class.\nThe killer feature though is the unique combination of safety and performance.\nIf you ship a game to a million players and they throw money at you, you'll better make sure that it doesn't crash... right?\nThat said, the first AAA Rust game might still be far in the future. Here's Blizzard's standpoint on Rust in 2017.\nSystems Engineering\nMaybe — eventually — we will also see formal verification of the Rust core. Projects like RustBelt would then open new opportunities in safety-focused industries like the Space industry. Wouldn't it be nice to safely land a Spacecraft on Mars that is controlled by Rust? (Or by one of its spiritual successors.)\nI wonder if SpaceX is experimenting with Rust already...\nIntegrating with other languages\nThere are many other areas I haven't even mentioned yet. For example, financial and medical software or Scientific Computing, just to name a few.\nIn all cases, Rust might be a good fit. Right now the biggest barrier to entry \nis probably the huge amount of legacy code. Many industries maintain large codebases in Cobol,\nC or Fortran that are not easily rewritten.\nFortunately, Rust has been proven to work very nicely with other languages. \nPartly because of strong C-compatibility and partly because there is no Runtime or Garbage Collector.\nA typical pattern is to optimize some core part of an application in Rust that has hard safety/performance\nrequirements, while leaving the rest untouched.\nI think this symbiosis will only become stronger in the long run.\nThere are even ambitious projects like Corrode which attempt to translate C code to Rust automatically.\nSummary\nOverall I see huge potential for Rust in areas where safety, performance or total control over the machine are essential. With languages like Rust and Crystal, a whole class of errors is a thing of the past. No null pointers, no segmentation faults, no memory leaks, no data races.\nI find it encouraging that future generations of programmers will take all that for granted.\n"
  },
  {
    "title": "Launching a URL Shortener in Rust using Rocket",
    "url": "https://endler.dev/2017/rust-url-shortener/",
    "body": "One common Systems Design task in interviews is to sketch the software architecture of a URL shortener (a bit.ly clone, so to say).\nSince I was playing around with Rocket, why not give it a try?\n\n\n    \n    \n        A rocket travelling through space\n    \n\nRequirements\nA URL shortener has two main responsibilities:\n\nCreate a shorter URL from a longer one (d'oh)\nRedirect to the longer link when the short link is requested.\n\nLet's call our service rust.ly (Hint, hint: the domain is still available at the time of writing...).\nFirst, we create a new Rust project:\n\ncargo new --bin rustly\n\nNext, we add Rocket to our Cargo.toml:\n\n[dependencies]\nrocket = "0.2.4"\nrocket_codegen = "0.2.4"\n\nWarning: Most likely you need to get the very newest Rocket version.\nOtherwise, you might get some entertaining error messages. Check out the newest\nversion from crates.io.\nSince Rocket requires cutting-edge Rust features, we need to use a recent nightly\nbuild. Rustup provides a simple way to switch between stable and nightly.\n\nrustup update && rustup override set nightly\nA first prototype\nNow we can start coding our little service.\nLet's first write a simple "hello world" skeleton to get started.\nPut this into src/main.rs:\n\n#![feature(plugin)]\n#![plugin(rocket_codegen)]\n\nextern crate rocket;\n\n#[get("/<id>")]\nfn lookup(id: &str) -> String {\n    format!("⏩ You requested {}. Wonderful!", id)\n}\n\n#[get("/<url>")]\nfn shorten(url: &str) -> String {\n    format!("💾 You shortened {}. Magnificient!", url)\n}\n\nfn main() {\n    rocket::ignite().mount("/", routes![lookup])\n                    .mount("/shorten", routes![shorten])\n                    .launch();\n}\n\nUnder the hood, Rocket is doing some magic to enable this nice syntax.\nMore specifically, we use the rocket_codegen crate for that.\n(It's implemented as a compiler plugin, which is also the reason why we need to use nightly Rust.)\nIn order to bring the rocket library into scope, we write extern crate rocket;.\nWe defined the two routes for our service. Both routes will respond to a GET request.\nThis is done by adding an attribute named get to a function.\nThe attribute can take additional arguments.\nIn our case, we define an id variable for the lookup endpoint and a url variable for the shorten endpoint. \nBoth variables are Unicode string slices. Since Rust has awesome Unicode support, we respond with a nice emoji just to show off. 🕶\nLastly, we need a main function which launches Rocket and mounts our two routes. This way, they become publicly available.\nIf you want to know even more about the in-depth details, I may refer you to the official Rocket documentation.\nLet's check if we're on the right track by running the application.\n\ncargo run\n\nAfter some compiling, you should get some lovely startup output from Rocket:\n\n🔧  Configured for development.\n    => address: localhost\n    => port: 8000\n    => log: normal\n    => workers: 8\n🛰  Mounting '/':\n    => GET /<hash>\n🛰  Mounting '/shorten':\n    => GET /shorten/<url>\n🚀  Rocket has launched from http://localhost:8000...\n\nSweet! Let's call our service.\n\n> curl localhost:8000/shorten/www.matthias-endler.de\n💾 You shortened www.matthias-endler.de. Magnificient!\n\n> curl localhost:8000/www.matthias-endler.de\n⏩ You requested www.matthias-endler.de. Wonderful!\n\nSo far so good.\nData storage and lookup\nWe need to keep the shortened URLs over many requests... but how?\nIn a production scenario, we could use some NoSQL data store like Redis for that.\nSince the goal is to play with Rocket and learn some Rust, we will simply use an\nin-memory store.\nRocket has a feature called managed state.\nIn our case, we want to manage a repository of URLs.\nFirst, let's create a file named src/repository.rs:\n\nuse std::collections::HashMap;\nuse shortener::Shortener;\n\npub struct Repository {\n    urls: HashMap<String, String>,\n    shortener: Shortener,\n}\n\nimpl Repository {\n    pub fn new() -> Repository {\n        Repository {\n            urls: HashMap::new(),\n            shortener: Shortener::new(),\n        }\n    }\n\n    pub fn store(&mut self, url: &str) -> String {\n        let id = self.shortener.next_id();\n        self.urls.insert(id.to_string(), url.to_string());\n        id\n    }\n\n    pub fn lookup(&self, id: &str) -> Option<&String> {\n        self.urls.get(id)\n    }\n}\n\nWithin this module we first import the HashMap implementation from the standard library.\nWe also include shortener::Shortener;, which will help us to shorten the URLs in the next step. Don't worry too much about that for now.\nBy convention, we implement a new() method to create a new Repository struct with an empty HashMap and a new Shortener. Additionally, we have two methods, store and lookup. \nstore takes a URL and writes it to our in-memory HashMap storage. It uses our yet to be defined shortener to create a unique id. It returns the shortened ID for the entry.\nlookup gets a given ID from the storage and returns it as an Option. If the ID is found, the return value will be Some(url), if there is no match it will return None. \nNote that we convert the string slices (&str) to String using the to_string() method. This way we don't need to deal with lifetimes. As a beginner, don't think too hard about them.\nAdvanced remarks (can safely be skipped)\nA seasoned (Rust) Programmer might do a few things differently here. Did you notice the tight coupling between the repository and the shortener? In a production system, Repository and Shortener might simply be concrete implementations of traits (which are a bit like interfaces in other languages, but more powerful). For example, Repository would implement Cache trait:\n\ntrait Cache {\n    // Store an entry and return an ID\n    fn store(&mut self, data: &str) -> String;\n    // Look up a previously stored entry\n    fn lookup(&self, id: &str) -> Option<&String>;\n}\n\nThis way we get a clear interface, and we can easily switch to a different implementation (e.g. a RedisCache). Also, we could have a MockRepository to simplify testing. Same for Shortener.\nYou might also want to use the Into trait to support both, &str and String as a parameter of store:\n\npub fn store<T: Into<String>>(&mut self, url: T) -> String {\n\t\tlet id = self.shortener.shorten(url);\n\t\tself.urls.insert(id.to_owned(), url.into());\n\t\tid\n}\n\nIf you're curious about this, read this article from Herman J. Radtke III.\nFor now, let's keep it simple.\nActually shortening URLs\nLet's implement the URL shortener itself.\nYou might be surprised how much was written about URL shortening all over the web.\nOne common way is to create short urls using base 62 conversion.\nAfter looking around some more, I found this sweet crate called harsh, which perfectly fits the bill.\nTo use harsh, we add it to the dependency section of our Cargo.toml:\n\nharsh = "0.1.2"\n\nNext, we add the crate to the top of to our main.rs:\n\nextern crate harsh;\n\nLet's create a new file named src/shortener.rs and write the following:\n\nuse harsh::{Harsh, HarshBuilder};\n\npub struct Shortener {\n    id: u64,\n    generator: Harsh,\n}\n\nimpl Shortener {\n    pub fn new() -> Shortener {\n        let harsh = HarshBuilder::new().init().unwrap();\n        Shortener {\n            id: 0,\n            generator: harsh,\n        }\n    }\n\n    pub fn next_id(&mut self) -> String {\n        let hashed = self.generator.encode(&[self.id]).unwrap();\n        self.id += 1;\n        hashed\n    }\n}\n\nWith use harsh::{Harsh, HarshBuilder}; we bring the required structs into scope. Then we define our own Shortener struct, which wraps Harsh. It has two fields: id stores the next id for shortening. (Since there are no negative ids, we use an unsigned integer for that.) The other field is the generator itself, for which we use Harsh. \nUsing the HarshBuilder you can do a lot of fancy stuff, like setting a custom alphabet for the ids. For more info, check out the official docs.\nWith next_id we retrieve a new String id for our URLs.\nAs you can see, we don't pass the URL to next_id. That means we actually don't shorten anything. We merely create a short, unique ID. That's because most hashing algorithms produce fairly long URLs and having short URLs is the whole idea.\nWiring it up\nSo we are done with our shortener and the repository.\nWe need to adjust our src/main.rs again to use the two.\nThis is the point where it gets a little hairy.\nI have to admit that I struggled a bit here.\nMainly because I was not used to multi-threaded request handling. In Python or\nPHP you don't need to think about shared-mutable access.\nInitially I had the following code in my main.rs:\n\n#[get("/<url>")]\nfn store(repo: State<Repository>, url: &str) {\n    repo.store(url);\n}\n\nfn main() {\n    rocket::ignite().manage(Repository::new())\n                    .mount("/store", routes![store])\n                    .launch();\n}\n\nState is the built-in way to save data across requests in Rocket. Just tell it what belongs to your application state with manage() and Rocket will automatically inject it into the routes.\nBut the compiler did not like that:\n\nerror: cannot borrow immutable borrowed content as mutable\n  --> src/main.rs\n   |\n   |     repo.store(url);\n   |     ^^^^ cannot borrow as mutable\n\nWhat would happen if two requests wanted to modify our repository at the same time?\nRust prevented a race condition here!\nAdmittedly the error message could have been a bit more user-friendly, though.\nFortunately, Sergio Benitez helped me out on the Rocket IRC channel (thanks again!).\nThe solution was to put the repository behind a Mutex.\nHere is the full src/main.rs in its entirety:\n\n#![feature(plugin, custom_derive)]\n#![plugin(rocket_codegen)]\n\nextern crate rocket;\nextern crate harsh;\n\nuse std::sync::RwLock;\nuse rocket::State;\nuse rocket::request::Form;\nuse rocket::response::Redirect;\n\nmod repository;\nmod shortener;\nuse repository::Repository;\n\n#[derive(FromForm)]\nstruct Url {\n    url: String,\n}\n\n#[get("/<id>")]\nfn lookup(repo: State<RwLock<Repository>>, id: &str) -> Result<Redirect, &'static str> {\n    match repo.read().unwrap().lookup(id) {\n        Some(url) => Ok(Redirect::permanent(url)),\n        _ => Err("Requested ID was not found.")\n    }\n}\n\n#[post("/", data = "<url_form>")]\nfn shorten(repo: State<RwLock<Repository>>, url_form: Form<Url>) -> Result<String, String> {\n    let ref url = url_form.get().url;\n    let mut repo = repo.write().unwrap();\n    let id = repo.store(&url);\n    Ok(id.to_string())\n}\n\nfn main() {\n    rocket::ignite().manage(RwLock::new(Repository::new()))\n                    .mount("/", routes![lookup, shorten])\n                    .launch();\n}\n\nAs you can see we're using a std::sync::RwLock here, to protect our repository from shared mutable access. This type of lock allows any number of readers or at most one writer at the same time.\nIt makes our code a bit harder to read because whenever we want to access our repository, we need to call the read and write methods first.\nIn our lookup method, you can see that we are returning a Result type now. It has two cases: if we find an id in our repository, we return Ok(Redirect::permanent(url)), which will take care of the redirect. If we can't find the id, we return an Error.\nIn our shorten method, we switched from a get to a post request.\nThe advantage is, that we don't need to deal with URL encoding. We just create a struct Url and derive FromForm for it, which will handle the deserialization for us. Fancy!\nWe're done. Let's fire up the service again and try it out!\n\ncargo run\n\nIn a new window, we can now store our first URL:\n\ncurl --data "url=https://www.matthias-endler.de" http://localhost:8000/\n\nWe get some ID back that we can use to retrieve the URL again. In my case, this was gY.\nPoint your browser to http://localhost:8000/gY.\nYou should be redirected to my homepage.\nSummary\nRocket provides fantastic documentation and a great community.\nIt really feels like an idiomatic Rustlang web framework.\nI hope you had some fun while playing with Rocket.\nYou can find the full example code on Github.\n"
  },
  {
    "title": "The Essence of Information",
    "url": "https://endler.dev/2017/the-essence-of-information/",
    "body": "People look confused when I tell them about my passion for algorithms and data-structures.\nMost of them understand what a Programmer is doing, but not what Computer Science is good for.\nAnd even if they do, they think it has no practical relevance.\nLet me show you with a simple example, that applied Computer Science can be found everywhere.\nImagine a pile of socks that need to get sorted.\nNot exactly the most exciting pastime.\nYou've put off this task for so long, that it will inevitably take an hour to be done.\n\n    \n    \n        Yes, there is a game about sorting socks.\n        \n        It's called Sort the Socks and you can get it for free on the App Store.\n    \n\nConsidering your options, you decide to get some help.\nTogether with a friend you get to work. You finish in roughly half the time.\nA Computer Scientist might call this pile of socks a resource.\nYou and your friend get bluntly degraded to workers.\nBoth of you can work on the problem at the same time — or in parallel.\nThis is the gist of Parallel Computing.\nNow, some properties make sock-sorting a good fit for doing in parallel.\n\nThe work can be nicely split up. It takes about the same time for every worker to find a pair of socks.\nFinding a different pair is a completely separate task that can happen at the same time.\n\nThe more workers you assign to this task, the faster you're done.\n\n1 worker takes 60 minutes.\n2 workers take 30 minutes.\n\nHow long will 3 workers take? Right! Around 20 minutes. We could write down\na simple formula for this relationship:\n \nWell, that is not quite correct. We forgot to consider the overhead: When Mary\ntries to pick up a sock, Stephen might reach for the same.\nThey both smile and one of them picks another sock.\nIn computing, a worker might do the same. Well, not smiling but picking another\ntask. When lots of workers share resources, these situations occur quite\nfrequently. And resolving the situation always takes a little extra time. So we are a\nbit away from our optimal sorting speed because of that.\nBut it gets worse! Let's say you have 100 workers for 100 socks.\nIn the beginning, every worker might take one sock and try to find a match for\nit. Here's the problem: As soon as they pick up one sock each, there are no\nsocks left. All workers are in a waiting state. The sorting takes forever.\nThat's a deadlock, and it's one of the most frightening scenarios of parallel computing.\nIn this case, a simple solution is to put down the sock again and wait for some time until trying to get a new sock.\nAnother way out of the dilemma would be, to enforce some kind of "protocol" for sorting. \nThink of a protocol as a silent agreement between the workers on how to achieve a common goal.\nSo, in our case, each worker might only be responsible for one color of socks.\nWorker one takes the green socks, worker two the gray ones and so on.\nWith this simple trick, we can avoid a deadlock, because we work on completely\nseparate tasks.\nBut there's still a catch. What if there are only four green socks and 4000 gray socks?\nWorker one would get bored fairly quickly. He would sort the two pairs of socks in\nno time and then watch worker two sort the rest.\nThat's not really team spirit, is it?\nSplitting up the work like this makes most sense, if we can assume that we\nhave around the same number of socks for every color.\nThis way we achieve roughly the same workload for\neveryone.\nThe following histogram gives you an idea of what I mean:\n\nIn this case, we have about equally sized piles for each color. Looks\nlike a fair workload for every worker to me.\n\nIn the second case, we don't have an equal distribution. I don't want to sort the\ngray socks in this example. We need to think a little harder here.\nWhat can we do?\nMost of the time it helps to think of other ways to split up work.\nFor example, we could have two workers sort the big gray pile together. One\nsorts the large socks; the other one sorts the small ones. We run into another problem, though: Who decides what "large" and "small" means in this case?\nSo, instead of thinking too hard about a smarter approach, we decide to be\npragmatic here. Everyone just grabs an equally sized pile of socks — no\nmatter the color or the size — and gets\nto work.\nMost likely, there will be some remaining socks in each pile, which have no match.\nThat's fine. We just throw them all together, mix the socks, create new piles from\nthat, and sort them again. We do so until we're done.\nWe call that a task queue. It has two advantages: First, you don't need any additional agreements between the workers and second, it scales reasonably\nwell with the number of workers without thinking too hard about the problem\ndomain.\nThe tricky part about distributed systems is, that seemingly straightforward solutions can fail\nmiserably in practice.\nWhat if our small piles look like this?\n \nThe number of pairs in each pile is... sobering.\nWhat we could do is run a very quick presorting step to increase the number of matches. Or maybe you come up with an even better idea?\nThe cool thing is, once you have found a faster approach, it works for similar tasks, too.\nProblems like this have their roots in Computer Science, and they can be found everywhere.\nPersonally, I don't like the term Computer Science too much. I prefer\nthe German term "Informatik", which I would roughly translate as "Information Science".\nBecause the real essence of what we're doing here is to find a general way to solve a\nwhole class of problems. We think of the nature of objects and their properties.\nWe don't sort socks; we try to answer the fundamental questions of information. Maybe now you can understand why I'm so passionate about this subject.\nOh, and here's a related post about why I love programming.\n"
  },
  {
    "title": "Why I Love Programming",
    "url": "https://endler.dev/2017/why-i-love-programming/",
    "body": "Programming has many faces. It is the science of structured thinking.\nIt is the art of eloquent expression.\nIt teaches you to be humble when you look at other peoples' fascinating work. \nMost of all, it teaches you a lot about yourself.\nWhile the syntax may change, the concepts will not.\n\nThis post is split into two parts.\nIn the first part, I will talk about the joy of programming.\nThe second part will deal with the notion of being a professional programmer.\nIf you're not sure yet whether you want to learn how to program, this article is for you.\nAutomating stuff gives you superhero strengths\nBeing able to program is infinitely rewarding. You can help your sister sort a\nthousand pictures in a few seconds. You write a little backup\nscript for your grandma. The possibilities are endless.\nCoding is fun!\nCoding something is more fun than using it. It's even better than playing games.\nWhy? Learn how to program a computer and get the best games for free — your own.\nYou're in total control. It's your idea, your logic, even your laws of physics.\nIt's like building a house but without paying anything for the\nbuilding materials. You can build a mansion for free.\nSharing is fun, too!\nTo get new inspiration for your next project, read the programs of others.\nThis will give you an idea of how they think and how they solve problems.\nMany great programmers share their best code with you.\nYou can do the same and share your project - or just the prettiest parts of it - with other programmers.\nWatching somebody else use your work is one\nof the most satisfying things you will ever experience.\nIt's very fulfilling to see your tool serve a purpose it wasn't built for.\nElegant, creative solutions\nIt's very appealing to work so hard on your vision that everything unnecessary peels off.\nAll these little ideas and fundamental insights suddenly fall into place.\nWhat's remaining is the distilled truth, the result of an ambitious but rewarding thought process\nand when you write it down as a program you can see all the little pieces working together.\nThis makes it so gratifying to figure stuff out on your own.\nProgramming is about understanding a problem so thoroughly, that you can teach a\npiece of metal how to solve it.\nEven the way your program is structured can be a piece of art.\nIt can be concise, witty and fast all at the same time.\nTalk to a machine\nIt's fascinating that something is understood by machines and humans using the same language.\nI'm baffled when I realize that these circuits can actually "understand" and interpret words - in a way.\nStanding on the shoulders of giants\nTalking to other programmers and watching them work is a fascinating inspiration. \nThe very system you are using to read this text relies on their work.\nEven if you're far apart, you can study their work on Open Source projects online.\nBut if you get a chance, watch them giving talks at conferences and meet them at local user groups.\nBecoming part of a community is gratifying.\nTo exchange ideas and to collaborate on projects helps you push your boundaries and learn something new every day.\nHave fun, forget the rest\nThe machine is agnostic to your skin color. It doesn't matter if you're a twelve-year-old girl or a lecturer at University.\nIf you keep making the same mistake for ten hours straight, your computer won't scream at you. It won't punish you. It will happily await your commands. Also, the hurdles of entry are pretty low. An old computer is enough; even pen and paper and a book will suffice to work on cool programming ideas.\nGet started!\nYou choose your own projects; nobody else.\nDon't let anybody tell you that you're not smart enough for this stuff. Ever.\nEach program is a wonderful journey so join us and code the world around you.\n"
  },
  {
    "title": "Tools",
    "url": "https://endler.dev/2011/tools/",
    "body": "For as long as I can think, religious flamewars have infected computer science.\nHaving arguments about technical topics can be healthy, but flamewars are not. I'm sick of it.\nI'm fed up with people telling me that their work environment is oh-so better,\nfaster and so on. That's fine, but it doesn't matter. Your equipment only plays a supporting role. You don't even need\na computer to do programming. Donald Knuth wrote algorithms on a\nnotepad. Alan Turing wrote the first chess computer on a piece of\npaper. And it worked. Beat that!\nFor an average user, the next best system is probably good enough. Just a few bucks and you get an excellent piece of hardware which is completely sufficient to surf the web, chat, archive photos, write documents, listen to music and watch movies. You can do that with a Pentium IV, 256 MB RAM and any recent Operating System (you will likely get that one for free). Heck, you can use your old Commodore for most of that. Computers have been mature and reliable enough to do all that for ages. There's no need to upgrade your system for Farmville, just like there's no reason to buy a new car if the old one works perfectly fine. When it comes to software, many of us still use Office 2000 or Photoshop 8 or VisiCalc without feeling the urge to upgrade.\nProfessionals find themselves in a similar situation. Well, maybe we invest a bit more money, but still, our hardware is incredibly cheap compared to our salary (hopefully). Nothing is perfect, but most of the time it's good enough. That compiler you were using a decade ago? Still does the job. We are still using slightly modified descendants of programming languages from computing stone-age. Even if you're doing numerical computing for NASA, your primary work environment is a black box running a text editor or an IDE.\nI don't care what you are using to get things done. Find an environment that suits your needs and be happy with it. Maybe you use Emacs on a Lemote Yeelong netbook (hello Richard Stallman) or Vim on your workstation. It's the same thing: A text editor running on a piece of metal.\nYou're not a worse programmer for using Nano, ed or TextMate. Notepad works just fine, too. It loads files, saves files and lets you edit them in between. That's a hell lot more functionality than Bill Gates and Paul Allen had when they wrote a BASIC interpreter for the Altair. If you find something you're happy with, just stick with it but don't start arguing. It isn't worth your time.\nDon't feed the trolls. When it comes to software, don't fall into the old FreeBSD vs. Linux vs. Windows vs. mum cliche. Instead, talk about your code. Let's look at your problem-solving skills. Let's be pragmatic here.\n\nTalk is cheap. Show me the code. - Linus Torvalds\n\nI don't care which programming language you are using. Java? Fine. Visual Basic? Great! Scala, Cobol, PHP, C++? All fine. Write in Assembler or lolcode. Don't moan about the fact that language X is missing feature Y. Write a library or use something different. Stop saying JavaScript is a toy language. It just doesn't fit your needs. Instead, show me your Lisp adventure game. Write an interpreter for Brainfuck. Do something. Move things.\nConcerning PHP, nir wrote on Hackernews:\n\nAny idiot can write a snarky comment about PHP. Very few get to write code that has anywhere near the impact it had.\n\nWill you fall off your chair when I admit that I like the PHP syntax?\nOK, it has its rough edges (do we really need the $ sign?) but what's\nmore important is how much I can get done with it. PHP was my long time\ngo-to language for off the hook, one time scripts. It looks a bit ugly\nbut it runs on any server and comes with an enormous amount of built-in\nfunctionality. It's great for rapid prototyping and gluing things together.\nIn fact, when you write a piece of software, what you should strive for is to produce quite good software and what you really need to accomplish is good enough software to make your users happy.\nZed A. Shaw puts it quite nicely in the afterword to Learn Python the hard way\n\nI have been programming for a very long time. So long that it is incredibly boring to me. At the time that I wrote this book I knew about 20 programming languages and could learn new ones in about a day to a week depending on how weird they were. Eventually though this just became boring and couldn't hold my interest. What I discovered after this journey of learning was that the languages didn't matter, it was what you did with them. Actually, I always knew that, but I'd get distracted by the languages and forget it periodically. The programming language you learn and use does not matter. Do not get sucked into the religion surrounding programming languages as that will only blind you to their real purpose of being your tool for doing interesting things.\n\nDon't get emotional for any tool you use. An iPhone - I'm sorry to disappoint you -  is just a phone. No magic. No "think different". "But it's evil!", the ether says, "it's not open source". Well, Android just exists because Google needed to rapidly develop a mobile platform. It's simply part of their business. There is no moral behind that. Google is a yet another company just like Microsoft or Apple.\nMy MacBook serves me as a solid tool, but if something "better" comes around, I will happily kick it out. I've ditched Firefox after five years just because Chrome is faster and I will get rid of Chrome when I find a worthy successor.\nVim is quite good in my opinion but if there's a faster way to do things I'm not afraid to dump it. Instead get your hands dirty and fix the problems or craft something new.\n"
  },
  {
    "title": "Are you a Programmer?",
    "url": "https://endler.dev/2011/are-you-a-programmer/",
    "body": "My geography teacher once told the story of her first lecture at University.\nAs an introduction, her professor asked the class to draw\na map of Germany without any help and as accurate as possible. To her surprise, she was not\nable to fill the map with much detail. Even the shape of the country was a bit vague.\nShe had seen thousands of images of Germany (her mother country) but\nwasn't able to reproduce it from her blurry memory. She would have to look it up.\nDoesn't this sound familiar? We rely on machines to manage large portions\nof our knowledge. There's hard work involved to learn something by heart.\nHere is a similar test for programmers:\n\nUsing a programming language of your choice, write a correct sorting\nalgorithm with an average runtime complexity of O(n*log n) (Heapsort,\nQuicksort, Bucketsort, you name it) on a piece of paper without the help of any\nexternal tools.\n\nAnd by correct I mean it must be free of bugs without any modifications when you type it in.\nYou would be surprised by the large percentage of professional software\nengineers who can't pull this off.\nSome might argue that knowledge about details of programming language\nsyntax is unimportant: "Why learn all the little nitpicks when you know\nhow to use a search engine? Why start with a clean slate when you can easily\ncopy, paste and modify an example from a tutorial?\nEvery few years/months I have to completely relearn the syntax for a different language anyway."\nBut that is a myth. If you know only\none programming language really well - even if it is something\noutdated like Fortran or COBOL - you could easily earn a fortune with\nthat knowledge. Suppose you started with C in 1975. You could still\nuse the same syntax today - almost four decades later.\nSame for text editors. Emacs and Vim are both decades\nold. They are battle-hardened. I don't care which one you prefer, but you\nwill spend a large part of your life with your tools so invest the time to master them.\nAs a side note, it appears that very few people strive for perfection in anything they do.\nThey happily settle for "good enough". This can have many different reasons, and I'm not\nblaming anybody for not doing his homework but maybe I'm not alone with\nthat observation.\nIf you don't know how to use your tools without a manual, you are a lousy craftsman.\nIf you need a dictionary to write a simple letter, you will have a hard\ntime becoming a writer because it would already be challenging for you to form elegant, fluent\nsentences -- let alone engaging and original stories.\nI don't want to read these books.\nWhat makes a programmer?\n\nShe has at least one programming language she knows inside out.\nShe can implement standard algorithms (i.e. for sorting, searching)\nand data-structures (i.e. trees, linked lists) which are robust and\nreasonably fast on the fly.\nShe has at least a basic understanding of complexity theory and\nprogramming concepts like recursion and pointers.\n\nBut, to be a good programmer, you should\n\nBe able to code in at least two fundamentally different programming\nparadigms (i.e. declarative, functional).\nHave experience with big software architectures.\nBe familiar with your programming environment like the operating system and a sophisticated text editor of your choice. Preferably one, that is\neasily extendable.\n\nAnd that is just the tip of the iceberg.\n"There's too much to learn!", I hear some of you say.\nStart slowly.\nYou need only three commands to start with Vim: i, ESC, :wq.\nThat's enough for day one.\nI realize that most of these essentials won't be taught during lectures.\nYou have to learn a vast portion on your own.\nBut let's face it: If you don't know this stuff, you are not a programmer, you're a freshman.\n"
  },
  {
    "title": "On Hard Work",
    "url": "https://endler.dev/2011/on-hard-work/",
    "body": "Great people get shaped by their achievements\n\nThere's Thomas Edison who developed countless prototypes before selling a single light bulb.\nThe unemployed Joanne K. Rowling writing Harry Potter in a Cafe while caring for her child.\nSteve Wozniak creating the first personal computer in his spare time while working at HP.\n\nWhat do they have in common?\nThey all lived through frustration and contempt but still reached their goals, even though the chances for success were\nlow. These people are stemming their strong will from an intrinsic curiosity.\nDedication\nSure, I love what I do. I want to be a programmer for the rest of my life, but sometimes it seems simply too hard to finish a project.\nI get scared by the big picture and fear that I won't finish on time. What I need is a different mindset.\nDhanji R. Prasanna, a former Google Wave team member made this observation\n\nAnd this is the essential broader point--as a programmer you must have a series of wins, every single day. It is the Deus Ex Machina of hacker success. It is what makes you eager for the next feature, and the next after that.\n\nWhile Google Wave has not been commercially successful, it sure was a\ntechnical breakthrough -- and it was a drag to push it out into public.\nWe always have to see our goal right in front of us, as we take a billion baby steps to reach it.\nThis is true for any profession. Winners never give up.\nDirection\nToday it is easier to accomplish something meaningful than ever before.\nIf you are reading this, you have access to a powerful instrument -- a\ncomputer with an Internet connection. We live in a time where a single\nperson can accomplish miracles without hard physical labor.\nA time where billions of people can grow a business from their desk, get famous in minutes,\npublish books in seconds and have instant access to large amounts of\ndata. The most potent development over the last 100\nyears has been the reduction of communication costs. Transferring a bit of\ninformation to the other end of the world is virtually free and takes\nfractions of a second. While proper education was a privilege of a lucky few\nwell into the 20th century, learning new things is now mostly a question of\nwill.\nNevertheless, learning is still a tedious task,\nrequiring patience and determination.\nAs the amount of information has increased, so have the ways of distraction.\nLosing focus is just a click away.\nDevotion\nEverybody can start something. Few will finish anything.\nThat's because getting things done is hard, even if you love what\nyou're doing. (Watch the beginnings of There Will Be Blood and Primer for a\ndefinition of hard work.)\nNo matter what they tell you, achieving anything sustainable means hustling. It means making\nsacrifices. It means pushing through.\nIt means selling something even though it isn't perfect. Your beautiful project might turn into an ugly groundhog in\nthe end. Put makeup on it and get it out the door.\nOn a report about Quake's 3D-Engine, developer Michael Abrash says:\n\nBy the end of a project, the design is carved in stone, and most of the work involves fixing bugs, or trying to figure out how to shoehorn in yet another feature that was never planned for in the original design.  All that is a lot less fun than starting a project, and often very hard work--but it has to be done before the project can ship. As a former manager of mine liked to say, "After you finish the first 90% of a project, you have to finish the other 90%."  It's that second 90% that's the key to success.\n\n\nA lot of programmers get to that second 90%, get tired and bored and frustrated, and change jobs, or lose focus, or find excuses to procrastinate.  There are a million ways not to finish a project, but there's only one way to finish:  Put your head down and grind it out until it's done.  Do that, and I promise you the programming world will be yours.\n\nThat last part has influenced me a lot.\nThe dedication, the urgency to reach your aims must come from within you.\nIt's your raw inner voice speaking -- don't let it fade away.\nAnd when you are close to giving up, stop thinking so hard. Just try to\npush forward and make a tiny step in the right direction.\nYou can't lose.\n"
  },
  {
    "title": "Overkill – Java as a First Programming Language",
    "url": "https://endler.dev/2010/overkill-java-as-a-first-programming-language/",
    "body": "I recently talked to a student in my neighborhood about his first programming\nexperiences. They started learning Java at school, and it soon turned out to be\nhorrible.\nA lot of us learned to code in languages like BASIC or Pascal. There was no\nobject orientation, no sophisticated file I/O and almost no\nmodularization... and it was great. In BASIC you could just write\n\nPRINT "HELLO WORLD"\n\nand you were done. This was actually a running program solving a basic and\nreoccurring problem: Output some text on a screen.\nIf you wanted to do the same thing in Java you just write:\n\npublic class Main { \n  public static void main (String[] args) {\n    System.out.println("Hello, world!"); \n  }\n}\n\nDo you see how much knowledge about programming you must have to achieve the\neasiest task one could think of? Describing the program to a novice programmer\nmay sound like this:\n\nCreate a Main class containing a main-method returning void expecting a\nstring array as a single argument using the println method of the out object of\nclass PrintStream passing your text as a single argument.\n\n— please just don't forget your brackets. This way your first programming hours are guaranteed to\nbe great fun.\nOK. So what are the alternatives? I admit that nobody wants to write BASIC\nanymore because of its lack of a sophisticated standard library for graphics\n(Java doesn't have one either) and its weak scalability. The language has to\nbe clean and straightforward. It should be fast enough for numerical tasks but not as\nwordy as the rigid C-type bracket languages (sorry C++ guys). It should have a\nsmooth learning curve and provide direct feedback (compiled languages often\nsuck at that point). It should encourage clean code and reward best practices.\nOne language that provides all that is Python.\nAnd Python has even more: hundreds of libraries that help you with almost\neverything, good integration into common IDEs (PyDev in Eclipse, IDLE...), a\nprecise and elegant syntax.\nHere is our program from above written in Python:\n\nprint("Hello World")\n\nThere's no need to know about object orientation, scopes and function\narguments at this point. No householding or book-keeping. Yes, it's an\ninterpreted language, but that's not a deal breaker for beginners.\nIf you aren't convinced yet, printing and formatting text output in Java is\nrelatively easy for an advanced programmer but the gruesome stuff begins\nwith file input:\n\nimport java.io.BufferedReader; \nimport java.io.FileNotFoundException; \nimport java.io.FileReader; \nimport java.io.IOException; \n \npublic class fileIO { \n    public static void main(String[] args) { \n        String filename = "test.txt", line; \n        try { \n            BufferedReader myFile = \n                new BufferedReader(new FileReader(filename)); \n            \n            while ( ( line = myFile.readLine()) != null) { \n                System.out.println(line);\n            } \n        } catch (FileNotFoundException e) { \n            e.printStackTrace(); \n        } catch (IOException e) { \n            e.printStackTrace(); \n        } \n    } \n} \n\nI hear you say: "Dude, file I/O is pretty complex. It's just the way it is".\nThat's true... internally . But a beginner should get an easy interface. Python\nshows how it's done:\n\nfile = open("test.txt")\ntext = file.read()\nprint(text);\n\nThe code goes hand in hand with the natural understanding of how the process\nworks: "The computer opens a file, reads it and prints it". Even a five-year-old kid can understand that. Nobody would start to explain: "Before you can\nread a file you need a BufferedReader that works on a FileReader..." even if\nthis is precisely how it works internally. You want to explain the big picture\nat first. The elementary principles of teaching a computer how to do useful\nstuff. Otherwise, you will start frustrating beginners and fool them into\nthinking that they are not bright enough for programming. Programming is fun\nand starting with it is the most crucial step. So don't spoil that\nexperience with layers of unneeded abstraction.\n"
  },
  {
    "title": "Howto Sort a Vector or a List in C++ using STL",
    "url": "https://endler.dev/2010/howto-sort-a-vector-or-a-list-in-c-using-stl/",
    "body": "A little code snippet that people need very often.\n\n\n/*\n* Howto sort a vector or a list in C++ using STL\n*/\n\n#include <algorithm>  // Needed for sort() method\n#include <vector>     // STL vector class\n#include <list>       // STL list class\n#include <iostream>   // Needed for cout,endl\n\nusing namespace std;  // Save us some typing\n\n/*\n* This is a comparison function. It can be used to tell sort()\n* how to order the elements in our container (the vector or list).\n* You can write a comparator for every data type (i.e. double, string...).\n*/\nbool comp(const int& num1, const int& num2) {\n    return num1 > num2;\n}\n\nint main() {\n    // SORTING WITH VECTORS //\n\n    // A vector containing integers\n    vector<int> v;\n\n    // Insert some values\n    v.push_back(5);\n    v.push_back(12);\n    v.push_back(1);\n\n    // The generic STL sort function uses three parameters:\n    // \n    // v.begin()  Iterator pointing at the _beginning_ of the container\n    // v.end()    Iterator pointing at the _end_ of it\n    // comp       [Optional] A comparison function (see above)\n    // \n    // The above mentioned iterators must be random access iterators because\n    // sort() takes advantage of clever tricks that require direct access to\n    // all elements of the vector. This makes it really fast.\n    // (Currently introsort is used with O(n*log n) even in worst case).\n\n    sort(v.begin(), v.end(), comp);\n\n    cout << "Vector: ";\n\n    // Iterate over vector elements\n    vector<int>::iterator vIt;\n    for (vIt = v.begin(); vIt != v.end(); vIt++) {\n        // Print current element to standard output\n        cout << *vIt << " ";\n    }\n    cout << endl;\n\n    // SORTING WITH LISTS //\n    // A list containing integers\n    list<int> l;\n\n    // Insert some values\n    l.push_back(5);\n    l.push_back(12);\n    l.push_back(1);\n\n    // Here is the major difference between vectors and lists in general:\n    // Vectors offer fast random access to every element\n    // but inserting a new element at the beginning or in the middle is slow.\n    // On the other hand inserting into a list is fast but searching for\n    // a specific element is slow.\n    //\n    // Vectors behave much like an array, while lists only allow slow sequential access.\n    // Therefore we need a different function to sort all elements that does\n    // not need random access iterators.\n    //  \n    // comp  [Optional] A comparison function (see above)\n    //  \n    // Note that sort() is specific for the list and is implemented as a\n    // member function of list<>. This is feels more object oriented than the vector.\n    \n    l.sort(comp);\n\n    cout << "List: ";\n\n    // A pointer to a list element\n    list<int>::iterator lIt;\n    for (lIt = l.begin(); lIt != l.end(); lIt++) {\n        cout << *lIt << " ";\n    }\n    cout << endl;\n\n    return 0;\n}\nCompilation and execution\nSave the above code inside a file, e.g. list_vector.cpp and compile it like so:\n\nclang++ list_vector.cpp\n\nTo run it, execute the resulting binary.\n\n./a.out\nProgram output\n\nVector: 12 5 1\nList: 12 5 1\n"
  },
  {
    "title": "Why I Love Text Files",
    "url": "https://endler.dev/2010/why-i-love-text-files/",
    "body": "Text files are the single most important way we can communicate with computers. It's no coincidence that they are also the most vital way to interact with other human beings. What we can achieve with text files is invaluable: Write it once and refer to it whenever you want to get the message across in the future. Write a program (it's just text), save it and let the machine execute it whenever you like. Write another text file which contains the rules for the execution of your program and the computer runs your application exactly as you specified (cron files do that on Unix).\nText files can be structured in any way you can imagine. Some flavours are JSON, Markdown and SVG. It's all just text. There exist a billion of programs and algorithms to access, modify and distribute text files. You can write them with Emacs, print them on a terminal, pipe them through sed and send them via email to a friend who publishes them on the web. Because text files are so important we have good support for them on any computing system. On Unix, everything is a file and HTML is just structured text. It's a simple and powerful tool to make a contribution to society that outlasts our lives. \nI have a single text file in my mac dock bar which is called TODO.txt. I open it every day, and after years of experimenting with different task management apps from simple command line tools to sophisticated online information storage systems, I always came back to plain text files. And the explanation is simple: If humanity will still be around a thousand years from now, chances are that plain text files are one of the very few file formats that will still be readable.\nThey are an incremental part of how we can modify our environment without even leaving our desk. They have no overhead and can contain a single thought or the complete knowledge of our species. Distributing textual information is so vital for us that we permanently develop faster distribution networks – the fastest by now being the internet.\nOn the web, you have instant access to a virtually endless amount of information and data distributed as plain text files. New web services made accessing the data even easier offering APIs and feeds. You can pull down the data from their servers and make statistics with a programming language of your choice. As you may have noticed, my affinity to text files partially comes from my programming background. As Matt Might correctly points out on his blog: \n\nThe continued dominance of the command line among experts is a testament to the power of linguistic abstraction: when it comes to computing, a word is worth a thousand pictures.\n\nWhenever you like a text on the web, just link to it and create a wonderful chain of ideas. Want to read it later or recommend it to a friend? Just share the text or print it on paper. The fact that we all take such things for granted is a testament for the power of text files and their importance for the information age.\nLinks\n\nUnix Text Processing (PDF)\n\n"
  },
  {
    "title": "Running Legacy Code",
    "url": "https://endler.dev/2009/running-legacy-code/",
    "body": "This short article deals with a severe problem in software development: bit rot.\nWhen switching to a new platform (for instance from Windows XP to Windows Vista/7), the programmers need to make sure that old bits of code run flawlessly. There are several ways to achieve this goal that will be discussed in the next paragraphs:\nPorting the code\nThis is generally considered a hard path to follow. For non-trivial legacy code-blocks, chances are high that they contain side-effects and hacks to make them work in different environments. Porting code means replacing parts of the program that use functions and methods that don't exist anymore with new ones which make use of the modern libraries  and routines of the new platform. The significant advantages are maintainable software and sometimes faster running programs. But it may be needed to hack the new platform libraries in order to preserve the whole functionality of an old application. When changing an algorithm inside legacy code, the ported version may become unstable. Thus there may be better ways of maintaining obsolete code today.\nEmulators\nEmulators work much the same like porting the code. You replace old function calls with new ones to make everything work again. However you don't alter the old codebase itself (because you may not have the source code available) but you create a new compatibility layer that "translates" the communication between the underlying operating system and software (our new platform) and our old software. Emulation can also be very fast and run stable for many years but writing an emulator can be even harder than porting the code because an educational guess may be needed to figure out how the program works internally. Additionally, the emulator itself may become obsolete in the future and might eventually  be replaced by a new one.\nVirtual machines\nDuring the last years, a new approach was gaining popularity. The idea is simple: Don't touch anything. Take the whole platform and copy it in order to run old software. The old software runs on top of the old operating system within a virtual machine that runs on the new platform. \nFrom a sane software developers view, this method is ridiculous. A lot of resources are wasted along the way. The system is busier switching contexts from an old platform to the new one and back than running the actual legacy program. However, with cheap and capable hardware everywhere this idea gets more and more interesting. As Steve Atwood coined it:\n\nAlways try to spend your way out of a performance problem first by throwing faster hardware at it.\n\nAnd he's right. The Microsoft developers did the same on their new NT 6.0 platform (Vista, Windows 7, Windows Server 2008...): Windows XP is running on a virtual machine. This way everything behaves just like one would run the software on the old system. And by optimizing the performance bottlenecks (input/output, context switches), one gets a fast and stable, easy to maintain product.\nEvery method has its major advantages and disadvantages. It's on the developer to select the appropriate strategy.\n"
  }
]