Skip to content

Files

Latest commit

 

History

History

LearningGo-AnIdiomaticApproachToRealWorldGoProgramming

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
ch10_ConcurrencyInGo
ch10_ConcurrencyInGo
 
 
README.md
README.md
 
 

README.md

While reading and doing exercises, etc of this book, I also look at quite a few other resources, as it helps to solidify concepts for me. When I use other resources, I specify what they are.

Chapter 1

The Go Workspace

"Since the introduction of Go in 2009, there have been several changes in how Go developers organize their code and their dependencies. Because of this churn, there’s lots of conflicting advice, and most of it is obsolete."

"Go still expects there to be a single workspace for third-party Go tools installed via go install"

"By default, this workspace is located in $HOME/go, with source code for these tools stored in $HOME/go/src and the compiled binaries in $HOME/go/bin. You can use this default or specify a different workspace by setting the $GOPATH environment variable."

"Whether or not you use the default location, it’s a good idea to explicitly define GOPATH and to put the $GOPATH/bin directory in your executable path. Explicitly defining GOPATH makes it clear where your Go workspace is located and adding $GOPATH/bin to your executable path makes it easier to run third-party tools installed via go install"

If using asdf, the GOPATH is defined for you. For example if you are using golang 1.20 (as specified in your .tool-versions file), then your GOPATH will look like:

> go env GOPATH
GOPATH="/home/<you>/.asdf/installs/golang/1.20/packages"

If using zsh, there are are couple of tweaks that need to be made to the .zshrc file, including adding the GOPATH/bin to your system path, this is done with the following line in the .zshrc:
export PATH=$(go env GOPATH)/bin:$PATH

If using asdf, the following advice does not apply:

(If you are using zsh, add these lines to .zshrc instead):

export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

The go Command

go run and go build

  1. Builds the binary
  2. Executes the binary
  3. Deletes the binary after your program finishes

go run compiles your code into a binary and saves the file to a temporary directory, by default the systems default temporary directory. You can override this directory by modifying the $GOTMPDIR environment variable. To view all of your go environment variables, just run: go env.

If you want to locate the exact temporary directory where Go is storing these artifacts on your system, you can use the os.TempDir() function in your Go code to retrieve the path to the system's temporary directory. Here's an example:

package main

import (
	"fmt"
	"os"
)

func main() {
	tempDir := os.TempDir()
	fmt.Printf("Temporary directory: %s\n", tempDir)
}

go build

go build <packages>
or
go build -o <custom-binary-name> <packages>

Getting Third-Party Go Tools

Pg 5 of "Learning Go an Idiomatic approach..." says:
The go install command takes an argument, which is the location of the source code repository of the project you want to install, followed by an @ and the version of the tool you want (if you just want to get the latest version, use @latest). It then downloads, compiles, and installs the tool into your $GOPATH/bin directory.
To view all of your go environment variables, just run: go env.

It looks like the source is downloaded from a proxy (as defined by the value of the $GOPROXY environment variable) if it exists into:

  1. $GOPATH/pkg/mod/cache/download/github.com/rakyll/hey/@v/v0.1.4.zip as an archive file
  2. $GOPATH/pkg/mod/github.com/rakyll/hey@v0.1.4/ as the decompressed archive (the hydrated source)
  3. and then compiled into the $GOPATH/bin directory

Formatting Your Code

Also discussed in Effective Go: Formatting.

"Go programs use tabs to indent, and it is a syntax error if the opening brace is not on the same line as the declaration or command that begins the block."

go fmt "reformats your code to match the standard format"

VS Code with the official Golang plugin uses goimports which is better.

Like JavaScript, the Go lexer uses semicolon insertion (ASI). Unlike JavaScript, Go developers never add semicolons other than places such as for loop clauses, to separate the initialiser, condition, and continuation elements. They are also necessary to separate multiple statements on a line, should you write code that way.

Linting and Vetting

golint is deprecated, drop-in replacement is https://github.com/mgechev/revive .
Use go vet instead.
golangci-lint runs multiple (configurable which) tools in parallel

The recommendation is: start off using `go vet`` as a required part of your automated build process and revive as part of your code review process (since revive might have false positives and false negatives, you can’t require your team to fix every issue it reports). Once you are used to their recommendations, try out golangci-lint and tweak its settings until it works for your team.

Choose Your Tools

The Go wiki Go IDEs and Editors Discusses:

  • Visual Studio Code
    Uses:
    • the Delve debugger
    • gopls, the official Go language server
  • GoLand IDE from JetBrains
  • The Go Playground

Makefiles

Go developers typically use Makefiles.

Staying Up to Date

If using asdf, just follow the asdf documentation. You can have as many versions of Go installed as you like, and specific versions can be run from specific directories and below simply by adding a .tool-versions file (containing the binary's name and version) in the directory of your choosing.

Chapter 2. Primitive Types and Declarations

Built-in Types

The Zero Value

Go assigns a default zero value to any variable that is declared but not assigned a value.

Literals

The following lower or upper case prefixes are used to indicate different bases:

  • 0b: binary (base two)
  • 0o: octal (base eight). 0 can also be used, but obviously it can cause confusion
  • 0x: hexadecimal (base sixteen)

Underscores _ can be used to separate parts of a number, for example you can separate thousands in base ten (1_234). The underscores have no effect on the value of the number. Underscores can't be at the beginning, end, or next to each other in a number.

  • 6.03e23 is equivalent to 6.03 x 1023 in decimal notation
  • 0x6.03p23 represents a hexadecimal floating-point number, equivalent to 6.03 × 223. The use of "p" in this context is specific to Go's syntax for hexadecimal floating-point literals

Single and double quotes are not interchangeable in Go.
Rune literals represent characters and are surrounded by single quotes.

Rune literals can be written as:

  • 'a': single unicode character
  • '\141': 8-bit octal number
  • '\x61': 8-bit hexadecimal number
  • '\u0061': 16-bit hexadecimal number
  • '\U00000061': 32-bit unicode number

The most useful backslash escape rune literals:

  • '\n': newline
  • '\t': tab
  • '\'': single quote
  • '\"': double quote
  • '\\': backslash

"`Practically speaking, use base ten to represent your number literals and, unless the context makes your code clearer, try to avoid using any of the hexadecimal escapes for rune literals. Octal representations are rare, mostly used to represent POSIX permission flag values (such as 0o777 for rwxrwxrwx). Hexadecimal and binary are sometimes used for bit filters or networking and infrastructure applications."

The interpreted string literal: "Greetings and\n\"Salutations\"" ends up being:

Greetings and
"Salutations"

"If you need to include backslashes, double quotes, or newlines in your string, use a raw string literal. These are delimited with backquotes (`) and can contain any literal character except a backquote. When using a raw string literal, we write our multiline greeting like so:"

`Greetings and
"Salutations"`

Numeric Types

Integer types

There are signed and unsigned integers from one to four bytes:

Type name Alias Value range
int8 -128 to 127
int16 -32768 to 32767
int32 int (most 32-bit CPU), rune -2147483648 to 2147483647
int64 int (most 64-bit CPU) -9223372036854775808 to 9223372036854775807
uint8 byte 0 to 255
uint16 0 to 65536
uint32 uint (most 32-bit CPU) 0 to 4294967295
uint64 uint (most 64-bit CPU) 0 to 18446744073709551615

int, on a 32-bit CPU is a 32-bit signed integer like an int32. On most 64-bit CPUs, int is a 64-bit signed integer like an int64. Because int isn’t consistent from platform to platform, it is a compile-time error to assign, compare, or perform mathematical operations between an int and an int32 or int64 without a type conversion.

Go supports 32-bit signed integers as int on CPU architectures: amd64p32, mips64p32, and mips64p32le.

rune (covered on pg 26) and uintptr (covered in ch 14) are special names for integer types also.

Go has all the usual bit-manipulation operators found in other languages.

Floating point types

The floating point types are:

  • float32
  • float64

Unless you have to be compatible with an existing format, use float64. Floating point literals have a default type of float64. In most cases you shouldn't use a floating point number at all. Go floating point numbers have a huge range, but they cannot store every value in that range; they store the nearest approximation. Because floats aren’t exact, they can only be used in situations where inexact values are acceptable or the rules of floating point are well understood.

If you have to use a float, you can use == and != to compare floats, but don’t do it. Due to the inexact nature of floats, two floating point values might not be equal when you think they should be. Instead, define a maximum allowed variance and see if the difference between two floats is less than that.

A Taste of Strings and Runes

Explicit Type Conversion

These are built-in conversion expressions, not functions as they look like.

Declaring Variables

var Versus :=

:= can be used to assign values to new and existing variables, as long as there is one new variable on the lefthand side of the :=. Beware of creating new shadow variables with := when you think you are reusing an existing variable. In situations like this, explicitly declare all of your new variables with var to make it clear which variables are new, and then use the assignment operator (=) to assign values to both new and old variables.

While var and := allow you to declare multiple variables on the same line, only use this style when assigning multiple values returned from a function or the comma ok idiom.

Using const

Also covered in:

Created at compile time, as opposed to variables at runtime.

const in Go is very limited. Constants in Go are a way to give names to literals. They can only hold values that the compiler can figure out at compile time.

Typed and Untyped Constants

Here’s what an untyped constant declaration looks like:

const x = 10

All of the following assignments are legal:

var y int = x
var z float64 = x
var d byte = x

Here’s what a typed constant declaration looks like:

const typedX int = 10

This constant can only be assigned directly to an int.

Unused Variables

Effective Go: Unused imports and variables discusses temporarily reading and storing to the _, this allows the compilation to succeed.

Naming Variables and Constants

Also discussed in The Go wiki CodeReviewComments#variable-names, which to an extent goes against what history has taught us about non-descriptive variable names. Bob Martin, Steve McConnell, other greats, even myself, have written about short, unmemorable names many times, favouring write-time convenience over read-time convenience is often a mistake. Use descriptive names rather than adding non-executable comments to explain what something does. Although I agree that short variable names are fine in some cases, such cases are mentioned in this Go wiki section.

Initialisms are covered in:

MixedCaps is also discussed in:

No snake_case, use PascalCase for exported identifiers, use camelCase for unexported identifiers.

Unicode characters are allowed for variable naming, but don't use them.

Chapter 3. Composite Types

Effective Go: Constructors and composite literals discusses composite literals.

Using [n] or [...] makes an array. Using [] makes a slice.

In arrays and slices you can create what's known as a sparse array or slice.

Array:

var x = [12]int{1, 5: 4, 6, 10: 100, 15}
Creates an array of 12 ints with the following values: [1, 0, 0, 0, 0, 4, 6, 0, 0, 0, 100, 15].

Slice:

var x = []int{1, 5: 4, 6, 10: 100, 15}
Creates a slice of 12 ints with the following values: [1, 0, 0, 0, 0, 4, 6, 0, 0, 0, 100, 15].

Arrays—Too Rigid to Use Directly

Also covered in:

== and != can be used for equality checking in arrays but not slices, deep equality checking also works.

"Go only has one-dimensional arrays and slices, but you can simulate multidimensional arrays or slices:"
var x [2][3]int
"This declares x to be an array of length 2 whose type is an array of ints of length 3."

"Go considers the size of the array to be part of the type of the array. This makes an array that’s declared to be [3]int a different type from an array that’s declared to be [4]int. This also means that you cannot use a variable to specify the size of an array, because types must be resolved at compile time, not at runtime."

"What’s more, you can’t use a type conversion to convert arrays of different sizes to identical types. Because you can’t convert arrays of different sizes into each other, you can’t write a function that works with arrays of any size and you can’t assign arrays of different sizes to the same variable."

Slices

Also covered in:

"The length is not part of the type for a slice. This removes the limitations of arrays."

Inspiration for the following example was from Effective Go: Two-dimensional slices:

package main

import (
	"fmt"
)

func main() {
	var slice [][][]byte
	slice = [][][]byte{
		{
			{'c'},
			{'a', 't'},
		},
		{
			{'d'},
			{'o', 'g'},
		},
	}
	fmt.Println("Contents of the three-dimensional slice:")
	for i := 0; i < len(slice); i++ {
		fmt.Printf("Level %d:\n", i+1)
		for j := 0; j < len(slice[i]); j++ {
			fmt.Println(string(slice[i][j]))
		}
		fmt.Println()
	}
}

The Go wiki CodeReviewComments#declaring-empty-slices also discusses nil slices vs non-nil, and when to use which.

// Declares a nil slice:
var t []string

vs

//Declares a non-nill slice:
t := []string{}

Slices can not be compared with == or !=. "The only thing you can compare a slice with is nil".

"The reflect package contains a function called DeepEqual that can compare almost anything, including slices. It’s primarily intended for testing, but you could use it to compare slices if you needed to."

len

append

Also covered in:

Capacity

make

The built-in function make creates slices, maps, and channels only.

Effective Go: Allocation with make, has some low value examples.

If you use a variable to specify a capacity with make that’s smaller than the length, your program will panic at runtime.

Declaring Your Slice

Declaring a slice that might stay nil:

var data []int

Create a slice using an empty slice literal

var x = []int{}
// or
x := []int{}

If you have some starting values, or if a slice’s values aren’t going to change, then a slice literal is a good choice

data := []int{2, 4, 6, 8}

There are several permutations using make. Specifying nonzero length, zero length and nonzero capacity. Check the three possibilities in the book.

Slicing Slices

Another not-so-great part of the Go language...

"When you take a slice from a slice, you are not making a copy of the data. Instead, you now have two variables that are sharing memory."

// ...
x := []int{1, 2, 3, 4}
y := x[:2] // y == [1, 2]
z := x[1:] // z == [2, 3, 4]
x[1] = 20
// x now == [1, 20, 3, 4]
// y now == [1, 20]
// z now == [20, 3, 4]
y[0] = 10
// x now == [10, 20, 3, 4]
// y now == [10, 20]
// z now = [20, 3, 4]
z[1] = 30
// x now == [10, 20, 30, 4]
// y now == [10, 20]
// z == [20, 30, 4]
// ...

Changing x modified both y and z, while changes to y and z modified x.

Slicing slices gets extra confusing when combined with append:

x := []int{1, 2, 3, 4}
y := x[:2]                                  // y == [1, 2]
fmt.Println(len(x), len(y), cap(x), cap(y)) // 4, 2, 4, 4
y = append(y, 30)
// x now == [1, 2, 30, 4]
// y now == [1, 2, 30]

"To avoid complicated slice situations, you should either never use append with a sub‐slice or make sure that append doesn’t cause an overwrite by using a full slice expression."

"The full slice expression (x[:2:2], x[2:4:4]) includes a third part , which indicates the last position in the parent slice’s capacity that’s available for the subslice."

It's well worth reading and understanding "Example 3-7. Even more confusing slices" on pg 45.

Converting Arrays to Slices

"If you have an array, you can take a slice from it using a slice expression."

"be aware that taking a slice from an array has the same memory-sharing properties as taking a slice from a slice."

copy

Using the built-in copy function rectifies the above problems with copying the memory locations.

Strings and Runes and Bytes

Maps

Also covered in:

Chapter 4. Blocks, Shadows, and Control Structures

Blocks

This section should have discussed Redeclaration and reassignment, thankfully we have coverage in Effective Go: Redeclaration and reassignment, be sure to read it.

if

Also discussed in Effective Go: if

for, Four Ways

Also discussed in Effective Go: for

switch

Also covered in:

Chapter 5. Functions

Declaring and Calling Functions

Multiple Return Values

Also covered in:

Named Return Values

Also covered in:

defer

Also covered in:

Blank Returns—Never Use These!

Also called naked returns, and covered in The Go Wiki CodeReviewComments#naked-returns. Doesn't say anything about not using them though.

Go is Call By Value

The Go Wiki CodeReviewComments#pass-values says: "Don't pass pointers as function arguments just to save a few bytes. If a function refers to its argument x only as *x throughout, then the argument shouldn't be a pointer."

"Go for Javascript Developers" says: "In Go, there are value types, reference types, and pointers. References types are slices, maps, and channels. All the rest are value types, but have the ability "to be referenced" with pointers."

Because Slices in Go are reference types, a slice's value is a reference. Any modifications made to the slice within the function will affect the original slice in the calling function. However, when you append an element to a slice, a new underlying array is created if the capacity of the current array is exceeded, and the original slice's reference does not change. In the modSlice function, when you append 10 to s, it exceeds the capacity of the original array, and a new array is created to accommodate the additional element. The s slice inside the modSlice function now points to a new array, but the s slice in the main function remains unchanged because slices are passed by value (a copy of the slice header is passed) and not by reference.

Chapter 6. Pointers

A Quick Pointer Primer

"Learning Go an Idiomatic approach..." has an example (var x = new(int)) of the new built-in function, which is also covered in:

Chapter 7. Types, Methods, and Interfaces

Effective Go: Interface Names discusses how to name single method interfaces.

Methods

As Effective Go: Methods mentions:

  • "methods can be defined for any named type (except a pointer or an interface); the receiver does not have to be a struct."
  • "The rule about pointers vs. values for receivers is that value methods can be invoked on pointers and values, but pointer methods can only be invoked on pointers."

Pointer Receivers and Value Receivers

The following "100 Go Mistakes and How to Avoid Them" are also worth a read:

"do not write getter and setter methods for Go structs, unless you need them to meet an interface"

Also discussed in:

Use Embedding for Composition

"You can embed any type within a struct"
Effective Go: "Only interfaces can be embedded within interfaces"

A Quick Lesson on Interfaces

Also covered in:

Type Assertions and Type Switches

Also covered in:

  • Effective Go: Type switch has one example.
  • Pg 205, 7.10. Type Assertions of "The Go Programming Language"
  • Pg 210, 7.13. Type Switches of "The Go Programming Language"

Chapter 8. Errors

In-band errors are covered in the Go wiki CodeReviewComments#in-band-errors.

Errors Are Values

Error strings are:

panic and recover

Also covered in:

Chapter 9. Modules, Packages and Imports

Building Packages

The Go wiki CodeReviewComments#examples discusses adding examples.

Imports and Exports

A topic that is somewhat thin in the "Learning Go an Idiomatic approach..." book is printing, this has coverage in:

Effective Go: Unused imports and variables discusses temporarily reading and storing to the _, this allows the compilation to succeed.

Naming Packages

Also covered in:

Overriding a Package’s Name

The Go wiki:

Also covered in "100 Go Mistakes and How to Avoid Them" Ignoring package name collisions (#14)

Package Comments and godoc

Also covered in:

The init Function: Avoid if Possible

Also covered in the Go wiki CodeReviewComments#import-blank

Chapter 10. Concurrency in Go

The Go wiki CodeReviewComments#synchronous-functions discusses preferring synchronous functions. This of course means waiting/blocking.

Go has two styles of concurrent programming:

  • Goroutines and channels, which support communicating sequential processes or CSP. A go community saying: "Share memory by communicating; do not communicate by sharing memory". This style is covered in 8. Goroutines and Channels of "The Go Programming Language"
  • The more traditional model of shared memory multithreding. This style is covered in 9. Concurrency with Shared Variables of "The Go Programming Language"

When to Use Concurreny

"If you are not sure if concurrency will help, first write your code serially, and then write a benchmark to compare performance with a concurrent implementation."

Goroutines

Operating System schedules threads across CPU cores. A process can own multiple threads. a thread can own many goroutines (lightweight processes).

  • Pg 205 says "Goroutines are lightweight processes managed by the Go runtime" Just about every other source says that Goroutines are more like lightweight threads, including "The Go Programming Language" pg 218

The crux of this section is that goroutines are faster and lighter (in terms of resources) because they are managed by the Go runtime scheduler instead of the operating system (as in OS managed threads). The Go runtime scheduler is obviously closer to your code, and knows more about your code than the operating system.

I have a full example code for this.

Pg 218, 8.1. Goroutines of "The Go Programming Language" says "The go statement itself completes immediately:"

f() // call f(); wait for it to return
go f() // create a new goroutine that calls f(); don't wait

Channels

As per (https://go.dev/tour/concurrency/2) By default, sends and receives block until the other side is ready. This allows goroutines to synchronize without explicit locks or condition variables.

Being puzzled about channel size (#67) of "100 Go Mistakes and How to Avoid Them" says:

  • "Carefully decide on the right channel type to use, given a problem. Only unbuffered channels provide strong synchronization guarantees". Thus they are sometimes called "synchronous channels" as mentioned on pg 226 of "The Go Programming Language"
  • "You should have a good reason to specify a channel size other than one for buffered channels"

"Like maps, channels are reference types. When you pass a channel to a function, you are really passing a pointer to the channel"

8.4.2. Pipelines of "The Go Programming Language" is a pattern/technique which has a goroutine sending to a channel, that another goroutine receives. Pg 229: If the first goroutine closes the channel, after the receiving end of the channel has drained all messages, all subsequent receive operations will proceed without blocking but will yield a zero value, unless the receive operation checks the 2nd return value which is a boolean value, conventionally called ok, which is true for a successful receive and false for a receive on a closed and drained channel. Using a range loop provides the same behaviour in a more succinct manner.
Also discussed on pg 208, Closing a Channel of "Learning Go an Idiomatic approach...".

Pg 230 "You needn’t close every channel when you’ve finished with it. It’s only necessary to close a channel when it is important to tell the receiving goroutines that all data have been sent."

"A channel that the garbage collector determines to be unreachable will have its resources reclaimed whether or not it is closed.". This is also mentioned on pg 208, Closing a Channel of "Learning Go an Idiomatic approach...".

Pg 233, 8.4.4. Buffered Channels of "The Go Programming Language" "Novices are sometimes tempted to use buffered channels within a single goroutine as a queue, lured by their pleasingly simple syntax, but this is a mistake. Channels are deeply connected to goroutine scheduling, and without another goroutine receiving from the channel, a sender—and perhaps the whole program—risks becoming blocked forever. If all you need is a simple queue, make one using a slice.".

"Unlike garbage variables, leaked goroutines are not automatically collected, so it is important to make sure that goroutines terminate themselves when no longer needed.". The sub-section "Always Clean Up Your Goroutines" below covers this.

The last paragraph of this page (Pg 207) discusses when to use unbuffered vs buffered channels.

Pg 207, Reading, Writing, and Buffering of "Learning Go an Idiomatic approach..." discusses using the built-in cap and len functions to find out how many elements the channel can hold, and how many elements are currently in the channel respectively.

Pg 209, How Channels Behave of "Learning Go an Idiomatic approach..." discusses the synchronisation of multiple writing channels using sync.WaitGroup, then refers to pg 220 (Using WaitGroups). Pg 227, 8.4.1. Unbuffered Channels of "The Go Programming Language" discusses using the "done" channel pattern. "Learning Go an Idiomatic approach..." also has a section on The Done Channel Pattern. "The Go Programming Language" also discusses and has examples for sync.WaitGroup.

Pg 226, 8.4.1. Unbuffered Channels of "The Go Programming Language" mentions "When a value is sent on an unbuffered channel, the receipt of the value happens before the reawakening of the sending goroutine."

select

Pg 211 of "Learning Go an Idiomatic approach..." mentions using "The Done Channel Pattern", and https://go.dev/tour/concurrency/5 actually uses it (quit <- 0). Also seen on pg 227 of "The Go Programming Language" (done <- struct{}{}).

Pg 212 of "Learning Go an Idiomatic approach..." mentions: "If you want to implement a nonblocking read or write on a channel, use a select with a default". There is also a warning that states: "Having a default case inside a for-select loop is almost always the wrong thing to do. It will be triggered every time through the loop when there’s nothing to read or write for any of the cases. This makes your for loop run constantly, which uses a great deal of CPU.". https://go.dev/tour/concurrency/5 states: "A select blocks until one of its cases can run, then it executes that case. It chooses one at random if multiple are ready (Pg 210 of "Learning Go an Idiomatic approach..." also mentions this)."... Unless of course there is a default. Pg 245 of "The Go Programming Language" states: "the other communications do not happen." within the select block.
Pg 245 of "The Go Programming Language" states "A select with no cases, select{}, waits forever.".

The following example is from pg 245 of "The Go Programming Language". It's behaviour is deterministic.

package main

import "fmt"

func main() {
	ch := make(chan int, 1)
	for i := 0; i < 10; i++ {
		fmt.Printf("Before select, iteration %d, ch now holds %d elements\n", i, len(ch)) // Added for debugging.
		select {
	  	case x := <-ch:
		  	fmt.Println(x) // "0" "2" "4" "6" "8"
		  case ch <- i:
			  fmt.Printf("ch now holds the value %d\n", i) // Added for debugging.
		}
		fmt.Printf("After select,  iteration %d, ch now holds %d elements\n", i, len(ch)) // Added for debugging.
	}
}

"Increasing the buffer size of the previous example makes its output nondeterministic, because when the buffer is neither full nor empty, the select statement figuratively tosses a coin."

The following is the same example as above, but with the channel (ch) being unbuffered. This deadlocks, why? Pg 207 of "Learning Go an Idiomatic approach..." says: "Every write to an open, unbuffered channel causes the writing goroutine to pause until another goroutine reads from the same channel. Likewise, a read from an open, unbuffered channel causes the reading goroutine to pause until another goroutine writes to the same channel. This means you cannot write to or read from an unbuffered channel without at least two concurrently running goroutines."

package main

import "fmt"

func main() {
	ch := make(chan int)
	for i := 0; i < 10; i++ {
		fmt.Printf("Before select, iteration %d, ch now holds %d elements\n", i, len(ch)) // Added for debugging.
		select {
	  	case x := <-ch:
		  	fmt.Println(x) // "0" "2" "4" "6" "8"
		  case ch <- i:
			  fmt.Printf("ch now holds the value %d\n", i) // Added for debugging.
		}
		fmt.Printf("After select,  iteration %d, ch now holds %d elements\n", i, len(ch)) // Added for debugging.
	}
}

Expecting a deterministic behavior using select and channels (#64) of "100 Go Mistakes and How to Avoid Them" is mostly about buffered channels with a size greater than 1.

The following example is made up from bits and pieces from 8.7. Multiplexing with select of "The Go Programming Language":

package main

import (
	"fmt"
	"os"
	"time"
)

func launch() {}

func main() {
	abort := make(chan struct{})
	fmt.Println("Commencing countdown. Press return to abort.")

	go func() {
		os.Stdin.Read(make([]byte, 1)) // read a single byte
		abort <- struct{}{}
	}()

	ticker := time.NewTicker(1 * time.Second)

	for countdown := 10; countdown > 0; countdown-- {
		fmt.Println(countdown)
		select {
		case <-ticker.C:
			// Do nothing.
		case <-abort:
			fmt.Println("Launch aborted!")
			return
		}
	}
	launch()
}

Concurrency Practices and Patterns

Goroutines, for Loops, and Varying Variables

Any time a goroutine depends on a variable in an outer scope whose value might change, you must pass the value into the goroutine.

I think the reason this example works is because the shadowing v in the for range loop's block is part of the goroutine's closure.

for _, v := range a {
  v := v
    go func() {
      ch <- v * 2
  }()
}

This is easier to read and makes more initial sense:

for _, v := range a {
  go func(val int) {
    ch <- val * 2
  }(v)
}

Go 1.22 fixes this. "For Go 1.22, we plan to change for loops to make these variables have per-iteration scope instead of per-loop scope."

Always Clean Up Your Goroutines

Whenever you launch a goroutine, you must make sure that it will eventually exit. Unlike variables, the Go runtime can’t detect that a goroutine will never be used again. If a goroutine doesn’t exit, the scheduler will still periodically give it time to do nothing, which slows down your program. This is called a goroutine leak.

func countTo(max int) <-chan int {
  ch := make(chan int)
  go func() {
    for i := 0; i < max; i++ {
      ch <- i // Goroutine blocks forever waiting for a listener.
    }
    close(ch)
  }()
  return ch
}

func main() {
  for i := range countTo(10) {
    if i > 5 { // If we exit the loop early.
      break
    }
    fmt.Println(i)
  }
}

Pg 219, 8.1. Goroutines of "The Go Programming Language" says that if there are goroutines still running when the main function returns, "all goroutines are abruptly terminated".

The Done Channel Pattern

I thought the example provided was incorrect as the case <-done: didn't provide a return, but it doesn't need to. select will block until either searcher (function) or done (channel) provides a value, and because all but the fastest search's do not provide values from the searcher, the done channel is closed, so the zero value of the done channel is read by the select, and the goroutine finishes.

8.9. Cancellation of "The Go Programming Language" covers the same topic.

Using a Cancel Function to Terminate a Goroutine

A function that contains a goroutine that returns it's goroutine's sending channel, along side a cancel function (closure containing code which closes a done channel).

When to Use Buffered and Unbuffered Channels

I'm not sure this section answers this question very well, at least it seems possibly a little naive? This is what it says:

"Buffered channels are useful when you know how many goroutines you have launched, want to limit the number of goroutines you will launch, or want to limit the amount of work that is queued up."

See the Channels subsection for quite a few comments around this decision.

Backpressure

If the number (concurrent requests) of tokens (a token being a struct{}) exceeds a specific number, the request is dropped and an error logged. I can see how this could keep resources within an acceptable/agreed limit.

Turning Off a case in a select

If you're using a select statement, and one or more of it's cases reads from a channel that's been closed, that isn't being iterated on by a for range loop, then the case will continue to be successful (return the zero value) and waste time. Instead set the channel's variable to nil, this will stop the case from being run because a nil channel doesn't return a value.

How to Time Out Code

Use a select statement with a case that reads a done channel (work has finished within time), and a case that reads a Time channel returned by time.After(time.Duration).

Pg 247, 8.7. Multiplexing with select of "The Go Programming Language" has an exercise for the reader to pretty much do the example just mentioned in Pg 220 of "Learning Go an Idiomatic approach...".

There is another timeout strategy on Pg 130 of 5.4.1. Error-Handling Strategies of "The Go Programming Language".

Using WaitGroups

Sometimes one goroutine needs to wait for multiple goroutines (or more specifically, the writing channels within the multiple goroutines) to complete their work, and their writing channels closed. If you are waiting for a single goroutine, you can use the done channel pattern that we saw earlier. But if you are waiting on several goroutines, you need to use a WaitGroup.

Also covered in:

  • Pg 237-239, 8.5. Looping in Parallel of "The Go Programming Language". The code for this can be found here, as well as a copy of it here. I also used this pattern as a start for the Gitlab interview I did. You can see the pattern in the files: bytes.go, checksum.go, which the find.go file consumes
  • Pg 250, 8.8. Example: Concurrent Directory Traversal of "The Go Programming Language" also has an example
  • Pg 274, 9.7. Example: Concurrent Non-Blocking Cache of "The Go Programming Language" also has an example
  • "100 Go Mistakes and How to Avoid Them" Misusing sync.WaitGroup (#71)

"A sync.WaitGroup doesn’t need to be initialized, just declared, as its zero value is" where we start from.

Basically we just use the following statements in different places:

var wg sync.WaitGroup
wg.Add(3) 
defer wg.Done() // Within a goroutine
defer wg.Done() // Within another goroutine
defer wg.Done() // Within another goroutine
wg.Wait()

We don't pass the sync.WaitGroup instance to each goroutine, because unless you pass a pointer to it, the copy is modified, which doesn't decrement the original. "By using a closure to capture the sync.WaitGroup, we are assured that every goroutine is referring to the same instance.".

"While WaitGroups are handy, they shouldn’t be your first choice when coordinating goroutines. Use them only when you have something to clean up (like closing a channel they all write to) after all of your worker goroutines exit."

The golang.org/x/sync/errgroup type builds on top of WaitGroup "to create a set of goroutines that stop processing when one of them returns an error". Also covered in "100 Go Mistakes and How to Avoid Them" Not using errgroup (#73).

"100 Go Mistakes and How to Avoid Them" Forgetting about sync.Cond (#72) allows you to "send repeated notifications to multiple goroutines"

Running Code Exactly Once

sync.Once allows for initialisation (or any) code to run exactly once. Like sync.WaitGroup, we declare a variable, no need to initialise it, because we want it to have its zero value.

"Also like sync.WaitGroup, we must make sure not to make a copy of an instance of sync.Once, because each copy has its own state to indicate whether or not it has already been used. Declaring a sync.Once instance inside a function is usually the wrong thing to do, as a new instance will be created on every function call and there will be no memory of previous invocations." It's usually best to declare the sync.Once instance at the package level.

Also covered in:

  • Pg 268, 9.5. Lazy Initialization: sync.Once of "The Go Programming Language"

Putting Our Concurrent Tools Together

Covers the example of 3 web services being called, waiting for the result of the first 2 services, then sending them to the third service. The entire process must complete in 50 milliseconds.

When to Use Mutexes Instead of Channels

A mutual exclusion limits the concurrent execution of some code or access to a shared (critical section) piece of data.

Being puzzled about when to use channels or mutexes (#57) of "100 Go Mistakes and How to Avoid Them" also discusses this. "Being aware of goroutine interactions can also be helpful when deciding between channels and mutexes. In general, parallel goroutines require synchronization and hence mutexes. Conversely, concurrent goroutines generally require coordination and orchestration and hence channels."

Effective Go: Concurrency - Share by communicating is along similar lines.

Go by Example: Mutexes

I have an example code for this. The easiest way to see the differences is to put the files side by side.

Atomics

"Like sync.WaitGroup and sync.Once, mutexs must never be copied. If they are passed to a function or accessed as a field on a struct, it must be via a pointer. If a mutex is copied, its lock won’t be shared."

"Never try to access a variable from multiple goroutines unless you acquire a mutex for that variable first."

sync.Map is only appropriate in situations:

  • When you have a shared map where key/value pairs are inserted once and read many times
  • When goroutines share the map, but don’t access each other’s keys and values

"sync.Map uses interface{} as the type for its keys and values; the compiler cannot help you ensure that the right data types are used."
"Given these limitations, in the rare situations where you need to share a map across multiple goroutines, use a built-in map protected by a sync.RWMutex."

Chapter 12. The Context

The Go wiki CodeReviewComments#contexts also touches on this.

Misunderstanding Go contexts (#60) of "100 Go Mistakes and How to Avoid Them" also discusses this.
"In general, a function that users wait for should take a context, as doing so allows upstream callers to decide when calling this function should be aborted."

Chapter 13. Writing Tests

The Basics of Testing

Reporting Test Failures

The Go wiki CodeReviewComments#useful-test-failures has some thoughts around what test failures should look like.

Finding Concurrency Problems with the Race Checker

A data race is a type of race condition.

Pg 257, 9.1. Race Conditions of "The Go Programming Language"

"100 Go Mistakes and How to Avoid Them" Not understanding race problems (data races vs. race conditions and the Go memory model) (#58)