Linux Administration

Commands

avahi-browse: browse for mDNS/DNS-SD services using the Avahi daemon.
- -a: browse for all services on the LAN.
- -r: resolve services found automatically.
- -t: terminate after dumping a list.
- Examples:
- avahi-browse -art: pull and resolve all mDNS/DNS-SD devices on the local network and terminate.
dmidecode: reads the system DMI table to display hardware and BIOS information.
ecryptfs-unwrap-passphrase: displays the encryption key for a user's home directory.
genisoimage: make ISO files, based on cdrkit.
- -l: allow full 31 character filenames.
- -J: generate Joliet directory records in addition to regular iso9660 file names.
- -R: generate System Use Sharing Protocol (SUSP) and Rock Ridge (RR) records using the Rock Ridge protocol.
- Examples:
- genisoimage -lJR -o [output name].iso [source directory path]: create an ISO with Rock Ridge and Joilet extensions with full character filenames from [source directory path] as [output name].iso.
journalctl: SystemD interface for viewing and interacting with logs.
- -f: "follow" logs as they are created, analogous to tail -f.
- -u [service]: view logs for a given service only
- Examples:
- journalctl -u bind9.service -f: follow logs for the bind9 service as they are created.
logrotate -vf [path to config]: manually runs logrotate on a configuration.
lsmod: list kernel modules in use.
lsof: list open files.
- Examples
- lsof -i -n -P | grep LISTEN: display all currently open and listening ports.
- lsof -i :[port]: show process using a port.
mkisofs: make ISO files, based on cdrtools.
- -l: allow full 31 character filenames.
- -J: generate Joliet directory records in addition to regular iso9660 file names.
- -R: generate System Use Sharing Protocol (SUSP) and Rock Ridge (RR) records using the Rock Ridge protocol.
- Examples:
- mkisofs -lJR -o [output name].iso [source directory path]: create an ISO with Rock Ridge and Joilet extensions with full character filenames from [source directory path] as [output name].iso.
modinfo: list the capabilities for a kernel module
- Examples:
- modinfo i915: list the capabilities and options for the Intel i915 graphics driver.
pacman: manage packages using Pacman (Arch Linux).
- Examples:
- pacman -Syu: update and upgrade system packages.
- pacman -S [package]: install a package.
rsync: copies files from one place to another.
- Examples:
- rsync --bwlimit=[speed in kb/second] [source] [dest]: rate limits a transfer to the speed listed.
showmount: display all exported file systems from an NFS server.
- Examples:
- showmount -e [IP|hostname]: display mounts exported from a host.
ssh-keygen: generate keys for OpenSSH.
- -t: specify key type
- -b: specify bit strength
- Examples:
- ssh-keygen -t ecdsa -b 521: generate a 521-bit key using the ECDSA algorithm.
- ssh-keygen -t rsa -b 4096: generate a 4096-bit key using the RSA algorithm.
systemctl: SystemD interface for managing services.
- no arguments: list all services.
- -u: specify an individual unit ("service").
systool: show currently-loaded options.
- Examples:
- systool -m i915 -av: show all currently loaded options for the i915 module.
tcpdump: network traffic dumper.
- -A: print packets in ASCII.
- -i [interface]: limit capture to a given interface (or any for all).
- -n: don't convert addresses to names.
- -n dst host [IP]: filter for packets only toward a destination host IP.
- -p: do NOT put the interface into promiscuous mode.
update-ca-certificates: update /etc/ssl/certs and certificates.crt. Reads /usr/share/ca-certificates and /usr/local/share/ca-certificates to generate the list.
wget: downloads files from URIs.
- --content-disposition: uses Content-Disposition headers which let a server dictate the resultant filename.
- --read-timeout=X: sets a number of X seconds for read data before considering an attempt failed.
- --retry-connrefused: considers connection refused a transient error and tries again.
- -t X: sets a maximum number of X retries.
- --timeout=X: sets a number of X seconds of no data before considering an attempt failed. Sets several types.
- --waitretry=X: sets X seconds for retries.

Firewall

Linux firewall is typically maintained by iptables.

Allow a port in: iptables -A INPUT -p [tcp/udp] --dport [port] -j ACCEPT
Allow a port range in: iptables -A INPUT -p [tcp/udp] --match multiport --dports [start]:[end] -j ACCEPT
Allow ICMP (ping) in: iptables -A INPUT -p icmp -j ACCEPT
Delete chain: iptables -X
Flush rules: iptables -F

Procedures

Add a Service to SystemD

Create a servicename.service file and place it in /etc/systemd/user/. You can technically put the service file anywhere but this will differentiate it from files maintained by packages.

ln -s /etc/systemd/user/[servicename].service /etc/systemd/system/
systemctl daemon-reload

Add an SSL Certificate to be Trusted

Ubuntu

$ sudo cp [certificate to be trusted] /usr/local/share/ca-certificates/
$ sudo update-ca-certificates

Apt Repository Management

add repository key: wget -qO - http://deb.project.net/keyname.key | sudo apt-key add -
list repository keys: apt-key list

Automount NFS Shares

Raspbian

/etc/fstab

[server]:[path to export] [local mountpoint] nfs defaults,x-systemd.automount 0 0

Configure OpenSSH to Use a Key for a Host or Domain

Create a suitable key, note the storage location

ssh-keygen -t ecdsa -b 521

Add the key and domain/host to the user's ~/.ssh/config.

~/.ssh/config

Host [name, e.g. github]
  HostName [domain or hostname, e.g. github.com]
  User git
  IdentityFile ~/.ssh/[secret key file name]

Creating Large (> 2TB) Partitions

fdisk -l /dev/disk/by-uuid
parted /dev/disk-to-modify
mklabel gpt
unit TB
mkpart primary 0 <target TB>
print
quit
mkfs.ext4 /dev/disk-just-made

Generate Password Hashes

Sometimes you need to be able to generate a password hash for the /etc/shadow file manually. You can easily do this with Python.

md5:

python -c "import random,string,crypt; randomsalt = ''.join(random.sample(string.ascii_letters,8)); print crypt.crypt('[PASSWORD STRING HERE]', '\$1\$%s\$' % randomsalt)"

sha256:

python -c "import random,string,crypt; randomsalt = ''.join(random.sample(string.ascii_letters,8)); print crypt.crypt('[PASSWORD STRING HERE]', '\$5\$%s\$' % randomsalt)"

sha512:

python -c "import random,string,crypt; randomsalt = ''.join(random.sample(string.ascii_letters,8)); print crypt.crypt('[PASSWORD STRING HERE]', '\$6\$%s\$' % randomsalt)"

Remove Services From SystemD

systemctl stop [servicename]
systemctl disable [servicename]
rm /etc/systemd/system/[servicename]
rm /etc/systemd/system/[servicename] symlinks that might be related
systemctl daemon-reload
systemctl reset-failed

Upgrade Ubuntu

sudo apt-get install update-manager-core`
do-release-upgrade

Recipes

Examine traffic to a port on an interface: tcpdump -pni [interface] port [port] -A
Find old files: find [directory] -mtime +[days]
Spy on process output: strace -e trace=write -s1000 -fp [pid] 2>&1 | grep -o '".\+[^"]"'

Status

Top Memory Consumers

Issue: frequently you need to know the top memory-consuming processes on a system at a point in time, typically for scripting.
Solution 1: dump processes and sort off the memory resident field, then reverse to show in descending order.

$ ps aux | sort -nk +4 | tail | sort -rnk +4

Solution 2: dump processes and sort off the memory resident field, then reverse to show in descending order for the top twenty processes.

$ ps aux | sort -nk +4 | tail -20 | sort -rnk +4

Raw Copy Status Update

Issue: sometimes when doing a raw copy with dd it appears that the process has either taken too long or hung. Before completely killing it (since there is no resume) it would be helpful to obtain some status information.
Solution 1: sending a SIGUSR1 signal to the dd process will get a status update.

 $ ps -ef | grep dd | awk '{print $2}' | xargs kill --signal SIGUSR1
 $ kill -SIGUSR1 `pidof dd`

Troubleshooting

Bind

Issue: receiving errors in the log about insecurity proof failed or no valid RRSIG resolving.
Solution: one of the DNS resolvers in the /etc/bind/named.conf.options is not set up to handle DNSSEC but BIND is trying to use DNSSEC. Either disable DNSSEC or remove the offending resolver.
Issue: DNS queries are not resolving though the service is running.
Solution:
- bind may be trying to use an IPv6 address to do lookups but no IPv6 is supported on your network. Adjust the default options for bind to use only IPv4 for lookups.
- /etc/default/bind9: (Ubuntu/Debian)

# run resolvconf?
RESOLVCONF=no

# startup options for the server
OPTIONS="-u bind -4"

Python

Issue: pip or pip3 fails to install a package, citing an error message like:

[...]
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 139: ordinal not in range(128)
[...]

Solution: this has to do with an unset system locale encoding. Depending on your needs your solution will vary.

If you only need the settings for the current session, simply change the LC_ALL variable to the default system locale. Assuming your locale has been set to en_US.UTF-8 you can set it as such:

$ export LC_ALL="en_US.UTF-8"

To persist those changes, you'll want to modify your system settings in addition to the above.

Ubuntu: /etc/default/locale

In some systems you may not have the desired locale built. On those systems (such as Docker images) you'll need to generate and then set the locale: