hank / life
watch download tarball
public
Forks0Right
Watchers3Right
Description: Good code.
Homepage: http://www.ralree.com
Clone URL: git://github.com/hank/life.git
life / oscon / 2008 / tutorials / ProPostgres.rdoc
raw normal view history
c3d2ec2e » Erik 2008-07-21 Adding Pro PostgreSQL tut 1 = OSCON 2008, Tutorial 1: Pro Postgres
2 == Some useful notes
3
4 Slides: http://www.slideshare.net/xzilla/pro-postgresql-oscon-2008
5
6 Consider Book: Beginning PHP & Postgres 8
7
8 pgfoundry is a postgres dedicated projects
9
10 auth methods:
11 - TRUST means you're leaving the box pretty open
12 - md5 uses hashing (of course)
13 - Don't use IDENT
14
15 == Contributed code
16 pgcrypto = widely used contrib lib
17
18 == Procedural plugins
19 plperl apparently rocks your socks off.
20
21 Transitioning versions of pg:
22 - -Fc is your friend (compressed) (90% compression is avg!)
23 - dump with the new version of pg_dump
24 - restore form files with pg_restore
25
26 Look into slony for upgrading. Slave-copy system.
27
28 == Options
29
30 === effective_cache_size
31 - "here's how much fs cache I have to work with."
32 - Maybe 75% of RAM on a dedicated machine.
33 - Doesn't preallocate.
34
35 === shared_buffers
36 Does preallocate. Read more.
37
38 === default_statistics_target
39 - samples the tables depending on the size of the table and makes a decision on when to index first.
40 - "set it to 100 and let it go."
41 - The higher you set it, the more statistics it does.
42
43 === work_mem
44 - amount of memory available IN an SQL query for hashing/sorting.
45 - For a big box, set a LOT higher.
46 - It defaults to about 1M. It can multiply very fast.
47 - Depending on max num of queries, set to something reasonable (10M?).
48
49 === checkpoint_segments
50 - as it's going along, it makes checkpoints (flush, check).
51 - If you have a lot of writes, turn up this option.
52 - Putting it at something like 30 is not unheard of.
53 - If there's a crash, it takes a long time the higher the number is.
54
55 === maintenance_work_mem
56 - Running vacuum or reindexing.
57 - Like work_mem.
58 - Big tables it makes a difference. He's gone as high as 1G.
59
60 === update_process_title
61 Might remove some overhead...
62
63 === max_fsm_pages
64 - Not Flying Spaghetti Monster.
65 - Freespace map keeps track of dead rows, and it sizes this.
66
67 === syncronous_commit
68 - New in 8.3. At the end of any commit, we push data to disk.
69 - That's good, but expensive. This lets you put it into memory for a second, and then chunk commits together.
70 - Good for lots of fast concurrent transactions.
71 - You can lose transactions to crashes, but you don't get data corruption.
72
73 == Logging
74
75 === log_min_error_statement
76 Gives the SQL query that caused the error.
77
78 === log_line_prefix
79 Keeps a PID in the error.
80
81 == Vacuuming
82 Lazy vacuuming is awesome if you want no downtime.
83
84 === Autovacuuming
85 - Do it.
86 - track_activities and track_counts are your friends
87 - autovacuum_max_freeze_age: for n transactions, make sure everything's cleaned up.
88 - pg_autovacuum allows you to set vacuuming per table.
89
90 == Hardware
91 Some general recommendations. He's "not a hardware guy."
92
93 - Get Lots of (ECC) RAM
94 - Prefer SCSI, Accept SATA
95 - RAID Z (JBOD) ?
96 - Plateau of CPU linear improvement at about 8
97
98 === Disk
99 - Put WAL on it's own disk ?
100 - More spindles = better
101 - More controllers = even better (tablespaces, dedicated disk, insert-only)
102 - write cache if you care about crash recovery.
103 - NFS = bad
104 - RAID 5 no. Not optimal for DB performance. Use RAID 1 or 10.
105 - Someone said "after 8 spindles you won't notice the diff between RAID 5 and RAID 10"
106
107 == Availability
108 === pg_dump LOL
109 - Dump it and copy it to another server
110 - constantly run restore
111 - large time/IO constraints
112
113 === bucardo
114 - asynchronous multi-master or master-slave
115 - low IO, time constraints
116 - other benefits - upgrades, scaling
117 - fewer people in the community
118
119 === Shared disk
120 - one copy of PGDATA on shared storage
121 - standby takes over when db crashes
122 - STONITH (Shoot the other node in the head): When the original side comes up after the second node takes over, and pwns everything.
123
124 === Filesys replication
125 - drbd is an example
126 - Shipping filesystem crap around
127 - sync, ordered writes
8bb536bd » Erik 2008-07-21 Finished out the notes 128
129 === pg_pool
130 - dual-master, statement based
131 - bad for random, now, sequences
132 - has to run as root
133
134 === Scaling
135 - pg scales well
136 - more disks (tablespaces)
137 - more cpus, more ram
138 - connection pooling
139 - 1000+ connections, TBs of data
140
141 === pg_bouncer
142 - simple connection pooler (gets rid of process limitations)
143 - 10/1 - 40/1 ratio of processes
144 - caveats: prepared statements, temp tables
145 - skype, myyearbook.com
146
147 == Optimization
148 === Queries
149 - pgfouine and pqa
150 - command line reports!
151 - IO load
152
153 ==== explain analyze
154 - complicated
155 - good for large queries
156 - http://wiki.postgresql.org/Using_EXPLAIN
157
158 === Internal
159 ==== pg_stat_all_tables
160 - number of inserts, update, deletes, hot updates (not too interesting)
161 - sequential scans are when the whole table is scanned. BAD.
162 - Live and dead tuples. New in 8.3. Need to vacuum/vac_full? Ratio dead/live is a good indicator.
163 - timestamps of when crap happened
164
165 ==== pg_stat_all_indexes
166 Good for looking at index perf.
167
168 ==== Other
169 - in-memory-joins are nice for dual-col indexing
170 - restrain index to rows that are queried a lot (maybe for SB /8, /16, /24)
171 - where clause of the index should match the where of the query
172 - push expensive functions into your index (it allows a lot)
173
174 === Fulltext search
175 - lexemes, word stemming
176 - _looks nice. We should try._
177 - gist is oldschool, fast insert/update, slow queries
178 - gin is better for query, worse for indexing
179
180 == Tablespaces
181 - logical for locations for object placement
182 - point to locations on disk (symlinks)
183 - size is by disk size
184 - dedicate per db, split on disk
185 - _maybe use fast flash for indexes_
186
187 == Partitioning
188 - Look at pgcon talk from 2007.
189 - http://www.pgcon.org/2007/schedule/events/41.en.html