/
varnish_book.rst
7917 lines (5483 loc) · 323 KB
/
varnish_book.rst
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.. include:: util/frontpage.rst
.. include:: util/printheaders.rst
.. include:: build/version.rst
.. contents::
:class: handout
.. include:: util/control.rst
.. include:: util/param.rst
.. raw:: pdf
PageBreak coverPage
.. class:: heading1
Abstract
The Varnish Book is the training material for Varnish Plus courses.
This book teaches such concepts to understand the theory behind Varnish Cache 4.
Covered are the Varnish finite state machine, design principles, HTTP, cache invalidation and more.
With these foundations, the book builds practical knowledge on Varnish Configuration Language (VCL), Varnish Test Code (VTC) and Varnish utility programs such as ``varnishlog``, ``varnishstat`` and ``varnishtest``.
Examples and exercises develop the needed skills to administrate and extend the functionality of Varnish.
Also included are appendices that explain how to develop Varnish Modules (VMODs) and how to use selected modules of Varnish Plus.
.. raw:: pdf
PageBreak
.. class:: heading1
Preface
- Course for Varnish Plus
- Learn specific features depending the course and your needs
- Necessary Background
- How to Use the Book
- Acknowledgments
.. TODO for trainer:
- Tailor this slide!
- Inform trainees what type of subscription they have and the services that they have access to
- Inform trainees how long the amazon VM will last after the training session
.. container:: handout
.. Common goal for Varnish Plus customers:
**After finishing this course, you will be able to install and configure the Varnish server, and write effective VCL code.**
The Varnish Book is designed for attendees of Varnish Plus courses.
Most of the presented material in this book applies to both, the open source Varnish Cache and the commercial edition Varnish Cache Plus.
Therefore, you can also refer to the Varnish Cache documentation at https://www.varnish-cache.org/docs/4.0/.
Varnish Plus is a commercial suite by Varnish Software that offers products for scalability, customization, monitoring, and expert support services.
The engine of Varnish Plus is Varnish Cache Plus, which is the enhanced commercial edition of Varnish Cache.
Varnish Cache Plus should not be confused with Varnish Plus, a product offering by Varnish Software.
Varnish Cache Plus is one of the software components available for Varnish Plus customers.
For simplicity, the book refers to *Varnish Cache* or *Varnish Cache Plus* as **Varnish** when there is no difference between them.
There is more information about differences between Varnish Cache and Varnish Cache Plus in the `Varnish Cache and Varnish Plus`_ chapter.
.. instructors goal:
The goal of this book is to make you confident when using Varnish.
Varnish instructors focus on your area, needs or interest.
Varnish courses are usually flexible enough to make room for it.
The instructor will cover selected material for the course you take.
The System Administration (Admin) course provides attendees with the necessary knowledge to troubleshoot and tune common parameters of a Varnish server.
The Web Developer (Webdev) course teaches how to adapt web applications so that they work with Varnish, which guarantees a fast experience for visitors of any website.
Besides that, other courses may also be taught with this book.
.. TODO for author: "as visitors' requests scales": can we state a magnitude number? or at what level of scalability are we talking about?
.. raw:: pdf
PageBreak
.. class:: heading2
Necessary Background
.. TODO for instructor: tailor this slide!
The Admin course requires that you:
- have expertise in a shell on a Linux/UNIX machine, including editing text files and starting daemons,
- understand HTTP cache headers,
- understand regular-expressions, and
- be able to install the software listed below.
The Webdev course requires that you:
- have expertise in a shell on a Linux/UNIX machine, including editing text files and starting daemons,
- understand HTTP cache headers,
- understand regular-expressions, and
- be able to install the software listed below.
.. container:: handout
You do not need background in theory or application behind Varnish to complete this course.
However, it is assumed that you have experience and expertise in basic UNIX commands, and that you can install the following software:
.. TODO: to update this list after introducing varnishtest throughout the book
It might not be needed to install Apache, HTTPie, PHP or curl.
- Varnish Cache 4.x or Varnish Cache Plus 4.x,
- Apache/2.4 or later,
- HTTPie 0.8.0 or later,
- PHP 5.4 or later, and
- curl – command line tool for transferring data with URL syntax
More specific required skills depend on the course you take.
The book starts with the installation of Varnish and navigation of some of the common configuration files.
This part is perhaps the most UNIX-centric part of the course.
.. Todo for the author: This organization is for the book as self instructive.
.. At the moment, the book is designed to be used under training.
Organization of the Book and Course
-----------------------------------
TODO for instructor: tailor this slide
Each chapter has the following structure:
1) background,
2) checklist,
3) example(s),
4) exercise(s), and
5) feedback.
At the end of each chapter there is a **Fast Track** section, and it contains three parts:
1) a review,
2) exercise(s), and
3) feedback.
.. container:: handout
.. TODO presentation of outline
The introduction presents background information of the relevant technologies behind Varnish.
It presents the history and main design principles needed to understand Varnish.
.. TODO for the author:
.. complete this paragraph
.. Each chapter teaches one **objective** and its **sub-objectives**, or component skills.
.. The format of each sub-objective is of a five-part **lesson** including:
The Fast Track can be used for various purposes:
- As a preliminary test. If you know the information in the Fast Track and can do its exercises, you may skip the chapter.
- As a test. If you can perform the exercises in the Fast Track, you have learned the important skills taught in the chapter.
- As a express lesson. If you do not have the time to complete the entire chapter, you can use the Fast Track instead.
- As a preview of the chapter. The Fast Track provides a concise overview of what you learn in the chapter.
- As a review. Once you have completed the Varnish Book, you can use the Fast Tracks as a quick review.
In Appendix A of the book, you will find the **Varnish Book Reference**.
The Varnish Book Reference is a complete listing, by chapter, of all the checklists.
Appendix x and C contain special purpose Varnish programs and supporting material respectively.
Appendix x lists the Varnish Three Letter Acronyms.
Appendix x describes what is new since Varnish 3.0.
.. TODO for the author: include a glossary based on:
.. https://www.varnish-cache.org/trac/wiki/VTLA
.. TODO for the author: remove this comment when the organization has been completely implemented.
Note: This organization is still not completely implemented.
.. raw:: pdf
PageBreak
.. class:: heading2
How to Use the Book
.. At the moment is a section to exclude it from compiling in the Makefile
- Most of the material in this book applies to both: Varnish Cache and Varnish Cache Plus.
Parts that apply only to Varnish Cache Plus are clearly stated.
- Varnish caching mechanisms are different than in other caching technologies.
Open your mind and try to think different when using Varnish.
- The instructor guides you through the book.
- Use the *manual pages* and help options.
- See `Appendix E: Varnish Three Letter Acronyms`_ for a list of acronyms.
.. - `Appendix E: Regular-Expressions in Varnish`_ explains basic concepts typically needed when administrating Varnish.
.. - To practice the course, use the Fast Track in the following way:
.. - Read the Review and Exercise parts of the Fast Track.
.. - If you can complete the exercise(s), you remember the most important parts.
.. - If not, you should repeat the chapter.
.. container:: handout
.. Varnish Cache Course
The Varnish Book is designed to be used as training material under the Varnish Plus course taught by a certified instructor.
Under the course, the instructor guides you and selects the relevant sections to learn.
However, you can also use this book as self-instructional material.
.. If you use the book as self-instructional tutorial, it is recommended to complete the Fast Track of the chapter you want to read.
.. Exercises
There are almost always many ways to do an exercise.
The solutions provided in `Appendix G: Solutions`_ are not necessarily better than yours.
.. man pages and help commands
Varnish installs several reference manuals that are accessible through the manual page command ``man``.
You can issue the command ``man -k varnish`` to list the manual pages that mention Varnish in their short description.
In addition, the ``vsl`` man page that explains the Varnish Shared memory Logging (VSL).
This man page does not come out when issuing ``man -k varnish``, because it does not contain the word `varnish` in its short description.
The command ``man varnishd``, for example, retrieves the manual page of the Varnish HTTP accelerator daemon.
Also, some commands have a help option to print the usage of the command.
For example, ``varnishlog -h`` prints the usage and options of the command with a short description of them.
.. reference to Varnish Cache docs
In addition, you should refer to the documentation of Varnish Cache and Varnish Cache Plus.
This documentation provides you extended details on the topics covered in this book and more.
To access to this documentation, please visit https://www.varnish-software.com/resources.
.. Ubuntu
The Varnish installation described in this book uses Ubuntu Linux 14.04 LTS (trusty), therefore most of the commands instructed in this book are for this Linux distribution.
We point out some differences on how to configure Varnish for other Linux distributions, but you should reference your Linux distribution's documentation for more details.
.. formats
The book is written with different formatting conventions.
Varnish Configuration Language (VCL) code uses the mono-spaced font type inside boxes::
vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
# Do request header transformations here.
if (req.url ~ "^/admin") {
return(pass);
}
}
The first occurrence of a new term is usually its *definition*, and appears in italics.
File names are indicated like this: ``/path/to/yourfile``.
Important notes, tips and warnings are also inside boxes, but they use the normal body text font type.
.. TODO: To uncomment this when Appendix E is ready
.. regex
..
If you have some regular-expression experience, must of the overview in `Appendix E: Regular-Expressions in Varnish`_ will not be new.
Even if this is the case, please glance over it anyway.
Although you may be aware of the basic meaning of certain meta-characters, perhaps some of the ways of thinking at regular-expressions will be new for you.
.. raw:: pdf
PageBreak
.. class:: heading2
Resources, and Errata
- https://varnish-cache.org
- https://varnish-software.com/academy
- `#varnish-hacking` and `#varnish` on `irc.linpro.net`.
- https://github.com/varnish/Varnish-Book/
- https://www.varnish-cache.org/docs/trunk/users-guide/troubleshooting.html
- https://www.varnish-cache.org/trac/wiki/VCLExamples
.. container:: handout
This book is meant to be understandable to everyone who takes a Varnish Plus course and has the required skills.
If you find something unclear, do not be shy and blame yourself, ask your instructor for help.
You can also contact the Varnish open source community at https://varnish-cache.org.
To book training, please look at https://varnish-software.com/academy.
Additional examples from different Varnish versions are available at https://www.varnish-cache.org/trac/wiki/VCLExamples.
These examples are maintained by the community.
For those interested in development, the developers arrange weekly bug washes were recent tickets and development is discussed.
This usually takes place on Mondays around 13:00 CET on the IRC channel `#varnish-hacking` on `irc.linpro.net`.
Errata, updates and general improvements of this book are available at its repository https://github.com/varnish/Varnish-Book.
.. class:: heading2
Acknowledgments
In addition to the authors, the following deserve special thanks (in no particular order):
- Rubén Romero
- Dag Haavi Finstad
- Martin Blix Grydeland
- Reza Naghibi
- Federico G. Schwindt
- Dridi Boukelmoune
- Lasse Karstensen
- Per Buer
- Sevan Janiyan
- Kacper Wysocki
- Magnus Hagander
- Arianna Aondio
- Poul-Henning Kamp
- Guillaume Quintard
- Everyone who has participated on the training courses
.. raw:: pdf
PageBreak oneColumn
Introduction
============
Table of contents:
- What is Varnish?
- Benefits of Varnish
- Open source / Free software
- Varnish Software: The company
- What is Varnish Plus?
- Varnish: more than a cache server
- History of Varnish
- Varnish Governance Board (VGB)
.. TODO Comparison of related software solutions such as: Apache mod_security, Squid, Nginx, and Apache Traffic Server (ATS) (reverse and forward proxy, generally comparable to Nginx and Squid).
What is Varnish?
----------------
.. figure 1
.. figure:: ui/img/reverse_proxy.svg
:alt: Reverse Proxy
:align: center
:width: 50%
Figure :counter:`figure`: Varnish is more than a reverse proxy
.. container:: handout
.. What is Varnish?:
Varnish is a reverse HTTP proxy, sometimes referred to as an HTTP accelerator or a web accelerator.
A reverse proxy is a proxy server that appears to clients as an ordinary server.
Varnish stores (caches) files or fragments of files in memory that are used to reduce the response time and network bandwidth consumption on future, equivalent requests.
Varnish is designed for modern hardware, modern operating systems and modern work loads.
.. Varnish use
Varnish is more than a reverse HTTP proxy that caches content to speed up your server.
Depending on the installation, Varnish can also be used as:
- web application firewall,
- DDoS attack defender,
- load balancer,
- integration point,
- single sign-on gateway,
- authentication and authorization policy mechanism,
- quick fix for unstable backends, and
- HTTP router.
Varnish is Flexible
...................
Example of Varnish Configuration Language (**VCL**)::
vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
# Do request header transformations here.
if (req.url ~ "^/admin") {
return(pass);
}
}
.. container:: handout
Varnish is flexible because you can configure it and write your own caching policies in its Varnish Configuration Language (VCL).
VCL is a domain specific language based on C.
VCL is then translated to C code and compiled, therefore Varnish executes lightning fast.
Varnish has shown itself to work well both on large (and expensive) servers and tiny appliances.
Varnish Cache and Varnish Plus
------------------------------
.. table 1
.. csv-table:: Table :counter:`table`: Topics Covered in This Book and Their Availability in Varnish Cache and Varnish Plus
:name: Topics Covered in This Book and Their Availability in Varnish Cache and Varnish Plus
:delim: ,
:header-rows: 1
:widths: 40,30,30
:file: tables/varnish_cache_plus_offer_diff.csv
.. container:: handout
.. Open Source / Free Software:
Varnish Cache is an open source project, and free software.
The development process is public and everyone can submit patches, or just take a peek at the code if there is some uncertainty on how does Varnish Cache work.
There is a community of volunteers who help each other and newcomers.
The BSD-like license used by Varnish Cache does not place significant restriction on re-use of the code, which makes it possible to integrate Varnish Cache in virtually any solution.
Varnish Cache is developed and tested on GNU/Linux and FreeBSD.
The code-base is kept as self-contained as possible to avoid introducing out-side bugs and unneeded complexity.
Therefore, Varnish uses very few external libraries.
.. Varnish Software:
Varnish Software is the company behind Varnish Cache.
Varnish Software and the Varnish community maintain a package repository of Varnish Cache for several common GNU/Linux distributions.
.. Varnish Plus:
Varnish Software also provides a commercial suite called Varnish Plus with software products for scalability, customization, monitoring and expert support services.
The engine of the Varnish Plus commercial suite is the enhanced commercial edition of Varnish Cache.
This edition is proprietary and it is called *Varnish Cache Plus*.
.. Covered in this book:
`Table 1 <#table-1>`_ shows the components covered in this book and their availability for Varnish Cache users and Varnish Plus customers.
The covered components of Varnish Plus are described in the `Varnish Plus Software Components`_ chapter.
For more information about the complete Varnish Plus offer, please visit https://www.varnish-software.com/what-is-varnish-plus.
.. Supported platforms
At the moment of writing this book, Varnish Cache supports the operating systems and Linux distributions listed in `Table 2 <#table-2>`_.
.. table 2
.. csv-table:: Table :counter:`table`: Varnish Cache and Varnish Plus supported platforms
:name: Varnish Cache and Varnish Plus supported platforms
:delim: ,
:header-rows: 1
:widths: 50,25,25
:file: tables/supported_platforms.csv
Varnish Cache and Varnish Plus support only 64-bit systems.
.. note::
Varnish Cache Plus should not be confused with Varnish Plus, a product offering by Varnish Software.
Varnish Cache Plus is one of the software components available for Varnish Plus customers.
Varnish Cache and Varnish Software Timeline
-------------------------------------------
- 2005: Ideas! Verdens Gang (www.vg.no, Norway's biggest newspaper) were looking for alternative cache solutions
- 2006: Work began:
Redpill Linpro was in charge of project management, infrastructure and supporting development.
Poul-Henning Kamp did the majority of the actual development.3
- 2006: Varnish 1.0 is released
- 2008: Varnish 2.0 is released
- 2008: ``varnishtest`` is introduced
- 2009: The first Varnish User Group Meeting is held in London
Roughly a dozen people participate from all around the world
- 2010: Varnish Software is born as a spin-off to Redpill Linpro AS
- 2011: Varnish 3.0 is released
- 2012: The fifth Varnish User Group Meeting is held in Paris
Roughly 70 people participate on the User-day and around 30 on the developer-day!
- 2012: The Varnish Book is published
- 2013: Varnish Software chosen as a 2013 Red Herring Top 100 Europe company
- 2013: BOSSIE award winner
- 2013: Varnish Software receives World Summit on Innovation & Entrepreneurship Global Hot 100 award
- 2014: Varnish Plus is launched
- 2014: Varnish 4.0 is released
- 2015: Varnish API Engine is released
- 2015: Gartner names Varnish Software as a 2015 ‘Cool Vendor’ in Web-Scale Platforms
- 2015: Varnish Plus supports SSL/TLS
.. container:: handout
VG, a large Norwegian newspaper, initiated the Varnish project in cooperation with Linpro.
The lead developer of the Varnish project, Poul-Henning Kamp, is an experienced FreeBSD kernel hacker.
Poul-Henning Kamp continues to bring his wisdom to Varnish in most areas where it counts.
From 2006 throughout 2008, most of the development was sponsored by VG, API, Escenic and Aftenposten, with project management, infrastructure and extra man-power provided by Redpill Linpro.
At the time, Redpill Linpro had roughly 140 employees mostly centered around consulting services.
Today Varnish Software is able to fund the core development with income from service agreements, in addition to offering development of specific features on a case-by-case basis.
The interest in Varnish continues to increase.
An informal study based on the list of most popular web sites in Norway indicates that about 75% or more of the web traffic that originates in Norway is served through Varnish.
.. TODO for the author: reference for the informal study?
.. VGB
Varnish development is governed by the Varnish Governance Board (VGB), which thus far has not needed to intervene.
The VGB consists of an architect, a community representative and a representative from Varnish Software.
.. TODO for the editor: confirm the VGB positions
As of November 2015, the VGB positions are filled by Poul-Henning Kamp (Architect), Rogier Mulhuijzen (Community) and Lasse Karstensen (Varnish Software).
On a day-to-day basis, there is little need to interfere with the general flow of development.
What Is New in Varnish 4?
-------------------------
- Version statement ``vcl 4.0;``
- ``req.request`` is now ``req.method``
- ``vcl_fetch`` is now ``vcl_backend_response``
- Directors have been moved to the ``vmod_directors``
- Hash directors as a client directors
- ``vcl_error`` is now ``vcl_backend_error``
- ``error()`` is now ``synth()``, and you must explicitly return it: ``return (synth(999, "Response"));``
- Synthetic responses in ``vcl_synth``
- Setting headers on synthetic response bodies made in ``vcl_synth`` are now done on ``resp.http`` instead of ``obj.http``.
- ``obj.*`` in ``vcl_error`` replaced by ``beresp.*`` in ``vcl_backend_error``
- ``hit_for_pass`` objects are created using ``beresp.uncacheable``
- ``req.*`` not available in ``vcl_backend_response``
- ``bereq.*`` in ``vcl_backend_response``
- ``vcl_*`` prefix reserved for builtin subroutines
- ``req.backend.healthy`` replaced by ``std.healthy(req.backend_hint)``
- ``client.port`` and ``server.port`` replaced by ``std.port(client.ip)`` and ``std.port(server.ip)``
- Cache invalidation with purges is now done via ``return(purge)`` in ``vcl_recv``
- ``obj.*`` is now read-only
- ``obj.last_use`` is retired
- ``vcl_recv`` must now return ``hash`` instead of ``lookup``
- ``vcl_hash`` must now return ``lookup`` instead of ``hash``
- ``vcl_pass`` must now return ``fetch`` instead of ``pass``
- ``restart`` in the backend is now ``retry``, this is now called ``return(retry)``, and jumps back up to ``vcl_backend_fetch``
- `default` VCL is now called `builtin` VCL
- The builtin VCL now honors ``Cache-Control: no-cache`` (and friends) to indicate uncacheable content from the backend
- ``remove`` keyword replaced by ``unset``
- ``X-Forwarded-For`` is now set before ``vcl_recv``
- ``session_linger`` has been renamed to ``timeout_linger`` and it is in seconds now (previously was milliseconds)
- ``sess_timeout`` is renamed to ``timeout_idle``
- Increasing ``sess_workspace`` is not longer necessary, you may need to increase either `workspace_backend` or `workspace_client`
- ``thread_pool_purge_delay`` is renamed to ``thread_pool_destroy_delay`` and it is in seconds now
- ``thread_pool_add_delay`` and ``thread_pool_fail_delay`` are in seconds now
- New parameter ``vcc_allow_inline_c`` to disable inline C in your VCL
- New query language to filter logs: ``-m`` option replaced by ``-q``
.. container:: handout
The above list tries to summarize the most important changes from Varnish Cache 3 to Varnish Cache 4.
For more information, please visit: https://www.varnish-cache.org/docs/trunk/whats-new/upgrading.html
If you want to migrate your VCL code from Varnish 3 to Varnish 4, you may be interested in looking at the *varnish3to4* script.
See the `VCL Migrator from Varnish 3 to Varnish 4`_ section for more information.
Design Principles
=================
Varnish is designed to:
- Solve real problems
- Run on modern hardware (64-bit multi-core architectures)
- Work with the kernel, not against it
- Translate Varnish Configuration Language (VCL) to C programming language
- Be extendible via Varnish Modules (VMODs)
- Reduce lock-contention via its workspace-oriented shared memory model
.. - Innovation, not regurgitation
.. container:: handout
.. run on modern hardware
The focus of Varnish has always been performance and flexibility.
Varnish is designed for hardware that you buy today, not the hardware you bought 15 years ago.
This is a trade-off to gain a simpler design and focus resources on modern hardware.
Varnish is designed to run on 64-bit architectures and scales almost proportional to the number of CPU cores you have available.
Though CPU-power is rarely a problem.
.. memory
32-bit systems, in comparison to 64-bit systems, allow you to allocate less amount of virtual memory space and less number of threads.
The theoretical maximum space depends on the operating system (OS) kernel, but 32-bit systems usually are bounded to 4GB.
You may get, however, about 3GB because the OS reserves some space for the kernel.
.. work with the kernel
Varnish uses a workspace-oriented memory-model instead of allocating the exact amount of space it needs at run-time.
Varnish does not manage its allocated memory, but it delegates this task to the OS because the kernel can normally do this task better than a user-space program.
Event filters and notifications facilities such as ``epoll`` and ``kqueue`` are advanced features of the OS that are designed for high-performance services like Varnish.
By using these, Varnish can move a lot of the complexity into the OS kernel which is also better positioned to decide which threads are ready to execute and when.
.. VCL
Varnish uses the *Varnish Configuration Language* (VCL) that allows you to specify exactly how to use and combine the features of Varnish.
VCL is translated to C programming language code.
This code is compiled with a standard C compiler and then dynamically linked directly into Varnish at run-time.
When you need functionalities that VCL does not provide, e.g., look for an IP address in a database, you can write raw C code in your VCL.
That is in-line C in VCL.
However, in-line C is strongly discouraged because in-line C is more difficult to debug, maintain and develop with other developers.
Instead in adding in-line C, you should modularized your C code in Varnish modules, also known as VMODs.
.. VMODs
VMODs are typically coded in VCL and C programming language.
A VMOD is a shared library, which can be called from VCL code.
The standard (``std``) VMOD, for instance, is a VMOD included in Varnish Cache.
These modules let you extend the functionality of VCL by pulling in custom-written features.
Some examples include non-standard header manipulation, access to *memcached* or complex normalization of headers.
`Appendix D: VMOD Development`_ explains in more details how VMODs work and how to develop yours.
.. shared memory
The Varnish Shared memory Log (VSL) allows Varnish to log large amounts of information at almost no cost by having other applications parse the data and extract the useful bits.
This design and other mechanisms decrease lock-contention in the heavily threaded environment of Varnish.
.. TODO for the author: Move to fast track
To summarize: Varnish is designed to run on modern hardware
under real work-loads and to solve real problems. Varnish does not
cater to the "I want to make Varnish run on my 486 just
because"-crowd. If it does work on your 486, then that's fine, but
that's not where you will see our focus. Nor will you see us
sacrifice performance or simplicity for the sake of niche use-cases
that can easily be solved by other means -- like using a 64-bit OS.
How objects are stored
----------------------
- Object: local store of HTTP response message
- Objects in Varnish are stored in memory and addressed by hash keys
- You can control the hashing
- Multiple objects can have the same hash key
.. container:: handout
`Objects` are local stores of response messages as defined in https://tools.ietf.org/html/rfc7234.
They are mapped with a hash key and they are stored in memory.
References to objects in memory are kept in a hash tree.
.. Cache control
A rather unique feature of Varnish is that it allows you to control the input of the hashing algorithm.
The key is by default made out of the HTTP Host header and the URL, which is sufficient and recommended for typical cases.
However, you are able to create the key from something else.
For example, you can use cookies or the user-agent of a client request to create a hash key.
.. multiple objects
HTTP specifies that multiple objects can be served from the same URL, depending on the preferences of the client.
For instance, content in *gzip* format is sent only to clients that indicate *gzip* support.
Varnish stores various objects under one key.
Upon a client request, Varnish selects the object that matches the client preferences.
Object Lifetime
---------------
.. add this figure to the beginning of the book as reference
.. figure 2
.. figure:: ui/img/objectlifetime.svg
:align: center
:width: 100%
Figure :counter:`figure`: Object Lifetime
.. container:: handout
`Figure 2 <#figure-2>`_ shows the lifetime of cached objects.
A cached object has an origin timestamp ``t_origin`` and three duration attributes: 1) ``TTL``, 2) ``grace``, and 3) ``keep``.
``t_origin`` is the time when an object was created in the backend.
An object lives in cache until ``TTL + grace + keep`` elapses.
After that time, the object is removed by the Varnish daemon.
In a timeline, objects within the time-to-live ``TTL`` are considered *fresh objects*.
*Stale objects* are those within the time period ``TTL`` and ``grace``.
Objects within ``t_origin`` and ``keep`` are used when applying conditions with the HTTP header field ``If-Modified-Since``.
The `VCL – vcl_backend_fetch`_ and
`VCL – vcl_backend_response`_ sections explain how Varnish handles backend responses and how these duration attributes affect subsequent actions.
Getting Started
===============
In this chapter, you will:
- learn about the Varnish distribution,
- install Varnish,
- learn about ``varnishtest``, and
- cover basic Varnish configuration.
.. container:: handout
Most of the commands you will type in this course require root privileges.
You can get temporary root privileges by typing ``sudo <command>``, or permanent root privileges by typing ``sudo -i``.
.. backend definition
In Varnish terminology, a *backend* is the origin server.
In other words, it is whatever server Varnish talks to fetch content.
This can be any sort of service as long as it understands HTTP.
Most of the time, Varnish talks to a web server or an application frontend server.
In this book, we use *backend*, *origin server*, *web server* or *application frontend server* depending the context.
Varnish Distribution
--------------------
Utility programs part of the Varnish distribution:
- ``varnishd``
- ``varnishtest``
- ``varnishadm``
- ``varnishlog``
- ``varnishstat``
- and more
.. container:: handout
The Varnish distribution includes several utility programs that you will use in this course.
You will learn how to use these programs as you progress, but it is useful to have a brief introduction about them before we start.
.. varnishd
The central block of Varnish is the Varnish daemon ``varnishd``.
This daemon accepts HTTP requests from clients, sends requests to a backend, caches the returned objects and replies to the client request.
``varnishd`` is further explained in the `Varnish Architecture`_ section.
.. varnishtest
``varnishtest`` is a script driven program used to test your Varnish installation.
``varnishtest`` is very powerful because it allows you to create client mock-ups, fetch content from mock-up or real backends, interact with your actual Varnish configuration, and assert the expected behavior.
``varnishtest`` is also very useful to learn more about the behavior of Varnish.
Therefore, ``varnishtest`` is used throughout the book as main testbed.
.. varnishadm
``varnishadm`` controls a running Varnish instance.
The ``varnishadm`` utility establishes a command line interface (CLI) connection to ``varnishd``.
This utility is the only one that may affect a running instance of Varnish.
You can use ``varnishadm`` to:
- start and stop ``varnishd``,
- change configuration parameters,
- reload the Varnish Configuration Language (VCL),
- view the most up-to-date documentation for parameters, and
- more.
`The Management Interface varnishadm`_ section explains in more detail this utility.
.. varnishlog
The Varnish log provides large amounts of information, thus it is usually necessary to filter it.
For example, "show me only what matches X".
``varnishlog`` does precisely that.
You will learn more about ``varnishlog`` in the `Examining Varnish Server's Output`_ chapter.
.. varnishstat
``varnishstat`` is used to access **global counters**.
It provides overall statistics, e.g the number of total requests, number of objects, and more.
``varnishstat`` is particularly useful when using it together with ``varnishlog`` to analyze your Varnish installation.
The `varnishstat`_ section explains in detail this utility.
.. others
In addition, there are other utility programs such as ``varnishncsa``, ``varnishtop`` and ``varnishhist``.
`Appendix B: Varnish Programs`_ explains them.
Exercise: Install Varnish
-------------------------
- Install Varnish
- Use packages provided by
- varnish-software.com for Varnish Cache Plus
- varnish-cache.org for Varnish Cache
- When you are done, verify your Varnish version, run ``varnishd -V``
.. container:: handout
You may skip this exercise if already have a well configured environment to test Varnish.
In case you get stuck, you may look at the proposed solution.
.. table 3
.. csv-table:: Table :counter:`table`: Different Locations of the Varnish Configuration File
:name: Different Locations of the Varnish Configuration File
:delim: ;
:header-rows: 2
:file: tables/varnish_configuration_files.csv
[1] The file does not exist by default.
Copy it from ``/lib/systemd/system/`` and edit it.
[2] There is no configuration file.
Use the command ``chkconfig varnishlog/varnishncsa on/off`` instead.
[3] There is no configuration file.
Use the command ``systemctl start/stop/enable/disable/ varnishlog/varnishncsa`` instead.
.. container:: handout
The configuration file is used to give parameters and command line arguments to the Varnish daemon.
This file also specifies the location of the VCL file.
Modifications to this file require to run ``service varnish restart`` for the changes to take effect.
The location of the Varnish configuration file depends on the operating system and whether it uses the ``init`` system of `SysV`, or `systemd`.
`Table 3 <#table-3>`_ shows the locations for each system installation.
.. Introduction to apt-get and yum
To install packages on Ubuntu or Debian, use the command ``apt-get install <package>``, e.g., ``apt-get install varnish``.
For CentOS, RHEL or Fedora, use ``yum install <package>``.
You might want to look at `Solution: Install Varnish`_, if you need help.
``varnishtest``
---------------
.. This subsection is based on http://blog.zenika.com/index.php?post/2012/08/27/Introducing-varnishtest.
- Script driven program used to test the configuration of Varnish, run regression tests, and develop VMODs
- Useful for system administrators, web developers, and VMODs developers
.. container:: handout
Varnish is distributed with many utility programs.
``varnishtest`` is a script driven program that allows you create client mock-ups, simulate transactions, fetch content from mock-up or real backends, interact with your actual Varnish configuration and assert expected behaviors.
You can use ``varnishtest`` when configuring your Varnish installation, i.e., writing VCL code, or developing VMODs.
In fact, we recommend you first to write *Varnish Case Tests* (VTCs) as part of your design.
``varnishtest`` is also useful to reproduce bugs when filing a bug report.
This section develops basic knowledge about ``varnishtest``, and you will learn more about it throughout the book.
Another way to learn how to create VTCs is by reading and running the ``.vtc`` files included in Varnish Cache under ``bin/varnishtest/tests/``.
Further documentation of ``varnishtest`` is found in its man page, ``bin/varnishtest/tests/README`` and https://www.varnish-cache.org/docs/trunk/reference/varnishtest.html.
``varnishtest`` has its own language: the *Varnish Test Case (VTC) language*.
This language is fairly simple to understand as we shall see next.
The Varnish Test Case (VTC) Language
....................................
- Test against simulated or real backends
- Starts real instance of ``varnishd``
- Simulates clients
- Asserts using ``expect``
**b00001.vtc**
.. include:: vtc/b00001.vtc
:literal:
.. container:: handout
.. introduction
``varnishtest`` does not follow the unit testing framework (up/test/assert/tear down) nor behavior-driven development (given/when/then).
Depending on your use case, there might be test preparations, executions and assertions all over the place.
VTC is not compiled but simply interpreted on the fly.
.. file naming convention
There is a naming convention for VTC files.
Files starting with ``b`` as the example above contain basic functionality tests.
The naming scheme is in ``Varnish-Cache/bin/varnishtest/tests/README`` or https://raw.githubusercontent.com/varnish/Varnish-Cache/master/bin/varnishtest/tests/README.
.. varnish test name
All VTC programs start by naming the test::
varnishtest "Varnish as Proxy"
.. origin server (backend)
In this example we declare a simulated origin server::
server s1 {
rxreq
txresp
} -start
All server declarations must start with ``s``.
In the code above, ``s1`` receives a request ``rxreq``, and transmits a response ``txresp``.
``-start`` boots ``s1`` and makes available the macros ``${s1_addr}`` and ``${s1_port}`` with the IP address and port of your simulated backend.
You may also start a declaration at a later point in your code, for example ``server s1 -start``.
.. Varnish server
To declare an instance of your real Varnish server::
varnish v1 -arg "-b ${s1_addr}:${s1_port}" -start
``varnish v1`` declares an instance of your real Varnish server, i.e., ``varnishd``.
The names for Varnish servers must start with ``v``.
This instance is controlled by the manager process, and ``-start`` forks a child, which is the actual cacher process.
You will learn about the manager and cacher in `The Parent Process: The Manager`_ and `The Child Process: The Cacher`_ sections.
There are many ways to configure ``varnishd``.
On way is by passing arguments with ``-arg`` as in ``-arg "-b ${s1_addr}:${s1_port}"``.
``-b`` is a ``varnishd`` option to define the backend.
In this case, we use the IP address and port of the simulated backend ``s1``, but you can also use a real backend.
Therefore, ``varnishtest`` can be used as integration tool when testing your real backend.
There are other ways to define backends.
The most common one is perhaps by defining them in your VCL code, as we shall see in the next section.
To simulate a client::
client c1 {
txreq
rxresp
expect resp.http.via ~ "varnish"
} -run
Simulated clients in ``varnishtest`` start with ``c``.
In this example, ``c1`` transmits one request and receives one response.
Since Varnish is a proxy, we expect to receive the response from the backend via Varnish.
Therefore, ``c1`` expects ``varnish`` in the ``via`` HTTP header field.
We use tilde ``~`` as match operator of regular expressions because the exact text in ``resp.http.via`` depends on the Varnish version you have installed.
Finally, you start client ``c1`` with the ``-run`` command.
Synchronization in Varnish Tests
................................
- Varnish is a multi-threaded program
- Use ``-wait`` as synchronization mechanism
- ``-run``: ``-start -wait``
.. container:: handout
You might have noticed that we used ``-start`` for ``v1``, but ``-run`` for ``c1``.
The difference between these commands is that ``-run`` executes ``-start -wait``.
Varnish is a multi-threaded program.
Therefore, each instance in ``varnishtest``, i.e., ``s1``, ``v1`` and ``c1``, is executed by a different thread.
Sometimes, you will need some sort of synchronization mechanism to ensure you avoid race conditions or other non-intuitive behaviors.
For those cases, you can use the ``-wait`` command.
``-wait`` tells the executor of ``varnishtest`` to wait for a given instance to complete before proceeding to the next instruction in your VTC program.
To illustrate this, see the difference between::
varnishtest "Synchronized"
server s1 {
rxreq
txresp
}
server s1 -start
client c1 -connect ${s1_sock} {
txreq
rxresp
}
# -run: -start - wait
client c1 -run
server s1 -wait
and::
varnishtest "Unsynchronized"
server s1 {
rxreq
txresp
}
server s1 -start -wait
client c1 -connect ${s1_sock} {
txreq
rxresp
}
client c1 -run
The second test fails in comparison to the first one, because ``varnishtest`` times out while waiting for ``s1`` to receive a request and transmit a response.
Therefore, you typically start Varnish servers with the ``-start`` command, but start clients with the ``-run`` command.
.. note::
You will learn more about the `Threading Model`_ of Varnish in its own section.