/
04.markmin
2556 lines (1942 loc) · 126 KB
/
04.markmin
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
## The core
### Command line options
It is possible to skip the GUI and start web2py directly from the command line by typing something like:
``password``:inxx
``
python web2py.py -a 'your password' -i 127.0.0.1 -p 8000
``:code
When web2py starts, it creates a file called "parameters_8000.py" where it stores the hashed password. If you use "<ask>" as the password, web2py prompts you for it.
For additional security, you can start web2py with:
``
python web2py.py -a '<recycle>' -i 127.0.0.1 -p 8000
``:code
In this case web2py reuses the previously stored hashed password. If no password is provided, or if the "parameters_8000.py" file is deleted, the web-based administrative interface is disabled.
``PAM``:inxx
On some Unix/Linux systems, if the password is
``
<pam_user:some_user>
``:code
web2py uses the PAM password of the Operating System account of ``some_user`` to authenticate the administrator, unless blocked by the PAM configuration.
-------
web2py normally runs with CPython (the C implementation of the Python interpreter created by Guido van Rossum), but it can also run with Jython (the Java implementation of the interpreter). The latter possibility allows the use of web2py in the context of a J2EE infrastructure. To use Jython, simply replace "python web2py.py ..." with "jython web2py.py". Details about installing Jython, zxJDBC modules required to access the databases can be found in Chapter 14.
-------
The "web2py.py" script can take many command-line arguments specifying the maximum number of threads, enabling of SSL, etc. For a complete list type:
``command line``:inxx
``
>>> python web2py.py -h
Usage: python web2py.py
web2py Web Framework startup script. ATTENTION: unless a password
is specified (-a 'passwd'), web2py will attempt to run a GUI.
In this case command line options are ignored.
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-i IP, --ip=IP ip address of the server (127.0.0.1)
-p PORT, --port=PORT port of server (8000)
-a PASSWORD, --password=PASSWORD
password to be used for administration (use -a
"<recycle>" to reuse the last password))
-c SSL_CERTIFICATE, --ssl_certificate=SSL_CERTIFICATE
file that contains SSL certificate
-k SSL_PRIVATE_KEY, --ssl_private_key=SSL_PRIVATE_KEY
file that contains SSL private key
--ca-cert=SSL_CA_CERTIFICATE
Use this file containing the CA certificate to
validate X509 certificates from clients
-d PID_FILENAME, --pid_filename=PID_FILENAME
file to store the pid of the server
-l LOG_FILENAME, --log_filename=LOG_FILENAME
file to log connections
-n NUMTHREADS, --numthreads=NUMTHREADS
number of threads (deprecated)
--minthreads=MINTHREADS
minimum number of server threads
--maxthreads=MAXTHREADS
maximum number of server threads
-s SERVER_NAME, --server_name=SERVER_NAME
server name for the web server
-q REQUEST_QUEUE_SIZE, --request_queue_size=REQUEST_QUEUE_SIZE
max number of queued requests when server unavailable
-o TIMEOUT, --timeout=TIMEOUT
timeout for individual request (10 seconds)
-z SHUTDOWN_TIMEOUT, --shutdown_timeout=SHUTDOWN_TIMEOUT
timeout on shutdown of server (5 seconds)
--socket-timeout=SOCKET_TIMEOUT
timeout for socket (5 second)
-f FOLDER, --folder=FOLDER
folder from which to run web2py
-v, --verbose increase --test verbosity
-Q, --quiet disable all output
-D DEBUGLEVEL, --debug=DEBUGLEVEL
set debug output level (0-100, 0 means all, 100 means
none; default is 30)
-S APPNAME, --shell=APPNAME
run web2py in interactive shell or IPython (if
installed) with specified appname (if app does not
exist it will be created). APPNAME like a/c/f (c,f
optional)
-B, --bpython run web2py in interactive shell or bpython (if
installed) with specified appname (if app does not
exist it will be created). Use combined with --shell
-P, --plain only use plain python shell; should be used with
--shell option
-M, --import_models auto import model files; default is False; should be
used with --shell option
-R PYTHON_FILE, --run=PYTHON_FILE
run PYTHON_FILE in web2py environment; should be used
with --shell option
-K SCHEDULER, --scheduler=SCHEDULER
run scheduled tasks for the specified apps: expects a
list of app names as -K app1,app2,app3 or a list of
app:groups as -K app1:group1:group2,app2:group1 to
override specific group_names. (only strings, no
spaces allowed. Requires a scheduler defined in the
models
-X, --with-scheduler run schedulers alongside webserver
-T TEST_PATH, --test=TEST_PATH
run doctests in web2py environment; TEST_PATH like
a/c/f (c,f optional)
-W WINSERVICE, --winservice=WINSERVICE
-W install|start|stop as Windows service
-C, --cron trigger a cron run manually; usually invoked from a
system crontab
--softcron triggers the use of softcron
-Y, --run-cron starts cron
-J, --cronjob identify cron-initiated command
-L CONFIG, --config=CONFIG
config file
-F PROFILER_FILENAME, --profiler=PROFILER_FILENAME
profiler filename
-t, --taskbar use web2py gui and run in taskbar (system tray)
--nogui text-only, no GUI
-A ARGS, --args=ARGS should be followed by a list of arguments to be passed
to script, to be used with -S, -A must be the last
option
--no-banner Do not print header banner
--interfaces=INTERFACES
listen on multiple addresses: "ip:port:cert:key:ca_cer
t;ip2:port2:cert2:key2:ca_cert2;..." (:cert:key
optional; no spaces)
--run_system_tests runs web2py tests
``:code
Lower-case options are used to configure the web server. The ``-L`` option tells web2py to read configuration options from a file, ``-W`` installs web2py as a windows service, while ``-S``, ``-P`` and ``-M`` options start an interactive Python shell. The ``-T`` option finds and runs controller doctests in a web2py execution environment. For example, the following example runs doctests from all controllers in the "welcome" application:
``
python web2py.py -vT welcome
``:code
if you run web2py as Windows Service, ``-W``, it is not convenient to pass the configuration using command line arguments. For this reason, in the web2py folder there is a sample "options_std.py" configuration file for the internal web server:
``
import socket
import os
ip = '0.0.0.0'
port = 80
interfaces=[('0.0.0.0',80)]
#interfaces.append(('0.0.0.0',443,'ssl_private_key.pem','ssl_certificate.pem'))
password = '<recycle>' # ## <recycle> means use the previous password
pid_filename = 'httpserver.pid'
log_filename = 'httpserver.log'
profiler_filename = None
minthreads = None
maxthreads = None
server_name = socket.gethostname()
request_queue_size = 5
timeout = 30
shutdown_timeout = 5
folder = os.getcwd()
extcron = None
runcron = False
``:code
This file contains the web2py defaults. If you edit this file, you need to import it explicitly with the ``-L`` command-line option. It only works if you run web2py as a Windows Service.
### Workflow
The web2py workflow is the following:
- An HTTP requests arrives to the web server (the built-in Rocket server or a different server connected to web2py via WSGI or another adapter). The web server handles each request in its own thread, in parallel.
- The HTTP request header is parsed and passed to the dispatcher (explained later in this chapter).
- The dispatcher decides which of the installed application will handle the request and maps the PATH_INFO in the URL into a function call. Each URL corresponds to one function call.
- Requests for files in the static folder are handled directly, and large files are automatically streamed to the client.
- Requests for anything but a static file are mapped into an action (i.e. a function in a controller file, in the requested application).
- Before calling the action, a few things happen: if the request header contains a session cookie for the app, the session object is retrieved; if not, a session id is created (but the session file is not saved until later); an execution environment for the request is created; models are executed in this environment.
- Finally the controller action is executed in the pre-built environment.
- If the action returns a string, this is returned to the client (or if the action returns a web2py HTML helper object, it is serialized and returned to the client).
- If the action returns an iterable, this is used to loop and stream the data to the client.
- If the action returns a dictionary, web2py tries to locate a view to render the dictionary. The view must have the same name as the action (unless specified otherwise) and the same extension as the requested page (defaults to .html); on failure, web2py may pick up a generic view (if available and if enabled). The view sees every variable defined in the models as well as those in the dictionary returned by the action, but does not see global variables defined in the controller.
- The entire user code is executed in a single transaction unless specified otherwise.
- If the user code succeeds, the transaction is committed.
- If the user code fails, the traceback is stored in a ticket, and a ticket ID is issued to the client. Only the system administrator can search and read the tracebacks in tickets.
There are some caveats to keep in mind:
- Models in the same folder/subfolder are executed in alphabetical order.
- Any variable defined in a model will be visible to other models following alphabetically, to the controllers, and to the views.
- Models in subfolders are executed conditionally. For example, if the user has requested "/a/c/f" where "a" is the application, "c" is the controller, and "f" is the function (action), then the following models are executed:
``
applications/a/models/*.py
applications/a/models/c/*.py
applications/a/models/c/f/*.py
``
- The requested controller is executed and the requested function is called. This means all top-level code in the controller is also executed at every request for that controller.
- The view is only called if the action returns a dictionary.
- If a view is not found, web2py tries to use a generic view. By default, generic views are disabled, although the 'welcome' app includes a line in /models/db.py to enable them on localhost only. They can be enabled per extension type and per action (using ``response.generic_patterns``). In general, generic views are a development tool and typically should not be used in production. If you want some actions to use a generic view, list those actions in ``response.generic_patterns`` (discussed in more detail in the chapter on Services).
The possible behaviors of an action are the following:
**Return a string**
``
def index(): return 'data'
``
**Return a dictionary for a view**:
``
def index(): return dict(key='value')
``
**Return all local variables**:
``
def index(): return locals()
``
**Redirect the user to another page**:
``
def index(): redirect(URL('other_action'))
``
**Return an HTTP page other than "200 OK"**:
``
def index(): raise HTTP(404)
``
**Return a helper** (for example, a FORM):
``
def index(): return FORM(INPUT(_name='test'))
``
(this is mostly used for Ajax callbacks and components, see chapter 12)
When an action returns a dictionary, it may contain code generated by helpers, including forms based on database tables or forms from a factory, for example:
``
def index(): return dict(form=SQLFORM.factory(Field('name')).process())
``
(all forms generated by web2py use postbacks, see chapter 3)
### Dispatching
``url mapping``:inxx ``dispatching``:inxx
web2py maps a URL of the form:
``
http://127.0.0.1:8000/a/c/f.html
``:code
to the function ``f()`` in controller "c.py" in application "a". If ``f`` is not present, web2py defaults to the ``index`` controller function. If ``c`` is not present, web2py defaults to the "default.py" controller, and if ``a`` is not present, web2py defaults to the ``init`` application. If there is no ``init`` application, web2py tries to run the ``welcome`` application. This is shown schematically in the image below:
(The names of the default application, controller and function can be overridden in routes.py; see ''[[Default Application, Controller and Function #Default-Application,-Controller-and-Function]]'' below.
[[image @///image/en5700.png center 480px]]
By default, any new request also creates a new session. In addition, a session cookie is returned to the client browser to keep track of the session.
The extension ``.html`` is optional; ``.html`` is assumed as default. The extension determines the extension of the view that renders the output of the controller function ``f()``. It allows the same content to be served in multiple formats (html, xml, json, rss, etc.).
-------
Functions that take arguments or start with a double underscore are not publicly exposed and can only be called by other functions.
-------
``static files``:inxx
There is an exception made for URLs of the form:
``
http://127.0.0.1:8000/a/static/filename
``:code
There is no controller called "static". web2py interprets this as a request for the file called "filename" in the subfolder "static" of the application "a".
``PARTIAL CONTENT``:inxx ``IF_MODIFIED_SINCE``:inxx
When static files are downloaded, web2py does not create a session, nor does it issue a cookie or execute the models. web2py always streams static files in chunks of 1MB, and sends PARTIAL CONTENT when the client sends a RANGE request for a subset of the file.
web2py also supports the IF_MODIFIED_SINCE protocol, and does not send the file if it is already stored in the browser's cache and if the file has not changed since that version.
When linking to an audio or video file in the static folder, if you want to force the browser to download the file instead of streaming the audio/video via a media player, add ``?attachment`` to the URL. This tells web2py to set the ``Content-Disposition`` header of the HTTP response to "attachment". For example:
``
<a href="/app/static/my_audio_file.mp3?attachment">Download</a>
``:code
When the above link is clicked, the browser will prompt the user to download the MP3 file rather than immediately streaming the audio. (As discussed [[below #response]], you can also set HTTP response headers directly by assigning a ``dict`` of header names and their values to ``response.headers``.)
``request.application``:inxx ``request.controller``:inxx ``request.function``:inxx ``GET``:inxx ``POST``:inxx ``request.args``:inxx
web2py maps GET/POST requests of the form:
``
http://127.0.0.1:8000/a/c/f.html/x/y/z?p=1&q=2
``:code
to function ``f`` in controller "c.py" in application ``a``, and it stores the URL parameters in the ``request`` variable as follows:
``
request.args = ['x', 'y', 'z']
``:code
and:
``
request.vars = {'p':1, 'q':2}
``:code
and:
``
request.application = 'a'
request.controller = 'c'
request.function = 'f'
``:code
In the above example, both ``request.args[i]`` and ``request.args(i)`` can be used to retrieve the i-th element of the ``request.args``, but while the former raises an exception if the list does not have such an index, the latter returns None in this case.
``request.url``:inxx
``
request.url
``:code
stores the full URL of the current request (not including GET variables).
``request.ajax``:inxx ``request.cid``:inxx
``
request.ajax
``:code
defaults False but it is True if web2py determines that the action was called by an Ajax request.
If the request is an Ajax request and it is initiated by a web2py component, the name of the component can be found in:
``
request.cid
``:code
Components are discussed in more detail in Chapter 12.
``request.get_vars``:inxx ``request.post_vars``:inxx ``request.vars``:inxx
If the HTTP request is a GET, then ``request.env.request_method`` is set to "GET"; if it is a POST, ``request.env.request_method`` is set to "POST". URL query variables are stored in the ``request.vars`` Storage dictionary; they are also stored in ``request.get_vars`` (following a GET request) or ``request.post_vars`` (following a POST request).
web2py stores WSGI and web2py environment variables in ``request.env``, for example:
``
request.env.path_info = 'a/c/f'
``:code
and HTTP headers into environment variables, for example:
``
request.env.http_host = '127.0.0.1:8000'
``:code
-------
Notice that web2py validates all URLs to prevent directory traversal attacks.
-------
URLs are only allowed to contain alphanumeric characters, underscores, and slashes; the ``args`` may contain non-consecutive dots. Spaces are replaced by underscores before validation. If the URL syntax is invalid, web2py returns an HTTP 400 error message``http:w,http:o``:cite .
If the URL corresponds to a request for a static file, web2py simply reads and returns (streams) the requested file.
If the URL does not request a static file, web2py processes the request in the following order:
- Parses cookies.
- Creates an environment in which to execute the function.
- Initializes ``request``, ``response``, ``cache``.
- Opens the existing ``session`` or creates a new one.
- Executes the models belonging to the requested application.
- Executes the requested controller action function.
- If the function returns a dictionary, executes the associated view.
- On success, commits all open transactions.
- Saves the session.
- Returns an HTTP response.
Notice that the controller and the view are executed in different copies of the same environment; therefore, the view does not see the controller, but it sees the models and it sees the variables returned by the controller action function.
If an exception (other than HTTP) is raised, web2py does the following:
- Stores the traceback in an error file and assigns a ticket number to it.
- Rolls back all open transactions.
- Returns an error page reporting the ticket number.
If the exception is an ``HTTP`` exception, this is assumed to be the intended behavior (for example, an ``HTTP`` redirect), and all open database transactions are committed. The behavior after that is specified by the ``HTTP`` exception itself. The ``HTTP`` exception class is not a standard Python exception; it is defined by web2py.
### Libraries
The web2py libraries are exposed to the user applications as global objects. For example (``request``, ``response``, ``session``, ``cache``), classes (helpers, validators, DAL API), and functions (``T`` and ``redirect``).
These objects are defined in the following core files:
``
web2py.py
gluon/__init__.py gluon/highlight.py gluon/restricted.py gluon/streamer.py
gluon/admin.py gluon/html.py gluon/rewrite.py gluon/template.py
gluon/cache.py gluon/http.py gluon/rocket.py gluon/storage.py
gluon/cfs.py gluon/import_all.py gluon/sanitizer.py gluon/tools.py
gluon/compileapp.py gluon/languages.py gluon/serializers.py gluon/utils.py
gluon/contenttype.py gluon/main.py gluon/settings.py gluon/validators.py
gluon/dal.py gluon/myregex.py gluon/shell.py gluon/widget.py
gluon/decoder.py gluon/newcron.py gluon/sql.py gluon/winservice.py
gluon/fileutils.py gluon/portalocker.py gluon/sqlhtml.py gluon/xmlrpc.py
gluon/globals.py gluon/reserved_sql_keywords.py
``:code
The tar gzipped scaffolding app that ship with web2py is
``
welcome.w2p
``:code
It is created upon installation and overwritten on upgrade.
-------
The first time you start web2py, two new folders are created: deposit and applications. The "welcome" app is zipped into a "welcome.w2p" file to be used as a scaffolding app.
The first time you start web2py, two new folders are created: deposit and applications. The "welcome" app is zipped into a "welcome.w2p" file to be used as scaffolding app.
The deposit folder is used as temporary storage for installing and uninstalling applications.
-------
web2py unit-tests are in
``
gluon/tests/
``:code
There are handlers for connecting with various web servers:
``
cgihandler.py # discouraged
gaehandler.py # for Google App Engine
fcgihandler.py # for FastCGI
wsgihandler.py # for WSGI
isapiwsgihandler.py # for IIS
modpythonhandler.py # deprecated
``:code
("fcgihandler" calls "gluon/contrib/gateways/fcgi.py" developed by Allan Saddi) and
``
anyserver.py
``
which is a script to interface with many different web servers, described in Chapter 13.
There are three example files:
``
options_std.py
routes.example.py
router.example.py
``:code
The former is an optional configuration file that can be passed to web2py.py with the ``-L`` option. The second is an example of a URL mapping file. It is loaded automatically when renamed "routes.py". The third is an alternative syntax for URL mapping, and can also be renamed (or copied to) "routes.py".
The files
``
app.yaml
index.yaml
queue.yaml
``:code
are configuration files used for deployment on the Google App Engine. You can read more about them in the Deployment Recipes chapter and on the Google Documentation pages.
There are also additional libraries, usually developed by a third party:
**feedparser**``feedparser``:cite by Mark Pilgrim for reading RSS and Atom feeds:
``
gluon/contrib/__init__.py
gluon/contrib/feedparser.py
``:code
**markdown2**``markdown2``:cite by Trent Mick for wiki markup:
``
gluon/contrib/markdown/__init__.py
gluon/contrib/markdown/markdown2.py
``:code
**markmin** markup:
``
gluon/contrib/markmin.py
``:code
**fpdf** created my Mariano Reingart for generating PDF documents:
``
gluon/contrib/fpdf
``
This is not documented in this book but it is hosted and documented here:
``
http://code.google.com/p/fpdf/
``
**pysimplesoap** is a lightweight SOAP server implementation created by Mariano Reingart:
``
gluon/contrib/pysimplesoap/
``:code
**simplejsonrpc** is a lightweight JSON-RPC client also created by Mariano Reingart: ``jsonrpc``:inxx
``
gluon/contrib/simplejsonrpc.py
``
**memcache**``memcache``:cite Python API by Evan Martin:
``
gluon/contrib/memcache/__init__.py
gluon/contrib/memcache/memcache.py
``
**redis_cache** ``redis``:inxx is a module to store cache in the redis database:
``
gluon/contrib/redis_cache.py
``
**gql**, a port of the DAL to the Google App Engine:
``
gluon/contrib/gql.py
``:code
**memdb**, a port of the DAL on top of memcache:
``
gluon/contrib/memdb.py
``:code
**gae_memcache** is an API to use memcache on the Google App Engine:
``
gluon/contrib/gae_memcache.py
``:code
**pyrtf**``pyrtf``:cite for generating Rich Text Format (RTF) documents, developed by Simon Cusack and revised by Grant Edwards:
``
gluon/contrib/pyrtf
gluon/contrib/pyrtf/__init__.py
gluon/contrib/pyrtf/Constants.py
gluon/contrib/pyrtf/Elements.py
gluon/contrib/pyrtf/PropertySets.py
gluon/contrib/pyrtf/README
gluon/contrib/pyrtf/Renderer.py
gluon/contrib/pyrtf/Styles.py
``:code
**PyRSS2Gen**``pyrss2gen``:cite developed by Dalke Scientific Software, to generate RSS feeds:
``
gluon/contrib/rss2.py
``:code
**simplejson**``simplejson``:cite by Bob Ippolito, the standard library for parsing and writing JSON objects:
``
gluon/contrib/simplejson/__init__.py
gluon/contrib/simplejson/decoder.py
gluon/contrib/simplejson/encoder.py
gluon/contrib/simplejson/jsonfilter.py
gluon/contrib/simplejson/scanner.py
``:code
**Google Wallet** ``googlewallet``:cite
provides "pay now" buttons which link Google as payment processor:
``
gluon/contrib/google_wallet.py
``:code
**Stripe.com** ``stripe``:cite provides a simple API for accepting credit card payments:
``
gluon/contrib/stripe.py
``:code
**AuthorizeNet** ``authorizenet``:cite provides API to accept credit card payments via Authorize.net network
``
gluon/contrib/AuthorizeNet.py
``:code
**Dowcommerce** ``dowcommerce``:cite is yet another credit cart processing API:
``
gluon/contrib/DowCommerce.py
``:code
**PAM**``PAM``:cite authentication API created by Chris AtLee:
``
gluon/contrib/pam.py
``:code
A Bayesian classifier to populate the database with dummy data for testing purposes:
``
gluon/contrib/populate.py
``:code
A file that allows interaction with the taskbar in windows, when web2py is running as a service:
``
gluon/contrib/taskbar_widget.py
``:code
Optional **login_methods** and login_forms to be used for authentication:
``
gluon/contrib/login_methods/__init__.py
gluon/contrib/login_methods/basic_auth.py
gluon/contrib/login_methods/cas_auth.py
gluon/contrib/login_methods/dropbox_account.py
gluon/contrib/login_methods/email_auth.py
gluon/contrib/login_methods/extended_login_form.py
gluon/contrib/login_methods/gae_google_account.py
gluon/contrib/login_methods/ldap_auth.py
gluon/contrib/login_methods/linkedin_account.py
gluon/contrib/login_methods/loginza.py
gluon/contrib/login_methods/oauth10a_account.py
gluon/contrib/login_methods/oauth20_account.py
gluon/contrib/login_methods/openid_auth.py
gluon/contrib/login_methods/pam_auth.py
gluon/contrib/login_methods/rpx_account.py
gluon/contrib/login_methods/x509_auth.py
``:code
web2py also contains a folder with useful scripts including
``
scripts/setup-web2py-fedora.sh
scripts/setup-web2py-ubuntu.sh
scripts/setup-web2py-nginx-uwsgi-ubuntu.sh
scripts/update-web2py.sh
scripts/make_min_web2py.py
...
scripts/sessions2trash.py
scripts/sync_languages.py
scripts/tickets2db.py
scripts/tickets2email.py
...
scripts/extract_mysql_models.py
scripts/extract_pgsql_models.py
...
scripts/access.wsgi
scripts/cpdb.py
``:code
The first three are particularly useful because they attempt a complete installation and setup of a web2py production environment from scratch.
Some of these are discussed in Chapter 14, but all of them contain a documentation string inside that explains their purpose and usage.
Finally web2py includes these files required to build the binary distributions.
``
Makefile
setup_exe.py
setup_app.py
``:code
These are setup scripts for **py2exe** and **py2app**, respectively, and they are only required to build the binary distributions of web2py. YOU SHOULD NEVER NEED TO RUN THEM.
In summary, web2py libraries provide the following functionality:
- Map URLs into function calls.
- Handle passing and returning parameters via HTTP.
- Perform validation of those parameters.
- Protect the applications from most security issues.
- Handle data persistence (database, session, cache, cookies).
- Perform string translations for various supported languages.
- Generate HTML programmatically (e.g. from database tables).
- Generate SQL via a Database Abstraction Layer (DAL).
- Generate Rich Text Format (RTF) output.
- Generate Comma-Separated Value (CSV) output from database tables.
- Generate Really Simple Syndication (RSS) feeds.
- Generate JavaScript Object Notation (JSON) serialization strings for Ajax.
- Translate wiki markup (Markdown) to HTML.
- Expose XML-RPC web services.
- Upload and download large files via streaming.
web2py applications contain additional files, particularly third-party JavaScript libraries, such as jQuery, calendar/datepicker, EditArea and nicEdit. Their authors are acknowledged in the files themselves.
### Applications
Applications developed in web2py are composed of the following parts:
- **models** describe a representation of the data as database tables and relations between tables.
- **controllers** describe the application logic and workflow.
- **views** describe how data should be presented to the user using HTML and JavaScript.
- **languages** describe how to translate strings in the application into various supported languages.
- **static files** do not require processing (e.g. images, CSS stylesheets, etc).
- **ABOUT** and **README** documents are self-explanatory.
- **errors** store error reports generated by the application.
- **sessions** store information related to each particular user.
- **databases** store SQLite databases and additional table information.
- **cache** store cached application items.
- **modules** are other optional Python modules.
- **private** files are accessed by the controllers but not directly by the developer.
- **uploads** files are accessed by the models but not directly by the developer (e.g., files uploaded by users of the application).
- **tests** is a directory for storing test scripts, fixtures and mocks.
Models, views, controllers, languages, and static files are accessible via the web administration [design] interface. ABOUT, README, and errors are also accessible via the administration interface through the corresponding menu items. Sessions, cache, modules and private files are accessible to the applications but not via the administration interface.
Everything is neatly organized in a clear directory structure that is replicated for every installed web2py application, although the user never needs to access the filesystem directly:
``about``:inxx ``license``:inxx ``cache``:inxx ``controllers``:inxx ``databases``:inxx ``errors``:inxx ``languages``:inxx ``models``:inxx ``modules``:inxx ``private``:inxx ``session``:inxx ``static``:inxx ``tests``:inxx ``uploads``:inxx ``views``:inxx ``__init__.py``:inxx
``
__init__.py ABOUT LICENSE models views
controllers modules private tests cron
cache errors upload sessions static
``:code
"__init__.py" is an empty file which is required in order to allow Python (and web2py) to import the modules in the ``modules`` directory.
Notice that the **admin** application simply provides a web interface to web2py applications on the server file system. web2py applications can also be created and developed from the command-line; you don't have to use the browser **admin** interface. A new application can be created manually by replicating the above directory structure under ,e.g., "applications/newapp/" (or simply untar the ``welcome.w2p`` file into your new application directory). Application files can also be created and edited from the command-line without having to use the web **admin** interface.
### API
Models, controllers, and views are executed in an environment where the following objects are already imported for us:
**Global Objects:** ``request``:inxx ``response``:inxx ``session``:inxx ``cache``:inxx
``
request, response, session, cache
``:code
**Internationalization:** ``T``:inxx ``internationalization``:inxx
``
T
``:code
**Navigation:** ``redirect``:inxx ``HTTP``:inxx
``
redirect, HTTP
``:code
**Helpers:** ``helpers``:inxx
``
XML, URL, BEAUTIFY
A, B, BODY, BR, CENTER, CODE, COL, COLGROUP,
DIV, EM, EMBED, FIELDSET, FORM, H1, H2, H3, H4, H5, H6,
HEAD, HR, HTML, I, IFRAME, IMG, INPUT, LABEL, LEGEND,
LI, LINK, OL, UL, META, OBJECT, OPTION, P, PRE,
SCRIPT, OPTGROUP, SELECT, SPAN, STYLE,
TABLE, TAG, TD, TEXTAREA, TH, THEAD, TBODY, TFOOT,
TITLE, TR, TT, URL, XHTML, xmlescape, embed64
CAT, MARKMIN, MENU, ON
``:code
**Forms and tables**
``
SQLFORM (SQLFORM.factory, SQLFORM.grid, SQLFORM.smartgrid)
``
**Validators:** ``validators``:inxx
``
CLEANUP, CRYPT, IS_ALPHANUMERIC, IS_DATE_IN_RANGE, IS_DATE,
IS_DATETIME_IN_RANGE, IS_DATETIME, IS_DECIMAL_IN_RANGE,
IS_EMAIL, IS_EMPTY_OR, IS_EXPR, IS_FLOAT_IN_RANGE, IS_IMAGE,
IS_IN_DB, IS_IN_SET, IS_INT_IN_RANGE, IS_IPV4, IS_LENGTH,
IS_LIST_OF, IS_LOWER, IS_MATCH, IS_EQUAL_TO, IS_NOT_EMPTY,
IS_NOT_IN_DB, IS_NULL_OR, IS_SLUG, IS_STRONG, IS_TIME,
IS_UPLOAD_FILENAME, IS_UPPER, IS_URL
``:code
**Database:** ``DAL``:inxx
``
DAL, Field
``:code
For backward compatibility ``SQLDB=DAL`` and ``SQLField=Field``. We encourage you to use the new syntax ``DAL`` and ``Field``, instead of the old syntax.
Other objects and modules are defined in the libraries, but they are not automatically imported since they are not used as often.
The core API entities in the web2py execution environment are ``request``, ``response``, ``session``, ``cache``, ``URL``, ``HTTP``, ``redirect`` and ``T`` and are discussed below.
A few objects and functions, including **Auth**, **Crud** and **Service**, are defined in "gluon/tools.py" and they need to be imported as necessary:
``
from gluon.tools import Auth, Crud, Service
``:code
#### Accessing the API from Python modules
Your models or controller may import python modules, and these may need to use some of the web2py API. The way for them to do it is importing them:
``
from gluon import *
``
In fact, any Python module, even if not imported by a web2py application, can import the web2py API as long as web2py is in the ``sys.path``.
There is one caveat, though. Web2py defines some global objects (request, response, session, cache, T) that can only exist when an HTTP request is present (or is faked). Therefore, modules can access them only if they are called from an application. For this reasons they are placed into a container caller ``current``, which is a thread local object. Here is an example.
Create a module "/myapp/modules/test.py" that contains:
``
from gluon import *
def ip(): return current.request.client
``
Now from a controller in "myapp" you can do
``
import test
def index():
return "Your ip is " + test.ip()
``
Notice a few things:
- ``import test`` looks for the module first in the current app's modules folder, then in the folders listed in ``sys.path``. Therefore, app-level modules always take precedence over Python modules. This allows different apps to ship with different versions of their modules, without conflicts.
- Different users can call the same action ``index`` concurrently, which calls the function in the module, and yet there is no conflict because ``current.request`` is a different object in different threads. Just be careful not to access ``current.request`` outside of functions or classes (i.e., at the top level) in the module.
- ``import test`` is a shortcut for ``from applications.appname.modules import test``. Using the longer syntax, it is possible to import modules from other applications.
For uniformity with normal Python behavior, by default web2py does not reload modules when changes are made. Yet this can be changed. To turn on the auto-reload feature for modules, use the ``track_changes`` function as follows (typically in a model file, before any imports):
``
from gluon.custom_import import track_changes; track_changes(True)
``:code
From now on, every time a module is imported, the importer will check if the Python source file (.py) has changed. If it has changed, the module will be reloaded. This applies to all Python modules, even Python modules outside web2py. The mode is global and applies to all applications. Changes made to models, controllers, and views are always reloaded regardless of the mode used. To turn the mode off, use the same function with ``False`` as the argument. To know the actual tracking state, use the ``is_tracking_changes()`` function, also from ``gluon.custom_import``.
Modules that import ``current`` can access:
- ``current.request``
- ``current.response``
- ``current.session``
- ``current.cache``
- ``current.T``
and any other variable your application chooses to store in current. For example a model could do
``
auth = Auth(db)
from gluon import current
current.auth = auth
``
and now all modules imported can access
- ``current.auth``
``current`` and ``import`` create a powerful mechanism to build extensible and reusable modules for your applications.
-------
There is one major caveat. Given ``from gluon import current``, it is correct to use ``current.request`` and any of the other thread local objects but one should never assign them to global variables in the module, such as in
``
request = current.request # WRONG! DANGER!
``
nor one should use it assign class attributes
``
class MyClass:
request = current.request # WRONG! DANGER!
``
This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.
-------
### ``request``
``request``:inxx ``Storage``:inxx ``request.cookies``:inxx ``user_agent``:inxx
The ``request`` object is an instance of the ubiquitous web2py class that is called ``gluon.storage.Storage``, which extends the Python ``dict`` class. It is basically a dictionary, but the item values can also be accessed as attributes:
``
request.vars
``:code
is the same as:
``
request['vars']
``:code
Unlike a dictionary, if an attribute (or key) does not exist, it does not raise an exception. Instead, it returns ``None``.
-----
It is sometimes useful to create your own Storage objects. You can do so as follows:
``
from gluon.storage import Storage
my_storage = Storage() # empty storage object
my_other_storage = Storage(dict(a=1, b=2)) # convert dictionary to Storage
``:code
-----
``request`` has the following items/attributes, some of which are also an instance of the ``Storage`` class:
- ``request.cookies``: a ``Cookie.SimpleCookie()`` object containing the cookies passed with the HTTP request. It acts like a dictionary of cookies. Each cookie is a Morsel object.
- ``request.env``: a ``Storage`` object containing the environment variables passed to the controller, including HTTP header variables from the HTTP request and standard WSGI parameters. The environment variables are all converted to lower case, and dots are converted to underscores for easier memorization.
- ``request.application``: the name of the requested application (parsed from ``request.env.path_info``).
- ``request.controller``: the name of the requested controller (parsed from the ``request.env.path_info``).
- ``request.function``: the name of the requested function (parsed from the ``request.env.path_info``).
- ``request.extension``: the extension of the requested action. It defaults to "html". If the controller function returns a dictionary and does not specify a view, this is used to determine the extension of the view file that will render the dictionary (parsed from the ``request.env.path_info``).
- ``request.folder``: the application directory. For example if the application is "welcome", ``request.folder`` is set to the absolute path "/path/to/welcome". In your programs, you should always use this variable and the ``os.path.join`` function to build paths to the files you need to access. Although web2py always uses absolute paths, it is a good rule never to explicitly change the current working folder (whatever that is) since this is not a thread-safe practice.
- ``request.now``: a ``datetime.datetime`` object storing the datetime of the current request.
- ``request.utcnow``: a ``datetime.datetime`` object storing the UTC datetime of the current request.
- ``request.args``: A list of the URL path components following the controller function name; equivalent to ``request.env.path_info.split('/')[3:]``
- ``request.vars``: a ``gluon.storage.Storage`` object containing the HTTP GET and HTTP POST query variables.
- ``request.get_vars``: a ``gluon.storage.Storage`` object containing only the HTTP GET query variables.
- ``request.post_vars``: a ``gluon.storage.Storage`` object containing only the HTTP POST query variables.
- ``request.client``: The ip address of the client as determined by, if present, ``request.env.http_x_forwarded_for`` or by ``request.env.remote_addr`` otherwise. While this is useful it should not be trusted because the ``http_x_forwarded_for`` can be spoofed.
- ``request.is_local``: ``True`` if the client is localhost, ``False`` otherwise. Should work behind a proxy if the proxy supports ``http_x_forwarded_for``.
- ``request.is_https``: ``True`` if the request is using the HTTPS protocol, ``False`` otherwise.
- ``request.body``: a read-only file stream that contains the body of the HTTP request. This is automatically parsed to get the ``request.post_vars`` and then rewinded. It can be read with ``request.body.read()``.
- ``request.ajax`` is True if the function is being called via an Ajax request.
- ``request.cid`` is the ``id`` of the component that generated the Ajax request (if any). You can read more about components in Chapter 12.
- ``request.restful`` this is a new and very useful decorator that can be used to change the default behavior of web2py actions by separating GET/POST/PUSH/DELETE requests. It will be discussed in some detail in Chapter 10.
- ``request.user_agent()`` parses the user_agent field from the client and returns the information in the form of a dictionary. It is useful to detect mobile devices. It uses "gluon/contrib/user_agent_parser.py" created by Ross Peoples. To see what it does, try to embed the following code in a view:
``
{{=BEAUTIFY(request.user_agent())}}
``:code
- ``request.wsgi`` is a hook that allows you to call third party WSGI applications from inside actions
The latter includes:
- ``request.wsgi.environ``
- ``request.wsgi.start_response``
- ``request.wsgi.middleware``
their usage is discussed at the end of this Chapter.
As an example, the following call on a typical system:
``
http://127.0.0.1:8000/examples/default/status/x/y/z?p=1&q=2
``:code
results in the following ``request`` object:
``request``:inxx ``env``:inxx
----------
variable | value
request.application | examples
request.controller | default
request.function | index
request.extension | html
request.view | status
request.folder | applications/examples/
request.args | ['x', 'y', 'z']
request.vars | <Storage {'p': 1, 'q': 2}>
request.get_vars | <Storage {'p': 1, 'q': 2}>
request.post_vars | <Storage {}>
request.is_local | False
request.is_https | False
request.ajax | False
request.cid | None
request.wsgi | hook
request.env.content_length | 0
request.env.content_type | ````
request.env.http_accept | text/xml,text/html;
request.env.http_accept_encoding | gzip, deflate
request.env.http_accept_language | en
request.env.http_cookie | session_id_examples=127.0.0.1.119725
request.env.http_host | 127.0.0.1:8000
request.env.http_max_forwards | 10
request.env.http_referer | http://web2py.com/
request.env.http_user_agent | Mozilla/5.0
request.env.http_via | 1.1 web2py.com
request.env.http_x_forwarded_for | 76.224.34.5
request.env.http_x_forwarded_host | web2py.com
request.env.http_x_forwarded_server | 127.0.0.1
request.env.path_info | /examples/simple_examples/status
request.env.query_string | remote_addr:127.0.0.1
request.env.request_method | GET
request.env.script_name | ````
request.env.server_name | 127.0.0.1
request.env.server_port | 8000
request.env.server_protocol | HTTP/1.1
request.env.web2py_path | /Users/mdipierro/web2py
request.env.web2py_version | Version 1.99.1
request.env.web2py_runtime_gae | (optional, defined only if GAE detected)
request.env.wsgi_errors | <open file, mode 'w' at >
request.env.wsgi_input | ````
request.env.wsgi_multiprocess | False
request.env.wsgi_multithread | True
request.env.wsgi_run_once | False
request.env.wsgi_url_scheme | http
request.env.wsgi_version | 10
--------
Which environment variables are actually defined depends on the web server. Here we are assuming the built-in Rocket wsgi server. The set of variables is not much different when using the Apache web server.
The ``request.env.http_*`` variables are parsed from the request HTTP header.
The ``request.env.web2py_*`` variables are not parsed from the web server environment, but are created by web2py in case your applications need to know about the web2py location and version, and whether it is running on the Google App Engine (because specific optimizations may be necessary).
Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi adaptor.
### ``response``
``response``:inxx
``response.body``:inxx
``response.cookies``:inxx
``response.download``:inxx
``response.files``:inxx
``response.flash``:inxx
``response.headers``:inxx
``response.meta``:inxx
``response.menu``:inxx
``response.postprocessing``:inxx
``response.render``:inxx
``response.status``:inxx
``response.stream``:inxx
``response.subtitle``:inxx
``response.title``:inxx
``response.toolbar``:inxx
``response.view``:inxx
``response.delimiters``:inxx
``response.js``:inxx
``response.write``:inxx
``response.include_files``:inxx
``response.include_meta``:inxx
``response.optimize_css``:inxx
``response.optimize_js``:inxx
``response`` is another instance of the ``Storage`` class. It contains the following:
``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.
``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.
``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).
``response.files``: a list of .css, .js, coffee, and .less files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is significant.
``response.include_files()`` generates html head tags to includes all ``response.files`` (used in "views/web2py_ajax.html").
``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.
``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header removing its key from the response.headers dict, e.g.``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behaviour, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``
``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.
``response.meta``: a storage object (like a dict) that contains optional meta information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".
``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").