/
NEWS
2299 lines (1705 loc) · 101 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
User visible changes in Foolscap -*- outline -*-
* Release 0.7.0 (23-Sep-2014)
** Security Fixes
The "flappserver" feature was found to have a vulnerability in the
service-lookup code which, when combined with an attacker who has the ability
to write files to a location where the flappserver process could read them,
would allow that attacker to obtain control of the flappserver process.
Users who run flappservers should upgrade to 0.7.0, where this was fixed as
part of #226.
Each flappserver runs from a "base directory", and uses multiple files within
the basedir to track the services that have been configured. The format of
these files has changed. The flappserver tool in 0.7.0 remains capable of
reading the old format (safely), but will upgrade the basedir to the new
format when you use "flappserver add" to add a new service. Brand new
servers, created with "flappserver create", will use the new format.
The flappserver tool in 0.6.5 (or earlier) cannot handle this new format, and
will believe that no services have been configured. Therefore downgrading to
an older version of Foolscap will require manual reconstruction of the
configured services.
** Major Changes
UnauthenticatedTub has been deprecated, and will be removed in the next
release (0.8.0). This seldom-used feature provides Foolscap's RPC semantics
without any of the security, and was included to enable the use of Foolscap
without depending upon the (challenging-to-install) PyOpenSSL library.
However, in practice, the lack of a solid dependency on PyOpenSSL has made
installation more difficult for applications that *do* want the security, and
UnauthenticatedTub is a footgun waiting to go off. Foolscap's code and
packaging will be simpler without it. (#67)
** Minor Changes
The "git-foolscap" tools, which make it possible to publish and clone Git
repositories over a Foolscap (flappserver) connection, have been moved from
their hiding place in doc/examples/ into their own project, hosted at
https://github.com/warner/git-foolscap . They will also be published on PyPI,
to enable "pip install git-foolscap".
The documentation was converted from Lore to ReStructuredText (.rst). Thanks
to Koblaid for the patient work. (#148)
The connection-hint parser in 0.7.0 has been changed to handle all TCP forms
of Twisted's "Client Endpoint Descriptor" syntax, including the short
"tcp:127.0.0.1:9999" variant. A future version should handle arbitrary
endpoint descriptors (including Tor and i2p, see #203), but this small step
should improve forward compatibility. (#216, #217)
* Release 0.6.5 (12-Aug-2014)
** Compatibility Fixes
This release is compatible with Twisted-14.0.0.
Foolscap no longer claims compatability with python-2.4.x or 2.5.x . These
old versions might still work, but there are no longer automated tests to
ensure this. Future versions will almost certainly *not* work with anything
older than python-2.6.x . Foolscap remains incompatible with py3, sorry.
** Forward Compatibility
When parsing FURLs, the connection hints can now use TCP sockets described
with the Twisted Endpoints syntax (e.g. "tcp:host=127.0.0.1:port=9999"), in
addition to the earlier host:port "127.0.0.1:9999" form. Foolscap-0.6.5
ignores any hint that is not in one of these two forms. This should make it
easier to introduce new hint types in the future.
** Minor Changes
The "ChangeLog" file is no longer updated.
Violation reports now include the method name. (#201)
The "flappserver" tool explicitly rejects unicode input, rather than
producing hard-to-diagnose errors later. (#209)
* Release 0.6.4 (18-Jun-2012)
** Minor Changes
The unreliable 'extras_require' property in setup.py, which allowed other
python programs to declare a dependency on foolscap's "secure_connections"
feature, was removed. See README.packagers for alternate instructions. (#174)
'flogtool' log-dumping commands (dump, tail, web-viewer) now accept a
consistent --timestamps= argument to control how event times are displayed
(UTC, local, seconds-since-epoch, etc). (#192, #193)
Certain invalid "location" strings (accepted by Tub.setLocation and put into
FURLs) are rejected earlier, and with better error messages. The error
message produced when 'flogtool dump' is given a FURL-file (instead of an
event log file) has been improved.
The Incident Gatherer will tolerate incident-file errors better, fetching
remaining incidents instead of halting. (#190)
The git-over-foolscap tools were cleaned up, and the documentation was
brought into line with the implementation. (#197)
Other minor bugs were fixed: #179, #191, #194, #195, #196
* Release 0.6.3 (05-Jan-2012)
** Compatibility Fixes
This release really is compatible with Twisted-11.1.0 . The previous Foolscap
release (0.6.2), despite the changes described below, suffered mild
incompatibilites with the new TLS code in the final Twisted-11.1.0 release.
The most common symptom is a DirtyReactorError in unit tests that use
Tub.stopService() in their tearDown() method (to coordinate shutdown and
cleanup). Another symptom is tests overlapping with one another, causing
port-already-in-use errors.
This incompatibility did not generally affect normal operation, but only
impacted unit tests.
** Other Changes
The Debian packaging tools in misc/ were removed, as they were pretty stale.
These days, both Debian and Ubuntu make their own Foolscap packages.
* Release 0.6.2 (15-Oct-2011)
** Compatibility Fixes
Foolscap-0.6.2 will be compatible with future versions of Twisted (>11.0.0).
The 0.6.1 release will not: a TLS change went into Twisted trunk recently
(after the 11.0.0 release) which broke Foolscap 0.6.1 and earlier.
This release also fixes a minor incompatibility with newer versions of
OpenSSL (0.9.8o was ok, 1.0.0d was not), which caused errors in the test
suite (but normal runtime operation) on e.g. Ubuntu 11.10 "Oneiric".
** Git-Over-Foolscap Tools
The doc/examples/ directory contains two executables (git-foolscap and
git-remote-pb) which, when placed in your $PATH, make it easy to use Foolscap
to access a Git repository. These use the flappserver/flappclient tools and
let you build a FURL that provides read-only or read-write access to a single
repository. This is somewhat like providing SSH access to a repo, but with a
much smaller scope: the client will only be able to manipulate the one
repository, and gets no other authority on the target system. See the tool's
inline comments for usage instructions.
** Minor Fixes
Using 'flappserver upload-file FILE1 FILE2 FILE3..' (with three or more
files) now correctly uploads all files: previously it only managed to upload
the first and last.
'flappserver' argument handling was improved slightly. A workaround was added
to handle a Twisted stdio-closing bug which affected flappserver's
run-command function and broke the git-foolscap tool. Several changes were
made for the benefit of Windows: log filenames all use hyphens (not colons),
log filtering tools tolerate the lack of atomic-rename filesystem operations,
and some unixisms in the test suite were removed.
The Tub.setLogGathererFURL() method can now accept a list (iterable) of log
gatherer FURLs, not just a single one.
* Release 0.6.1 (16-Jan-2011)
** Minor Fixes
The old "sets" module is no longer imported without wrapping the import in a
DeprecationWarning suppressor. We still import it from slicers.set for
compatibility with older code, but that import will not produce a warning.
This should make Foolscap quieter when used with Python 2.6 or later.
A new RemoteReference method named getDataLastReceivedAt() was added, which
will tell you when data was most recently received on the connection
supporting that reference. This can be compared against time.time() to see
how "live" the connection is. For performance reasons, this is only enabled
when keepalives are turned on, otherwise it returns None. (#169)
Some unreachable code was removed. (#165)
* Release 0.6.0 (28-Dec-2010)
** API Changes
*** "foolscap.api" now mandatory
The old import names from foolscap/__init__.py have been removed, finishing
the transition begun with 0.5.0 . Applications must now import Tub,
Referenceable, and so on from "foolscap.api". (#122)
** Compatibility Fixes
Foolscap-0.6.0 is compatible with Twisted-10.2 (released 29-Nov-2010). The
0.5.1 release was not: pb.Listener was depending upon the behavior of an
internal Twisted function that changed, causing an AttributeError in
"StreamServerEndpointService". This is fixed, but the code is still using an
undocumented internal attribute to handle port=0 which will need to be
replaced eventually. (#167)
The first unit test ("test__versions") spuriously failed against Twisted-10.1
and 10.2, mistakenly believing that 10.1 was older than 8.1.0 due to a
lexicographic comparison that should have been numeric.
** Other Changes
Incident filenames are now like "2008-08-22--16:20:28Z.flog" which are in UTC
and mostly ISO-8601 format (the real ISO-8601 would use "_" instead of "--").
This is also used for log-gatherer filenames. (#111)
The logging code now honors FLOGLEVEL= when using FLOGTOTWISTED=1; previously
FLOGLEVEL= was ignored when deciding which log events should be bridged to
the twisted logger. (#154)
Some minor packaging bugs were fixed.
* Release 0.5.1 (25 Mar 2010)
** Bugfixes
This release fixes a significant performance problem, causing receivers a
very long time (over 10 seconds) to process large (>10MB) messages, for
example when receiving a large string in method arguments. Receiver CPU time
was quadratic in the size of the message. (#149)
** Other Changes
This release removes some unused code involved in the now-abandoned
resource-exhaustion defenses. (#127)
* Release 0.5.0 (18 Jan 2010)
** Compatibility
The wire format remains the same as in earlier releases. The preferred API
import path has changed, see below.
** API changes: import statements, foolscap.api
To reduce circular dependencies in Foolscap's internal code, a new
"foolscap.api" module has been created. Applications should use:
from foolscap.api import Tub
instead of e.g. "from foolscap import Tub". Deprecation warnings will be
raised for code which imports symbols directly from the "foolscap" module.
These warnings will turn into errors in the 0.6.0 release. (see ticket #122
for details)
The nearly-useless getRemoteURL_TCP() function was removed.
** setup.py is more windows-friendly
The main setup.py script has been modified to use setuptools "entry_points="
on windows, which should help create runnable executables of "flogtool" and
"flappserver", with proper extensions. Entry-point scripts are not used on
non-windows platforms, but setuptools still creates fairly opaque executable
scripts (which makes it hard to figure out that e.g. /usr/bin/flogtool wants
to import the "foolscap" module). To get non-opaque scripts, install with
"setup.py install --single-version-externally-managed". (#109)
** tool changes
*** flappserver
"flappserver create" now records the umask value from its environment, and
uses it later when the server is started (since normally twistd resets the
umask to a very restrictive value). A new --umask argument was added to
override this. The server's base directory is chmod go-rwx to protect the
private key from other users.
The "flappserver start" command uses twisted.scripts.twistd.run(), instead of
spawning an intermediate "twistd" process with os.execvp(). This should make
things work better in environments where Twisted is not fully installed
(especially on windows) and correctly launching "twistd" is non-trivial, such
as when some other package is installing it as a setuptools dependency.
"flappclient upload-file ~/foo.txt" will use os.path.expanduser() on the
filename, even if your shell does not. This should make it easier to use from
e.g. buildbot upload commands. (#134)
*** logging
The "flogtool dump" and "flogtool web-viewer" commands now have a
--timestamps argument, which controls how timestamps are expressed (UTC vs
localtime, ISO-9601, etc). The web-viewer HTML pages now have more timestamp
and sorting options, and hyperlinks to select each. (#100)
"flogtool web-viewer --open" will tell your local web browser to open to the
correct page, using the Python stdlib "webbrowser" module.
"flogtool dump" now emits a better error when told to open a missing file.
*** examples
Examples of running the Git version-control-system over a flappserver-based
secure connection have been added to doc/examples/ . This enables
remote-update authority to be expressed as a FURL with no other shell
privileges. To accomplish the same with ssh "authorized_keys" command
restrictions is annoying and error-prone. See
doc/examples/git-proxy-flappclient for setup instructions. This will probably
be simplified to a single "git-furl" executable in a later release.
The xfer-client/xfer-server/command-client examples have been removed,
obsoleted by the flappserver/flappclient tools.
** Other changes
The DeprecationWarning for the obsolete "sets" module is now removed on
python2.6 (#124)
When a getReference() call fails because the remote Tub does not recognize
the FURL, it now only emits the first two letters of the secret swissnum in
the exception, instead of the whole thing. This reduces information leakage
into e.g. stderr logs from a "flappclient --furlfile=X upload-file" command.
DeadReferenceError now includes the remote tubid, interfacename, and remote
method name of the message that was being sent when the lost connection was
discovered, so log.err() calls which record a DeadReferenceError should
include this information. This may make it easier to locate the code that
provoked the error.
* Release 0.4.2 (16 Jun 2009)
** Compatibility
Same as 0.4.1
** the Foolscap Application Server
The big new feature in this release is the "Foolscap Application Server".
This is both a demo of what you can do with Foolscap, and an easy way to
deploy a few simple services that run over secure connections. You create and
start a "flappserver" on one machine, and then use the new "flappclient" on
the other side. The server can contain multiple services, each with a
separate FURL. You give the client a FURL for a specific services, it
connects, does a job, and shuts down.
See doc/flappserver.xhtml for details.
Two service types are provided in this release. The first is a simple
file-uploader: the holder of the FURL gets to upload arbitrary files into a
specific target directory, and nowhere else. The second is a pre-configured
command runner: the service is configured with a shell command, and the
client gets to make it run (but doesn't get to influence anything about what
gets run). The run-command service defaults to sending stdout/stderr/exitcode
to the client program, which will behave as if it were the command being run
(stdout and stderr appear at right time, and it exits with the same
exitcode). The service can be configured to accept stdin, or to turn off
stdout or stderr. The service always runs in a preconfigured working
directory.
To do this with SSH, you'd need to create a new keypair, then set up an
authorized_keys entry to limit that pubkey to a single command, and hope that
environment variables and the working directory don't cause any surprises.
Implementing the fixed-directory file-uploader would probably require a
specialized helper program.
The flappserver provides an easy-to-configure capability-based replacement
those sorts of SSH setups. The first use-case is to allow buildslaves to
upload newly-created debian packages to a central repository and then trigger
a package-index rebuild script. By using FURLs instead of raw SSH keys, the
buildslaves will be unable to affect any .debs in other directories, or any
other files on the repository host, nor will they be able to run arbitrary
commands on that host. By storing the FURLs in a file and using the
--furlfile argument to "flappclient", a buildbot transcript of the upload
step will not leak the upload authority.
** new RemoteReference APIs
RemoteReference now features two new methods. rref.isConnected() returns a
boolean, True if the remote connection is currently live, False if it has
been lost. This is an immediate form of the rref.notifyOnDisconnect()
callback-registration mechanism, and can make certain types of
publish-subscribe code easier to write.
The second is rref.getLocationHints(), which returns a list of location hints
as advertised by the host Tub. Most hints are a ("ipv4",host,portnum) tuple,
but other types may be defined in the future. Note that this is derived from
the FURL that each Tub sends with its my-reference sequence (i.e. it is
entirely controlled by the Tub in which that Referenceable lives), so
getLocationHints() is quite distinct from rref.getPeer() (which returns an
IPv4Address or LoopbackAddress instance describing the other end of the
actual network connection). getLocationHints() indicates what the other Tub
wants you to use for new connections, getPeer() indicates what was used for
the existing connection (which might not accept new connections due to NAT or
proxy issues).
getLocationHints() is meant to make it easier to write connection-status
display code, for example in a server which holds connections to a number of
peers. A status web page can loop over the peer RemoteReferences and display
location information for each one without needing to look deep inside the
hidden RemoteReferenceTracker instance to find it.
** giving up on resource-consumtion defenses
Ticket #127 contains more detail, but beginning with this release, Foolscap
will be slowly removing the code that attempted to prevent memory-exhaustion
attacks. Doing this in a single process is just too hard, and the limits that
were enforced provided more problems than protection. To this end, an
internal 200-byte limit on FURL length (applied in Gifts) has been removed.
Later releases will remove more code, hopefully simplifying the deserization
path.
** other bugfixes
Previous releases would throw an immediate exception when Tub.getReference()
or Tub.connectTo() was called with an unreachable FURL (one with a corrupted
or empty set of location hints). In code which walks a list of FURLs and
tries to initiate connections to all of them, this synchronous exception
would bypass all FURLs beyond the troublesome one.
This has been improved: Tub.getReference() now always returns a Deferred,
even if the connection is doomed to fail because of a bad FURL. These
problems are now indicated by a Deferred that errbacks instead of a
synchronous exception.
* Release 0.4.1 (22 May 2009)
** Compatibility
Same as 0.4.0
** Bug fixes
The new RemoteException class was not stringifiable under python2.4 (i.e.
str(RemoteException(f)) would raise an AttributeError), causing problems
especially when callRemote errbacks attempted to record the received Failure
with log.msg(failure=f). This has been fixed.
* Release 0.4.0 (19 May 2009)
** Compatibility
The wire protocol remains the same as before, unchanged since 0.2.6 .
The main API entry point has moved to "foolscap.api": e.g. you should do
"from foolscap.api import Tub" instead of "from foolscap import Tub".
Importing symbols directly from the "foolscap" module is now deprecated.
(this makes it easier to reorganize the internal structure of Foolscap
without causing circular dependencies). (#122)
A near-future release (probably 0.4.1) will add proper
DeprecationWarnings-raising wrappers to all classes and functions in
foolscap/__init__.py . The next major release (probably 0.5.0) will remove
these symbols from foolscap/__init__.py altogether.
Logging functions are still meant to be imported from foolscap.logging.* .
** expose-remote-exception-types (#105)
Remote exception reporting is changing. Please see the new document
docs/failures.xhtml for full details. This release adds a new option:
tub.setOption("expose-remote-exception-types", BOOL)
The default is True, which provides the same behavior as previous releases:
remote exceptions are presented to look as much as possible like local
exceptions.
If you set it to False, then all remote exceptions will be collapsed into a
single "foolscap.api.RemoteException" type, with an attribute named .failure
that can be used to get more details about the remote exception. This means
that callRemote will either fire its Deferred with a regular value, or
errback with one of three exception types from foolscap.api:
DeadReferenceError, Violation, or RemoteException. (When the option is True,
it could errback with any exception type, limited only by what the remote
side chose to raise)
A future version of Foolscap may change the default value of this option.
We're not sure yet: we need more experience to see which mode is safer and
easier to code with. If the default is changed, the deprecation sequence will
probably be:
0.5.0: require expose-remote-exception-types to be set
0.6.0: change the default to False, stop requiring the option to be set
0.7.0: remove the option
** major bugs fixed:
Shared references now work after a Violation (#104)
The tubid returned by rref.getSturdyRef() is now reliable (#84)
Foolscap should work with python-2.6: Decimal usage fixed, sha/md5
deprecation warnings fixed, import of 'sets' still causes a warning. (#118,
#121)
Foolscap finally uses new-style classes everywhere (#96)
bin/flogtool might work better on windows now (#108)
logfiles now store library versions and process IDs (#80, #97)
The "flogtool web-viewer" tool listens at a URL of "/" instead of "/welcome",
making it slightly easier to use (#120)
You can now setOption() on both log-gatherer-furl and log-gatherer-furlfile
on the same Tub. Previously this caused an error. (#114)
* Release 0.3.2 (14 Oct 2008)
** Compatibility: same as 0.2.6
Incident classifier functions (introduced in 0.3.0) have been changed: if you
have written custom functions for an Incident Gatherer, you will need to
modify them upon upgrading to this release.
** Logging Changes
The log.msg counter now uses a regular Python integer/bigint. The counter in
0.3.1 used itertools.count(), which, despite its documentation, stores the
counter in a C signed int, and thus throws an exception when the message
number exceeds 2**31-1 . This exception would pretty much kill any program
which ran long enough to emit this many messages, a situation which was
observed in a busy production server with an uptime of about three or four
weeks. The 0.3.2 counter will be promoted to a bigint when necessary,
removing this limitation. (ticket #99)
The Incident-Gatherer now imports classification functions from files named
'classify_*.py' in the gatherer's directory. This effectively adds
"classifier plugins". The signature of the functions has changed since the
0.3.0 release, making them easier to use. If you have written custom
functions (and edited the gatherer.tac file to activate them, using
gs.add_classifier()), you will need to modify the functions to take a single
'trigger' argument.
These same 'classify_*.py' plugins are used by a new "flogtool
classify-incident" subcommand, which can be pointed at an incident file, and
performs the same kind of classification as the Incident Gatherer. (#102).
The logfiles produced by the "flogtool tail" command and the internal
incident-reporter now include the PID of the reporting process. This can be
seen by passing the --verbose option to "flogtool dump", and will be made
more visible in later releases. (#80).
The "flogtool web-viewer" tool now marks Incident triggers (#79), and
features a "Reload Logfile" button to re-read the logfile on disk (#103).
This is most useful when running unit tests, in conjunction with the
FLOGFILE= environment variable.
** Other Changes
When running unit tests, if the #62 bug is encountered (pyopenssl >= 0.7 and
twisted <= 8.1.0 and selectreactor), the test process will emit a warning and
pause for ten seconds to give the operator a chance to halt the test and
re-run it with --reactor=poll. This may help to reduce the confusion of a
hanging+failing test run.
The xfer-client.py example tool (in doc/listings/) has been made more
useable, by calling os.path.expanduser() on its input files, and by doing
sys.exit(1) on failure (instead of hanging), so that external programs can
react appropriately.
* Release 0.3.1 (03 Sep 2008)
** Compatibility: same as 0.2.6
** callRemote API changes: DeadReferenceError
All partitioning exceptions are now mapped to DeadReferenceError. Previously
there were three separate exceptions that might indicate a network partition:
DeadReferenceError, ConnectionLost, and ConnectionDone. (a network partition
is when one party cannot reach the other party, due to a variety of reasons:
temporary network failure, the remote program being shut down, the remote
host being taken offline, etc).
This means that, if you want to send a message and don't care whether that
message makes it to the recipient or not (but you *do* still care if the
recipient raises an exception during processing of that message), you can set
up the Deferred chain like this:
d = rref.callRemote("message", args)
d.addCallback(self.handle_response)
d.addErrback(lambda f: f.trap(foolscap.DeadReferenceError))
d.addErrback(log.err)
The first d.addErrback will use f.trap to catch DeadReferenceError, but will
pass other exceptions through to the log.err() errback. This will cause
DeadReferenceError to be ignored, but other errors to be logged.
DeadReferenceError will be signalled in any of the following situations:
1: the TCP connection was lost before callRemote was invoked
2: the connection was lost after the request was sent, but before
the response was received
3: when the active connection is dropped because a duplicate connection was
established. This can occur when two programs are simultaneously
connecting to each other.
** logging improvements
*** bridge foolscap logs into twistd.log
By calling foolscap.logging.log.bridgeLogsToTwisted(), or by setting the
$FLOGTOTWISTED environment variable (to anything), a subset of Foolscap log
events will be copied into the Twisted logging system. The default filter
will not copy events below the log.OPERATIONAL level, nor will it copy
internal foolscap events (i.e. those with a facility name that starts with
"foolscap"). This mechanism is careful to avoid loops, so it is safe to use
both bridgeLogsToTwisted() and bridgeTwistedLogs() at the same time. The
events that are copied into the Twisted logging system will typically show up
in the twistd.log file (for applications that are run under twistd).
An alternate filter function can be passed to bridgeLogsToTwisted().
This feature provides a human-readable on-disk record of significant events,
using a traditional one-line-per-event all-text sequential logging structure.
It does not record parent/child relationships, structured event data, or
causality information.
*** Incident Gatherer improvements
If an Incident occurs while a previous Incident is still being recorded (i.e.
during the "trailing log period"), the two will be folded together.
Specifically, the second event will not trigger a new Incident, but will be
recorded in the first Incident as a normal log event. This serves to address
some performance problems we've seen when incident triggers occur in
clusters, which used to cause dozens of simultaneous Incident Recorders to
swing into action.
The Incident Gatherer has been changed to only fetch one Incident at a time
(per publishing application), to avoid overloading the app with a large
outbound TCP queue.
The Incident Gatherer has also been changed to scan the classified/* output
files and reclassify any stored incidents it has that are not mentioned in
one of these files. This means that you can update the classification
functions (to add a function for some previously unknown type of incident),
delete the classified/unknown file, then restart the incident gatherer, and
it will only reclassify the previously-unknown incidents. This makes it much
easier to iteratively develop classification functions.
*** Application Version data
The table of application versions, previously displayed only by the 'flogtool
tail' command, is now recorded in the header of both Incidents and the
'flogtool tail --save-to' output file.
The API to add application versions has changed: now programs should call
foolscap.logging.app_versions.add_version(name, verstr).
* Release 0.3.0 (04 Aug 2008)
** Compatibility: same as 0.2.6
The wire-level protocol remains the same as other recent releases.
The new incident-gatherer will only work with applications that use Foolscap
0.3.0 or later.
** logging improvements
The "incident gatherer" has finally been implemented. This is a service, like
the log-gatherer, which subscribes to an application's logport and collects
incident reports: each is a dump of accumulated log messages, triggered by
some special event (such as those above a certain severity threshold). The
"flogtool create-incident-gatherer" command is used to create one, and twistd
is used to start it. Please see doc/logging.xhtml for more details.
The incident publishing API was changed to support the incident-gatherer. The
incident-gatherer will only work with logports using foolscap 0.3.0 or newer.
The log-publishing API was changed slightly, to encourage the use of
subscription.unsubscribe() rather than publisher.unsubscribe(subscription).
The old API remains in place for backwards compatibility with log-gatherers
running foolscap 0.2.9 or earlier.
The Tub.setOption("log-gatherer-furlfile") can accept a file with multiple
FURLs, one per line, instead of just a single FURL. This makes the
application contact multiple log gatherers, offering its logport to each
independently, e.g. to connect to both a log-gatherer and an
incident-gatherer.
** API Additions
RemoteReferences now have a getRemoteTubID() method, which returns a string
(base32-encoded) representing the secure Tub ID of the remote end of the
connection. For any given Tub ID, only the possessor of the matching private
key should be able to provide a RemoteReference for which getRemoteTubID()
will return that value. I'm not yet sure if getRemoteTubID() is a good idea
or not (the traditional object-capability model discourages making
access-control decisions on the basis of "who", instead these decisions
should be controlled by "what": what objects do they have access to). This
method is intended for use by application code that needs to use TubIDs as an
index into a table of some sort. It is used by Tahoe to securely compute
shared cryptographic secrets for each remote server (by hashing the TubID
together with some other string).
Note that the rref.getSturdyRef() call (which has been present in Foolscap
since forever) is *not* secure: the remote application controls all parts of
the sturdy ref FURL, including the tubid. A future version of foolscap may
remedy this.
** Bug fixes
The log-gatherer FURL can now be set before Tub.setLocation (the connection
request will be deferred until setLocation is called), and
getLogPort/getLogPortFURL cannot be called until after setLocation. These two
changes, in combination, resolve a problem (#55) in which the gatherer
connection could be made before the logport was ready, causing the
log-gatherer to fail to subscribe to receive log events.
** Dependent Libraries
Foolscap uses PyOpenSSL for all of its cryptographic routines. A bug (#62)
has been found in which the current version of Twisted (8.1.0) and the
current version of PyOpenSSL (0.7) interact badly, causing Foolscap's unit
tests to fail. This problem will affect application code as well
(specifically, Tub.stopService will hang forever). The problem only appears
to affect the selectreactor, so the current recommended workaround is to run
unit tests (and applications that need to shut down Tubs) with --reactor=poll
(or whatever other reactor is appropriate for the platform, perhaps iocp). A
less-desireable workaround is to downgrade PyOpenSSL to 0.6, or Twisted to
something older. The Twisted maintainers are aware of the problem and intend
to fix it in an upcoming Twisted release.
* Release 0.2.9 (02 Jul 2008)
** Compatibility: exactly the same as 0.2.6
** logging bugs fixed
The foolscap.logging.log.setLogDir() option would throw an exception if the
directory already existed, making it unsuitable for use in an application
which is expected to be run multiple times. This has been fixed.
** logging improvements
'flogtool tail' now displays the process ID and version information about the
remote process. The tool will tolerate older versions of foolscap which do
not offer the get_pid interface. (foolscap ticket #71)
The remote logport now uses a size-limited queue for messages going to a
gatherer or 'flogtool tail', to prevent the monitored process from using
unbounded amounts of memory during overload situations (when it is generating
messages faster than the receiver can handle them). This solves a runaway
load problem we've seen in Tahoe production systems, in which a busy node
sends log messages to a gatherer too quickly for it to absorb, using lots of
memory to hold the pending messages, which causes swapping, which causes more
load, making the problem worse. We frequently see an otherwise well-behaved
process swell to 1.4GB due to this problem, occasionally failing due to VM
exhaustion. Of course, a bounded queue means that new log events will be
dropped during this overload situation. (#72)
** serialization added for the Decimal type (#50)
** debian packaging targets added for gutsy and hardy
The Makefile now has 'make debian-gutsy' and 'make debian-hardy' targets.
These do the same thing as 'make debian-feisty'. (#76)
* Release 0.2.8 (04 Jun 2008)
** Compatibility: exactly the same as 0.2.6
** setuptools dependencies updated
Foolscap (when built with setuptools) now uses the "extras_require" feature
to declare that it needs pyOpenSSL if you want to use the
"secure_connections" feature. This makes easy_install easier to use in
projects that depend upon Foolscap (and also insist upon using secure
connections): they do not need to declare a dependency on pyOpenSSL
themselves, instead they declare a dependency on
"Foolscap[secure_connections]". See the following documentation for more
details:
http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies
** New RemoteReference.getPeer() method
The new rref.getPeer() method will return address information about the far
end of the connection, allowing you to determine their IP address and port
number. This may be useful for debugging or diagnostic purposes.
** Minor bugs fixed
Tub.registerReference() with both name= and furlFile= arguments now works
even when the furlFile= already exists.
Tubs which have been shutdown now give more useful error messages when you
(incorrectly) try to use them again. Previously a bug caused them to emit a
TypeError.
* Release 0.2.7 (13 May 2008)
** Compatibility: exactly the same as 0.2.6
** flogtool works again
The "flogtool" utility was completely non-functional in 0.2.6 . This has been
fixed.
** Known Issues
*** some debian packages are wrong
When using the 'make debian-dapper' target (to build a .deb for a dapper
system), the resulting .deb sometimes includes a full copy of Twisted, and is
probably unsuitable for installation. This appears to be a result of
installation behavior changing due to setuptools being imported (even though
it is not explicitly used). No other platforms .deb files seem to be affected
this way. Package builders are advised to examine the generated .deb closely
before using it.
* Release 0.2.6 (06 May 2008)
** Compatibility
All releases between 0.1.3 and 0.2.6 (inclusive) are fully wire-compatible.
The saved logfiles produced (by e.g. 'flogtool tail --save-to' and the
log-gatherer) in 0.2.6 and beyond are not readable by tools (e.g. 'flogtool
dump' and 'flogtool filter') from 0.2.5 and earlier.
FURLs which contain "extensions" (described below) will not be tolerated by
foolscap 0.2.5 or earlier. If, at some point in the future, we add such
extensions to published FURLs, then such an application will require
foolscap-0.2.6 or later to interpret those FURLs.
** Logging Changes
*** "Incident" Logging
This release finally implements the "strangeness-triggered logging" espoused
in doc/logging.xhtml . By giving the foolscap logging code a directory to
work with, the logging system will automatically save a compressed pickled
logfile containing the messages that lead up to sufficiently-severe log
event. The docs explain how to control what "sufficiently-severe" means.
These events are retrievable through the logport, although no tools have been
written yet to actually extract them. They are also retrievable by using
'flogtool dump' directly on the incident files.
*** 'flogtool as a subcommand
The implementation of the 'flogtool' executable has been rearranged to make
it possible to add a 'flogtool' subcommand to some other executable.
*** 'flogtool filter' now has --above LEVEL and --from TUBID options
*** 'flogtool dump' has --rx-time option, also shows failure tracebacks
*** gatherer: don't add erroneous UTC-implying "Z" suffix to filename timestamps
*** 'flogtool tail': don't add spurious "0" to timestamps
** constraints no longer reject ('reference',) sequences
The foolscap/banana serialization protocol responds to sending two separate
copies of the same object in the same callRemote message by emitting one
serialized object sequence and one 'reference' sequence: this is the standard
way by which cycles are broken in the serialized data. Unfortunately, the
wire-level constraint checkers in 0.2.5 and earlier would reject reference
sequences with a Violation exception: if they were expecting a tuple, they
would reject anything else, even a reference sequence that pointed at a
tuple.
Worse yet, python's normal constant-object folding code can produce shared
references where you might not expect. In the following example, the two
tuples are identical objects (and result in a 'reference' sequence on the
wire), despite the programmer having typed them separately:
rref.callRemote("haveTwoTuples", (0,1), (0,1) )
Foolscap-0.2.6 now allows reference sequence in all wire-level constraint
checks, to avoid this false-negative Violation. The post-deserialization
check will still enforce the constraint properly. It just does it late enough
to be able to tell what the reference points to.
** Twisted/pyopenssl compatibility
*** compatible with Twisted-8.0.x
Some unit test failures under Twisted-8.0.x (the most recent release) were
fixed: tests now pass against Twisted-8.0.x, and a buildbot is used to make
sure compatibility is maintained in the future.
*** incompatible with pyOpenSSL-0.7
An incompatibility has been discovered with the most recent version of
PyOpenSSL. pyopenssl 0.6 works correctly, but pyopenssl 0.7 causes modern
versions of Twisted (both 2.5.x and 8.0.x) to follow a code path that breaks
the Foolscap unit tests. This may or may not cause a problem in actual
application use (the symptom is that the non-winning parallel connections are
not disconnected properly, and several negotiation timers are left running).
Until a fix is developed for either Twisted or PyOpenSSL, the recommended
workaround is to downgrade to PyOpenSSL-0.6 . Twisted bug #3218 and Foolscap
bug #62 exist to track this problem.
** setup.py is more setuptools-friendly
The foolscap version string is located with a regular expression rather than
an import, allowing setuptools to install Foolscap as a build-dependency of
some other project without having Twisted available first. If setuptools is
available, we also declare a dependency upon Twisted (at least 2.4.0), to
give more information to the setuptools dependency-auto-installer.
** Unauthenticated FURLs can now contain multiple connection hints
Previously they were limited to a single one
** FURLs can now contain extensions, providing forwards-compatibility
The parsing of FURLs has been refined to tolerate (and ignore) certain kinds
of extensions. The "tubid" section will be able to have additional
identifiers (perhaps stronger hashes for the public key, or an
encryption-ready EC-DSA public key). In addition, the "connection hints"
section will be able to contain alternate protocol specifiers (for TCP over
IPv6, or a less connection-oriented UDP transport).
By ignoring such extensions, foolscap-0.2.6 will tolerate FURLs produced
(with extensions) by some future version of foolscap. This marks the
beginning of a "transition period": when such extensions are introduced,
0.2.6 will be the oldest version still capable of interoperating with the
extension-using new version.
* Release 0.2.5 (25 Mar 2008)
** Compatibility
All releases between 0.1.3 and 0.2.5 (inclusive) are fully wire-compatible.
The new 'flogtool tail --catch-up' command requires a log publisher running
0.2.5 or later. 'flogtool tail' without the --catch-up option will work with
earlier publishers.
** Licensing clarification
Foolscap is distributed under the (very liberal) terms of the MIT license,
which is the same license that Twisted uses. It's been like this since the
beginning, but this is the first release to make this obvious by including a
LICENSE file.
** foolscap.logging Changes
'flogtool tail' now has a --catch-up option, which prompts the remote
publisher to deliver stored historical events to the subscribe, in proper
sequential order. This allows you to connect to a process that has just done
something interesting and grab a copy of the log events relevant to that
event.
'flogtool tail' also has a --save-to option, which specifies a filename to
which all captured events should be saved. This file can be processed further
with 'flogtool dump', 'flogtool filter', or 'flogtool web-viewer'. This
behaves much like the unix 'tee' utility, except that the saved data is
recorded in a lossless binary format (whereas the text emitted to stdout is
not particularly machine-readable).
'flogtool tail' and 'flogtool dump' both emit human-readable log messages by
default. The --verbose option will emit raw event dictionaries, which contain
slightly more information but are harder to read.
'flogtool create-gatherer' will create a log gatherer .tac file in a new
working directory. This .tac file can be launched with 'twistd', the standard
Twisted daemon-launching program. This is significantly easier to work with
than the previous 'flogtool gather' command (which has been removed). The new
--rotate option will cause the log-gatherer to switch to a new output file
every N seconds. The --bzip option will make it compress the logfiles after
rotating them. For example, a log gatherer that rotates and compresses log
files once per day could be created and launched with:
flogtool create-gatherer --rotate 86400 --bzip ./workdir
(cd workdir && twistd -y gatherer.tac)
** New sample programs
doc/listings/command-server.py and command-client.py are a pair of programs
that let you safely grant access to a specific command. The server is
configured with a command line to run, and a directory to run it from. The
client gets a FURL: when the client is executed, the server will run its
preconfigured command. The client gets to see stdout and stderr (and the exit
status), but does not get to influence the command being run in any way.
This is much like setting up an ssh server with a restricted command, but
somewhat easier to configure.
doc/listings/xfer-server.py and xfer-client.py are similar, but provide file
transfer services instead of command execution.
** New Features
Tub.setLocationAutomatically() will try to determine an externally-visible IP
address and feed it to Tub.setLocation(). It does this by preparing to send a