forked from warner/foolscap
/
ChangeLog
4078 lines (3310 loc) · 177 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2009-05-19 Brian Warner <warner@lothar.com>
* NEWS: update for the next release
* README: mention python2.6 compatibility
* foolscap/test/test__versions.py (Versions.test_record): record
versions of platform/python/twisted/foolscap in the test logs
* foolscap/api.py: add some more symbols: ChoiceOf, IntegerConstraint
* foolscap/banana.py (Banana.handleData): increment the inbound
objectCounter as soon as we see the OPEN token, rather than doing
it in handleOpen(), since the latter is skipped if we're dropping
tokens due to a Violation. This avoids the loss-of-sync that
occurred when the dropped tokens included more OPEN
sequences (e.g. when a remote method call used a bad method name,
and its arguments contained lists or tuples or other sequences).
This fixes the bug in which subsequent method calls that used
shared arguments saw their "reference" sequences point to the
wrong value. Closes #104.
* foolscap/test/test_call.py (ReferenceCounting): test it
* foolscap/referenceable.py (RemoteReferenceTracker.__init__):
require the proposed per-my-reference FURL to match the TubID that
we're using for the parent Broker. This prevents the far end from
sending a bogus FURL and thus breaking the security properties of
rref.getSturdyRef(). (they couldn't confuse Tub.getReference,
because the spoofed FURL would only be added to the per-Broker
table, but they could confuse someone who relied upon the
correctness of getSturdyRef). This closes #84.
(RemoteReferenceTracker.__init__): oops, don't compare tubids when
using an UnauthenticatedTub, since that fails the non-crypto tests
* foolscap/test/test_reference.py (TubID): test it
* foolscap/test/test_schema.py (Interfaces.test_remotereference):
update to match
* foolscap/hashutil.py: new file to smoothly use hashlib when
available and fall back to sha1/md5 when not (i.e. python2.4).
When we eventually drop py2.4 support, we'll get rid of this file.
Thanks to Stephan Peijnik for the patch. Closes #118.
* foolscap/sslverify.py: use hashutil instead of the md5 module
* foolscap/vocab.py (hashVocabTable): use hashutil instead of the
sha module
* foolscap/schema.py (PolyConstraint.checkToken): remove unused
code, thanks to Stephan Peijnik for the patch.
* foolscap/test/test_banana.py (VocabTest1.test_table_hashes):
make sure we don't change the pre-agreed vocab tables, or the hash
algorithm that we use to confirm their contents. This is in
preparation for the conditional switch to 'hashlib', from
Stephan's patch in ticket #118
2009-05-18 Stephan Peijnik <stephan@peijnik.at>
* foolscap/call.py, foolscap/constraint.py, foolscap/eventual.py,
foolscap/observer.py, foolscap/promise.py,
foolscap/reconnector.py, foolscap/remoteinterface.py,
foolscap/slicer.py: Convert all old-style classes to new-style
classes. Should close #96.
* foolscap/reconnector.py, foolscap/referenceable.py,
foolscap/remoteinterface.py: Import each module with a separate
import statement.
2009-05-18 Brian Warner <warner@lothar.com>
* foolscap/test/test_tub.py (TestCertFile.test_tubid): confirm
that we get the TubID right when using a pre-recorded cert. If we
were to accidentally change the TubID-computing hash function in
the future, this test should catch it.
(CERT_DATA): stash the contents of a pre-recorded cert
* bin/flogtool: let 'base' default to abs("."), to help windows and
systems where somebody has managed to copy it to /tmp/flogtool or
/sbin/flogtool . Closes #108.
2009-05-13 Brian Warner <warner@lothar.com>
* foolscap/test/test_call.py (ExamineFailuresMixin): rearrange
code to try and fix a weird py2.4 failure
* foolscap/api.py (RemoteException): hush pyflakes
2009-05-12 Brian Warner <warner@lothar.com>
* foolscap/logging/dumper.py: emit PID and embedded versions,
except when using --just-numbers or --verbose. Closes #97.
* foolscap/logging/web.py (Welcome.render): same
* foolscap/logging/log.py (LogFileObserver): store app_versions in
the log file created by $FLOGFILE
* foolscap/test/test_logging.py: test it all
* foolscap/tokens.py (RemoteException): add a Tub option that
causes remote exceptions to be reported with a special exception
type, named foolscap.api.RemoteException, so that they won't
be confused with locally-generated exceptions. Closes #105.
* foolscap/api.py: make RemoteException available through api.py
* doc/failures.xhtml: explain the new option
* foolscap/pb.py (Tub.setOption): handle the new
expose-remote-exception-types option, default remains True to
retain the old behavior
* foolscap/broker.py (Broker.setTub): copy the flag from the Tub
into the Broker
* foolscap/call.py (ErrorUnslicer.receiveClose): wrap exceptions
if the mode says we should
* foolscap/test/test_call.py (Failures): test RemoteException
wrapping
(TubFailures): test setOption() too
* foolscap/test/common.py (ShouldFailMixin.shouldFail.done):
return the Failure for further testing, but wrapped in an array to
avoid triggering the errback
2009-05-11 Brian Warner <warner@lothar.com>
* Makefile (LORE): parameterize "lore" binary
* doc/stylesheet.css: update to new version from Lore
* doc/schema.xhtml: fix mis-nested H3 header, to appease new Lore
2009-05-04 Brian Warner <warner@lothar.com>
* foolscap/api.py: new preferred entry point. Application code
should import symbols from "foolscap.api" instead of directly from
foolscap/__init__.py . Importing from the __init__.py will be
deprecated in the next major release, and removed in the
subsequent one. Addresses #122 (but it won't be closed until we
remove everything from __init__.py).
* foolscap/__init__.py: add comment about preferring foolscap.api
* doc/using-foolscap.xhtml: update examples to use e.g.
"from foolscap.api import Tub"
* doc/listings/*.py: update examples
* foolscap/pb.py (Tub.setLogGathererFURL): allow users to call
both Tub.setOption("log-gatherer-furl") and
Tub.setOption("log-gatherer-furlfile") on the same Tub. Previously
this was disallowed. Also avoid making multiple connections to the
same gatherer. Closes #114.
* foolscap/test/test_logging.py
(Gatherer.test_log_gatherer_furlfile_multiple): test it
* foolscap/test/test_logging.py (Publish.test_logpublisher_catchup):
don't use an absolute delay, since it causes spurious test
failures on slow systems.
* foolscap/logging/web.py (WebViewer.start): put the welcome page
at the root URL, in addition to /welcome . This simplifies the
starting URL. Closes #120.
(Reload.render_POST): make the "Reload Logfile" button point at
the new URL too.
* foolscap/logging/web.py (Welcome.render): display PID of
logfile, if available. This will be present in incident records
that were written out by the original process or gathered by a log
gatherer, in the files written by "flogtool tail --save-to=", and
in the file written by creating an explicit FileLogObserver (such
as done by setting the FLOGFILE environment variable). This should
close #80.
* foolscap/logging/log.py (LogFileObserver): record PID in header
* foolscap/test/test_logging.py (Web.test_basic): test PID
* foolscap/test/test_banana.py (ThereAndBackAgain.test_decimal):
use Decimal("Inf") instead of decimal.Inf, to unbreak python2.6.2
which privatized both Inf and Nan. Thanks to ivank for the patch.
Closes #121.
2008-10-15 Brian Warner <warner@allmydata.com>
* misc/*/debian/rules (install/python-foolscap): include
misc/classify_foolscap.py in the debian package, uncompressed, so
it goes into /usr/share/doc/python-foolscap/classify_foolscap.py ,
so you can make a symlink to it from an incident-gatherer.
* misc/classify_foolscap.py (TUBCON_RE): update to match
foolscap-tubconnector messages for both old and new versions of
Foolscap, and for python2.4 and 2.5 .
* foolscap/logging/incident.py (IncidentClassifier): add --verbose
option to 'flogtool classify-incident' to show the trigger
dictionary for any unclassifiable incidents. This is useful when
developing classification functions.
* foolscap/test/test_logging.py (Incidents.test_classify): test it
* Makefile (test-figleaf): oops, fix the test-figleaf-poll target,
by making the test-figleaf target use $(TRIAL), not hardcoded
'trial'.
* foolscap/_version.py: bump revision to 0.3.2+ while between releases
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
2008-10-14 Brian Warner <warner@lothar.com>
* foolscap/_version.py: release Foolscap-0.3.2
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
* MANIFEST.in: add misc/classify_foolscap.py to source distribution
* NEWS: update for the next release
* Makefile (test-poll): add a convenience target, for running
tests on a system with the broken combination of pyopenssl and
twisted that requires the pollreactor
(test-figleaf-poll): same
2008-10-14 Brian Warner <warner@allmydata.com>
* foolscap/logging/tail.py (LogTail): include the retrieved PID in
the saved logfile. Part of #80.
* foolscap/logging/incident.py (IncidentReporter.incident_declared):
also include the PID in the incident report's header.
* doc/specifications/logfiles.xhtml: document the ['versions']
and ['pid'] fields in incident reports and the 'flogtool tail'
savefile.
* foolscap/logging/web.py: mark any incident triggers in the
logfile, and add links to them from the welcome page. Closes #79.
* foolscap/logging/web.py (Reload): add a 'Reload Logfile' button
to the web-viewer's Welcome page. Not automatic, but it means you
don't have to get back to the shell and restart the viewer. Works
well-enough to say Closes #103.
* misc/classify_foolscap.py: plugin to classify some
foolscap-internal incidents
* foolscap/logging/cli.py: new "flogtool classify-incident"
subcommand: given an incident, say what categories it falls into.
Closes #102.
* foolscap/logging/gatherer.py (IncidentGathererService): factor
out the classification pieces into IncidentClassifierBase
* foolscap/logging/incident.py (IncidentClassifierBase): same
(IncidentClassifier.run): support for the new CLI command
* foolscap/test/test_logging.py (Incidents.test_classify): test it
* foolscap/logging/gatherer.py
(IncidentGathererService.classify_incident): change the classifier
function signature: now it just takes a single 'trigger' dict.
* foolscap/test/test_logging.py: update to match
* doc/logging.xhtml (gatherer): same
* foolscap/logging/gatherer.py
(IncidentGathererService.add_classify_files): make the incident
gatherer look in its base directory for "classify_*.py" files, use
them as plugins with classification functions.
(INCIDENT_GATHERER_TACFILE): update example text to match
* foolscap/test/test_logging.py (IncidentGatherer): test it
(Gatherer.test_wrongdir): exercise another error case
* doc/logging.xhtml (gatherer): document it
2008-10-13 Brian Warner <warner@allmydata.com>
* foolscap/logging/publish.py (IncidentSubscription.catch_up):
Don't tell the gatherer about incidents that don't have triggers:
these are malformed logfiles, such as the zero-length files that
result from a process being terminated before it manages to write
anything.
* foolscap/logging/log.py (Count): replace itertools.count with a
version that doesn't overflow at 2**31-1 (thanks to Zooko for the
patch). Closes #99.
(FoolscapLogger.__init__): use it
* foolscap/pb.py (Listener.startFactory): add log facility
identifiers
(Listener.stopFactory): same
(Listener.buildProtocol): same
(Tub.getReference): same
2008-09-20 Brian Warner <warner@lothar.com>
* doc/listings/xfer-client.py (_failure): do sys.exit(1) upon
failure, so the caller can detect it via the exit code
2008-09-10 Brian Warner <warner@allmydata.com>
* foolscap/test/test__versions.py (Versions.test_required): check
for bug #62 (openssl>0.7, twisted<=8.1.0, selectreactor) and print
a warning if it is likely that the tests would fail, to remind the
user to re-run with -r poll.
(Versions.test_required): oops, guard import of OpenSSL on having
crypto available, otherwise this test gets an error when OpenSSL
is not installed.
2008-09-08 Brian Warner <warner@lothar.com>
* doc/listings/xfer-client.py: use os.path.expanduser() on the
filenames passed in by the user, to enable things like
--furlfile ~/.upload.furl
* foolscap/_version.py: bump revision to 0.3.1+ while between releases
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
2008-09-03 Brian Warner <warner@lothar.com>
* foolscap/_version.py: release Foolscap-0.3.1
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
* NEWS: update for next release
2008-09-02 Brian Warner <warner@lothar.com>
* foolscap/negotiate.py (TubConnector.__repr__): add more
information to the repr, to help track down #81. Made TubConnector
a new-style class in the process, to make upcalling easier.
(TubConnector.checkForIdle): improve the log message
* foolscap/pb.py (Tub._log_gatherer_connected): use callRemoteOnly
to pass the logport to the gatherer: we don't need to hear about
any problems it has.
* foolscap/logging/gatherer.py
(IncidentGathererService.classify_stored_incidents): reclassify
everything that isn't already present in one of the classified/*
files. This makes it a lot easier to iterate over the [start
gatherer; see what is unknown; update classifiers; remove
classified/unknown; repeat] loop. Also log classification events
better. Closes #94.
* foolscap/test/test_logging.py (IncidentGatherer.test_emit): test it
* foolscap/test/common.py (PollMixin): replace the use of chained
Deferreds with a task.LoopingCall-based version, from Tahoe. This
avoids the weird and annoying maximum-recursion-depth-exceeded
error that occurs when the check function is called more than
about 600 times. Closes #95.
2008-08-29 Brian Warner <warner@lothar.com>
* foolscap/logging/app_versions.py (versions): move the
application version dict (which gets reported to remote
subscribers, and copied into logfiles) to a separate module. It
was causing circular import problems when it lived in an attribute
of LogPublisher.
(add_version): provide a setter method
* foolscap/logging/publish.py (LogPublisher.versions): same
(LogPublisher.remote_get_versions): same
* foolscap/logging/incident.py (IncidentReporter.incident_declared):
same
* foolscap/test/test_logging.py (IncidentGatherer.test_emit):
improve test shutdown a little bit
2008-08-28 Brian Warner <warner@lothar.com>
* foolscap/logging/incident.py (IncidentReporter.incident_declared):
put versions in the incident header too, also for #80.
* foolscap/test/test_logging.py (Incidents.test_basic): test it
* foolscap/logging/tail.py (LogPrinter.got_versions): record
remote application versions to the --save-to file, in the header.
Part of #80.
(LogPrinter.got_versions): oops, control where stdout goes
* foolscap/logging/dumper.py (LogDumper.start): show those
versions with --verbose
* foolscap/test/test_logging.py (Dumper.test_dump): update to match
(Tail.test_logprinter): same, make sure got_versions is called
* foolscap/logging/gatherer.py (IncidentObserver): only fetch one
incident at a time, to limit the size of the sender's outbound
queue. This should help close #85.
(IncidentGathererService.new_incident): include the classification
results in the per-incident log messages
* foolscap/test/test_logging.py (IncidentGatherer.test_emit): tests
* foolscap/logging/log.py (FoolscapLogger.declare_incident):
combine overlapping incidents, by passing new triggers to an
existing reporter instead of creating a new one. Helps with #85.
* foolscap/logging/incident.py (IncidentReporter.new_trigger): new
method to add subsequent triggers to an existing reporter. This
doesn't do anything yet. It should be improved to record the other
triggers in a trailer (since it's too late to add it to the
header). Every triggering event will make it into an incident
somewhere, but the report-file analysis tools may not know how to
pay attention to the subsequent triggers.
* foolscap/logging/interfaces.py (IIncidentReporter.new_trigger): same
* foolscap/test/test_logging.py (Incidents.test_overlapping): test it
* doc/logging.xhtml: mention incident-coalescing
* foolscap/logging/log.py: add bridge-foolscap-logs-to-twisted
functionality, set up by either calling bridgeLogsToTwisted(), or
by setting the FLOGTOTWISTED environment variable (to anything).
The default filter will exclude events below the OPERATIONAL
severity level, and those generated by foolscap internals (i.e.
facility.startswith("foolscap") ). Closes #93.
* foolscap/pb.py (Tub.setOption): update to match
* foolscap/test/test_logging.py (Bridge): tests for it
(Publish): update to use new APIs
* doc/logging.xhtml: docs
2008-08-25 Brian Warner <warner@lothar.com>
* foolscap/broker.py (Broker.abandonAllRequests): map both
ConnectionLost and ConnectionDone into DeadReferenceError, so that
application code only needs to check for one exception type.
* foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1):
when an existing connection is dropped in favor of a new one, drop
it with DeadReferenceError instead of ConnectionDone. The mapping
in Broker.abandonAllRequests doesn't seem to quite catch
everything in unit tests.
(Negotiation.acceptDecisionVersion1): same, when we're the slave
* foolscap/test/test_call.py (TestCall.test_connection_lost_is_deadref):
test it
(TestCall.test_connection_done_is_deadref): same
(TestCall.testChoiceOf): switch to use ShouldFailMixin
* foolscap/test/test_gifts.py (Gifts): remove
ignoreConnectionDone, just look for DeadReferenceError
2008-08-21 Brian Warner <warner@lothar.com>
* foolscap/_version.py (verstr): bump revision to 0.3.0+ while
between releases
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
2008-08-04 Brian Warner <warner@lothar.com>
* foolscap/_version.py (verstr): release Foolscap-0.3.0
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
2008-08-04 Brian Warner <warner@lothar.com>
* foolscap/logging/interfaces.py
(RILogPublisher.subscribe_to_incidents): small edit, make it more
clear that since= is used for catch_up=True
* NEWS: update for the upcoming release
* foolscap/referenceable.py (RemoteReferenceOnly.getRemoteTubID):
make this secure, by using the broker's .remote_tubref field,
instead of the remote-side-controlled sturdyref.
(RemoteReferenceOnly.getSturdyRef): add a note about the
insecurity of this method
* foolscap/test/test_pb.py (TestCallable.testGetSturdyRef): add a
test for getRemoteTubID
* doc/logging.xhtml: change filenames in the incident-gatherer to
have fewer files starting with incident*, so tab-completion works
better.
* foolscap/logging/gatherer.py (IncidentGathererService): same
(create_incident_gatherer): same
* foolscap/test/test_logging.py (Filter): test 'flogtool filter'
* foolscap/logging/filter.py: control stdout/stderr better
* foolscap/test/test_logging.py (Dumper): test 'flogtool dump'
* foolscap/logging/cli.py (dispatch): pass stdout/stderr through
attributes on the subcommand's Options instance, rather than as
separate function arguments.. makes testing easier.
* foolscap/logging/gatherer.py (create_log_gatherer): same
(create_incident_gatherer): same
(CreateGatherOptions): same
(CreateIncidentGatherOptions): same
* foolscap/logging/dumper.py (DumpOptions): same
2008-08-01 Brian Warner <warner@lothar.com>
* foolscap/logging/log.py (LogFileObserver): make these more
useful, by not doing the addSystemEventTrigger in __init__
(LogFileObserver.stop_on_shutdown): do it here instead
(FLOGFILE): when using $FLOGFILE, add call to stop_on_shutdown
* foolscap/logging/gatherer.py (GathererService.do_rotate): oops,
implement the precautions claimed by the comment in startService:
test self._savefile before doing anything.
(GathererService.__init__): set self._savefile to None
* foolscap/test/test_logging.py (Gatherer.test_log_gatherer2): add
a timed rotator, which caught the problem in do_rotate
* foolscap/test/test_logging.py (IncidentGatherer.test_emit): add
test coverage for re-classifying existing incidents
(IncidentGatherer.test_emit._update_classifiers_again): and verify
that leaving the classified/ directory in place properly inhibits
reclassification at startup
* foolscap/referenceable.py (RemoteReferenceOnly.getRemoteTubID):
new method to extract the remote tubid from a RemoteReference. I'm
not sure this is a good idea, but it fixes the immediate problem
I'm dealing with.
* foolscap/logging/gatherer.py: use "tubid_s" instead of "nodeid_s"
(IncidentGathererService.remote_logport): use getRemoteTubID()
instead of trying to sanitize the tubid we receive, since we
use it as a directory name
(IncidentGathererService.add_classifier): rename addClassifier to
add_classifier, I'm more fond of the latter form these days. Also
remove a pyflakes warning.
* foolscap/test/test_logging.py (IncidentGatherer.test_emit): add
test of incident generation, publish, recording, and default
classification
* foolscap/logging/gatherer.py (IncidentGathererService): get
control over stdout, so we can exercise more code during tests
(IncidentGathererService.startService): oops, this needs to be
startService instead of start
* foolscap/test/test_logging.py (IncidentGatherer): basic test of
an incident gatherer, just setup and connection so far
* foolscap/test/common.py (StallMixin): factor stall() out into a
separate class
(TargetMixin): same
* foolscap/test/test_logging.py (LogfileReaderMixin): refactor
(Gatherer.test_log_gatherer): turn on bzip=True, for more coverage
* foolscap/test/test_logging.py (Gatherer._check_gatherer): ignore
internal foolscap messages (like connection negotiation), since
they occur at unpredictable times (specifically in
test_log_gatherer_furlfile_multiple, which has establishes
multiple connections)
* foolscap/test/common.py (GoodEnoughTub): factor this and
crypto_available out of all the other unit tests, make
GoodEnoughTub(certFile=) work even if crypto is unavailable and
we must therefore discard the certFile= argument.
* foolscap/test/test_crypto.py: same
* foolscap/test/test_gifts.py: same
* foolscap/test/test_keepalive.py: same
* foolscap/test/test_logging.py: same
* foolscap/test/test_loopback.py: same
* foolscap/test/test_negotiate.py: same
* foolscap/test/test_pb.py: same
* foolscap/test/test_serialize.py: same
* foolscap/test/test_tub.py: same
* foolscap/logging/gatherer.py (GatheringBase): rewrite Gatherers
to make them easier to test: now they are a Service (with a
subordinate Tub), meant to be run by a .tac file or manually. The
intermediate class has been removed. All .tac files are unchanged:
gatherers created by old versions of 'flogtool create-gatherer'
will continue to work.
(GathererService.do_rotate): return the name of the logfile that
was just closed and/or compressed, so tests can know where to
look.
* foolscap/test/test_logging.py (Gatherer): update to match
* foolscap/util.py (get_local_ip_for): update reactor comment
2008-07-30 Brian Warner <warner@lothar.com>
* setup.py: update comment about bug #62 (pyopenssl problems)
* foolscap/test/test_logging.py (CLI.test_create_gatherer):
improve test coverage a little bit, by recording stdout
(CLI.test_create_incident_gatherer): exercise the 'flogtool
create-incident-gatherer' command
* foolscap/logging/cli.py (run_flogtool): capture stdout+stderr
(dispatch): same
* foolscap/logging/gatherer.py (create_log_gatherer): same
(create_incident_gatherer): same
* trees/tahoe/Makefile (figleaf-output): exclude foolscap/test/ from
the HTML results
* doc/logging.xhtml (Running an Incident Gatherer): describe the
Incident Gatherer, like the Log Gatherer but it only gathers
incidents. It also does classification, and will eventually do
reporting. No unit tests yet, but some manual system-level tests
have been run.
* foolscap/logging/gatherer.py (IncidentGatherer): implement it
* foolscap/logging/cli.py (Options.subCommands): add the CLI
command, named 'flogtool create-incident-gatherer'
* foolscap/logging/incident.py
(IncidentReporter.incident_declared): clean up incident naming
* foolscap/logging/interfaces.py (RILogPublisher.list_incidents): same
* foolscap/logging/log.py (FoolscapLogger.incident_recorded): same
* foolscap/logging/publish.py (LogPublisher.list_incident_names): same
* foolscap/test/test_logging.py
(IncidentPublisher.test_list_incident_names): test the
incident-naming cleanup
2008-07-29 Brian Warner <warner@lothar.com>
* foolscap/logging/gatherer.py (GatheringBase): refactor the log
gatherer to share code with the upcoming incident gatherer
(LogGatherer.__init__): add a basedir= argument, rather than
using os.getcwd
* foolscap/test/test_logging.py (Gatherer): add basedir= argument
* doc/logging.xhtml (Setting up the logport): New feature (well,
it didn't work before, and now it does, so make it explicit): the
log-gatherer FURL can be configured (but will not be connected)
until after setLocation. This should resolve a crash I've seen in
Tahoe (which runs a slow /sbin/ifconfig command to figure out the
addresses to pass to setLocation) in which the app connects to the
log gatherer before it figures out its own location, and then gets
an exception during registerReference. Closes #55.
* foolscap/pb.py (Tub._maybeConnectToGatherer): don't initiate the
gatherer connection until locationHints is set
(Tub.setLocation): call _maybeConnectToGatherer after the location
is set. Also, don't let setLocation be called multiple times.
* foolscap/test/test_logging.py
(Publish.test_logport_furlfile2): test it
(Gatherer.test_log_gatherer2): same
(Gatherer.test_log_gatherer_furlfile2): same
* doc/logging.xhtml (Setting up the logport): slight API
restriction: the logport and its FURL are not available until
after Tub.setLocation is called. This results in a better error
message than the usual one inside registerReference.
* foolscap/pb.py (Tub.getLogPort): enforce the API restriction by
throwing an exception when it is violated
(Tub.getLogPortFURL): same
* foolscap/tokens.py (NoLocationError): new exception for it
* foolscap/test/test_tub.py (SetLocation.test_set_location): test it
* doc/logging.xhtml (Configuring a Log Gatherer): allow multiple
log-gatherer furls in the log-gatherer-furlfile
* foolscap/pb.py (Tub._maybeConnectToGatherer): same
* foolscap/logging/gatherer.py (LogGatherer.remote_logport):
return the subscribe_to_all Deferred, for testing
* foolscap/test/test_logging.py (Gatherer.test_log_gatherer):
refactor, to accomodate new test
(Gatherer.test_log_gatherer_furlfile_multiple): test it
2008-07-28 Brian Warner <warner@lothar.com>
* foolscap/logging/interfaces.py (RILogPublisher.list_incidents):
change signature to remove tubid/incarnation from the response.
(RILogPublisher.subscribe_to_incidents): add pubsub interface for
incidents, including catch_up= and since=
(RILogObserver.new_incident): same
(RILogObserver.done_with_incident_catchup): same
* foolscap/logging/publish.py (IncidentSubscription): same
* foolscap/logging/incident.py
(IncidentReporter.finished_recording): tell the logger about the
incident name, so it can publish it.
* foolscap/logging/log.py
(FoolscapLogger.addImmediateIncidentObserver): same
(FoolscapLogger.incident_recorded): same
* foolscap/test/test_logging.py (IncidentPublisher._check_listed):
same
(IncidentPublisher.test_subscribe): test it
* foolscap/logging/interfaces.py (RISubscription.unsubscribe):
move the RILogPublisher.unsubscribe() method to the RISubscription
object, since that's a better place for it.
(RILogPublisher.unsubscribe): deprecate this one
* foolscap/logging/publish.py (Subscription.remote_unsubscribe): same
* foolscap/logging/publish.py (LogPublisher.remote_get_incident):
reject invalid incident names
* doc/logging.xhtml: add details about running a Log Gatherer
(Python 'logging' module): comment out this section, it is wrong
2008-07-09 Brian Warner <warner@lothar.com>
* doc/specifications/logfiles.xhtml: fix typos
* doc/logging.xhtml (Remote log aggregation): same
2008-07-07 Brian Warner <warner@lothar.com>
* foolscap/logging/dumper.py (LogDumper): when dumping an Incident
Report, mark the triggering event with "[INCIDENT-TRIGGER]"
* foolscap/_version.py (verstr): bump revision to 0.2.9+ while
between releases
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
2008-07-02 Brian Warner <warner@lothar.com>
* foolscap/_version.py (verstr): release Foolscap-0.2.9
* misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog:
same
* NEWS: update for the upcoming release
* MANIFEST.in: add gutsy/hardy directories
2008-07-02 Brian Warner <warner@allmydata.com>
* foolscap/test/test_logging.py
(Publish.test_logpublisher_overload): change poller condition to
avoid spurious failures on slow systems
* misc/*/debian/watch: update to point at foolscap.lothar.com
* Makefile (debian-gutsy, debian-hardy): add .deb targets for
gutsy and hardy. The rules are the same as for feisty. Closes #76.
* misc/gutsy/*, misc/hardy/*: same
* foolscap/logging/publish.py (Subscription.send): change the
discard policy to discard-new instead of discard-random: in tests
with a busy Tahoe node, this seems to give good enough
behavior (probably since the busyness is bursty), although I can't
say we fully understand what's really going on. discard-new lets
us use a faster and simpler deque instead of requiring random
access to the list.
* foolscap/test/test_logging.py
(Publish.test_logpublisher_overload): update to match
2008-07-02 Brian Warner <warner@lothar.com>
* foolscap/logging/publish.py (Subscription.send): add a
size-limited queue for sending messages to subscribers, with a
default queue size of 2000 entries. This should probably keep the
memory footprint bounded to perhaps 1MB. When the queue gets full,
we randomly discard old messages, so recent messages are more
likely to survive than earlier ones. We allow 10 messages to be
outstanding on the wire at once, to pipeline them a bit and
improve network utilization. Any errors during sending will cause
the subscription to be dropped. This should close #72.
* foolscap/logging/log.py (FoolscapLogger.addImmediateObserver):
add a new kind of observer, so that the logport publisher can also
avoid unboundedness in the eventual-send queue. If we can't throw
away messages fast enough, callers to log.msg will block, slowing
down the inlet rate.
* foolscap/test/test_logging.py
(Publish.test_logpublisher_overload): test it, by throwing 10k
messages at log.msg and counting how many make it through.
* foolscap/test/common.py (PollMixin): add a comment: this
chained-Deferred pattern will run up against python's recursion
limit if the check function is called more than about 300 times.
2008-07-01 Brian Warner <warner@lothar.com>
* foolscap/logging/publish.py (Subscription): refactor a bit, in
preparation for #72 limit log-publishing queue size
2008-07-01 Brian Warner <warner@allmydata.com>
* doc/logging.xhtml ("That Was Weird" Buttons): provide a simple
example of giving the user a way to trigger Incident logging.
Closes #75.
* foolscap/logging/interfaces.py (RILogPublisher.get_pid): new
interface, to retrieve the process ID through the logport
* foolscap/test/test_logging.py
(Publish.test_logpublisher_catchup._got_logport._check_pid): test it
* foolscap/logging/publish.py (LogPublisher.remote_get_pid):
implement it
* foolscap/logging/tail.py (LogTail._got_logpublisher._announce):
Use it, but tolerate old logports that don't offer it. Closes #71.
* foolscap/slicers/decimal_slicer.py: handle Decimal objects,
serializing them as a string. No constraints yet. Closes #50.
* foolscap/slicers/allslicers.py: import decimal_slicer
* foolscap/test/test_banana.py (ThereAndBackAgain.test_decimal):
test it
* foolscap/logging/tail.py (LogTail._got_logpublisher): exit if we
get an error while connecting or subscribing. Closes #63.
(LogTail._print_versions): print remote versions immediately after
connecting to the logport. Closes #70.
* foolscap/logging/log.py (FoolscapLogger.setLogDir): oops, allow
the incident directory to be re-used: os.makedirs throws an
exception if the directory already exists.
* foolscap/test/test_logging.py (Incidents.test_basic): test it
* foolscap/_version.py (verstr): bump revision to 0.2.8+ while
between releases
* misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same
2008-06-04 Brian Warner <warner@lothar.com>
* foolscap/_version.py (verstr): release Foolscap-0.2.8
* misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same
* NEWS: update for the upcoming release
2008-06-04 Brian Warner <warner@allmydata.com>
* foolscap/pb.py (Tub._tubsAreNotRestartable): fix args so that
these methods actually get run (and produce a useful error
message, instead of TypeError). Thanks to Brian Granger for the
patch. Closes #65.
* foolscap/test/test_tub.py (Shutdown.test_doublestop): test it
* foolscap/test/common.py (ShouldFailMixin.shouldFail): add some
support code
* foolscap/referenceable.py (RemoteReferenceOnly.getPeer): new
method to get the IP address and port number of the other end of a
connection. This returns a twisted.internet.interfaces.IAddress
provider. Loopback connections give
foolscap.broker.LoopbackAddress instances. Real remote connections
give twisted.internet.address.IPv4Address instances, so you can use
rref.getPeer().host and rref.getPeer().port on them. Closes #45.
* setup.py: add an "extras_require" clause to the
setuptools-specific setup args, to declare that our
"secure_connections" feature requires pyOpenSSL. This helps other
packages, which can declare a dependency on
"Foolscap[secure_connections]", rather than claiming to require
pyOpenSSL themselves. Addresses #66.
* foolscap/pb.py (Tub.registerReference): fix an exception that
occurs if you call this with both name= and furlFile= and the
furlFile already exists. Also prohibit attempts to change the name
to something other than what is in the furlFile. Thanks to Brian
Granger for the patch. Closes #64.
* foolscap/_version.py (verstr): bump revision to 0.2.7+ while
between releases
* misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same
2008-05-13 Brian Warner <warner@lothar.com>
* foolscap/_version.py (verstr): release Foolscap-0.2.7
* misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same
* NEWS: update for new "OMG flogtool is broken" release.
2008-05-12 Brian Warner <warner@allmydata.com>
* foolscap/logging/cli.py (run_flogtool): fix use of sys.argv,
the previous version was completely broken
* foolscap/_version.py (verstr): bump revision to 0.2.6+ while
between releases
* misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same
2008-05-06 Brian Warner <warner@lothar.com>
* foolscap/_version.py (verstr): release Foolscap-0.2.6
* misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same
* NEWS: update for the upcoming release
2008-05-05 Brian Warner <warner@allmydata.com>
* foolscap/test/test_logging.py (Web): add tests for 'flogtool
web-viewer'
* foolscap/logging/web.py (WebViewer.start): make this class more
amenable to being tested
* foolscap/logging/interfaces.py: misc cleanups
(RILogPublisher.list_incidents): provide an interface to retrieve
stored incident reports from a logport.
(RILogPublisher.get_incident): same
* foolscap/logging/publish.py (LogPublisher): same
* foolscap/test/test_logging.py (IncidentPublisher): test it
* foolscap/pb.py (Tub.setup): make the Tub's logger a bit easier
to override, for tests
* foolscap/logging/incident.py (IncidentReporter): make the
compressed logfile use a temporary name until we've closed it, so
later observers don't get tricked into using an incomplete
compressed logfile (they should use the more-complete uncompressed
logfile instead).
(IncidentReporter.incident_declared): make it easier to turn off
the gather-trailing-events behavior
(NonTrailingIncidentReporter): convenience subclass that does that
* doc/logging.xhtml: update to match current reality, remove some
TODO warnings
* foolscap/logging/log.py (FoolscapLogger): rename
setIncidentReporterClass to setIncidentReporterFactory
* foolscap/test/test_logging.py (Incidents.test_customize): same
* foolscap/logging/interfaces.py (RISubscription): remove spurious
'pass', figleaf thought it was real code
* bin/flogtool: split out the CLI dispatcher to..
* foolscap/logging/cli.py (run_flogtool): here, to help with #51
(run_flogtool): make it possible to run with a wrapper, closes #51.
* foolscap/test/test_logging.py (CLI.test_wrapper): test a wrapper,
provide an example of how to build one
* foolscap/logging/gatherer.py (CreateGatherOptions.optFlags): add
a --quiet flag to make the unit test less noisy
* misc/testutils/trial_figleaf.py: make this compatible with
twisted-8.0.x: the earlier version didn't write out any coverage
data on newer twisteds
* foolscap/logging/log.py (FoolscapLogger.setLogDir): wire up
Incident handling. We still need incident publishing. This gets
us most of the way to #61.
(setIncidentQualifier): this customizes the qualification function
(setIncidentReporterClass): this customizes the reporter/recorder
* foolscap/logging/levels.py: move log.WEIRD and friends here to
avoid circular import problems elsewhere. They are still available
from log.py .
* foolscap/logging/incident.py (IncidentReporter.incident_declared):
flush the uncompressed logfile just before we switch into
gather-trailing-events mode
(IncidentReporter.stop_recording): oops, fix typo
* foolscap/test/test_logging.py (Incidents): test most
incident-handling functionality
2008-05-02 Brian Warner <warner@allmydata.com>
* foolscap/logging/incident.py: start to implement Incident
handling, for ticket #61. Not complete yet, might be completely
broken, needs tests and to be wired up.
* foolscap/logging/log.py (FoolscapLogger.setIncidentQualifier):
start to add the new interfaces, not complete yetxo
* foolscap/logging/interfaces.py (IIncidentReporter): same
* doc/logging.xhtml (Incidents): document the new features
2008-05-01 Brian Warner <warner@allmydata.com>
* doc/logging.xhtml: fix discussion of format=, since the
instructions and examples were simply wrong
* doc/specifications/logfiles.xhtml (Logfile Headers): define
"headers", a separate dictionary at the start of the logfile that
contains metadata. The specific use for this will be the
"Triggering Event" that gets put into incident reports, once we
implement those. This induces a backwards compatibility break:
logfiles produced after this change will probably not be tolerated
by tools like 'flogtool dump' from before this point. The other
direction is ok: newer tools can handle either format.
* foolscap/logging/gatherer.py (LogGatherer._open_savefile): emit
a header, with type="gatherer", and a starting timestamp in
"start". Refactor a bit to make this easier.
* foolscap/logging/tail.py (LogPrinter): emit a header, with
type="tail"
* foolscap/logging/log.py (LogFileObserver): emit a header, with
type="log-file-observer", and a "threshold" key
* foolscap/logging/dumper.py (LogDumper.start): tolerate headers
* foolscap/logging/filter.py (Filter.run): same
* foolscap/logging/web.py (WebViewer.process_logfiles): same
* foolscap/test/test_logging.py: update tests to match
* doc/specifications/logfiles.xhtml: document the current saved
logfile format (event dictionaries, pickled wrapper dicts)
* foolscap/referenceable.py (SturdyRef): accept multiple
connection hints in unauthenticated FURLs. This fixes a test
failure induced by the #60 changes when pyOpenSSL is unavailable.
* doc/using-foolscap.xhtml: document the new feature.
* foolscap/referenceable.py (SturdyRef): tolerate extensions in
tubid and location-hints fields, change FURL parsing, store
structured hints in the SturdyRef instead of just strings. This
should give us some wiggle room in the future to gracefully
transition applications to using new features while retaining
backwards compatibilty. Many thanks to Zooko for the suggestion.
Closes #60.
* foolscap/negotiate.py (TubConnector): same
* foolscap/test/test_sturdyref.py (URL.testTubIDExtensions): new
tests for it
(URL.testLocationHintExtensions): same
* foolscap/test/test_tub.py (SetLocation.test_set_location):
update to match
* foolscap/test/test_gifts.py (Bad): same
* foolscap/test/test_negotiate.py
(Versus.testVersusHTTPServerAuthenticated): same
* foolscap/base32.py (is_base32): new utility function
* foolscap/test/test_util.py (Base32.test_is_base32): test it
2008-04-22 Brian Warner <warner@lothar.com>
* doc/listings/command-client.py: don't emit extra newlines, use
/usr/bin/env on shbang line
* doc/listings/command-server.py: use /usr/bin/env on shbang line
2008-04-22 Brian Warner <warner@allmydata.com>
* foolscap/logging/filter.py: add "--above UNUSUAL" option (to
discard events below the given level", and "--from [TUBID]" (to
discard events that weren't recorded by the given tubid).
* foolscap/logging/dumper.py (LogDumper.print_event): use
--rx-time to show both event-generation time and event-receive
time in the logs, useful if you suspect the application is getting
bogged down and events are being delivered slowly.
(LogDumper.print_event): Use a different format to display failure
tracebacks.
* foolscap/logging/web.py (WebViewer.get_events): catch
ValueError, mention it as a possible truncated pickle file
* foolscap/logging/log.py (FoolscapLogger.msg): if something goes
wrong, print both str() and repr() in case it helps figure out the
problem
* foolscap/logging/dumper.py (LogDumper.print_event): include
failure tracebacks in 'flogtool dump' output
2008-04-09 Brian Warner <warner@allmydata.com>
* foolscap/constraint.py (Constraint.checkOpentype): always
accept ('reference',) sequences. The per-token constraint checking
system is a defense against resource-exhaustion attacks, and
shared reference don't consume any more memory or stack frames
than any other object. The check-all-args that CallUnslicer does
just before delivering the arguments is responsible for making
sure the final (resolved) types all match the constraint. By
allowing shared references here, we fix a bug in which a schema
violation was raised when python combined two equivalent tuples
into a single object.
* foolscap/test/test_call.py (TestCall.testMega3): test it
* foolscap/test/common.py (MegaSchema3): same
* setup.py: finally remove zip_safe=False
* foolscap/test/test_interfaces.py (TestInterface.testStack):