Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement PRIVCOUNT_HSDIR_CACHE_STORE and counters #336

Closed
teor2345 opened this issue Jun 5, 2017 · 2 comments
Closed

Implement PRIVCOUNT_HSDIR_CACHE_STORE and counters #336

teor2345 opened this issue Jun 5, 2017 · 2 comments
Assignees
@teor2345
Labels
enhancement
Milestone
1.1.0 - Circuit, ...

Comments

Assignees

@teor2345 teor2345

Labels
enhancement
Projects
None yet
Milestone
1.1.0 - Circuit, Cell, and HSDir Stor...
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
@teor2345
@teor2345
Copy link
Collaborator

@teor2345 teor2345 commented Jun 5, 2017 •
edited

We can produce the following statistics without storing HS addresses in RAM.
This is safer and simpler (because we don't need to decide on rotation periods).

  • How many descriptors uploaded per HSDir?
  • How many bytes ~~~/cells~~~ per descriptor? (Histogram)
  • How many descriptor bytes ~~~/cells~~~ per HSDir?
  • How many intro points per descriptor? (Histogram, HSv2 only)
  • How many failed descriptor uploads per HSDir?

Less interesting:

  • Circuit counts
  • Circuit lifetimes
  • Ratios? Connections? (No Client IPs)

Advanced (needs circuit info):
Like Tor trac #13792:

  • Upload time: Time of service upload

Advanced (needs storage in RAM?):

  • How many descriptor versions per HSDir? (Histogram, needs storage for HSv2)
  • What is the delay between version uploads?
  • Is the same version ever uploaded twice? Are there gaps between versions?
  • What is the inter-upload delay time?
  • How many uploads per hidden service? (Histogram)
  • Tor trac #13195:
    • How many unique hidden services are uploaded? (Histogram?)
  • Tor trac #15272, #13208, #13792:
    • What is the intro point churn rate?
    • How often do we change our IPs from service-side? How long do we keep a typical IP?
    • How often do we re-upload?

Research (needs to be turned into a series of statistical questions):
@teor2345 teor2345 added this to the 1.2.0/2.0.0 - Onion Service Stats milestone Jun 5, 2017
@teor2345 teor2345 self-assigned this Jun 5, 2017
@teor2345 teor2345 added this to the 1.2.0 - HSDir Upload Stats milestone Jun 5, 2017
@teor2345 teor2345 removed this from the 1.3.0 - More Onion Service Stats milestone Jun 5, 2017
@teor2345 teor2345 mentioned this issue Jun 5, 2017
Closed
@teor2345 teor2345 mentioned this issue Jun 5, 2017
Closed
@teor2345
Copy link
Collaborator Author

@teor2345 teor2345 commented Jun 7, 2017

It is not easy to find the circuit used to upload a particular descriptor. So we won't have any cell counts.

HSv2 descriptors do not have a version number: instead, they have a creation timestamp. So we can't do version histograms for v2 without storing descriptor creation times, and then creating our own order.

It also requires some work to produce a hidden service address from a v2 descriptor: the address is in the descriptor, but not parsed by existing code. Maybe we should leave it that way, and use the descriptor ID instead. I'll document the differences.

@teor2345 teor2345 changed the title Implement HSDir upload statistics Implement HSDir descriptor cache addition statistics Jun 7, 2017
@teor2345 teor2345 changed the title Implement HSDir descriptor cache addition statistics Implement HSDir descriptor cache add statistics Jun 7, 2017
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 7, 2017
@teor2345
Add a prefix string argument to privcount_timeval_*to_epoch_str_dup()
d7d9c27 
Preparation for privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 7, 2017
@teor2345
Add comments categorising existing event categories and existing events
4104edf 
Preparation for privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 7, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
7f0e47e 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the descriptor ID (v2) or blinded public key (v3), intro point
count (v2 only), times, sizes, and caching and validation information.

It does not contain the hidden service address, or any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/tor-privcount that referenced this issue Jun 8, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
cd37f8f 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the descriptor ID (v2) or blinded public key (v3), intro point
count (v2 only), times, sizes, and caching and validation information.

It does not contain the hidden service address, or any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 8, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
c36644d 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the descriptor ID (v2) or blinded public key (v3), intro point
count (v2 only), times, sizes, and caching and validation information.

It does not contain the hidden service address, or any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 8, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
d9d2a2a 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the descriptor ID (v2) or blinded public key (v3), intro point
count (v2 only), times, sizes, and caching and validation information.

It does not contain the hidden service address, or any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 8, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
278d849 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the descriptor ID (v2) or blinded public key (v3), intro point
count (v2 only), times, sizes, and caching and validation information.

It does not contain the hidden service address, or any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 8, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORED and tagged fields
06222bc 
Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 9, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORED and tagged fields
cb2c0d4 
Part of privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 9, 2017
@teor2345
Add a prefix string argument to privcount_timeval_*to_epoch_str_dup()
2704edf 
Preparation for privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 9, 2017
@teor2345
Add comments categorising existing event categories and existing events
c9a5985 
Preparation for privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 9, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
95bfa49 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the onion address (v2) or blinded public key (v3), intro point
count (v2 only), client auth flag, times, sizes, and caching and validation
information.

It does not contain any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 9, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORED and tagged fields
9f725be 
Part of privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 14, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
ee7bf29 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the onion address (v2) or blinded public key (v3), intro point
count (v2 only), client auth flag, times, sizes, and caching and validation
information.

It does not contain any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 14, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORED event
7c74fda 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the onion address (v2) or blinded public key (v3), intro point
count (v2 only), client auth flag, times, sizes, and caching and validation
information.

It does not contain any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 14, 2017
@teor2345
Refactor string summary code into summarise_string
ed11daf 
Preparation for privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 14, 2017
@teor2345
Update test/README.markdown for HSDir test events
3c6ca63 
Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 14, 2017
@teor2345
Log requested events in test_tor_ctl_event.py
54dd293 
Diagnostic, part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 14, 2017
@teor2345
Implement and document HSDirStore counters
feaf3ea 
These counters count:
* descriptor stores
* intro point counts (HSv2 only)
* descriptor and intro point bytes
* descriptor upload delays (HSv2 only), and
* descriptor revisions (HSv3 only).

In the following categories:
* cache add/reject,
* cache new/existing,
* cache action reason,
* client auth/no client auth (HSv2 only),

Requires a PrivCount Tor Patch that supports
EVENT_PRIVCOUNT_HSDIR_CACHE_STORED.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 14, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORED and tagged fields
bfa9e16 
Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 22, 2017
@teor2345
Release Notes for privcount#336 and related tickets
429ae12
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 22, 2017
@teor2345
Implement expected events for HSDir v2 and v3
a44edaf 
Part of privcount#360 and privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 22, 2017
@teor2345
Implement expected events for HSDir v2 and v3
94313b9 
Part of privcount#360 and privcount#336.
@robgjansen robgjansen mentioned this issue Jun 23, 2017
Closed
teor2345 added a commit to teor2345/tor-old that referenced this issue Jun 23, 2017
@teor2345
Add the EVENT_PRIVCOUNT_HSDIR_CACHE_STORE event
6308e19 
This event is emitted when a HS descriptor is uploaded to a v2 or v3 HSDir.
It contains the onion address (v2) or blinded public key (v3), intro point
fingerprints and count (v2 only), client auth flag, times, sizes, and caching
and validation information.

It does not contain any circuit information.

Part of privcount/privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORE and tagged fields
7d28221 
Using tagged fields for new counters is part of privcount#256.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Log requested events in test_tor_ctl_event.py
fedfba1 
Diagnostic, part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Implement and document HSDirStore counters
f44d4db 
These counters count:
* descriptor stores
* intro point counts (HSv2 only)
* descriptor and intro point bytes
* descriptor upload delays (HSv2 only), and
* descriptor revisions (HSv3 only).

In the following categories:
* cache add/reject,
* cache new/existing,
* cache action reason,
* client auth/no client auth (HSv2 only).

Requires a PrivCount Tor Patch that supports
EVENT_PRIVCOUNT_HSDIR_CACHE_STORE.

Creating counter variants from template strings is part of privcount#229.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Add test events and test configs for HSDir Stores
5138c42 
Add privcount/tools/add_counter.sh for initial counter configs.
(Bins for multi-bin counters need to be added manually.)

Fix the counter match tests for template counters, update to privcount#347.

Update test documentation and comments for HSDirStore counters.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Increase the maximum tor event size
c3dc3df 
With fingerprints, HS Store events can be up to 1kB long.
So we now warn at 2kB, reject at 20kB.
(Was warn at 1.6kB, reject at 16kB.)

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Release Notes for privcount#336 and related tickets
f79697b
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Implement expected events for HSDir v2 and v3
56116f8 
Part of privcount#360 and privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORE and tagged fields
34e8c8d 
Using tagged fields for new counters is part of privcount#256.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Log requested events in test_tor_ctl_event.py
3c46da1 
Diagnostic, part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Implement and document HSDirStore counters
c5494fc 
These counters count:
* descriptor stores
* intro point counts (HSv2 only)
* descriptor and intro point bytes
* descriptor upload delays (HSv2 only), and
* descriptor revisions (HSv3 only).

In the following categories:
* cache add/reject,
* cache new/existing,
* cache action reason,
* client auth/no client auth (HSv2 only).

Requires a PrivCount Tor Patch that supports
EVENT_PRIVCOUNT_HSDIR_CACHE_STORE.

Creating counter variants from template strings is part of privcount#229.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Add test events and test configs for HSDir Stores
078d5c3 
Add privcount/tools/add_counter.sh for initial counter configs.
(Bins for multi-bin counters need to be added manually.)

Fix the counter match tests for template counters, update to privcount#347.

Update test documentation and comments for HSDirStore counters.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Increase the maximum tor event size
4345269 
With fingerprints, HS Store events can be up to 1kB long.
So we now warn at 2kB, reject at 20kB.
(Was warn at 1.6kB, reject at 16kB.)

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Release Notes for privcount#336 and related tickets
e745bb3
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 23, 2017
@teor2345
Implement expected events for HSDir v2 and v3
dbced75 
Part of privcount#360 and privcount#336.
@teor2345 teor2345 mentioned this issue Jun 29, 2017
Open
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Document PRIVCOUNT_HSDIR_CACHE_STORE and tagged fields
c7a3692 
Using tagged fields for new counters is part of privcount#256.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Log requested events in test_tor_ctl_event.py
95704f3 
Diagnostic, part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Implement and document HSDirStore counters
a5b8b22 
These counters count:
* descriptor stores
* intro point counts (HSv2 only)
* descriptor and intro point bytes
* descriptor upload delays (HSv2 only), and
* descriptor revisions (HSv3 only).

In the following categories:
* cache add/reject,
* cache new/existing,
* cache action reason,
* client auth/no client auth (HSv2 only).

Requires a PrivCount Tor Patch that supports
EVENT_PRIVCOUNT_HSDIR_CACHE_STORE.

Creating counter variants from template strings is part of privcount#229.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Add test events and test configs for HSDir Stores
c33ca56 
Add privcount/tools/add_counter.sh for initial counter configs.
(Bins for multi-bin counters need to be added manually.)

Fix the counter match tests for template counters, update to privcount#347.

Update test documentation and comments for HSDirStore counters.

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Increase the maximum tor event size
b968e4b 
With fingerprints, HS Store events can be up to 1kB long.
So we now warn at 2kB, reject at 20kB.
(Was warn at 1.6kB, reject at 16kB.)

Part of privcount#336.
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Release Notes for privcount#336 and related tickets
695530f
teor2345 added a commit to teor2345/privcount that referenced this issue Jun 29, 2017
@teor2345
Implement expected events for HSDir v2 and v3
ffebb3a 
Part of privcount#360 and privcount#336.
@teor2345
Copy link
Collaborator Author

@teor2345 teor2345 commented Jun 29, 2017

This was closed in ffebb3a, remaining work is in #367.

@teor2345 teor2345 closed this Jun 29, 2017
teor2345 added a commit to teor2345/privcount that referenced this issue Jul 18, 2017
@teor2345
Make the get_*_value comments consistent in tagged_event.py
acffb05 
Bugfix on privcount#336.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment