-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge the satellite base data into the merged table #236
Conversation
resolver_name AS server_name, | ||
domain_is_control, | ||
domain, | ||
NULL as outcome, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calculate the outcome here, and remove received_rcode and answers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
answers | ||
FROM `PROJECT_NAME.BASE_DATASET.satellite_scan` AS a | ||
# Only include the last measurement in any set of retries | ||
JOIN `PROJECT_NAME.DERIVED_DATASET.satellite_last_measurement_ids` AS b |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move the filtering and the Outcome calculation to a previous WITH.. AS statement, so all the tables are pref-filtered and with outcome in this UNION ALL query. Easier to reason about.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
), Grouped AS ( | ||
SELECT | ||
date, | ||
source, | ||
server_country AS country_code, | ||
# As per https://docs.censoredplanet.org/dns.html#id2, some resolvers are named `special` instead of the real hostname. | ||
IF(server_name="special","special",NET.REG_DOMAIN(server_name)) as reg_hostname, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to the Satelite-specific query. Let's have only code that applies to all sources after the union.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
99a6cf0
to
af4f81e
Compare
@@ -42,8 +42,7 @@ | |||
# The test table is written into the <project>:test dataset | |||
BEAM_TEST_BASE_DATASET = 'test' | |||
BEAM_TEST_BASE_TABLE_SUFFIX = '_scan' | |||
DERIVED_TABLE_NAME_HYPERQUACK = 'merged_reduced_scans_v2' | |||
DERIVED_TABLE_NAME_SATELLITE = 'reduced_satellite_scans_v1' | |||
DERIVED_TABLE_NAME = 'merged_reduced_scans_v3' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not blocking, but ideally we would call it DASHBOARD_TABLE_NAME
and change the dataset name to dashboard
.
WHEN (SELECT LOGICAL_OR(answer.http_analysis_is_known_blockpage) | ||
FROM UNNEST(answers) answer) | ||
THEN CONCAT("❗️page:http_blockpage:", answers[OFFSET(0)].http_analysis_page_signature) | ||
WHEN (SELECT LOGICAL_OR(answer.https_analysis_is_known_blockpage) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can't possibly trigger, since we will trigger the cert cases first.
Remove?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
WHEN (SELECT LOGICAL_OR(answer.matches_control.asn) | ||
FROM UNNEST(answers) answer) | ||
THEN "✅answer:matches_asn" | ||
ELSE CONCAT("❓answer:not_validated:", AnswersSignature(answers)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: rename to "fetch failed" or something clearer. This indicates we were not able to establish a TLS connection which is clearly unexpected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this was worked on seperatly in #247
2e78ebf
to
3c781ed
Compare
ea13b45
to
15e914e
Compare
4529b4b
to
68e416b
Compare
68e416b
to
fa4d36d
Compare
28081d0
to
c370428
Compare
21eff76
to
1e41b5f
Compare
dashboard using this table: https://lookerstudio.google.com/c/u/0/reporting/d5506fb3-c9f7-42c6-894a-55e4b0528c8e/page/p_011unrxq4c
Currently the number if satellite rows is much larger than any other data source
![image](https://private-user-images.githubusercontent.com/1127209/239546091-b062e524-9281-4330-ab60-6534138c46ea.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNDgzNTQsIm5iZiI6MTcyNDI0ODA1NCwicGF0aCI6Ii8xMTI3MjA5LzIzOTU0NjA5MS1iMDYyZTUyNC05MjgxLTQzMzAtYWI2MC02NTM0MTM4YzQ2ZWEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjFUMTM0NzM0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YWE2YjQyMjg3MzBmMTdmZTM5MGExNzgxZmIyYmFmYTQ5ZDIzMDNmYmUyYjhlYTQ1Y2U3MWZhOTZjNDFmNzA4MiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.4PpoJ7FnYMu1DyWSBF9RuxjJb-MbSOFufPHX7mGzn5g)
probably because we're including the hostname, which splits potentially merged rows out into many.