Add basic support of IETF QUIC by maskit · Pull Request #2342 · apache/trafficserver

maskit · 2017-08-08T02:33:29Z

This PR is used to run CI jobs against quic-latest branch, and the branch is used for catching up new drafts. Also, this enables you to point out any issues on the latest code on this PR before we make another PR for supporting a new draft on master (since draft-20 support was merged the diff is smaller than before).

This PR will be merged to master when we agree on adding experimental support for QUIC to a future release. In the meanwhile, we are going to maintain quic-latest branch to implement QUIC collaboratively.

Please feel free to comment on this PR, and improve it with your PRs against quic-latest branch.

The whole activity will be tracked on this project below.
https://github.com/apache/trafficserver/projects/8

Some documentations are available on our wiki.
https://cwiki.apache.org/confluence/display/TS/QUIC

Committers, please do not merge / close this PR for a while.

bryancall

Comments for now. I am 1/4 of the way through the review.

bryancall · 2017-08-08T15:30:26Z

+//  class QUICNetProcessor
+//
+//////////////////////////////////////////////////////////////////
+struct QUICNetProcessor : public UnixNetProcessor {


Why not a class?

bryancall · 2017-08-08T15:33:31Z

+// (another 20-60 bytes on average, depending on the negotiated ciphersuite [2]).
+// All in all: 1500 - 40 (IP) - 20 (TCP) - 40 (TCP options) - TLS overhead (60-100)
+// For larger records, the size is determined by TLS protocol record size
+#define QUIC_DEF_TLS_RECORD_SIZE 1300  // 1500 - 40 (IP) - 20 (TCP) - 40 (TCP options) - TLS overhead (60-100)


Shouldn't this be based on the max size of a QUIC packet?

Created an issue for this (#2377).

bryancall · 2017-08-08T15:37:47Z

+// class QUICNextProtocolSet;
+// struct QUICCertLookup;
+
+typedef enum {


Would be better to use an enum class for type safety.

Created an issue for this (#2378).

bryancall · 2017-08-08T15:38:09Z

+//
+//////////////////////////////////////////////////////////////////
+
+typedef std::unique_ptr<uint8_t> ats_uint8_t_unique_ptr;


We switched over to use the C++11 using syntax.

bryancall · 2017-08-08T15:39:27Z

+ **/
+class QUICNetVConnection : public UnixNetVConnection, public QUICPacketTransmitter, public QUICFrameTransmitter
+{
+  typedef UnixNetVConnection super; ///< Parent type.


We switched over to use the C++11 using syntax.

bryancall · 2017-08-08T15:52:22Z

+  na->action_->server = &na->server;
+  na->init_accept();
+
+  udpNet.UDPBind((Continuation *)na, &na->server.accept_addr.sa, 1024000, 1024000);


Why not 1048576 for the send and recv buffers?

bryancall · 2017-08-08T16:06:59Z

+    this->_connections.put(packet.connection_id(), vc);
+  }
+
+  uint8_t udp_payload[65536];


Why are we creating a buffer of 64K when we the max size of a QUIC packet should be:
"The recommended default maximum packet size is 1350 bytes for IPv6 and 1370 bytes for IPv4"

The maximum QUIC packet size depends PMTU. So, it can be the same as the maximum payload size of UDP.

Please read draft-04. I googled the sentence and found on a very old draft.

I had to google this too... It says that for IPv6, jumbograms can be over >64K as well ...

Now it uses PMTU here. c6b0534

bryancall · 2017-08-08T16:08:04Z

+{
+  // TODO: remove a connection which is created by Client Initial
+  //       or update key to new one
+  if (!this->_connections.get(packet.connection_id())) {


Seems like there might be contention on connections. We might want to think about how to reduce this later.

What do you mean?

This is using a Map to store the connections. As I remember this it not thread safe and you will have to have a lock to access it. It will be expensive to access it.

I see. I created a issue for that. #2478

zizhong · 2017-08-08T18:21:58Z

@@ -0,0 +1,121 @@
+/** @file


One thing we're excited about is TLS 1.3 handshake. It should be available not only with QUIC but also HTTP.

QUIC use the TLS-1.3 to handshake and key deriving. But QUIC is not on TLS, because QUIC is on UDP:D So, this is really QUIC specific class, IMO.
The overview of QUIC and TLS 1.3 is in Section 3 of draft-ietf-quic-tls-04

For HTTP, current master branch can do 1-RTT handshake of TLS-1.3. But I'm not sure it can do 0-RTT handshake without any changes.

I see, thanks for the explanation

zwoop · 2017-08-09T15:49:51Z

For building, can we make it such that if there is no TLS v1.3 support, we don't build the pieces that needs it? This can be automatically detected, such that you get QUIC support IF your OpenSSL / BoringSSL supports it. You would still need to configure ATS to listen on the UDP port (443), but that doesn't affect compile time.

Fixing this would make it easier to pass this on the CI for now, and have people start testing it as well.

zwoop · 2017-08-09T15:52:30Z

I assume we're continuing development on the quic-latest branch, and will merge that branch to master when it's ready for it?

maskit · 2017-08-09T15:57:24Z

@zwoop

For building, can we make it such that if there is no TLS v1.3 support, we don't build the pieces that needs it? This can be automatically detected, such that you get QUIC support IF your OpenSSL / BoringSSL supports it. You would still need to configure ATS to listen on the UDP port (443), but that doesn't affect compile time.

That is exactly what in my mind. It would be done on #2344.

I assume we're continuing development on the quic-latest branch, and will merge that branch to master when it's ready for it?

Yes, we are going to do so.

scw00 · 2017-08-15T06:44:33Z

+  std::unique_ptr<QUICFrame, QUICFrameDeleterFunc> ack_frame = this->_ack_frame_creator.create_if_needed();
+  if (ack_frame != nullptr) {
+    this->transmit_frame(std::move(ack_frame));
+    eventProcessor.schedule_imm(this, ET_CALL, QUIC_EVENT_PACKET_WRITE_READY, nullptr);


Why do we schedule twice here ? The transmit_frame seems schedule the same vc.

void QUICNetVConnection::transmit_frame(std::unique_ptr<QUICFrame, QUICFrameDeleterFunc> frame) { Debug(tag, "Type=%s Size=%zu", QUICDebugNames::frame_type(frame->type()), frame->size()); this->_frame_buffer.push(std::move(frame)); eventProcessor.schedule_imm(this, ET_CALL, QUIC_EVENT_PACKET_WRITE_READY, nullptr); }

I guess schedule_imm in transmit_frame was added for convenience so we don't need to call schedule_imm after calling transmit_frame.

However, scheduling an event on every frame is too expensive, so I think we should add flush(), or bool transmit_now parameter.

maskit · 2017-08-16T02:14:04Z

Current status:

Works on Fedora 26, FreeBSD 11 and maxOS 10.12 (tested with ngtcp2)
All items for the first impl. have been implemented

scw00 · 2017-08-22T12:55:44Z

+{
+  this->_transmitter_mutex = new_ProxyMutex();
+  this->_udp_con           = udp_con;
+  this->_transmitter_mutex = new_ProxyMutex();


It seems we reallocated mutex !!

Definitely! Nice catch!

scw00 · 2017-08-30T11:46:47Z

+    this->_smoothed_rtt = latest_rtt;
+    this->_rttvar       = latest_rtt / 2;
+  } else {
+    this->_rttvar       = 3 / 4 * this->_rttvar + 1 / 4 * (this->_smoothed_rtt - latest_rtt);


Do we need to use absolutely value. this->_rttvar = 3 / 4 * this->_rttvar + 1 / 4 * ABS(this->_smoothed_rtt - latest_rtt);

(2.3) When a subsequent RTT measurement R' is made, a host MUST set RTTVAR <- (1 - beta) * RTTVAR + beta * |SRTT - R'| SRTT <- (1 - alpha) * SRTT + alpha * R'

Most of code in QUICLossDetector came from pseudo code on the specification. I'm not sure whether we need to use the values.
https://tools.ietf.org/html/draft-ietf-quic-recovery-04#section-3.2.5

https://github.com/quicwg/base-drafts/issues/753

scw00 · 2017-09-01T01:48:37Z

+  QUICPacketType packet_type() const;
+
+private:
+  std::unique_ptr<QUICFrame, QUICFrameDeleterFunc> _frame = std::unique_ptr<QUICFrame, QUICFrameDeleterFunc>(nullptr, nullptr);


This code causes compile error with gcc 4.9 .

scw00 · 2017-09-06T05:36:32Z

+    e->init(cont, 0, 0);
+
+    cont->handleEvent(event, e);
+  } else {


Mem leaking ??

scw00 · 2017-09-12T00:14:55Z

+  int bytes_added = this->_read_vio.buffer.writer()->write(frame->data(), frame->data_length());
+  this->_read_vio.nbytes += bytes_added;
+  this->_recv_offset += frame->data_length();
+  this->_local_flow_controller->forward_limit(frame->offset() + this->_flow_control_buffer_size);


Why don't use the this->_recv_offset instead of frame->offset(). We update the offset by frame->offset() + frame->data_length(). we will get a negative value in forward_limit in forward_limit if this->_flow_control_buffer_size is less than frame->data_length().(all this type of literal are unsigned)

(gdb) p this->_offset $5 = 3 (gdb) p this->_limit $6 = 0 (gdb) n 98 } (gdb) p this->_threshold $7 = 1024 (gdb) p this->_limit - this->_offset $8 = 18446744073709551613

Sounds correct. Probably I forgot writing + frame->data_length().

invalid now

maskit · 2017-11-16T23:48:00Z

It's broken right now. Trying to moving to draft-07 with pretty big changes but patches are always welcome.

maskit · 2017-11-28T08:53:35Z

Tested handshake with ngtcp2 and minq (draft-07).

bryancall · 2018-03-05T22:58:14Z

[approve ci debian]

* master: Make tls_conn_timeout test more reliable in CI (#7018) Remove deprecated verify.server for 9.0 (#7040) Updated GitHub description and homepage URL to be https (#7019) Add virtual destructor to QUICTPConfig. (#7036) Fix code to eliminate warning and enable feature (#7031) add a null check to avoid crashing (#7035) Squashed commit of the following: (#7000) Fixed problem with all "forced" volumes cache (#7028) Spacing tweaks to acl_filter_rule::print (#7026) Removes dead code from iocore/dns (#7025) Removes TODO (#7027) Add logic to resolve content-length transfer-encoding conflicts on response (#6992) Add memory_profile plugin (#7014) Fix typos relating to tls_bridge (#7011) slice: clean up of created 502 response header (#6919) Add new API / TSPluginDSOReloadEnable that overrides the configuration variable `proxy.config.plugin.dynamic_reload_mode` for a particular plugin. (#6880) Remove incorrect assert in inactivity timeout handling (#7012) Removes use of SPLIT_DNS macro (#7010) Fixed core when sending back a redirect and having an invalid server response (#7004) slice: fix throttle not work (#7008) Updates to thread scale factor (#7007) Added tasks and launch files for vscode, to configure, build and debug (#7005) NextHop Strategy Refactor and Fixes (#6782) Make the setting of the continuation handler safer. (#6996) ProtocolStack n -> count (#7006) Fix volume/stripe calcs when using forced volumes (#6995) Cleanup: Write error message on diags output instead of stderr (#6997) Conflicts: iocore/net/P_QUICNetVConnection.h iocore/net/P_QUICPacketHandler.h iocore/net/QUICNetProcessor.cc iocore/net/QUICNetVConnection.cc iocore/net/QUICPacketHandler.cc iocore/net/quic/Mock.h iocore/net/quic/QUICCongestionController.h iocore/net/quic/QUICContext.cc iocore/net/quic/QUICContext.h iocore/net/quic/QUICDebugNames.cc iocore/net/quic/QUICFrame.cc iocore/net/quic/QUICHandshake.cc iocore/net/quic/QUICKeyGenerator.h iocore/net/quic/QUICLossDetector.cc iocore/net/quic/QUICLossDetector.h iocore/net/quic/QUICNewRenoCongestionController.cc iocore/net/quic/QUICPacket.cc iocore/net/quic/QUICPacket.h iocore/net/quic/QUICPacketFactory.cc iocore/net/quic/QUICPacketFactory.h iocore/net/quic/QUICRetryIntegrityTag.cc iocore/net/quic/QUICRetryIntegrityTag.h iocore/net/quic/QUICTLS.h iocore/net/quic/QUICTLS_boringssl.cc iocore/net/quic/QUICTLS_openssl.cc iocore/net/quic/QUICTransportParameters.cc iocore/net/quic/QUICTransportParameters.h iocore/net/quic/QUICTypes.cc iocore/net/quic/QUICTypes.h iocore/net/quic/qlog/QLogListener.h iocore/net/quic/test/test_QUICHandshakeProtocol.cc iocore/net/quic/test/test_QUICLossDetector.cc iocore/net/quic/test/test_QUICPacket.cc iocore/net/quic/test/test_QUICPacketFactory.cc iocore/net/quic/test/test_QUICPacketHeaderProtector.cc iocore/net/quic/test/test_QUICStreamManager.cc iocore/net/quic/test/test_QUICVersionNegotiator.cc proxy/http/HttpProxyServerMain.cc src/traffic_quic/quic_client.cc src/tscore/ink_inet.cc

* master: Fixes spelling/license formatting in traffic_dump plugin (#7047) Fixes spelling in docs (#7048) Fixes spelling H3-related code (#7046) Cleans up various versions checks (#7049) Fix a typo (#7043) Conflicts: iocore/net/quic/QUICHandshake.cc iocore/net/quic/QUICLossDetector.cc iocore/net/quic/QUICNewRenoCongestionController.cc iocore/net/quic/test/test_QUICPacketFactory.cc

* master: Fix a crash on active timeout on QUIC connections (#7059) Don't make an error on receiving retransmitted handshake data (#7061) Document proxy.config.http.cache.post_method. (#7060) Quote out lists of servers and domains in splitdns.config example (#7057) Fix proxy.process.http.current_client_connections (#7056) Fixed CLIENT-URL to use the pristine client URL (#7050) Removes FIXME that is unlikely to be fixed at this point in the project history (#7058) Move to denylists and allowlists (#7034) Avoid unnecessary copying of STL map for QUICTPConfigQCP class. (#7039)

* master: Signal WRITE_COMPLETE regardless of transmission progress (#7062) Converts files to #pragma once (#7089) Fix eval_ip compile - missing const. (#7087) Fix a crash on connection migration to the advertised preferred address (#7080) Update and run the autopep8 make target (#7070) Fixes no_content_length status code description in docs (#7086) const-ify quic/http3 code (#7084) Fixes build warnings in maxmind_acl (#7085) Add TS_USE_QUIC to traffic_layout info (#7074) Added support for out of tree builds with vscode (#7072) constify Print() methods and other low hanging fruit (#7068) Updating to AuTest 1.8.1. (#7065) Use system include-style for STL and OpenSSL headers (#7066) tests: gitignore ssl-delay-server binary (#7067) Conflicts: iocore/net/quic/QUICLossDetector.cc iocore/net/quic/QUICLossDetector.h

* master: Adding autopep8 as a pre-commit hook. (#7071) Refresh proxy protocol diagram (#7095) Skip docs builds if there are no changes in the doc dir and files it includes (#7088) Remove more deadcode (#7098) destroy threads after job done (#7083) Fix compilation error - missing '&' operator (#7093) Adds description for ssl_ticket_number in ssl_multicert docs (#7091) Ran clang-tidy over the source tree (#7077) Move the direct self loop check later to HttpSM::do_http_server_open just before connection upstream. (#7069)

* master: Backing out my update of our jenkin's autest file. (#7118) Don't send image/webp responses from cache to broswers that don't support it (#7104) Updating our autest suite to require Python3.6 (#7113) Squashed commit of the following: (#7110) Supporting out of source builds for AuTests. (#7109) Fixes uninitialized variables found by Xcode (#7100) Add cross references between server session sharing match and upstream connection tracking match. (#7038)

* master: Add an autest testcase for HTTP3 (#7063) Fix TSHttpTxnServerPacket* API's to correctly update existing server connections (#7175) Do not lose original inactivity timeout on disable (#7134) Emits log when OCSP fails to connect to server (#7183) autopep8: avoid running on non-tracked files. (#7186) TextView: Add additional constructor tests. (#7189) Remove duplicate code (#7180) TextView: add constructor size values to enable strlen even for null pointers. (#7185) Add virtual destructor to QUICRTTProvider. (#7184) AuTest: Reuse venv if it exists already (#7178) TS_API for Note,Status,Warning,Alert,Fatal (#7181) Traffic Dump: Record HTTP/2 priority. (#7149) leaks in logs (#7172) Additions to enable loading qat_engine (#7150) Removes references to non-existent function handle_conditional_headers (#7162) Fix #7164 Chaning Warning to Debug and creating a stat for inserting duplicates to pending dns (#7166) Fix #7167, make autopep8 failure (#7168) MicroDNS Extension: handle different 'default' types (#7159) Traffic Dump documentation for post_process.py (#7161) Fix memory leaks in multiplexer plugin (#7160) rc: fixes systemd unit file stopping (#7157) Fix lua plugin mem leak problem (#7158) Don't make an error on duplicated RETIRE_CONNECTION frames (#7131) URL::parse fixes for empty paths (#7119) Replace ACTION_RESULT_NONE with nullptr (#7135) Add metric tracking async job pauses (#7153) PluginFactory - Remove unused code that was left from last PluginFactory change(TSPluginDSOReloadEnable) (#7155) Fix stale pointer due to SSL config reload (#7148) slice: check if vio is still valid before calling TSVIODone* on shutdown (#7147) Deprecate cqhv field (#7143) Don't return QUIC frame if the size exceeds maximum frame size (#7121) Check VIO availability before acquiring a lock for it (#7145) Fix #7116, skip the insertion of the same continuation to pending dns (#7117) Allow override of CA certs for cert from client based on SNI server name sent by client. (#7130) Fix typo in cache docs (#7144) remove useless shortopt (#7138) Protect TSActionCancel from null INKContInternal actions (#7128) Check VIO availability before checking whether the VIO has data (#7120) Accept NAT rebinding on a QUIC connection (#7123) Fixes garbled logs when using %<vbn> log tag (#7140) Removes duplicated listing of files in same Makefile target (#7137) Updated gdb mutex script to get process file for Fedora 32 (#7133) SSLConfig mem leak fix (#7125) Replaces "smart" quotes with ASCII equivalents (#7126) Comment out a wrong assertion in QUIC Loss Detection logic (#7129) Add member initialization to the Errata class. (#7132) Cancel active/inactive timeout on closing Http2Stream (#7111) Add modsecurity lua script to example (#7105) Expose remap config file callback (#7073) Make tls_hooks tests more likely to pass (#7122)

* master: Rename ambiguous log variable (#7199) KWF useless member function HttpSM::kill_this_async_hook(). (#7198) Fix the active_timeout test to work without quic enabled (#7197) Remove obsolete cdn_ HttpTransact vars (#7182) Remove unused HttpUpdate mechanism (#7194) Updates the list of supported / linked Docs versions (#7152) Make custom xdebug HTTP header name available to other plugins. (#7193) Update sni outbound policy to allow directly setting the outbound SNI. (#7188)

* master: Adds a shell script to help build the H3 toolchains (#7299) Remove unfinished h2c support (#7286) Allow disabling SO_MARK and IP_TOS usage (#7292) Enable all h2spec test (#7289) Fix bad HTTP/2 post client causing stuck HttpSM (#7237) Sticky server does not work with H2 client (#7261) 7096: Synchronize Server Session Management and Network I/O (#7278) HostDB: remove unused field in HostDBApplicationInfo, and update remaining types in http_data to fix broken padding. (#7264) Add support for a new (TSMgmtDataTypeGet) mgmt API function to retrieve the record data type (#7221) Fix example in default sni.yaml configuration. (#7277) Fix proxy.process.http.current_client_transactions (#7258) Add AuTest for HTTP/2 Graceful Shutdown (#7271) Fix truncated reponse on HTTP/2 graceful shutdown (#7267) url_sig add 'ignore_expiry = true' option for log replay testing (#7231) Respecting default rolling_enabled in plugins. (#7275) gracefully handle TSReleaseAsserts in statichit and generator plugins (#7269) Removes commented out code from esi plugin (#7273) Allow initial // in request targets. (#7266) Document external log rotation support via SIGUSR2 (#7265) Let Dedicated EThreads use `EThread::schedule` (#7228) HostDB: Fix cache data version checking to use full version, not major version. (#7263) Bugfix: set a default inactivity timeout only if a read or write I/O operation was set (#7226) Treat objects with negative max-age CC directives as stale. (#7260) Remove some usless defines, which just obsfucates code (#7252) Remove useless if for port set assertion. (#7250) Fix test_error_page_selection memory leaks and logic errors (#7248) [multiplexer] option to skip post/put requests (#7233) Incorporates the latest CI build changes (#7251) Add support for server protocol stack API (#7239) Fix for plugins ASAN suppression file (#7249) RolledLogDeleter: do not sort on each candidate consideration. (#7243) Make double Au test more reliable. (#7216) Ensure that ca override does not get lost (#7219) Stop crash on disk failure (#7218) Do not cache Transfer-Encoding header (#7234) clean up body factory tests (#7236) Revert "Create an explicit runroot.yaml for AuTests (#7177)" (#7235) New option to dead server to not retry during dead period (#7142) Increment ssl_error_syscall only if not EOF (#7225) Fix renamed setting in default config (#7224) Log config reload: use new config for initialization (#7215) Introduce proxy-verifier to AuTests (#7211) Follow redirection responses when refreshing stale cache objects. (#7213) Create an explicit runroot.yaml for AuTests (#7177) Support external log rotation tools via SIGUSR2 (#6806) Add support for TS API for Note, Status, Warning, Alert (#7208) If the weight is 0, the SRV record should be selected from the highest priority group (#7206) Cleanup: remove unnecessary memset() within dns_process() (#7209) Docs cleanup (#7210) Strip whitespaces after field-name and before the colon in headers from the origin (#7202) Adds new plugin: statichit (#7173) Add duplicate header field processing when creating outgoing response (#7207)

* master: Doc: Fix typo in negative_revalidating_lifetime (#7427) Change comment handling for long lines in url_sig plugin (#7421) Add unit tests for PROXY Protocol v1 parser (#7332) LGTM: Remove superfluous const qualifier in return type (#7412) Fix issue with unavailable server retry codes (#7410) Remove the warning statement (#7414) default to throttling and subsequently simplify the transfer code (#7257) Improvement to lua plugin (#7413) Make places to bind/unbind SSL object with/from NetVC (#7399) traffic_ctl - plugin msg now require only the tag as mandatory field data field is now optional. (#7364) API - Add new api function TSHttpTxnServerSsnTransactionCount() to retrieve the number of transactions between TS proxy and the origin server from a single session. (#7387) Fix clang compiler complaint about an unused parameter in SNIAction. (#7409) Add compression support to stats_over_http (#7393) Doc: Fix INPUT tag of Doxyfile (#7404) Remove unneeded variables in UnixNetVConnection (#7403) Correctly pass back errno to HttpSM (#7402) Reverting to old negative_caching conditional behavior (#7401) Remove unused MAYBE_ABORT state (#7400) traffic_manager should not retry on disk failure (#7397) Eliminate dangling pointer into stack space. (#7392) This PR aims to address some of the lock contention found and (#7377) Remove a special treatment for SSLNetVC in migrateToCurrentThread() (#7384) Replace ::exit() with _exit() to avoid secondary cleanup cores (#7395) [Doc] Fix build warnings (#7391) Clear call_sm on tunnel reset (#7352) Unused code: HostDBContinuation::removeEvent (#7383) Traffic Dump: Fix stream-id printing after first transaction. (#7311) Add comments to ink_queue.h. (#7376) Cleanup incoming PROXY Protocol v1 (#7331) In CI, only run autopep8 on branches that enforce autopep8 (#7270) Fix FreeBSD 12 link issue in test_libhttp2. (#7367) Adjust flags to ensure tunnel producer is cleaned up (#7336) Cleanup: Remove SSL Wire Trace releated code in UnixNetVConnection (#7368) Use EVP MAC API if available (#7363) Use EVP API instead of MD5_Init/Update/Final (secure_link plugin) (#7355) Use ERR_get_error_all if available (#7354) Use OpeSSL EVP API instead of SHA256_Init/Update/Final (#7342) Cleanup: Get rid of NetVConnection::outstanding() (#7366) Cleanup: Remove unused functions (#7365) Add a post case to the conn_timeout test (#7334) Fix sni ip_allow and host_sni_policy (#7349) AuTest for Split DNS (#7325) Make reloading client certificate configuration more reliable (#7313) Add negative caching tests and fixes. (#7361) ESI: Ensure gzip header is always initialized (#7360) Allow for regex_remap of pristine URL. (#7347) Set thread mutex to the DNSHandler mutex of SplitDNS (#7321) Fix lookup split dns rule with fast path (#7320) Add note to background fetch about include/exclude (#7343) AuTest for incoming PROXY Protocol v1 (#7326) Fix vc close migration race condition (#7337) TLS Session Reuse: Downgrade add_session messages to debug (#7345) TLS Session Reuse: Downgrade noisy log to debug (#7344) Remove the last remnants of the enable_url_expandomatic (#7276) Remove unnecessary cast from ReverseProxy. (#7329) Updates the Dockerfile with more packages (#7323) fixup in HttpSM to only set [TS_MILESTONE_SERVER_CLOSE if TS_MILESTONE_SERVER_CONNECT has been set (#7259) Add option for hybrid global and thread session pools (#6978) Get appropriate locks on SSN_START hook delays (#7295) s3_auth: demote noisy errors around configuration that doesn't affect plugin usability (#7306) Follow the comments in I_Thread.h, add an independent ink_thread_key for EThread. (#6288) Reduce the number of write operation on H2 (#7282)

* master: Fix a link error on traffi_quic command (#7433) Fix stall on outbound TLS handshake (#7432) Fix the Proxy Verifier AuTest extension to handle cert paths correctly (#7415) Update documentation for TSSslSessionInsert (#7420) Improve zlib detection logic (#7430) Fix parent connect fail segfault (#7429)

* master: Select lua context per thread (#7465) Fix out of bounds access error in jtest (#7526) Disable compiling Inline.cc on macOS (#7389) Makes sure the types are correct, avoiding compiler warnings (#7523) Move has_request_body to ProxyTransaction (#7499) Make the H3 build script work properly on Debian platforms (#7522) slice/handleFirstServerHeader: return sooner on requested range errors (#7486) Add new log field for negotiated ALPN Protocol ID with the client (#7491) Add Outbound PROXY Protocol (v1/v2) Support (#7446) Updates the Dockerfile for debian (#7518) Disable client inactivity timeout while server is processing POST request (#7309) Upgrade Catch.hpp to v2.13.4 (#7464) Move reopen_moved_log_files to log flushing thread (#7450) replace psutil.pid() with psutil.process_iter() for safer execution (#7515) Fix spacing in clang-analyzer.sh script (#7480) Fix out of bounds access error in ats_base64_decode (#7490) Updated to build lastest versions of Fedora and CentOS docker images (#7505) Fix QUIC unit tests build issue on GNU ld (#7496) Fix QUIC unit test failures (#7497) Fixed build issues with Fedora 34 (#7506) Fixing DNS local_ipv* config option (#7507) traffic_dump: AuTests to use Proxy Verifier. (#7502) Disable ja3 plugin when building with boringssl (#7500) Avoid -Warray-bounds on PROXY Protocol Builder (#7488) AuTest: Upgrade to Proxy Verifier 2.0.2 (#7493) fix certs (#7494) Add zlib1g-dev to Debian dependencies in README (#7495) Unit Test - Increase openssl's key size. Place test certs into a common test folder. (#7451) Add basic type aliases for std::chrono types to ink_time.h for future use. (#7482) traffic_ctl - Fix lookup key for run-root option (#7484) update thread config tests (#7370) Perf: Replace casecmp with memcmp in HPACK static table lookup (#6521) Add PROXY Protocol Builder (#7445) Adjust so transfer-encoding header can be treated hop-by-hop (#7473) Convert auxkey form 2 uint32_t to 1 uint64_t. (#7350) Remove the queuing option from proxy.config.http.per_server.connection (#7302) Remove unused function ink_microseconds. (#7481) use std::unordered_map to store sessions (#7405) drop use of BIO_f_base64 and EVP_PKEY_new_mac_key (#7106) Do not write to the cache if the plugin decides not to write to the cache (#7461) API to retrieve NoStore set by plugins (#7439) Update AuTest version update directions for pipenv (#7469) Add command line utility to help convert remap plugin usage to ATS9. (#7426) Cleanup: Get rid of MIMEFieldWrapper from HPACK encoding (#6520) Proxy Verifier: Making use of delay directives for caching tests. (#7468) Cleanup: Add SNIRoutingType (#7453) Updating to Proxy Verifier v2.0.0 (#7454) Adjust to actually try a server address more than once (#7288) Change atoi to atol, causing obvious issues on what needs to be int64's (#7466) Cleans up duplicated TSOutboundConnectionMatchType definition (#7090) Fixing compress expectation for new microserver (#7463) Update to the new MicroServer 1.0.6 release (#7460) CacheRead: clear dir entry if doc is found to be truncated (#7064) Do not provide a stale negative cache (#7422) Generalize SNI support (#6870) Add synchronization between UDPNetProcessor::UDPBind in main Thread and initialize_thread_for_udp_net in ET_UDP Thread (#7407) Fix heap use after free in DNSProcessor::getby() (#3871) Fix comment in include/tscore/Filenames.h. (#7457) Fix Makefile target for creating changelogs (#7455) Change squid log code for self looping (#7443) Enhancements for compress plugin (#7416) Add incoming PROXY Protocol v2 support (#7340) Cleanup: Remove unused members of NextHopProperty (#7436) Small fix to regex_remap PR # 7347. (#7437) PoolableSession (#6828) option to disable compression for range request's response (#7287) Make TSUrlSchemeGet() return scheme implied by URL type when there is no explicit scheme. (#7262)

* master: Fix the connection limit crash while using parents (#7604) Remove inline for detail::cache::CacheData::idAddr (#7592) Remove UnixNetVConnection::startEvent - not actually called. (#7596) Use return values to fix ubuntu release build error (#7591) Fix double destuct on Http2Stream termination (#7600) Add pointer/reference upcast function that is checked in debug builds. (#7582) Call constructors and destructors for H1/2 Session/Transaction via ClassAllocator (#7584) Add gold test for remap config .include directive. (#7589) Change the default value for verify.server.policy (#7587) Build the test library for tls_engine consistently (#7588) Generalize ALPN logic (#7555) Fix the final consumer write size from unchunked to chunked tunnel (#7577) Reactivate accept_no_activity_timeout (#7408) Tidy up session/transaction destruction process (#7571) Remove ProxyTransaction::set_proxy_ssn (#7567) Introduce TLSBasicSupport interface (#7556) Cleanup: Rename IOBufferReader of Http2ClientSession (#7569) Add a check for compress response, if from server and 304, then check cache for headers instead of the 304 response (#7564) Updates the STATUS file with all recent releases (#7566) Make Allocator.h less silly (no creepy "proto" object). (#6241) Cleanup: Remove unused member of Http2ClientSession (#7570) enable origin server session cache by default (#7537) Add tscontdestroy when transaction is closed and pacing rate is reset (#7572) Remove reference to CoreUtils (#7557) Remove unused enums from YamlSNIConfig struct. (#7565) Removes deprecated sni.yaml option: disable_h2 (#7547) This PR updates parent selection to limit the number of simultaneous (#7485) Fix KA header not checking strategy (#7483) Get rid of kruft LogObject copy constructor. (#7553) For TSHttpHdrEffectiveUrlBufGet(), include scheme for request to server URL. (#7545) Adding lower_ support to stats and bonding_slave data points for port status (#7560) Change cookie_remap plugin to allow use of pre-remap URL (and components). (#7519) check verify policy and properties (#7559) Fix parent.config to 504 not 502 on timeout (#7558) use SSL_CTX address as part of the lookup key (#7552) Add ALPN support on TLS Partial Blind Tunnel (#7511) Add server_name option to proxy.config.ssl.client.sni_policy (#7533) Fix a crash on origin session reuse (#7543) Removes the test plugins from the .spec file / RPM (#7551) Convert the inactive_client_timeout test to use Proxy Verifier (#7535) Fix ja3_fingerprint configure syntax (#7550) Fix asserts in multiplexer plugin. (#7532) parse expiration time and reload config at time out (#7281) Fix origin_session_reuse test (#7542) Fix tls_session_reuse test (#7541) Split SSL_CTX initialization logic into small functions (#7434) Remove dependency for SSL stuff from P_Net.h (#7531) Unify all the connect timeouts into one (#7335) Fix lua_states_stats Au test. (#7232) origin session reuse (#7479) Updating to use Proxy Verifier 2.1.0 (#7534) update the session reuse tests (#7529)

* master: Fix ALPN support on QUIC connections (#7593) fix mem leak in session cache (#7707) Parent Select Plugin (#7467) Add new TS API function TSUrlRawPortGet. (#7568) Add NixOS support (#7697) Remove support for --enable-remote-cov-commit (#7700) Remove configure-time loopback interface detection (#7702) Add sqpv log field for server protocol (#7680) Call do_io_close instead of HTTP2_SESSION_EVENT_FINI handler (#7594) Fix a bug in tspush that pushes corrupted content to cache (#7696) Automatically marks PRs and issues stale (#7675) New rate_limit plugin for simple resource limitations (#7623) Remove undefined method HttpSM::perform_nca_cache_action (#7692) Remove undefined method HttpSM::setup_client_header_nca (#7691) Scalar; Move "tag" struct to be inside the "ts" namespace to avoid collisions. (#7690) Rollback LAZY_BUF_ALLOC remove in HttpTunnel (#7583) Add class to normalize handling of pending action (#7667) Make HTTP/2 Curl AuTest gold files case insensitive (#7683) Add STL compliant field iteration to MIMEHdr. - rebase. (#7476) Fix use of -mcx16 flag - only use if it compiles cleanly. (#7684) Refine connection failure logging and messages and eliminate suprious connection errors (#7580) Add close header normalize openclose test (#7679) Fix has_consumer_besides_client to deal with no clients (#7685) create a new cache status RWW_HIT (#7670) Updating to AuTest 1.10.0 (#7682) sslheaders AuTest: Skip if plugin does not exist (#7678) Add AuTest for Background Fill (#7613) Do NOT kill tunnel if it has any consumer besides HT_HTTP_CLIENT (#7641) AuTest: address various permissions issues (#7668) Adding TCP Info header support to header rewrite (#7516) Refine Inline.cc carveout for arm64 darwin builds (#7662) Comment why log eviction isn't implemented via a log field. (#7648) Fixing Throttler.h for older clang and gcc compilers (#7651) Update -with-profile and add some profiling documentation (#7601) Use correct default value for verify.server.policy (#7636) Update server_response_body_bytes when background fill worked (#7621) Remove erroneous manager.log mesg with remap include file reload (#7646) Change ROUNDUP from function-like macro to function template. (#7614) Document http.default_buffer_water_mark (#7612) Add proxy.config.cache.log.alternate.eviction (#7629) Fix HttpSessionManager::acquireSession from previous rebase error (#7631) Fix tls_client_versions and tls_hooks18 tests (#7645) Updating documentation for negative_revalidating_lifetime (#7633) Remove reference to client.verify.server from tests and other bits (#7639) Add pooled_server_connections metric (#7627) Expose URL element methods through HTTPHdr (#7628) Add default implementation for allow_half_open (#7630) Add thread yeield to avoid busy waiting in LogObject::_checkout_write(). (#7576) Add proxy.process.http.background_fill_total_count (#7625) statichit: misc. fixes (#7634) Remove unused variables (#7626) Adding negative revalidating AuTests. (#7620) Add failed state to hostdb to better track failing origins (#7291) Use standard isdigit library function (#7619) Typo in output when forcing kqueue for configure (#7617) Implement log throttling (#7279) Increase Proxy Verifier caching delay. (#7616) Set pcre_malloc/free function pointers in core main() only. (#7608)

* master: Get rid of code for OpenSSL that has old QUIC API (#7599) Fixed warning in gcc 11 about array not being initalized (#7840) Don't call next next dup on destroyed mime field mloc. (#7833) build_h3_tools: use OpenSSL_1_1_1k+quic (#7836) Address assert on captive_action (#7807) Fix so EOS are delivered to sessions in the pool (#7828) Fix a format specifier for size_t (#7830) Fix stall on sending response for request with trailer header (#7831) Simplification dir_init_done (#7817) Remove unused member from HttpSM (#7835) AuTest: use exteneded help output to determin curl feature support (#7834) Apply fmt compile time argument checking to log functions (#7829) Adds new X-Cache-Info header to the xdebug plugin (#7784) Cleanup: Remove unused members of Http2Stream (#7813) Cleanup: unused functions of Http2ClientSession (#7812) Cancel cross_thread_event on clear_io_events (#7815) Cleanup: Remove a meaningless Http2Stream::do_io_close() call (#7814) Eliminate next dup call using stale mime field mloc is s3_auth plugin. (#7825) NetEvent cleanup - replace #define with constexpr (#7804) fix origin session related crashes (#7808) Update HTTP version info in HostDB on new outbound connection (#7816) Remove a redundant argument (#7811) SSL Cert lookup using PP dest ip when ProxyProtocol is enabled (#7802) Fix MLoc assert caused by s3auth (#7790) Fix cpu utilization problem in session cache (#7719) Fix to cookie_remap.cc tp avoid Intel compiler warning. (#7792) TSHttpTxnCacheDiskPathGet - tighten up the code a bit. (#7806) Doc: tcpinfo plugin table formatting (#7805) fix DNS spike issue for TCP_RETRY mode (#7307) Adds new TS API TSHttpTxnCacheDiskPathGet (#7783) tests: Fixes spelling (#7789) Traffic Dump: Add an HTTP/3 AuTest (#7758) use sendmsg and recvmsg (#7793) HTTP: clean up the http_hdr_describe format error (#7797) Fixes an issue where next hop unit tests crash when run on macOS. (#7787) Apply log throttling to HTTP/2 session error rate messages (#7772) Cleans up uninitialized warning in LogMessage.cc (#7788) Short circuit remap reload when a valid remap file is not specified (#7782) DNS: Clean up argument passing to DNS queries. (#7778) Remove extra verify-callback (#7540) Augment test cases for tls_verify_override test (#7736) Make when_to_revalidate setting available on HTTPS (#7753) Add traffic_server command line option for debugging in Au test. (#7762) Test: Update tls_partial_blind_tunnel to have a nameserver. (#7773) Test: update tls_forward_nonhttp to have a nameserver. (#7774) Test: add nameserver to log-filter test. (#7776) BWF: Add support for std::error_code. (#7777) Test: add nameserver to log-field test. (#7779) Test: add nameserver to regex_remap test. (#7775) Elevate privileges for traffic_manager during SSL cert reload (#7770) Clean up HTTP version processing (#7766) Remove proxy.config.http.down_server.abort_threshold (#7748) Remove undocumented keepalive_internal_vc setting (#7693) doc: header_rewrite random function not inclusive (#7760) Experimental Cache fill plugin (#7470) Remove references to removed options (#7756) Propagate TLS errors (#7714) AuTest extension: check for unrecognized configurations (#7752) Fixes errors in the strategies.yaml documentation. (#7745) Updates to Nexthop strategies to limit the number of simultaneous (#7744) Fixes Issue #7739 - Next hop strategy with bad 'to' URL causes TS crash. (#7749) header_rewrite: Various fixes for MaxMind support (#7746) Remove unused variable is_revalidation_necessary (#7747) Fix simple remapping in regex_remap plugin. (#7718) Adding DNS TTL AuTests. (#7742) Add a chunked disabled test. (#7743) Fix monitor threads in lib records to exit on system shutdown. (#7731) Add overload for memcpy to take a destination buffer and source string_view / TextView (#7732) Test: Add nameserver to TLS tunnel forward test. (#7733) AIO_NOT_IN_PROGRESS should not be 0 (#7734) if transaction status non-success, bypass intercept plugin (#7724) ink_utf8_to_latin1 is not defined, removing declaration (#7737) Fix build on FreeBSD 13 (#7730) Update VSCode CPP Standard (#7723) Updating to use Proxy Verifier 2.2.0 (#7729) header_rewrite: Allow for relative path to geo database files (#7727) Override proxy.config.ssl.client.sni_policy from sni.yaml (#7703) compress.test.py: Reference config file from Test.RunDirectory (#7725) Ran clang-tidy over the code (#7708) Deny unknown transfer encoding values (#7694) Fix doc for http2.no_activity_timeout_in (#7721) Add DynamicStats (#7704) header_rewrite: allow for use of maxminddb as source of geo truth (#7695) Include in parentselectdefs.h in install target (#7713) uri_signing: fix warning which affects ubuntu:20.04 builds (#7717) Increase the maximum slice block size from 32MB to 128MB (#7709)

* master: Don't rely on SSLNetVC when HttpSM gathers info about SSL (#7961) conf_remap: demote 'Invalid configuration' to warning (#7991) Cleans up the code bit, including milliseconds consistency (#7989) Pass through expect header and handle 100-continue response (#7962) Treat TRACE with body as bad request (#7905) Thread safe Mersenne Twister 64 using c++11 (#7859) ESI plugin documentation updates. (#7970) Add log name configuration and stderr/stdout support. (#7937) Cleanup: Constify MIMEHdr (#7949) Fixed compile error with Linux AIO unit test (#7958) Note YAML parser library bug, and work-around, in documentation. (#7963) Ensure that the content-length value is only digits (#7964) String the url fragment for outgoing requests (#7966) Fix for HTTP/2 frames (#7965) Improve parsing error messages for strategies.yaml. (#7948) fix the scheme of h2 0rtt tests (#7957) Fix double test flakiness due to EOS/TXN_CLOSE race (#7956) Use proxy.config.log.hostname for rotated log filenames (#7943) Fixed memory leak in the QUIC stream manager (#7951) Fixup TS_USE_LINUX_NATIVE_AIO AIO_MODE_NATIVE (#7832) Update GitHub stale action to auto close old PRs (#7952) Revert "Do not invalidate cached resources upon error responses to unsafe methods (#7864)" (#7954) regex_revalidate: add stats for miss/stale counts (#7950) Do not invalidate cached resources upon error responses to unsafe methods (#7864) Add an HTTP/2 304 "Not Modified" AuTest. (#7882) regex_revalidate: optionally retain rule epoch state across restarts (#7939) Fixed memory leak in QUIC ack frame unit test (#7947) cache_promote: Don't promote on uncacheable requests (#7942) Fix dynamic-stack-buffer-overflow of cachekey plugin (#7945) Compilation error fixes for QUIC unit tests (#7944) Adds bytes counting as a trigger to the cache_promote LRU (#7765) Add a JSON schema for strategies.yaml (#7932) Remove second call to TRANSACT_RETURN while handling cache write lock (#7873) Close connection after every bad request for HTTP/1.1 (#7885) Pin Sphinx to 3.x to unblock `make html` (#7940) Add support for Remap rule hit stats (#7936) Remove scrap log object dead code (#7935) Add STL forward iterators to DLL container. (#7934) Add log SQUID code testing to redirect.test.py Au test. (#7870) Fix race condition on server session state (#7921) regex_reval: bug where rule type is always reported as the first (#7928) Remove duplicate entry in overridable txn vars. (#7930) Satisfy ci/jenkins/bin/clang-format.sh (#7929) Add a basic Au test using strategies.yaml, with consistent hashing. (#7911) Add a chunked negative revalidating test. (#7907) Ensure that URL components are valid when alternate eviction is logged (#7924) fix grammar (#7927) AuTest: Enable h2spec generic test cases (#7926) Adjust vc read errors (#7923) Remove bucket search from IntrusiveHashMap::erase (#7848) Ensure TS_VCONN_CLOSE_HOOK hook is called during TS_EVENT_VCONN_CLOSE. (#7913) Update docs languages file to add 9.1.x for en and ja (#7917) * Adds a new peering ring mode to next hop selection strategies. (#7897) Add Au test for strategies.yaml, with consistent hashing, with fallover. (#7914) Make HttpSM server reference a Transaction instead of a Session (#7849) Set accept_options of Http1Transaction in Http1ClientSession::new_connection() (#7894) Reset Http1Transaction before adding vc to keep_alive_queue (#7892) Add dead server policy control and metric. Improve messages. (#7757) Ensure the HTTP protion of the protocol string is upper case (#7904) Fixed spelling mistakes in the docs (#7896) add MISS capability to the regex_revalidate plugin (#7899) docs: fix capitalization of Linux (#7898) Redirect - Make TS to honour the number_of_redirections configuration value (#7867) Clean up producer more regularly (#7386) Fix crash in open_close_h2 (#7586) Cleanup Http2ClientSession SessionHandler (#7876) Enforce HTTP parsing restrictions on HTTP versions supported (#7875) Do not delete the continuation twice (#7862) Cleanup: refer Http2ClientSession::mutex (#7853) Autest - Proxy Verifier Extension, add context template $-base string substitution in the replay file. (#7866) Fixed some spelling mistakes in comments (#7869) Fixed ASAN issues with MMH test (#7868) Cleanup: Move member functions defined inside of class definitions of Http2ConnectionState & Http2ConnectionSettings (#7854) Add URI Signing cdnistd Claim Implementation (#7822) Adds a new --enable-all-asserts configure option (#7858) Unifdef test code for MMH and moved it into its own test file (#7841) Clean up lua plugin doc for overridable configurations (#7844) Save and propagate epoll network error (#7809) Add method to write an IpAddr value to a sockaddr. (#7821) Add proxy.config.http.max_proxy_cycles (#7657) Update NextHop strategies so that unavailable server retry codes (#7837) generator: allow for POST requests (#7635) Fixed double declaration types for log buffer tracking (#7847) Extra braces for clang 5 / ubuntu 16.04 on array initialization (#7842) Conflicts: iocore/net/quic/QUICStreamFactory.cc

* master: Implement TLSBasicSupport for QUICNetVC (#7959) Reload server session inactivity timeout before placing a session into the pool (#7618) Use OpeSSL EVP API if SHA1 API is unavailable (cache_promote) (#7447) Cleanup: Get rid of HTTP2_SESSION_EVENT_RECV (#7879) Timing and permissions update for regex_revalidate test (#7998) limit m_current_range to max value in RangeTransform (#4843) Allow to TLS handshake to error out on TSVConnReenable (#7994) Cleanup: Get rid of HTTP2_SESSION_EVENT_INIT (#7878) Add hook for loading certificate and key data from plugin (#6609) Doc: Now's Minute invocation error (#7990) Fix typo in configure.ac (#7993)

* master: reuse multiple times (#7992) Test bad request behavior (#7884) Fix BoringSSL build (#8001) Update TSHttpTxnAborted API to distinguish client/server aborts (#7901) Enforce case for well known methods (#7886) Add null checks for http_load (#7995)

In udp_read_from_net, at least 32 IOBufferBlocks were allocated on every call, and some of them were unused and free-ed at the end of function. This is not a trivial thing if QUIC is enabled. This change adds a member variable to UDPNetProcessorInternal and keep unused IOBufferBlocks for next call. Although the kept IOBufferBlocks cannot be freed because the freelist is already not available when UDPNetProcessorInternal is destructed, it should be ok since TS is already in a process of exiting.

maskit · 2021-07-30T01:33:44Z

All the changes are merged into master.

maskit added QUIC WIP labels Aug 8, 2017

maskit added this to the 8.0.0 milestone Aug 8, 2017

maskit assigned maskit and masaori335 Aug 8, 2017

bryancall previously requested changes Aug 8, 2017

View reviewed changes

zizhong requested changes Aug 8, 2017

View reviewed changes

maskit mentioned this pull request Aug 9, 2017

Make TLS 1.3 support optional #2344

Closed

scw00 reviewed Aug 15, 2017

View reviewed changes

maskit mentioned this pull request Aug 15, 2017

Simplify UDP packet send queue #2373

Closed

scw00 reviewed Aug 22, 2017

View reviewed changes

scw00 reviewed Aug 30, 2017

View reviewed changes

scw00 reviewed Sep 1, 2017

View reviewed changes

scw00 reviewed Sep 6, 2017

View reviewed changes

scw00 reviewed Sep 12, 2017

View reviewed changes

masaori335 force-pushed the quic-latest branch from 88b5d7c to 2250a41 Compare October 12, 2017 07:42

maskit force-pushed the quic-latest branch from a404ce9 to df98d80 Compare May 14, 2018 01:23

zwoop modified the milestones: 8.0.0, 9.0.0 Jun 6, 2018

masaori335 modified the milestones: 9.0.0, QUIC Aug 1, 2018

scw00 force-pushed the quic-latest branch from fad804a to d924cd8 Compare August 9, 2018 00:19

maskit and others added 26 commits July 27, 2020 11:11

const-ify quic/http3 code (#7079)

d09a757

Fix docs

4d579f4

Simplify interface between H3 and QUIC, and remove memcopy between them

42e8898

Avoid unnecessary QUIC CID randomization

2a9887f

Fix link error

99ff68f

Add QUICStreamStateListener

f90e8dd

Fix unit tests for QUICStreamState

837bd0e

maskit closed this Jul 30, 2021

zwoop removed this from the QUIC milestone Sep 23, 2021

maskit deleted the quic-latest branch January 24, 2023 19:10

Conversation

maskit commented Aug 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bryancall left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryancall Sep 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masaori335 Aug 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zwoop commented Aug 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zwoop commented Aug 9, 2017

Uh oh!

maskit commented Aug 9, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maskit commented Aug 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scw00 Aug 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scw00 Sep 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

maskit commented Aug 8, 2017 •

edited

Loading

bryancall Sep 6, 2017 •

edited

Loading

masaori335 Aug 9, 2017 •

edited

Loading

zwoop commented Aug 9, 2017 •

edited

Loading

scw00 Aug 30, 2017 •

edited

Loading

scw00 Sep 12, 2017 •

edited

Loading