Add traffic_ops_ort.pl suuport for processing ##OVERRIDE## lines from…

CHANGELOG.md

@@ -19,6 +19,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added an API 1.4 endpoint, /api/1.4/cdns/dnsseckeys/refresh, to perform necessary behavior previously served outside the API under `/internal`.
 - Adds the DS Record text to the cdn dnsseckeys endpoint in 1.4.
 - Added monitoring.json snapshotting. This stores the monitoring json in the same table as the crconfig snapshot. Snapshotting is now required in order to push out monitoring changes.
+- To traffic_ops_ort.pl added the ability to handle ##OVERRIDE## delivery service ANY_MAP raw remap text to replace and comment out a base delivery service remap rules. THIS IS A TEMPORARY HACK until versioned delivery services are implemented.
 
 ### Changed
 - Issue 2821: Fixed "Traffic Router may choose wrong certificate when SNI names overlap"

docs/source/admin/traffic_ops/using.rst

@@ -503,7 +503,8 @@ One of the most important settings when creating the delivery service is the sel
 +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | DNS_LIVE        | DNS Content routing, same as DNS_LIVE_NATNL, but the MID tier is bypassed.                                                                                                                        |
 +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ANY_MAP         | ANY_MAP is not known to Traffic Router. For this deliveryservice, the "Raw remap text" field in the input form will be used as the remap line on the cache.                                       |
+| ANY_MAP         | ANY_MAP is not known to Traffic Router. For this Delivery Sevice, the "Raw Remap Text" field in the input form will be used as the remap line in the cache's :file:`remap.config`.                |
+|                 | For more information see `ANY_MAP Raw Remap Text`_                                                                                                                                                |
 +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | STEERING        | The Delivery Service will be used to route to other delivery services. The target delivery services and the routing weights for those delivery services will be defined by an admin or steering   |
 |                 | user. For more information see the `steering feature <../traffic_router.html#steering-feature>`_ documentation                                                                                    |
@@ -541,6 +542,60 @@ The deliveryservice screen also allows you to set the DSCP value of traffic sent
 
 .. Note:: The DSCP setting in the UI is *only* for setting traffic towards the client, and gets applied *after* the initial TCP handshake is complete, and the HTTP request is received (before that the cache can't determine what deliveryservice this request is for, and what DSCP to apply), so the DSCP feature can not be used for security settings - the TCP SYN-ACK is not going to be DSCP marked.
 
+.. _raw-remap-text:
+
+ANY_MAP Raw Remap Text
+----------------------
+The Raw Remap Text may contain the following special strings that will be replaced by :program:`traffic_ops_ort` at :abbr:`ATS (Apache Traffic Server)` edge and mid levels in `remap.config <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/files/remap.config.en.html>`_:
+
+.. table:: Traffic Ops ORT special strings
+
+	+---------------------+-------------------------------------------------+
+	| Traffic Ops Entry   | Gets Replaced with                              |
+	+=====================+=================================================+
+	| __CACHE_IPV4__      | The cache's IPv4 address                        |
+	+---------------------+-------------------------------------------------+
+	| __HOSTNAME__        | Short hostname (same as ``hostname -s``)        |
+	+---------------------+-------------------------------------------------+
+	| __FULL_HOSTNAME__   | Long hostname (same as :manpage:`hostname(1)`)  |
+	+---------------------+-------------------------------------------------+
+	| __SERVER_TCP_PORT__ | Server incoming TCP port number                 |
+	+---------------------+-------------------------------------------------+
+	| __RETURN__          | A newline                                       |
+	+---------------------+-------------------------------------------------+
+	| ##OVERRIDE##        | See below                                       |
+	+---------------------+-------------------------------------------------+
+
+ANY_MAP ##OVERRIDE##
+""""""""""""""""""""
+.. warning:: The ANY_MAP ``##OVERRIDE##`` special string is a temporary solution and will be deprecated once Delivery Service Versioning is implemented.
+
+A special ``##OVERRIDE##`` string has been added to allow an ANY_MAP rule to override another Delivery Service's remap rule, implemented by :program:`traffic_ops_ort`.  When present, the original Delivery Service remap rule is commented out with an ``##OVERRIDDEN##`` prefix and the ``##OVERRIDE##`` rule is activated in its place.
+
+:abbr:`ATS (Apache Traffic Server)` `remap.config <https://docs.trafficserver.apache.org/en/7.1.x/admin-guide/files/remap.config.en.html>`_:
+
+.. code-block:: text
+	:caption: Delivery Service :file:`remap.config` line:
+
+	map http://from.com/ http://to.com/
+
+.. code-block:: text
+	:caption: ANY_MAP Raw Remap Text
+
+	##OVERRIDE## map http://from.com/ http://to.com/ thundering_herd_mitigation.so
+
+.. code-block:: text
+	:caption: :program:`traffic_ops_ort` post process :file:`remap.config` lines after merge:
+
+	##OVERRIDE##
+	map http://from.com/ http://to.com/ thundering_herd_mitigation.so
+	##OVERRIDDEN## map http://from.com/ http://to.com/
+
+The ANY_MAP ``##OVERRIDE##`` may be used to incrementally deploy plugins by assigning a subset of caches to the ANY_MAP ``##OVERRIDE##`` Delivery Service in addition to the original Delivery Service.  This allows Traffic Router to send traffic to edges based on the original Delivery Service but serve them using the ANY_MAP override Raw Remap Text.
+
+.. warning:: The from endpoint must exactly match for this to properly work (ie: trailing URL '/'), otherwise :abbr:`ATS (Apache Traffic Server)` may fail to initialize or reload while processing :file:`remap.config`.
+
+.. note:: Any of these ANY_MAP ``##OVERRIDE##`` rules **should** be documented in the comment fields of the original Delivery Service to assist with troubleshooting.
 
 .. index::
 	Token Based Authentication

traffic_ops/bin/traffic_ops_ort.pl

@@ -78,8 +78,8 @@
 	else {
 		$traffic_ops_host = $ARGV[2];
 		$traffic_ops_host =~ s/\/*$//g;
-                # Stash to_url for later use...
-                $to_url = $traffic_ops_host;
+		# Stash to_url for later use...
+		$to_url = $traffic_ops_host;
 	}
 }
 else {
@@ -303,12 +303,12 @@ sub revalidate_while_sleeping {
 }
 
 sub os_version {
-  my $release = "UNKNOWN";
-  if (`uname -r` =~ m/.+(el\d)(?:\.\w+)*\.x86_64/)  {
-    $release = uc $1;
-  }
-  exists $supported_el_release{$release} ? return $release
-      : die("unsupported el_version: $release");
+	my $release = "UNKNOWN";
+	if (`uname -r` =~ m/.+(el\d)(?:\.\w+)*\.x86_64/)  {
+		$release = uc $1;
+	}
+	exists $supported_el_release{$release} ? return $release
+	    : die("unsupported el_version: $release");
 }
 
 sub usage {
@@ -377,6 +377,11 @@ sub process_cfg_file {
 		$result =~ s/\s*__RETURN__\s*/\n/g;
 	}
 
+	# Process ##OVERRIDE## remap rules (from anymap rawtext)
+	if ( $cfg_file eq "remap.config" ) {
+		$result = &adv_preprocessing_remap( \$result );
+	}
+
 	my @db_file_lines = @{ &scrape_unencode_text($result) };
 
 	my $file = $config_dir . "/" . $cfg_file;
@@ -1620,11 +1625,11 @@ sub get_cookie {
 	}
 
 	my $url = $to_host . "/api/1.3/user/login";
-    	my $json = qq/{ "u": "$u", "p": "$p"}/;
-    	my $lwp = LWP::UserAgent->new;
-    	my $response = $lwp->post($url, Content => $json);
+	my $json = qq/{ "u": "$u", "p": "$p"}/;
+	my $lwp = LWP::UserAgent->new;
+	my $response = $lwp->post($url, Content => $json);
 
-    	&check_lwp_response_code($response, $FATAL);
+	&check_lwp_response_code($response, $FATAL);
 
 	my $cookie;
 	if ( $response->header('Set-Cookie') ) {
@@ -1794,8 +1799,8 @@ sub get_cfg_file_list {
 		$to_rev_proxy_url = $ort_ref->{'info'}->{'toRevProxyUrl'};
 		if ( $to_rev_proxy_url && $rev_proxy_disable == 0 ) {
 			$to_rev_proxy_url =~ s/\/*$//g;
-                        # Note: If traffic_ops_url is changing, would be suggested to get a new cookie.
-                        #       Secrets might not be the same on all Traffic Ops instance.
+			# Note: If traffic_ops_url is changing, would be suggested to get a new cookie.
+			#       Secrets might not be the same on all Traffic Ops instance.
 			$traffic_ops_host = $to_rev_proxy_url;
 			$rev_proxy_in_use = 1;
 			( $log_level >> $INFO ) && printf("INFO Found Traffic Ops Reverse Proxy URL from Traffic Ops: $to_rev_proxy_url\n");
@@ -2657,7 +2662,7 @@ sub open_file_get_contents {
 		chomp($line);
 		( $log_level >> $TRACE ) && print "TRACE Line from cfg file on disk:\t$line.\n";
 		if ( $line =~ m/^\#/ || $line =~ m/^$/ ) {
-			if ( ( $line !~ m/DO NOT EDIT - Generated for / && $line !~ m/$header_comment/ ) && $line !~ m/TRAFFIC OPS NOTE\:/ ) {
+			if ( ( $line !~ m/DO NOT EDIT - Generated for / && $line !~ m/$header_comment/ ) && ( $line !~ m/TRAFFIC OPS NOTE\:/ ) && ( $line !~ m/^##OVERRID.*##/ ) ) {
 				next;
 			}
 		}
@@ -2834,6 +2839,75 @@ sub backup_file {
 
 }
 
+sub adv_preprocessing_remap {
+
+	my $buffer = ${ $_[0] };
+
+	( my @file_lines ) = split( /\n/, $buffer );
+
+	if ( 1 < $#file_lines ) { #header line is always present, so look for 2 lines or more
+		( $log_level >> $DEBUG ) && print "DEBUG Entering advanced pre-processing for remap.config.\n";
+
+		# key on the FROM remap
+		my %override_hash=();
+
+		# sweep for override lines
+		foreach my $line (@file_lines) {
+			if ( $line =~ m/^##OVERRIDE##\s*/ ) {
+				my $newline = $line =~ s/^##OVERRIDE##\s*//r;
+				( my @fields ) = split( /\s+/, $newline, 4 );
+				# ensure <rule> <from> <to>
+				if ( 2 < $#fields ) {
+					$override_hash{$fields[1]}=1;
+				}
+			}
+		}
+
+		# handle override lines, etc
+		my $overrides = keys %override_hash;
+		if (0 < $overrides) {
+			( $log_level >> $INFO ) && print "INFO Entering ##OVERRIDE## processing for remap.config.\n";
+
+			my @out_lines;
+			my $lineno = 0;
+			my $cnt = 0;
+
+			foreach my $line (@file_lines) {
+				if ( $line =~ m/DO NOT EDIT - Generated for/ ) {
+					push( @out_lines, $line );
+				}
+				elsif ( $line =~ m/^##OVERRIDE##\s*/ ) {
+					$line =~ s/^##OVERRIDE##\s*//;
+					( $log_level >> $DEBUG ) && print "DEBUG new override line $lineno: \'$line\'\n";
+					push( @out_lines, "##OVERRIDE##" );
+					push( @out_lines, $line );
+				}
+				elsif ( $line !~ m/^\s*#/ ) {
+					( my @fields ) = split( /\s+/, $line, 4 );
+					# ensure <rule> <from> <to>
+					if ( 2 < $#fields && exists($override_hash{$fields[1]}) ) {
+						( $log_level >> $DEBUG ) && print "DEBUG old overriden line $lineno: \'$line\'\n";
+						$cnt++;
+						$line = "##OVERRIDDEN## " . $line;
+					}
+
+					push( @out_lines, $line );
+				}
+				else {
+					push( @out_lines, $line );
+				}
+
+				$lineno++;
+			}
+
+			( $log_level >> $WARN ) && print "WARN Overrode $cnt old remap rule(s) with $overrides new remap rule(s).\n";
+			$buffer = join( "\n", @out_lines ) . "\n";
+		}
+	}
+
+	return $buffer;
+}
+
 sub adv_processing_udev {
 
 	my @db_file_lines = @{ $_[0] };