[TT-12355] added log format config, func and test cases

[LLe27]

FR Jira Ticket

https://tyktech.atlassian.net/browse/TT-12355

Description

• Added the TYK_GW_LOGFORMAT Gateway config option
• The Tyk Gateway will determine which format should be used for STDOUT logs
  • If the TYK_GW_LOGFORMAT is set to json the Gateway will output logs in JSON format
  • if the TYK_GW_LOGFORMAT is set to "" empty string or any other string value it will default the TEXT format

Additionally, TYK_LOGFORMAT is supported, as is TYK_LOGLEVEL (and TYK_GW_LOGLEVEL).

Note that there is some global use from the log package, I don't think we can set json output from json config, as it would be picked up too late. The one case to cover is to make sure this passes with only config enabled:

# TYK_GW_LOGFORMAT=json ./tyk 2>&1 | head -n 1
{"level":"info","msg":"Tyk API Gateway v5.3.0-dev","prefix":"main","time":"2024-07-10T15:16:11+02:00"}
# TYK_LOGFORMAT=json ./tyk 2>&1 | head -n 1
{"level":"info","msg":"Tyk API Gateway v5.3.0-dev","prefix":"main","time":"2024-07-10T15:16:14+02:00"}
# ./tyk 2>&1 | head -n 1
{"level":"info","msg":"Tyk API Gateway v5.3.0-dev","prefix":"main","time":"2024-07-10T15:45:02+02:00"}

The above confirms reading from env and from config.

Related Issue

Motivation and Context

The option to determine which STDOUT log format will open the opportunity for customers to easily parse the STDOUT logs and open up opportunity for our team to easily write tests and parse as needed.

How This Has Been Tested

• Manual testing
• Unit testing
• Benchmarks added

---

PR Type

Enhancement, Tests

---

Description

• Added the TYK_GW_LOGFORMAT Gateway config option to determine log format.
• Implemented NewFormatter function to switch between JSON and Text log formats.
• Modified log initialization to use the new formatter function.
• Added unit tests to verify the correct formatter is used based on the input format.

---

Changes walkthrough 📝

	Relevant files
Enhancement	log.go Add log format configuration and initialization                    log/log.go Added NewFormatter function to handle log format based on TYK_GW_LOGFORMAT environment variable. Modified the default log formatter initialization to use NewFormatter. +17/-6
Tests	log_test.go Add unit tests for log formatter configuration                      log/log_test.go Added unit tests for NewFormatter function. Tested JSON and Text formatters based on input format. +61/-0

---

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[github-actions]

PR Reviewer Guide 🔍

⏱️ Estimated effort to review [1-5]	2
🧪 Relevant tests	Yes
🔒 Security concerns	No
⚡ Key issues to review	None

[github-actions]

API Changes

--- prev.txt	2024-09-05 10:52:10.917725882 +0000
+++ current.txt	2024-09-05 10:52:07.753711925 +0000
@@ -5633,6 +5633,10 @@
 	// If not set or left empty, it will default to `info`.
 	LogLevel string `json:"log_level"`
 
+	// You can now configure the log format to be either the standard or json format
+	// If not set or left empty, it will default to `standard`.
+	LogFormat string `json:"log_format"`
+
 	// Section for configuring OpenTracing support
 	// Deprecated: use OpenTelemetry instead.
 	Tracer Tracer `json:"tracing"`
@@ -8554,8 +8558,6 @@
 
 func (gw *Gateway) InitHostCheckManager(ctx context.Context, store storage.Handler)
 
-func (gw *Gateway) InitializeRPCCache()
-
 func (gw *Gateway) LoadAPI(specs ...*APISpec) (out []*APISpec)
 
 func (gw *Gateway) LoadDefinitionsFromRPCBackup() ([]*APISpec, error)
@@ -10857,25 +10859,57 @@
 FUNCTIONS
 
 func Get() *logrus.Logger
+    Get returns the default configured logger.
+
 func GetRaw() *logrus.Logger
+    GetRaw is used internally. Should likely be removed first, do not rely on
+    it.
+
 func LoadTranslations(thing map[string]interface{})
     LoadTranslations takes a map[string]interface and flattens it to
-    map[string]string Because translations have been loaded - we internally
-    override log the formatter Nested entries are accessible using dot notation.
-    example: `{"foo": {"bar": "baz"}}` flattened: `foo.bar: baz`
+    map[string]string. Because translations have been loaded - we internally
+    override log the formatter. Nested entries are accessible using dot
+    notation.
+
+    Example: `{"foo": {"bar": "baz"}}` Flattened: `foo.bar: baz`
 
+func NewFormatter(format string) logrus.Formatter
 
 TYPES
 
+type JSONFormatter struct {
+	// TimestampFormat sets the format used for marshaling timestamps.
+	// The format to use is the same than for time.Format or time.Parse from the standard
+	// library.
+	// The standard Library already provides a set of predefined format.
+	TimestampFormat string
+
+	// DisableTimestamp allows disabling automatic timestamps in output.
+	DisableTimestamp bool
+
+	// DataKey allows users to put all the log entry parameters into a nested dictionary at a given key.
+	DataKey string
+}
+    JSONFormatter formats logs into parsable json.
+
+func (f *JSONFormatter) Format(entry *logrus.Entry) ([]byte, error)
+    Format renders a single log entry
+
 type RawFormatter struct{}
+    RawFormatter returns the logrus entry message as bytes.
 
 func (f *RawFormatter) Format(entry *logrus.Entry) ([]byte, error)
+    Format returns the entry.Message as a []byte.
 
 type TranslationFormatter struct {
-	*logrus.TextFormatter
+	logrus.Formatter
 }
+    TranslationFormatter handles message reformatting with translations.
 
 func (t *TranslationFormatter) Format(entry *logrus.Entry) ([]byte, error)
+    Format will translate the log message based on the message code. This is a
+    HTTP response code if provided. The message is usually just "Finished" for
+    those cases, this would likely produce a better log message.
 
 # Package: ./regexp
 

[github-actions]

PR Code Suggestions ✨

Category	Suggestion	Score
Error handling	Add error handling for unsupported formats in NewFormatter Add error handling in the NewFormatter function to manage unsupported formats gracefully by logging an error or returning a default formatter with an error message. log/log.go [78-89] switch strings.ToLower(format) { case "json": return &logrus.JSONFormatter{ TimestampFormat: time.RFC3339, } +case "": + return &logrus.TextFormatter{ + TimestampFormat: "Jan 02 15:04:05", + FullTimestamp: true, + DisableColors: true, + } default: + log.Errorf("Unsupported format: %s", format) return &logrus.TextFormatter{ TimestampFormat: "Jan 02 15:04:05", FullTimestamp: true, DisableColors: true, } Suggestion importance[1-10]: 10 Why: Adding error handling for unsupported formats is crucial for robustness. It ensures that the application logs meaningful error messages and defaults to a known state, improving overall reliability and debuggability.	10
Enhancement	Allow dynamic configuration of log formatter in init In the init function, replace the direct assignment to log.Formatter with a call to NewFormatter using an environment variable or a default value to allow dynamic configuration of the log formatter. log/log.go [54] -log.Formatter = NewFormatter("") +log.Formatter = NewFormatter(os.Getenv("LOG_FORMAT")) Suggestion importance[1-10]: 9 Why: Allowing dynamic configuration of the log formatter via an environment variable enhances flexibility and adaptability of the logging configuration, making it more suitable for different environments without code changes.	9
	Add explicit handling for empty string format in NewFormatter Consider adding a case for handling an empty string in the NewFormatter function to explicitly return a default formatter. This will improve code clarity and ensure that the default case is handled predictably. log/log.go [78-89] switch strings.ToLower(format) { case "json": return &logrus.JSONFormatter{ TimestampFormat: time.RFC3339, + } +case "": + // Explicitly handle the default case + return &logrus.TextFormatter{ + TimestampFormat: "Jan 02 15:04:05", + FullTimestamp: true, + DisableColors: true, } default: return &logrus.TextFormatter{ TimestampFormat: "Jan 02 15:04:05", FullTimestamp: true, DisableColors: true, } Suggestion importance[1-10]: 7 Why: This suggestion improves code clarity by explicitly handling the empty string case, making the code more readable and predictable. However, it does not introduce any functional changes since the default case already handles the empty string.	7
Maintainability	Use a constant for the default timestamp format in NewFormatter Replace the hardcoded timestamp format in the default case of NewFormatter with a constant. This will centralize the management of timestamp formats and facilitate changes or configurations in the future. log/log.go [84-89] +const defaultTimestampFormat = "Jan 02 15:04:05" default: return &logrus.TextFormatter{ - TimestampFormat: "Jan 02 15:04:05", + TimestampFormat: defaultTimestampFormat, FullTimestamp: true, DisableColors: true, } Suggestion importance[1-10]: 8 Why: Using a constant for the timestamp format improves maintainability by centralizing the format definition, making future changes easier and reducing the risk of inconsistencies.	8

[titpetric]

I have covered the change with some benchmarks.

# go test -count=1 -bench=. -benchtime=10s -benchmem ./...
goos: linux
goarch: amd64
pkg: github.com/TykTechnologies/tyk/log
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
BenchmarkFormatter/json-8         	 3895482	      3086 ns/op	    2483 B/op	      34 allocs/op
BenchmarkFormatter/json-ext-8     	 5285396	      2427 ns/op	    2076 B/op	      22 allocs/op
BenchmarkFormatter/default-8      	 4405042	      2740 ns/op	    1793 B/op	      27 allocs/op
BenchmarkFormatter/none-8         	12155498	      1079 ns/op	    1457 B/op	      12 allocs/op
PASS
ok  	github.com/TykTechnologies/tyk/log	59.213s

This shows improved performance under json-ext over both the json and the default text formatting, increasing throughput from ~390K (json), ~440K (standard) to ~528K logging calls per second.

[sonarcloud]

Quality Gate failed

Failed conditions
47.8% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[sonarcloud]

Quality Gate failed

Failed conditions
47.8% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[titpetric]

/release to release-5.3

[tykbot]

Working on it! Note that it can take a few minutes.

[tykbot]

@titpetric Succesfully merged PR