diff --git a/README.md b/README.md index f1730e1..8a446be 100644 --- a/README.md +++ b/README.md @@ -8,12 +8,15 @@ - All transformations support multibyte characters. ``` -Usage of Password Transformation Tool (ptt) version (0.1.3): +Usage of Password Transformation Tool (ptt) version (0.2.0): ptt [options] [...] Accepts standard input and/or additonal arguments. Options: + -b Bypass map creation and use stdout as primary output. + -d int + Enable debug mode with verbosity levels [0-2]. -f value Read additional files for input. -i value @@ -36,6 +39,8 @@ Options: Transformation to apply to input. -tf value Read additional files for transformations if applicable. + -tp value + Read a template file for multiple transformations and operations. -u value Read additional URLs for input. -v Show verbose output when possible. @@ -44,7 +49,7 @@ Options: -vvv Show verbose statistics output when possible. -The -f, -k, -r, -tf, and -u flags can be used multiple times and together. +The -f, -k, -r, -tf, -tp, and -u flags can be used multiple times and together. Transformation Modes: -t append @@ -65,10 +70,12 @@ Transformation Modes: Transforms input into insert rules starting at index. -t mask -rm [uldsb] -v Transforms input by masking characters with provided mask. + -t mask-match -tf [file] + Transforms input by keeping only strings with matching masks from a mask file. + -t mask-retain -rm [uldsb] -tf [file] + Transforms input by creating masks that still retain strings from file. -t mask-swap -tf [file] Transforms input by swapping tokens from a partial mask file and a input file. - -t match -tf [file] - Transforms input by keeping only strings with matching masks from a mask file. -t overwrite -i [index] Transforms input into overwrite rules starting at index. -t pop -rm [uldsb] @@ -81,8 +88,6 @@ Transformation Modes: Transforms input into prepend-shift rules. -t remove -rm [uldsb] Transforms input by removing characters with provided mask characters. - -t retain -rm [uldsb] -tf [file] - Transforms input by creating masks that still retain strings from file. -t swap -tf [file] Transforms input by swapping tokens with exact matches from a ':' separated file. -t toggle -i [index] diff --git a/docs/USAGE.md b/docs/USAGE.md index 5760cf7..2dfa6ea 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1,5 +1,5 @@ # Password Transformation Tool (PTT) Usage Guide -## Version 0.1.0 +## Version 0.2.0 ### Table of Contents 1. [Introduction](#introduction) @@ -65,11 +65,26 @@ There are some additional notes when importing data: - When reading from standard input, the tool can detect chaining `ptt` commands when the `-v` flag is used. This can be used to pipe multiple commands together. - When reading from files, the tool can detect when `ptt` JSON output is used as input and will parse the JSON data. +- The `-b` flag can be used to bypass map creation and use stdout as primary output. This can be useful for working with large amounts of data. + - If the `-b` flag is used, the final output will be empty and all + filtering and duplication removal will be disabled. +- The `-d [0-2]` flag can be used to enable debug output. This will show the data + object after all transformations have been applied. There are two (2) levels + of debug output that can be used. + - Level 1 will not print each iteration transformation but overall input and output. + - Level 2 will print each iteration transformation and overall input and output. +- The `-tp` flag can not be used with other transformations at the same time (`-t`). The + template file should contain a list of transformations and operations to apply + to the input data. The template file should be in JSON format. + - See `docs/template.json` ([link](https://github.com/JakeWnuk/ptt/blob/main/docs/template.json)) for an example. + - See `docs/templates/` ([link](https://github.com/JakeWnuk/ptt/blob/main/docs/templates/)) for more examples. -The `-f`, `-k`, `-r`, `-tf`, and `-u` flags can be used multiple times and have +The `-f`, `-k`, `-r`, `-tf`, `-tp`, and `-u` flags can be used multiple times and have their collective values combined. The rest of the flags can only be used once. #### Options: +- `-b`: Bypass map creation and use stdout as primary output. +- `-d`: Enable debug mode with verbosity levels [0-2]. - `-f`: Read additional files for input. - `-i`: Starting index for transformations if applicable. Accepts ranges separated by '-'. (default 0) - `-k`: Only keep items in a file. @@ -81,6 +96,7 @@ their collective values combined. The rest of the flags can only be used once. - `-rm`: Replacement mask for transformations if applicable. (default "uldsb") - `-t`: Transformation to apply to input. - `-tf`: Read additional files for transformations if applicable. +- `-tp`: Read a template file for multiple transformations and operations. - `-u`: Read additional URLs for input. - `-v`: Show verbose output when possible. - `-vv`: Show statistics output when possible. @@ -103,8 +119,8 @@ The following transformations can be used with the `-t` flag: - `dehex`: Transforms input by decoding $HEX[...] formatted - `mask`: Transforms input by masking characters with provided mask. - `remove`: Transforms input by removing characters with provided mask characters. -- `retain`: Transforms input by creating masks that still retain strings from file. -- `match`: Transforms input by keeping only strings with matching masks from a mask file +- `mask-retain`: Transforms input by creating masks that still retain strings from file. +- `mask-match`: Transforms input by keeping only strings with matching masks from a mask file - `swap`: Transforms input by swapping tokens with exact matches from a ':' separated file. - `pop`: Transforms input by generating tokens from popping strings at character boundaries. - `mask-swap`: Transforms input by swapping tokens from a partial mask file and a input file. @@ -149,6 +165,7 @@ keywords above: - `ptt -t [transformation] -rm ulds`: Apply a transformation with a custom mask. Default is all characters. - `ptt -t [transformation] -i 5`: Apply a transformation starting at a specific index. - `ptt -i 1-5 -t [transformation]`: Apply a transformation starting at a specific index. +- `ptt -tp template.json`: Apply multiple transformations and operations from a template file. #### Filter Formats: - `ptt -k keep.txt`: Keep only items in a file. diff --git a/docs/template.json b/docs/template.json new file mode 100644 index 0000000..093aec3 --- /dev/null +++ b/docs/template.json @@ -0,0 +1,18 @@ +[ + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "mask" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "dsb", + "Bypass": false, + "TransformationMode": "mask" + } +] diff --git a/main.go b/main.go index 8f09d11..9641e8e 100644 --- a/main.go +++ b/main.go @@ -15,7 +15,7 @@ import ( "github.com/jakewnuk/ptt/pkg/utils" ) -var version = "0.1.3" +var version = "0.2.0" var wg sync.WaitGroup var mutex = &sync.Mutex{} var retain models.FileArgumentFlag @@ -23,6 +23,7 @@ var remove models.FileArgumentFlag var readFiles models.FileArgumentFlag var readURLs models.FileArgumentFlag var transformationFiles models.FileArgumentFlag +var templateFiles models.FileArgumentFlag var intRange models.IntRange var lenRange models.IntRange var primaryMap map[string]int @@ -35,29 +36,29 @@ func main() { fmt.Fprintf(os.Stderr, "ptt [options] [...]\nAccepts standard input and/or additonal arguments.\n\n") fmt.Fprintf(os.Stderr, "Options:\n") flag.PrintDefaults() - fmt.Fprintf(os.Stderr, "\nThe -f, -k, -r, -tf, and -u flags can be used multiple times and together.\n") + fmt.Fprintf(os.Stderr, "\nThe -f, -k, -r, -tf, -tp, and -u flags can be used multiple times and together.\n") fmt.Fprintln(os.Stderr, "\nTransformation Modes:") modes := map[string]string{ - "append": "Transforms input into append rules.", - "append-remove": "Transforms input into append-remove rules.", - "append-shift": "Transforms input into append-shift rules.", - "prepend": "Transforms input into prepend rules.", - "prepend-remove": "Transforms input into prepend-remove rules.", - "prepend-shift": "Transforms input into prepend-shift rules.", - "insert -i [index]": "Transforms input into insert rules starting at index.", - "overwrite -i [index]": "Transforms input into overwrite rules starting at index.", - "toggle -i [index]": "Transforms input into toggle rules starting at index.", - "encode": "Transforms input by URL, HTML, and Unicode escape encoding.", - "decode": "Transforms input by URL, HTML, and Unicode escape decoding.", - "hex": "Transforms input by encoding strings into $HEX[...] format.", - "dehex": "Transforms input by decoding $HEX[...] formatted strings.", - "mask -rm [uldsb] -v": "Transforms input by masking characters with provided mask.", - "remove -rm [uldsb]": "Transforms input by removing characters with provided mask characters.", - "retain -rm [uldsb] -tf [file]": "Transforms input by creating masks that still retain strings from file.", - "pop -rm [uldsb]": "Transforms input by generating tokens from popping strings at character boundaries.", - "match -tf [file]": "Transforms input by keeping only strings with matching masks from a mask file.", - "swap -tf [file]": "Transforms input by swapping tokens with exact matches from a ':' separated file.", - "mask-swap -tf [file]": "Transforms input by swapping tokens from a partial mask file and a input file.", + "append": "Transforms input into append rules.", + "append-remove": "Transforms input into append-remove rules.", + "append-shift": "Transforms input into append-shift rules.", + "prepend": "Transforms input into prepend rules.", + "prepend-remove": "Transforms input into prepend-remove rules.", + "prepend-shift": "Transforms input into prepend-shift rules.", + "insert -i [index]": "Transforms input into insert rules starting at index.", + "overwrite -i [index]": "Transforms input into overwrite rules starting at index.", + "toggle -i [index]": "Transforms input into toggle rules starting at index.", + "encode": "Transforms input by URL, HTML, and Unicode escape encoding.", + "decode": "Transforms input by URL, HTML, and Unicode escape decoding.", + "hex": "Transforms input by encoding strings into $HEX[...] format.", + "dehex": "Transforms input by decoding $HEX[...] formatted strings.", + "mask -rm [uldsb] -v": "Transforms input by masking characters with provided mask.", + "remove -rm [uldsb]": "Transforms input by removing characters with provided mask characters.", + "mask-retain -rm [uldsb] -tf [file]": "Transforms input by creating masks that still retain strings from file.", + "pop -rm [uldsb]": "Transforms input by generating tokens from popping strings at character boundaries.", + "mask-match -tf [file]": "Transforms input by keeping only strings with matching masks from a mask file.", + "swap -tf [file]": "Transforms input by swapping tokens with exact matches from a ':' separated file.", + "mask-swap -tf [file]": "Transforms input by swapping tokens from a partial mask file and a input file.", } // Sort and print transformation modes @@ -82,10 +83,13 @@ func main() { transformation := flag.String("t", "", "Transformation to apply to input.") replacementMask := flag.String("rm", "uldsb", "Replacement mask for transformations if applicable.") jsonOutput := flag.String("o", "", "Output to JSON file in addition to stdout.") + bypassMap := flag.Bool("b", false, "Bypass map creation and use stdout as primary output.") + debugMode := flag.Int("d", 0, "Enable debug mode with verbosity levels [0-2].") flag.Var(&retain, "k", "Only keep items in a file.") flag.Var(&remove, "r", "Only keep items not in a file.") flag.Var(&readFiles, "f", "Read additional files for input.") flag.Var(&transformationFiles, "tf", "Read additional files for transformations if applicable.") + flag.Var(&templateFiles, "tp", "Read a template file for multiple transformations and operations.") flag.Var(&intRange, "i", "Starting index for transformations if applicable. Accepts ranges separated by '-'.") flag.Var(&lenRange, "l", "Keeps output equal to or within a range of lengths. Accepts ranges separated by '-'.") flag.Var(&readURLs, "u", "Read additional URLs for input.") @@ -97,9 +101,10 @@ func main() { removeMap := utils.ReadFilesToMap(fs, remove) readFilesMap := utils.ReadFilesToMap(fs, readFiles) transformationFilesMap := utils.ReadFilesToMap(fs, transformationFiles) + transformationTemplateArray := utils.ReadJSONToArray(fs, templateFiles) readURLsMap, err := utils.ReadURLsToMap(readURLs) if err != nil { - fmt.Println("Error reading URLs:", err) + fmt.Fprintf(os.Stderr, "[!] Error reading URLs: %s\n", err) return } @@ -108,7 +113,7 @@ func main() { if (stat.Mode() & os.ModeCharDevice) == 0 { primaryMap, err = utils.LoadStdinToMap(bufio.NewScanner(os.Stdin)) if err != nil { - fmt.Println("Error reading from stdin:", err) + fmt.Fprintf(os.Stderr, "[!] Error reading from stdin: %s\n", err) return } } @@ -123,16 +128,43 @@ func main() { primaryMap = utils.CombineMaps(primaryMap, readFilesMap, readURLsMap) } + // Bypass map creation if requested + if *bypassMap { + fmt.Fprintf(os.Stderr, "[*] Bypassing map creation and using stdout as primary output. Some features are disabled.\n") + } + // Apply transformation if provided - if *transformation != "" { - primaryMap = transform.TransformationController(primaryMap, *transformation, intRange.Start, intRange.End, *verbose, *replacementMask, transformationFilesMap) + if *transformation != "" && templateFiles == nil { + primaryMap = transform.TransformationController(primaryMap, *transformation, intRange.Start, intRange.End, *verbose, *replacementMask, transformationFilesMap, *bypassMap, *debugMode) + } else if templateFiles != nil && *transformation == "" { + fmt.Fprintf(os.Stderr, "[*] Using template files for multiple transformations.\n") + + // Make a copy of the primary map to avoid modifying the original + temporaryMap := make(map[string]int) + for k, v := range primaryMap { + temporaryMap[k] = v + } + + // Apply transformations from template files + for i, template := range transformationTemplateArray { + if i == 0 { + temporaryMap = transform.TransformationController(primaryMap, template.TransformationMode, template.StartIndex, template.EndIndex, template.Verbose, template.ReplacementMask, transformationFilesMap, template.Bypass, *debugMode) + } else { + temporaryMap = utils.CombineMaps(temporaryMap, transform.TransformationController(primaryMap, template.TransformationMode, template.StartIndex, template.EndIndex, template.Verbose, template.ReplacementMask, transformationFilesMap, template.Bypass, *debugMode)) + } + } + primaryMap = temporaryMap + + } else if *transformation != "" && templateFiles != nil { + fmt.Fprintf(os.Stderr, "[!] Transformation and template flags cannot be used together.\n") + return } // Process retain and remove maps if provided if len(retainMap) > 0 || len(removeMap) > 0 { primaryMap, err = format.RetainRemove(primaryMap, retainMap, removeMap) if err != nil { - fmt.Println("Error processing retain and remove flags:", err) + fmt.Fprintf(os.Stderr, "[!] Error processing retain and remove flags: %s\n", err) return } } @@ -160,7 +192,7 @@ func main() { if *jsonOutput != "" { err = format.SaveArrayToJSON(*jsonOutput, primaryMap) if err != nil { - fmt.Println("Error saving output to JSON:", err) + fmt.Fprintf(os.Stderr, "[!] Error saving output to JSON: %s\n", err) return } } diff --git a/pkg/format/format.go b/pkg/format/format.go index 0fd02e5..bc12cbc 100644 --- a/pkg/format/format.go +++ b/pkg/format/format.go @@ -80,6 +80,11 @@ func PrintStatsToSTDOUT(freq map[string]int, verbose bool, max int) { max = len(p) } + if len(p) == 0 { + fmt.Println("No items to print!") + return + } + // Print the statistics if verbose { fmt.Fprintf(os.Stderr, "[*] Starting statistics generation. Please wait...\n") @@ -443,25 +448,41 @@ func RemoveLengthRange(freq map[string]int, start int, end int) map[string]int { // Args: // // input (map[string]int): A map of input strings +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): A new map of encoded strings -func EncodeInputMap(input map[string]int) map[string]int { +func EncodeInputMap(input map[string]int, bypass bool, debug bool) map[string]int { output := make(map[string]int) for k, v := range input { urlEncoded, htmlEncoded, escapeEncoded := EncodeString(k) - if urlEncoded != "" { + if debug { + fmt.Fprintf(os.Stderr, "[?] EncodeInputMap:\n") + fmt.Fprintf(os.Stderr, "Input: %s\n", k) + fmt.Fprintf(os.Stderr, "URL Encoded: %s\n", urlEncoded) + fmt.Fprintf(os.Stderr, "HTML Encoded: %s\n", htmlEncoded) + fmt.Fprintf(os.Stderr, "Unicode Escaped: %s\n", escapeEncoded) + } + + if urlEncoded != "" && !bypass { output[urlEncoded] = v + } else if urlEncoded != "" && bypass { + fmt.Println(urlEncoded) } - if htmlEncoded != "" { + if htmlEncoded != "" && !bypass { output[htmlEncoded] = v + } else if htmlEncoded != "" && bypass { + fmt.Println(htmlEncoded) } - if escapeEncoded != "" { + if escapeEncoded != "" && !bypass { output[escapeEncoded] = v + } else if escapeEncoded != "" && bypass { + fmt.Println(escapeEncoded) } } return output @@ -505,25 +526,41 @@ func EncodeString(s string) (string, string, string) { // Args: // // input (map[string]int): A map of input strings +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): A new map of decoded strings -func DecodeInputMap(input map[string]int) map[string]int { +func DecodeInputMap(input map[string]int, bypass bool, debug bool) map[string]int { output := make(map[string]int) for k, v := range input { urlDecoded, htmlDecoded, escapeDecoded := DecodeString(k) - if urlDecoded != "" { + if debug { + fmt.Fprintf(os.Stderr, "[?] DecodeInputMap:\n") + fmt.Fprintf(os.Stderr, "Input: %s\n", k) + fmt.Fprintf(os.Stderr, "URL Decoded: %s\n", urlDecoded) + fmt.Fprintf(os.Stderr, "HTML Decoded: %s\n", htmlDecoded) + fmt.Fprintf(os.Stderr, "Unicode Escaped Decoded: %s\n", escapeDecoded) + } + + if urlDecoded != "" && !bypass { output[urlDecoded] = v + } else if urlDecoded != "" && bypass { + fmt.Println(urlDecoded) } - if htmlDecoded != "" { + if htmlDecoded != "" && !bypass { output[htmlDecoded] = v + } else if htmlDecoded != "" && bypass { + fmt.Println(htmlDecoded) } - if escapeDecoded != "" { + if escapeDecoded != "" && !bypass { output[escapeDecoded] = v + } else if escapeDecoded != "" && bypass { + fmt.Println(escapeDecoded) } } return output @@ -628,11 +665,13 @@ func DeASCIIEscapeUnicode(str string) string { // Args: // // input (map[string]int): A map of hex encoded strings +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): A new map of decoded strings -func DehexMap(input map[string]int) map[string]int { +func DehexMap(input map[string]int, bypass bool, debug bool) map[string]int { decodedMap := make(map[string]int) for k, v := range input { @@ -643,7 +682,18 @@ func DehexMap(input map[string]int) map[string]int { continue } decodedStr := string(decoded) - decodedMap[decodedStr] = v + + if debug { + fmt.Fprintf(os.Stderr, "[?] DehexMap:\n") + fmt.Fprintf(os.Stderr, "Input: %s\n", k) + fmt.Fprintf(os.Stderr, "Decoded: %s\n", decodedStr) + } + + if !bypass { + decodedMap[decodedStr] = v + } else { + fmt.Println(decodedStr) + } } return decodedMap @@ -655,15 +705,21 @@ func DehexMap(input map[string]int) map[string]int { // Args: // // input (map[string]int): A map of input strings +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): A new map of encoded strings -func HexEncodeMap(input map[string]int) map[string]int { +func HexEncodeMap(input map[string]int, bypass bool, debug bool) map[string]int { output := make(map[string]int) for k, v := range input { encoded := hex.EncodeToString([]byte(k)) - output["$HEX["+encoded+"]"] = v + if !bypass { + output["$HEX["+encoded+"]"] = v + } else { + fmt.Println("$HEX[" + encoded + "]") + } } return output } diff --git a/pkg/format/format_test.go b/pkg/format/format_test.go index a6738ab..b2419ee 100644 --- a/pkg/format/format_test.go +++ b/pkg/format/format_test.go @@ -171,7 +171,7 @@ func TestEncodeInputMap(t *testing.T) { // Run test cases for _, test := range tests { - result := EncodeInputMap(test.input) + result := EncodeInputMap(test.input, false, false) if utils.CheckAreMapsEqual(result, test.output) == false { t.Errorf("EncodeInputMap() failed - expected: %v, got: %v", test.output, result) } @@ -236,7 +236,7 @@ func TestDecodeInputMap(t *testing.T) { // Run test cases for _, test := range tests { - result := DecodeInputMap(test.input) + result := DecodeInputMap(test.input, false, false) if utils.CheckAreMapsEqual(result, test.output) == false { t.Errorf("DecodeInputMap() failed - expected: %v, got: %v", test.output, result) } @@ -333,7 +333,7 @@ func TestDehexMap(t *testing.T) { // Run test cases for _, test := range tests { - result := DehexMap(test.input) + result := DehexMap(test.input, false, false) if utils.CheckAreMapsEqual(result, test.output) == false { t.Errorf("DehexMap() failed - expected: %v, got: %v", test.output, result) } @@ -361,7 +361,7 @@ func TestHexEncodeMap(t *testing.T) { // Run test cases for _, test := range tests { - result := HexEncodeMap(test.input) + result := HexEncodeMap(test.input, false, false) if utils.CheckAreMapsEqual(result, test.output) == false { t.Errorf("HexEncodeMap() failed - expected: %v, got: %v", test.output, result) } diff --git a/pkg/mask/mask.go b/pkg/mask/mask.go index ea11aef..a7be6e1 100644 --- a/pkg/mask/mask.go +++ b/pkg/mask/mask.go @@ -3,6 +3,7 @@ package mask import ( "fmt" + "os" "regexp" "strings" "unicode" @@ -74,11 +75,13 @@ func ConstructReplacements(str string) []string { // input (map[string]int): Map to mask // replacements ([]string): Array of characters to replace // verbose (bool): Verbose information if true +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // maskedMap (map[string]int): Masked map -func MakeMaskedMap(input map[string]int, replacementMask string, verbose bool) map[string]int { +func MakeMaskedMap(input map[string]int, replacementMask string, verbose bool, bypass bool, debug bool) map[string]int { maskedMap := make(map[string]int) replacements := ConstructReplacements(replacementMask) replacer := strings.NewReplacer(replacements...) @@ -94,10 +97,24 @@ func MakeMaskedMap(input map[string]int, replacementMask string, verbose bool) m newKey = fmt.Sprintf("%s:%d:%d", newKey, len(key), TestMaskComplexity(newKey)) } - if oldValue, exists := maskedMap[newKey]; exists { - maskedMap[newKey] = oldValue + value - } else { - maskedMap[newKey] = value + if debug { + fmt.Fprintf(os.Stderr, "[?] MakeMaskedMap:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "New Key: %s\n", newKey) + fmt.Fprintf(os.Stderr, "Replacement Mask: %s\n", replacementMask) + } + + switch bypass { + case false: + + if oldValue, exists := maskedMap[newKey]; exists { + maskedMap[newKey] = oldValue + value + } else { + maskedMap[newKey] = value + } + + case true: + fmt.Println(newKey) } } return maskedMap @@ -111,11 +128,13 @@ func MakeMaskedMap(input map[string]int, replacementMask string, verbose bool) m // input (map[string]int): Map to mask // replacementMask (string): Mask characters to apply // retain (map[string]int): Map of keywords to retain +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // maskedMap (map[string]int): Masked retain map -func MakeRetainMaskedMap(input map[string]int, replacementMask string, retain map[string]int) map[string]int { +func MakeRetainMaskedMap(input map[string]int, replacementMask string, retain map[string]int, bypass bool, debug bool) map[string]int { maskedMap := make(map[string]int) replacements := ConstructReplacements(replacementMask) replacer := strings.NewReplacer(replacements...) @@ -144,10 +163,23 @@ func MakeRetainMaskedMap(input map[string]int, replacementMask string, retain ma continue } - if oldValue, exists := maskedMap[newKey]; exists { - maskedMap[newKey] = oldValue + value - } else { - maskedMap[newKey] = value + if debug { + fmt.Fprintf(os.Stderr, "[?] MakeRetainMaskedMap:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Retain Key: %s\n", retainKey) + fmt.Fprintf(os.Stderr, "New Key: %s\n", newKey) + fmt.Fprintf(os.Stderr, "Replacement Mask: %s\n", replacementMask) + } + + switch bypass { + case false: + if oldValue, exists := maskedMap[newKey]; exists { + maskedMap[newKey] = oldValue + value + } else { + maskedMap[newKey] = value + } + case true: + fmt.Println(newKey) } } } @@ -235,11 +267,13 @@ func TestMaskComplexity(str string) int { // Args: // // input (map[string]int): Input map +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): Masked map -func RemoveMaskedCharacters(input map[string]int) map[string]int { +func RemoveMaskedCharacters(input map[string]int, bypass bool, debug bool) map[string]int { maskedMap := make(map[string]int) replacer := strings.NewReplacer("?u", "", "?l", "", "?d", "", "?b", "", "?s", "") @@ -250,10 +284,21 @@ func RemoveMaskedCharacters(input map[string]int) map[string]int { newKey = ConvertMultiByteMask(newKey) } - if oldValue, exists := maskedMap[newKey]; exists { - maskedMap[newKey] = oldValue + value - } else { - maskedMap[newKey] = value + if debug { + fmt.Fprintf(os.Stderr, "[?] RemoveMaskedCharacters:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "New Key: %s\n", newKey) + } + + switch bypass { + case false: + if oldValue, exists := maskedMap[newKey]; exists { + maskedMap[newKey] = oldValue + value + } else { + maskedMap[newKey] = value + } + case true: + fmt.Println(newKey) } } return maskedMap @@ -272,10 +317,12 @@ func RemoveMaskedCharacters(input map[string]int) map[string]int { // input (map[string]int): Input map // replacementMask (string): Mask characters to apply // maskMap (map[string]int): Mask map +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // (map[string]int): Matched masked map -func MakeMatchedMaskedMap(input map[string]int, replacementMask string, maskMap map[string]int) map[string]int { +func MakeMatchedMaskedMap(input map[string]int, replacementMask string, maskMap map[string]int, bypass bool, debug bool) map[string]int { maskedMap := make(map[string]int) replacements := ConstructReplacements(replacementMask) replacer := strings.NewReplacer(replacements...) @@ -287,12 +334,24 @@ func MakeMatchedMaskedMap(input map[string]int, replacementMask string, maskMap newKey = ConvertMultiByteMask(newKey) } - if _, exists := maskMap[newKey]; exists { - if oldValue, exists := maskedMap[newKey]; exists { - maskedMap[key] = oldValue + value - } else { - maskedMap[key] = value + if debug { + fmt.Fprintf(os.Stderr, "[?] MakeMatchedMaskedMap:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "New Key: %s\n", newKey) + fmt.Fprintf(os.Stderr, "Replacement Mask: %s\n", replacementMask) + } + + switch bypass { + case false: + if _, exists := maskMap[newKey]; exists { + if oldValue, exists := maskedMap[newKey]; exists { + maskedMap[key] = oldValue + value + } else { + maskedMap[key] = value + } } + case true: + fmt.Println(newKey) } } return maskedMap @@ -306,11 +365,13 @@ func MakeMatchedMaskedMap(input map[string]int, replacementMask string, maskMap // // input (map[string]int): Input map // replacementMask (string): Mask characters to apply +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): Boundary split map -func BoundarySplitPopMap(input map[string]int, replacementMask string) map[string]int { +func BoundarySplitPopMap(input map[string]int, replacementMask string, bypass bool, debug bool) map[string]int { result := make(map[string]int) for s := range input { token := "" @@ -342,8 +403,20 @@ func BoundarySplitPopMap(input map[string]int, replacementMask string) map[strin } lastRuneType = runeType } + + if debug { + fmt.Fprintf(os.Stderr, "[?] BoundarySplitPopMap:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", s) + fmt.Fprintf(os.Stderr, "Token: %s\n", token) + fmt.Fprintf(os.Stderr, "Replacement Mask: %s\n", replacementMask) + } + if token != "" { - result[token]++ + if !bypass { + result[token]++ + } else { + fmt.Println(token) + } } } return result @@ -359,10 +432,12 @@ func BoundarySplitPopMap(input map[string]int, replacementMask string) map[strin // input (map[string]int): Input map // replacementMask (string): Mask characters to apply // swapMap (map[string]int): Items to swap with +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // (map[string]int): Shuffled map with swapped keys -func ShuffleMap(input map[string]int, replacementMask string, swapMap map[string]int) map[string]int { +func ShuffleMap(input map[string]int, replacementMask string, swapMap map[string]int, bypass bool, debug bool) map[string]int { shuffleMap := make(map[string]int) re := regexp.MustCompile(`^(\?u|\?l|\?d|\?s|\?b)*$`) reParser := regexp.MustCompile("(\\?[luds])") @@ -379,16 +454,45 @@ func ShuffleMap(input map[string]int, replacementMask string, swapMap map[string // Check if the new key is in the swap map for swapKey := range swapMap { - if MakeMaskedString(swapKey, replacementMask) == newKey { - shufKey := strings.Replace(key, newKey, swapKey, 1) + if debug { + fmt.Fprintf(os.Stderr, "[?] ShuffleMap:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "New Key: %s\n", newKey) + fmt.Fprintf(os.Stderr, "Swap Key: %s\n", swapKey) + fmt.Fprintf(os.Stderr, "Replacement Mask: %s\n", replacementMask) + } + + maskedSwapKey := MakeMaskedString(swapKey, replacementMask) + if maskedSwapKey == newKey || fmt.Sprintf("%s%s", maskedSwapKey, maskedSwapKey) == newKey { - if oldValue, exists := shuffleMap[shufKey]; exists { - shuffleMap[shufKey] = oldValue + value + var shufKey string + if fmt.Sprintf("%s%s", maskedSwapKey, maskedSwapKey) == newKey { + shufKey = strings.Replace(key, maskedSwapKey, swapKey, 2) } else { - shuffleMap[shufKey] = value + shufKey = strings.Replace(key, newKey, swapKey, 1) + } + + if debug { + fmt.Fprintf(os.Stderr, "Masked Swap Key: %s\n", maskedSwapKey) + fmt.Fprintf(os.Stderr, "Shuffle Key: %s\n", shufKey) + } + + if shufKey == key { + continue + } + + switch bypass { + case false: + if oldValue, exists := shuffleMap[shufKey]; exists { + shuffleMap[shufKey] = oldValue + value + } else { + shuffleMap[shufKey] = value + } + case true: + fmt.Println(shufKey) } - } + } } } return shuffleMap diff --git a/pkg/mask/mask_test.go b/pkg/mask/mask_test.go index 06861c6..12b63ca 100644 --- a/pkg/mask/mask_test.go +++ b/pkg/mask/mask_test.go @@ -83,7 +83,7 @@ func TestMakeMaskedMap(t *testing.T) { // Run test cases for _, test := range tests { - output := MakeMaskedMap(test.input, test.replacements, false) + output := MakeMaskedMap(test.input, test.replacements, false, false, false) if !reflect.DeepEqual(output, test.output) { t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output) } @@ -113,7 +113,7 @@ func TestMakeRetainMaskedMap(t *testing.T) { // Run test cases for _, test := range tests { - output := MakeRetainMaskedMap(test.input, test.replacements, test.retain) + output := MakeRetainMaskedMap(test.input, test.replacements, test.retain, false, false) if !reflect.DeepEqual(output, test.output) { t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output) } @@ -225,7 +225,7 @@ func TestRemoveMaskedCharacters(t *testing.T) { // Run test cases for _, test := range tests { - output := RemoveMaskedCharacters(test.input) + output := RemoveMaskedCharacters(test.input, false, false) if !reflect.DeepEqual(output, test.output) { t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output) } @@ -254,7 +254,7 @@ func TestMakeMatchedMaskedMap(t *testing.T) { // Run test cases for _, test := range tests { - output := MakeMatchedMaskedMap(test.input, test.replacements, test.masks) + output := MakeMatchedMaskedMap(test.input, test.replacements, test.masks, false, false) if utils.CheckAreMapsEqual(output, test.output) == false { t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output) } @@ -282,7 +282,7 @@ func TestBoundarySplitPopMap(t *testing.T) { // Run test cases for _, test := range tests { - output := BoundarySplitPopMap(test.input, test.replacements) + output := BoundarySplitPopMap(test.input, test.replacements, false, false) if !utils.CheckAreMapsEqual(output, test.output) { t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output) } @@ -305,11 +305,12 @@ func TestShuffleMap(t *testing.T) { // Define test cases tests := testCases{ {map[string]int{"abc?d?d?d": 1, "?u?u?u456": 2, "ABC?l?l?l123!!!": 3}, "luds", map[string]int{"DEF": 1, "321": 1, "zxc": 1}, map[string]int{"ABCzxc123!!!": 3, "DEF456": 2, "abc321": 1}}, + {map[string]int{"?d?d?dabc?d?d?d": 1, "123?l?l?l": 2, "ABC?l?l?l123!!!": 3}, "luds", map[string]int{"DEF": 1, "321": 1, "zxc": 1}, map[string]int{"123zxc": 2, "ABCzxc123!!!": 3, "321abc321": 1}}, } // Run test cases for _, test := range tests { - output := ShuffleMap(test.input, test.replacements, test.swaps) + output := ShuffleMap(test.input, test.replacements, test.swaps, false, false) if !utils.CheckAreMapsEqual(output, test.output) { t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output) } diff --git a/pkg/models/models.go b/pkg/models/models.go index de0352d..9d87410 100644 --- a/pkg/models/models.go +++ b/pkg/models/models.go @@ -52,6 +52,18 @@ func (i *IntRange) Set(value string) error { return nil } +// TemplateFileOperation is used to store the transformation operations loaded +// from JSON template files. The intention is to provide a way to define the +// operations in a structured way. +type TemplateFileOperation struct { + StartIndex int + EndIndex int + Verbose bool + ReplacementMask string + Bypass bool + TransformationMode string +} + // ---------------------------------------------------------------------------- // Output Sorting Models // ---------------------------------------------------------------------------- @@ -91,7 +103,7 @@ type MockFileSystem struct { Files map[string][]byte } -// Implements the ReadFile method of the FileSystem interface for the MockFileSystem +// ReadFile Implements the ReadFile method of the FileSystem interface for the MockFileSystem func (m *MockFileSystem) ReadFile(filename string) ([]byte, error) { if data, ok := m.Files[filename]; ok { return data, nil @@ -120,7 +132,7 @@ type MockScanner struct { Index int } -// Implements the Scan, Text, and Err methods of the Scanner interface for the MockScanner +// Scan implements the Scan, Text, and Err methods of the Scanner interface for the MockScanner func (m *MockScanner) Scan() bool { if m.Index < len(m.Lines) { m.Index++ @@ -129,12 +141,12 @@ func (m *MockScanner) Scan() bool { return false } -// Implements the Text method of the Scanner interface for the MockScanner +// Text implements the Text method of the Scanner interface for the MockScanner func (m *MockScanner) Text() string { return m.Lines[m.Index-1] } -// Implements the Err method of the Scanner interface for the MockScanner +// Err implements the Err method of the Scanner interface for the MockScanner func (m *MockScanner) Err() error { return nil } diff --git a/pkg/rule/rule.go b/pkg/rule/rule.go index d8dbce6..9741d0d 100644 --- a/pkg/rule/rule.go +++ b/pkg/rule/rule.go @@ -170,11 +170,13 @@ func FormatCharToIteratingRuleOutput(index int, strs ...string) (output string) // // items (map[string]int): Items to use in the operation // operation (string): Operation to use in the function +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // returnMap (map[string]int): Map of items to return -func AppendRules(items map[string]int, operation string) (returnMap map[string]int) { +func AppendRules(items map[string]int, operation string, bypass bool, debug bool) (returnMap map[string]int) { returnMap = make(map[string]int) switch operation { // remove will remove characters then append @@ -183,8 +185,19 @@ func AppendRules(items map[string]int, operation string) (returnMap map[string]i rule := CharToRule(key, "$") remove := LenToRule(key, "]") appendRemoveRule := FormatCharToRuleOutput(remove, rule) - if appendRemoveRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] AppendRules (remove):\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "Remove: %s\n", remove) + fmt.Fprintf(os.Stderr, "AppendRemoveRule: %s\n", appendRemoveRule) + } + + if appendRemoveRule != "" && !bypass { returnMap[appendRemoveRule] = value + } else if appendRemoveRule != "" && bypass { + fmt.Println(appendRemoveRule) } } return returnMap @@ -194,8 +207,19 @@ func AppendRules(items map[string]int, operation string) (returnMap map[string]i rule := CharToRule(key, "$") shift := LenToRule(key, "}") appendShiftRule := FormatCharToRuleOutput(shift, rule) - if appendShiftRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] AppendRules (shift):\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "Shift: %s\n", shift) + fmt.Fprintf(os.Stderr, "AppendShiftRule: %s\n", appendShiftRule) + } + + if appendShiftRule != "" && !bypass { returnMap[appendShiftRule] = value + } else if appendShiftRule != "" && bypass { + fmt.Println(appendShiftRule) } } return returnMap @@ -203,8 +227,18 @@ func AppendRules(items map[string]int, operation string) (returnMap map[string]i for key, value := range items { rule := CharToRule(key, "$") appendRule := FormatCharToRuleOutput(rule) - if appendRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] AppendRules:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "AppendRule: %s\n", appendRule) + } + + if appendRule != "" && !bypass { returnMap[appendRule] = value + } else if appendRule != "" && bypass { + fmt.Println(appendRule) } } return returnMap @@ -217,11 +251,13 @@ func AppendRules(items map[string]int, operation string) (returnMap map[string]i // // items (map[string]int): Items to use in the operation // operation (string): Operation to use in the function +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // returnMap (map[string]int): Map of items to return -func PrependRules(items map[string]int, operation string) (returnMap map[string]int) { +func PrependRules(items map[string]int, operation string, bypass bool, debug bool) (returnMap map[string]int) { returnMap = make(map[string]int) switch operation { // remove will remove characters then prepend @@ -230,8 +266,19 @@ func PrependRules(items map[string]int, operation string) (returnMap map[string] rule := CharToRule(utils.ReverseString(key), "^") remove := LenToRule(key, "[") prependRemoveRule := FormatCharToRuleOutput(remove, rule) - if prependRemoveRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] PrependRules (remove):\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "Remove: %s\n", remove) + fmt.Fprintf(os.Stderr, "PrependRemoveRule: %s\n", prependRemoveRule) + } + + if prependRemoveRule != "" && !bypass { returnMap[prependRemoveRule] = value + } else if prependRemoveRule != "" && bypass { + fmt.Println(prependRemoveRule) } } return returnMap @@ -241,8 +288,19 @@ func PrependRules(items map[string]int, operation string) (returnMap map[string] rule := CharToRule(utils.ReverseString(key), "^") shift := LenToRule(key, "{") prependShiftRule := FormatCharToRuleOutput(shift, rule) - if prependShiftRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] PrependRules (shift):\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "Shift: %s\n", shift) + fmt.Fprintf(os.Stderr, "PrependShiftRule: %s\n", prependShiftRule) + } + + if prependShiftRule != "" && !bypass { returnMap[prependShiftRule] = value + } else if prependShiftRule != "" && bypass { + fmt.Println(prependShiftRule) } } return returnMap @@ -250,8 +308,18 @@ func PrependRules(items map[string]int, operation string) (returnMap map[string] for key, value := range items { rule := CharToRule(utils.ReverseString(key), "^") prependRule := FormatCharToRuleOutput(rule) - if prependRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] PrependRules:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "PrependRule: %s\n", prependRule) + } + + if prependRule != "" && !bypass { returnMap[prependRule] = value + } else if prependRule != "" && bypass { + fmt.Println(prependRule) } } return returnMap @@ -265,10 +333,12 @@ func PrependRules(items map[string]int, operation string) (returnMap map[string] // items (map[string]int): Items to use in the operation // index (string): Index to insert at // end (string): Index to end at +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // returnMap (map[string]int): Map of items to return -func InsertRules(items map[string]int, index string, end string) (returnMap map[string]int) { +func InsertRules(items map[string]int, index string, end string, bypass bool, debug bool) (returnMap map[string]int) { returnMap = make(map[string]int) i, err := strconv.Atoi(index) if err != nil { @@ -286,8 +356,18 @@ func InsertRules(items map[string]int, index string, end string) (returnMap map[ for key, value := range items { rule := CharToIteratingRule(key, "i", i) insertRule := FormatCharToIteratingRuleOutput(i, rule) - if insertRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] InsertRules:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "InsertRule: %s\n", insertRule) + } + + if insertRule != "" && !bypass { returnMap[insertRule] = value + } else if insertRule != "" && bypass { + fmt.Println(insertRule) } } i++ @@ -302,11 +382,13 @@ func InsertRules(items map[string]int, index string, end string) (returnMap map[ // items (map[string]int): Items to use in the operation // index (string): Index to overwrite at // end (string): Index to end at +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // returnMap (map[string]int): Map of items to return -func OverwriteRules(items map[string]int, index string, end string) (returnMap map[string]int) { +func OverwriteRules(items map[string]int, index string, end string, bypass bool, debug bool) (returnMap map[string]int) { returnMap = make(map[string]int) i, err := strconv.Atoi(index) if err != nil { @@ -324,8 +406,18 @@ func OverwriteRules(items map[string]int, index string, end string) (returnMap m for key, value := range items { rule := CharToIteratingRule(key, "o", i) overwriteRule := FormatCharToIteratingRuleOutput(i, rule) - if overwriteRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] OverwriteRules:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "OverwriteRule: %s\n", overwriteRule) + } + + if overwriteRule != "" && !bypass { returnMap[overwriteRule] = value + } else if overwriteRule != "" && bypass { + fmt.Println(overwriteRule) } } i++ @@ -340,11 +432,13 @@ func OverwriteRules(items map[string]int, index string, end string) (returnMap m // items (map[string]int): Items to use in the operation // index (string): Index to start at // end (string): Index to end at +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // returnMap (map[string]int): Map of items to return -func ToggleRules(items map[string]int, index string, end string) (returnMap map[string]int) { +func ToggleRules(items map[string]int, index string, end string, bypass bool, debug bool) (returnMap map[string]int) { returnMap = make(map[string]int) i, err := strconv.Atoi(index) if err != nil { @@ -362,8 +456,18 @@ func ToggleRules(items map[string]int, index string, end string) (returnMap map[ for key, value := range items { rule := StringToToggleRule(key, "T", i) toggleRule := FormatCharToIteratingRuleOutput(i, rule) - if toggleRule != "" { + + if debug { + fmt.Fprintf(os.Stderr, "[?] ToggleRules:\n") + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "Rule: %s\n", rule) + fmt.Fprintf(os.Stderr, "ToggleRule: %s\n", toggleRule) + } + + if toggleRule != "" && !bypass { returnMap[toggleRule] = value + } else if toggleRule != "" && bypass { + fmt.Println(toggleRule) } } diff --git a/pkg/rule/rule_test.go b/pkg/rule/rule_test.go index 3c786ed..a5660bd 100644 --- a/pkg/rule/rule_test.go +++ b/pkg/rule/rule_test.go @@ -227,7 +227,7 @@ func TestAppendRules(t *testing.T) { // Run test cases for _, test := range tests { - given := AppendRules(test.items, test.operation) + given := AppendRules(test.items, test.operation, false, false) if !utils.CheckAreMapsEqual(given, test.output) { t.Errorf("Expected %v, but got %v", test.output, given) } @@ -256,7 +256,7 @@ func TestPrependRules(t *testing.T) { // Run test cases for _, test := range tests { - given := PrependRules(test.items, test.operation) + given := PrependRules(test.items, test.operation, false, false) if !utils.CheckAreMapsEqual(given, test.output) { t.Errorf("Expected %v, but got %v", test.output, given) } @@ -286,7 +286,7 @@ func TestInsertRules(t *testing.T) { // Run test cases for _, test := range tests { - given := InsertRules(test.items, test.startIndex, test.endIndex) + given := InsertRules(test.items, test.startIndex, test.endIndex, false, false) if !utils.CheckAreMapsEqual(given, test.output) { t.Errorf("Expected %v, but got %v", test.output, given) } @@ -316,7 +316,7 @@ func TestOverwriteRules(t *testing.T) { // Run test cases for _, test := range tests { - given := OverwriteRules(test.items, test.startIndex, test.endIndex) + given := OverwriteRules(test.items, test.startIndex, test.endIndex, false, false) if !utils.CheckAreMapsEqual(given, test.output) { t.Errorf("Expected %v, but got %v", test.output, given) } @@ -345,7 +345,7 @@ func TestToggleRules(t *testing.T) { // Run test cases for _, test := range tests { - given := ToggleRules(test.items, test.startIndex, test.endIndex) + given := ToggleRules(test.items, test.startIndex, test.endIndex, false, false) if !utils.CheckAreMapsEqual(given, test.output) { t.Errorf("Expected %v, but got %v", test.output, given) } diff --git a/pkg/transform/transform.go b/pkg/transform/transform.go index 5db0fe5..30518b4 100644 --- a/pkg/transform/transform.go +++ b/pkg/transform/transform.go @@ -29,73 +29,99 @@ import ( // replacementMask (string): The mask characters to use for masking operations // transformationFilesMap (map[string]int): A map of transformation files to // use for modes like retain-mask +// bypass (bool): If true, the map is not used for output or filtering +// debug (int): Different debug levels to use for debugging [0-2] // // Returns: // // (map[string]int): A map of transformed values -func TransformationController(input map[string]int, mode string, startingIndex int, endingIndex int, verbose bool, replacementMask string, transformationFilesMap map[string]int) (output map[string]int) { +func TransformationController(input map[string]int, mode string, startingIndex int, endingIndex int, verbose bool, replacementMask string, transformationFilesMap map[string]int, bypass bool, debug int) (output map[string]int) { + + functionDebug := false + if debug > 1 { + functionDebug = true + } + + if debug > 0 { + fmt.Fprintf(os.Stderr, "[?] TransformationController: Starting debug mode:\n") + fmt.Fprintf(os.Stderr, "[?] TransformationController: Running in mode %s\n", mode) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Starting index is %d\n", startingIndex) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Ending index is %d\n", endingIndex) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Replacement mask is %s\n", replacementMask) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Bypass is %t\n", bypass) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Verbose is %t\n", verbose) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Transformation files map is %v\n", transformationFilesMap) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Input map is %v\n", input) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Starting transformation...\n") + } + switch mode { case "append", "append-remove", "append-shift", "a": - output = rule.AppendRules(input, mode) + output = rule.AppendRules(input, mode, bypass, functionDebug) case "prepend", "prepend-remove", "prepend-shift", "ar": - output = rule.PrependRules(input, mode) + output = rule.PrependRules(input, mode, bypass, functionDebug) case "insert", "i": strIndex := fmt.Sprintf("%d", startingIndex) endIndex := fmt.Sprintf("%d", endingIndex) - output = rule.InsertRules(input, strIndex, endIndex) + output = rule.InsertRules(input, strIndex, endIndex, bypass, functionDebug) case "overwrite", "o": strIndex := fmt.Sprintf("%d", startingIndex) endIndex := fmt.Sprintf("%d", endingIndex) - output = rule.OverwriteRules(input, strIndex, endIndex) + output = rule.OverwriteRules(input, strIndex, endIndex, bypass, functionDebug) case "toggle", "t": strIndex := fmt.Sprintf("%d", startingIndex) endIndex := fmt.Sprintf("%d", endingIndex) - output = rule.ToggleRules(input, strIndex, endIndex) + output = rule.ToggleRules(input, strIndex, endIndex, bypass, functionDebug) case "encode", "e": - output = format.EncodeInputMap(input) + output = format.EncodeInputMap(input, bypass, functionDebug) case "decode", "de": - output = format.DecodeInputMap(input) + output = format.DecodeInputMap(input, bypass, functionDebug) case "mask", "partial-mask", "partial", "m": - output = mask.MakeMaskedMap(input, replacementMask, verbose) + output = mask.MakeMaskedMap(input, replacementMask, verbose, bypass, functionDebug) case "dehex", "unhex", "dh": - output = format.DehexMap(input) + output = format.DehexMap(input, bypass, functionDebug) case "hex", "rehex": - output = format.HexEncodeMap(input) + output = format.HexEncodeMap(input, bypass, functionDebug) case "remove", "remove-all", "delete", "delete-all", "rm": - input = mask.MakeMaskedMap(input, replacementMask, false) - output = mask.RemoveMaskedCharacters(input) - case "retain-mask", "retain", "r": + input = mask.MakeMaskedMap(input, replacementMask, false, false, false) + output = mask.RemoveMaskedCharacters(input, bypass, functionDebug) + case "retain-mask", "retain", "r", "mask-retain": if len(transformationFilesMap) == 0 { - fmt.Println("Retain masks require use of one or more -tf flags to specify one or more files") + fmt.Fprintf(os.Stderr, "[!] Retain masks require use of one or more -tf flags to specify one or more files\n") os.Exit(1) } - output = mask.MakeRetainMaskedMap(input, replacementMask, transformationFilesMap) - case "match-mask", "match", "mt": + output = mask.MakeRetainMaskedMap(input, replacementMask, transformationFilesMap, bypass, functionDebug) + case "match-mask", "match", "mt", "mask-match": if len(transformationFilesMap) == 0 { - fmt.Println("Match masks require use of one or more -tf flags to specify one or more files") + fmt.Fprintf(os.Stderr, "[!] Match masks require use of one or more -tf flags to specify one or more files\n") os.Exit(1) } - output = mask.MakeMatchedMaskedMap(input, replacementMask, transformationFilesMap) + output = mask.MakeMatchedMaskedMap(input, replacementMask, transformationFilesMap, bypass, functionDebug) case "swap", "replace", "s": if len(transformationFilesMap) == 0 { - fmt.Println("Swap operations require use of one or more -tf flags to specify one or more files") - fmt.Println("This transformation mode requres a ':' separated list of keys to swap") + fmt.Fprintf(os.Stderr, "[!] Swap operations require use of one or more -tf flags to specify one or more files\n") + fmt.Fprintf(os.Stderr, "[!] This transformation mode requires a ':' separated list of keys to swap\n") os.Exit(1) } - output = ReplaceKeysInMap(input, transformationFilesMap) + output = ReplaceKeysInMap(input, transformationFilesMap, bypass, functionDebug) case "pop", "split", "boundary-split", "boundary-pop", "pop-split", "split-pop", "po": - output = mask.BoundarySplitPopMap(input, replacementMask) + output = mask.BoundarySplitPopMap(input, replacementMask, bypass, functionDebug) case "mask-swap", "shuffle", "shuf", "token-swap", "ms": if len(transformationFilesMap) == 0 { - fmt.Println("Mask-swap operations require use of one or more -tf flags to specify one or more files") - fmt.Println("This transformation mode requres a retain mask file to use for swapping") + fmt.Fprintf(os.Stderr, "[!] Mask-swap operations require use of one or more -tf flags to specify one or more files") + fmt.Fprintf(os.Stderr, "[!] This transformation mode requres a retain mask file to use for swapping") os.Exit(1) } - output = mask.ShuffleMap(input, replacementMask, transformationFilesMap) + output = mask.ShuffleMap(input, replacementMask, transformationFilesMap, bypass, functionDebug) default: output = input } + if debug > 0 { + fmt.Fprintf(os.Stderr, "[?] TransformationController: Output map is %v\n", output) + fmt.Fprintf(os.Stderr, "[?] TransformationController: Transformation complete. Resuming output.\n") + } + return output } @@ -111,16 +137,28 @@ func TransformationController(input map[string]int, mode string, startingIndex i // // originalMap (map[string]int): The original map to replace keys in // replacements (map[string]int): The map of replacements to use +// bypass (bool): If true, the map is not used for output or filtering +// debug (bool): If true, print additional debug information to stderr // // Returns: // // (map[string]int): A new map with the keys replaced -func ReplaceKeysInMap(originalMap map[string]int, replacements map[string]int) map[string]int { +func ReplaceKeysInMap(originalMap map[string]int, replacements map[string]int, bypass bool, debug bool) map[string]int { newMap := make(map[string]int) for key, value := range originalMap { newKeyArray := utils.ReplaceSubstring(key, replacements) for _, newKey := range newKeyArray { - newMap[newKey] = value + + if debug { + fmt.Fprintf(os.Stderr, "Key: %s\n", key) + fmt.Fprintf(os.Stderr, "New Key: %s\n", newKey) + } + + if !bypass { + newMap[newKey] = value + } else { + fmt.Println(newKey) + } } } return newMap diff --git a/pkg/transform/transform_test.go b/pkg/transform/transform_test.go index adc0824..57d0290 100644 --- a/pkg/transform/transform_test.go +++ b/pkg/transform/transform_test.go @@ -39,7 +39,7 @@ func TestReplaceKeysInMap(t *testing.T) { // Run the test cases for _, test := range tests { - result := ReplaceKeysInMap(test.input, test.replace) + result := ReplaceKeysInMap(test.input, test.replace, false, false) if utils.CheckAreMapsEqual(result, test.output) == false { t.Errorf("Test case failed. Expected %v, got %v", test.output, result) } diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go index 1ff6cfd..5047008 100644 --- a/pkg/utils/utils.go +++ b/pkg/utils/utils.go @@ -41,7 +41,8 @@ func ReadFilesToMap(fs models.FileSystem, filenames []string) map[string]int { for _, filename := range filenames { data, err := fs.ReadFile(filename) if err != nil { - panic(err) + fmt.Fprintf(os.Stderr, "[!] Error reading file %s\n", filename) + os.Exit(1) } err = json.Unmarshal(data, &wordMap) @@ -209,7 +210,8 @@ func ProcessURL(url string, ch chan<- string, wg *sync.WaitGroup) { // Read Body body, err := io.ReadAll(resp.Body) if err != nil { - panic(err) + fmt.Fprintf(os.Stderr, "[!] Error reading response body from URL %s\n", url) + os.Exit(1) } text := string(body) text = html.UnescapeString(text) @@ -221,7 +223,8 @@ func ProcessURL(url string, ch chan<- string, wg *sync.WaitGroup) { // Parse the HTML doc, err := html.Parse(strings.NewReader(text)) if err != nil { - panic(err) + fmt.Fprintf(os.Stderr, "[!] Error parsing HTML from URL %s\n", url) + os.Exit(1) } // Traverse the HTML tree and extract the text @@ -273,6 +276,66 @@ func ProcessURL(url string, ch chan<- string, wg *sync.WaitGroup) { } } +// ReadJSONToAray reads the contents of a transformation template file and +// returns a slice of template structs. +// +// Args: +// +// fs (FileSystem): The filesystem to read the file from (used for testing) +// fileArray ([]string): The name of the files to read +// +// Returns: +// +// templates ([]models.TemplateFileOperation): The slice of template structs +func ReadJSONToArray(fs models.FileSystem, filenames []string) []models.TemplateFileOperation { + var templates []models.TemplateFileOperation + for _, filename := range filenames { + data, err := fs.ReadFile(filename) + if err != nil { + fmt.Fprintf(os.Stderr, "[!] Error reading file %s\n", filename) + os.Exit(1) + } + + err = json.Unmarshal(data, &templates) + if err != nil { + fmt.Fprintf(os.Stderr, "[!] Error unmarshalling JSON file %s\n", filename) + os.Exit(1) + } + + alphaRe := regexp.MustCompile(`[a-zA-Z]`) + numRe := regexp.MustCompile(`[0-9]`) + + for _, template := range templates { + if !numRe.MatchString(fmt.Sprintf("%v", template.StartIndex)) || !numRe.MatchString(fmt.Sprintf("%v", template.EndIndex)) { + fmt.Fprintf(os.Stderr, "[!] Error: StartIndex and EndIndex must be integers\n") + os.Exit(1) + } + + if !alphaRe.MatchString(fmt.Sprintf("%v", template.Verbose)) { + fmt.Fprintf(os.Stderr, "[!] Error: Verbose must be a boolean\n") + os.Exit(1) + } + + if !alphaRe.MatchString(fmt.Sprintf("%v", template.ReplacementMask)) { + fmt.Fprintf(os.Stderr, "[!] Error: ReplacementMask must be a string\n") + os.Exit(1) + } + + if !alphaRe.MatchString(fmt.Sprintf("%v", template.Bypass)) { + fmt.Fprintf(os.Stderr, "[!] Error: Bypass must be a boolean\n") + os.Exit(1) + } + + if !alphaRe.MatchString(fmt.Sprintf("%v", template.TransformationMode)) { + fmt.Fprintf(os.Stderr, "[!] Error: TransformationMode must be a string\n") + os.Exit(1) + } + } + } + + return templates +} + // ---------------------------------------------------------------------------- // Transformation Functions // ---------------------------------------------------------------------------- diff --git a/templates/append.json b/templates/append.json new file mode 100644 index 0000000..19298ef --- /dev/null +++ b/templates/append.json @@ -0,0 +1,26 @@ +[ + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "append" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "append-remove" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "append-shift" + } +] diff --git a/templates/inov1-6.json b/templates/inov1-6.json new file mode 100644 index 0000000..89f937d --- /dev/null +++ b/templates/inov1-6.json @@ -0,0 +1,18 @@ +[ + { + "StartIndex": 1, + "EndIndex": 6, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "insert" + }, + { + "StartIndex": 1, + "EndIndex": 6, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "overwrite" + } +] diff --git a/templates/mask.json b/templates/mask.json new file mode 100644 index 0000000..694ed97 --- /dev/null +++ b/templates/mask.json @@ -0,0 +1,26 @@ +[ + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "ul", + "Bypass": false, + "TransformationMode": "mask" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "dsb", + "Bypass": false, + "TransformationMode": "mask" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "mask" + } +] diff --git a/templates/prepend.json b/templates/prepend.json new file mode 100644 index 0000000..d5f628a --- /dev/null +++ b/templates/prepend.json @@ -0,0 +1,26 @@ +[ + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "prepend" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "prepend-remove" + }, + { + "StartIndex": 0, + "EndIndex": 0, + "Verbose": false, + "ReplacementMask": "uldsb", + "Bypass": false, + "TransformationMode": "prepend-shift" + } +]