Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added JSON format support for templates #3333

Merged
merged 6 commits into from
Feb 28, 2023
Merged

Conversation

CodFrm
Copy link
Contributor

@CodFrm CodFrm commented Feb 17, 2023

Proposed changes

fix some json deserialization issues

{
    "id": "self-signed-ssl",
    "info": {
        "name": "Self Signed SSL Certificate",
        "author": "righettod,pdteam",
        "severity": "low",
        "reference": [
            "https://www.rapid7.com/db/vulnerabilities/ssl-self-signed-certificate/"
        ],
        "description": "self-signed certificates are public key certificates that are not issued by a certificate authority. These self-signed\ncertificates are easy to make and do not cost money. However, they do not provide any trust value.\n",
        "remediation": "Purchase or generate a proper SSL certificate for this service.\n",
        "tags": "ssl"
    },
    "ssl": [
        {
            "address": "{{Host}}:{{Port}}",
            "matchers": [
                {
                    "type": "dsl",
                    "dsl": [
                        "self_signed == true"
                    ]
                }
            ]
        }
    ]
}

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Copy link
Member

@tarunKoyalwar tarunKoyalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CodFrm , I fixed a minor bug and added some missing tags along with a unit test .
currently Unit Test Fails because there is a marshal/unmarshal issue with enum types.

Problem/Reason

  • In some enums ex: severity , signature we use iota and iota+1 for initialisation. this causes empty string to be marshalled and other similar issue causes due to inconsistency
$ go test -v ./... -run "TestTemplateStruct"                                                                                                                                                                                                                                     
=== RUN   TestTemplateStruct
    templates_test.go:27: 
                Error Trace:    /Users/tarun/reviews/nuclei/v2/pkg/templates/templates_test.go:27
                Error:          Expected nil, but got: invalid attack type: 
                Test:           TestTemplateStruct
                Messages:       failed to unmarshal json template
--- FAIL: TestTemplateStruct (0.00s)

@tarunKoyalwar
Copy link
Member

Note:
This is a JSON Marshal issue and not Unmarshal.
@CodFrm are you only unmarshalling JSON templates ??

@CodFrm
Copy link
Contributor Author

CodFrm commented Feb 22, 2023

@tarunKoyalwar
I hope that json and yaml can be converted to each other, but it seems to be difficult now.

while parsing json templates is ok

func TestTemplateStruct(t *testing.T) {
	bin := `{
  "id": "valid-gmail-checker",
  "info": {
    "name": "Valid Google Mail Checker",
    "author": "dievus,dwisiswant0",
    "severity": "info",
    "reference": [
      "https://github.com/dievus/geeMailUserFinder"
    ]
  },
  "self-contained": true,
  "requests": [
    {
      "method": "HEAD",
      "path": [
        "https://mail.google.com/mail/gxlu?email={{email}}"
      ],
      "matchers": [
        {
          "type": "word",
          "part": "header",
          "words": [
            "COMPASS"
          ]
        }
      ]
    }
  ]
}`
	var jsonTemplate Template
	err := json.Unmarshal([]byte(bin), &jsonTemplate)
	require.Nil(t, err, "failed to unmarshal json template")
}

@CodFrm
Copy link
Contributor Author

CodFrm commented Feb 27, 2023

Added the ability to judge an empty string when deserializing, and do nothing if it is empty

	s := strings.Trim(string(data), `"`)
	if s == "" {
		return nil
	}

@tarunKoyalwar
Copy link
Member

@CodFrm , nice catch on the empty string deserialization issue , it seems like we almost have full support to JSON templates. currently it does not omit tags we are looking into a custom enum type so that will be fixed in near future.

Copy link
Member

@tarunKoyalwar tarunKoyalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm !

@tarunKoyalwar
Copy link
Member

tarunKoyalwar commented Feb 27, 2023

@CodFrm by the way is there a specific usecase for this json support ? maybe we can add some helper functions in a followup issue/PR ??

@tarunKoyalwar tarunKoyalwar requested review from ehsandeep and Mzack9999 and removed request for ehsandeep February 27, 2023 10:57
@CodFrm
Copy link
Contributor Author

CodFrm commented Feb 28, 2023

@tarunKoyalwar We can implement a Network Catalog to read templates from remote servers and JSON is better for network transportation.

Copy link
Member

@Mzack9999 Mzack9999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - xxxTypeHolder and iota offset will be reworked and simplified in the future

@Mzack9999 Mzack9999 merged commit d80cbef into projectdiscovery:dev Feb 28, 2023
@ehsandeep ehsandeep changed the title fix some json deserialization issues Added JSON format support for templates Mar 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants