Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yaml deserialization to update a key value, results in completely different serialization, without double-quotes #449

Open
DevOpsAzurance opened this issue Nov 7, 2019 · 6 comments
Labels

Comments

@DevOpsAzurance
Copy link

DevOpsAzurance commented Nov 7, 2019

I have the following yaml:

---
swagger: "2.0"
info:
  title: "AccountService"
  description: ""
  x-ibm-name: "accountservice"
  version: "1.0.0"
schemes:
- "https"
basePath: "/AccountService"
produces:
- "application/xml"
consumes:
- "text/xml"
securityDefinitions: {}
security: []
x-ibm-configuration:
  type: "wsdl"
  phase: "realized"
  enforced: true
  testable: true
  gateway: "datapower-gateway"
  cors:
    enabled: true
  wsdl-definition:
    wsdl: "http://somewsdldefinition.wsdl"
    service: "AccountService"
    port: "AccountPort"
    soap-version: "1.1"
  assembly:
    execute:
    - proxy:
        title: "proxy"
        target-url: "http://someurlendpoint"
    catch: []
  categories:
  - "- clientID: []"
paths:
  /getBalance:
    post:
      summary: "Operation getBalance"
      description: ""
      operationId: "getBalance"
      x-ibm-soap:
        soap-action: "getBalance"
        soap-operation: "{http://someotherstuff/}getBalance"
      parameters:
      - in: "body"
        name: "body"
        required: true
        schema:
          $ref: "#/definitions/getBalanceInput"
      responses:
        default:
          description: ""
          schema:
            $ref: "#/definitions/getBalanceOutput"
definitions:
  Security:
    xml:
      namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
      prefix: "wsse"
    description: "Header for WS-Security"
    type: "object"
    properties:
      UsernameToken:
        xml:
          namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
          prefix: "wsse"
        type: "object"
        properties:
          Username:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
              prefix: "wsse"
            type: "string"
          Password:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
              prefix: "wsse"
            type: "string"
          Nonce:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
              prefix: "wsse"
            type: "string"
            properties:
              EncodingType:
                xml:
                  namespace: ""
                  attribute: true
                type: "string"
          Created:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
              prefix: "wsu"
            type: "string"
      Timestamp:
        xml:
          namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
          prefix: "wsu"
        type: "object"
        properties:
          Created:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
              prefix: "wsu"
            type: "string"
          Expires:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
              prefix: "wsu"
            type: "string"
          Id:
            xml:
              namespace: "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
              prefix: "wsu"
              attribute: true
            type: "string"
  getBalanceInput:
    type: "object"
    properties:
      Envelope:
        xml:
          prefix: "soapenv"
          namespace: "http://schemas.xmlsoap.org/soap/envelope/"
        type: "object"
        properties:
          Header:
            $ref: "#/definitions/getBalanceHeader"
          Body:
            type: "object"
            properties:
              getBalance:
                $ref: "#/definitions/getBalance_tns"
            required:
            - "getBalance"
        required:
        - "Body"
    required:
    - "Envelope"
    example: "\n<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"\
      >\n <soapenv:Header>\n  <wsse:Security xmlns:wsse=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\"\
      \ xmlns:wsu=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\"\
      >\n   <wsse:UsernameToken>\n    <wsse:Username>string</wsse:Username>\n    <wsse:Password>string</wsse:Password>\n\
      \    <wsse:Nonce EncodingType=\"string\">string</wsse:Nonce>\n    <wsu:Created>string</wsu:Created>\n\
      \   </wsse:UsernameToken>\n   <wsu:Timestamp wsu:Id=\"string\">\n    <wsu:Created>string</wsu:Created>\n\
      \    <wsu:Expires>string</wsu:Expires>\n   </wsu:Timestamp>\n  </wsse:Security>\n\
      \ </soapenv:Header>\n <soapenv:Body>\n  <tns:getBalance xmlns:tns=\"http://someurl/\"\
      ><!-- mandatory -->\n   <arg0><!-- mandatory -->3</arg0>\n  </tns:getBalance>\n\
      \ </soapenv:Body>\n</soapenv:Envelope>"
  getBalanceHeader:
    type: "object"
    properties:
      Security:
        $ref: "#/definitions/Security"
  getBalanceOutput:
    type: "object"
    properties:
      Envelope:
        xml:
          prefix: "soapenv"
          namespace: "http://schemas.xmlsoap.org/soap/envelope/"
        type: "object"
        properties:
          Body:
            type: "object"
            properties:
              getBalanceResponse:
                $ref: "#/definitions/getBalanceResponse_tns"
        required:
        - "Body"
    required:
    - "Envelope"
    example: "\n<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"\
      >\n <soapenv:Body>\n  <tns:getBalanceResponse xmlns:tns=\"http://someurl/\"\
      >\n   <return><!-- mandatory -->3</return>\n  </tns:getBalanceResponse>\n </soapenv:Body>\n\
      </soapenv:Envelope>"
  getBalance_tns:
    xml:
      namespace: "http://someurl"
      prefix: "tns"
    type: "object"
    properties:
      arg0:
        type: "integer"
        format: "int32"
        xml:
          namespace: ""
    required:
    - "arg0"
    example: "\n<tns:getBalance xmlns:tns=\"http://someurl/\">\n <arg0><!--\
      \ mandatory -->3</arg0>\n</tns:getBalance>"
  getBalanceResponse_tns:
    xml:
      namespace: "http://someurl/"
      prefix: "tns"
    type: "object"
    properties:
      return:
        type: "integer"
        format: "int32"
        xml:
          namespace: ""
    required:
    - "return"
    example: "\n<tns:getBalanceResponse xmlns:tns=\"http://someurl/\"\
      >\n <return><!-- mandatory -->3</return>\n</tns:getBalanceResponse>"

I am able to use yamldotnet to deserialize it:

var d = new YamlDotNet.Serialization.DeserializerBuilder()
.Build();
var yamlObj = d.Deserialize(yamlTxtRdr)

I update the value that needs to be flipped to false and then serialize it back to write to a file, but the serialized output is the following:

swagger: 2.0
info:
  title: AccountService
  description: ''
  x-ibm-name: accountservice
  version: 1.0.0
schemes:
- https
basePath: /AccountService
produces:
- application/xml
consumes:
- text/xml
securityDefinitions: {}
security: []
x-ibm-configuration:
  type: wsdl
  phase: realized
  enforced: true
  testable: false
  gateway: datapower-gateway
  cors:
    enabled: true
  wsdl-definition:
    wsdl: http://banka.mybluemix.net/services/AccountService?wsdl
    service: AccountService
    port: AccountPort
    soap-version: 1.1
  assembly:
    execute:
    - proxy:
        title: proxy
        target-url: http://banka.mybluemix.net/services/AccountService
    catch: []
  categories:
  - '- clientID: []'
paths:
  /getBalance:
    post:
      summary: Operation getBalance
      description: ''
      operationId: getBalance
      x-ibm-soap:
        soap-action: getBalance
        soap-operation: '{http://bankA.sample.ibm.com/}getBalance'
      parameters:
      - in: body
        name: body
        required: true
        schema:
          $ref: '#/definitions/getBalanceInput'
      responses:
        default:
          description: ''
          schema:
            $ref: '#/definitions/getBalanceOutput'
definitions:
  Security:
    xml:
      namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd
      prefix: wsse
    description: Header for WS-Security
    type: object
    properties:
      UsernameToken:
        xml:
          namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd
          prefix: wsse
        type: object
        properties:
          Username:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd
              prefix: wsse
            type: string
          Password:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd
              prefix: wsse
            type: string
          Nonce:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd
              prefix: wsse
            type: string
            properties:
              EncodingType:
                xml:
                  namespace: ''
                  attribute: true
                type: string
          Created:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd
              prefix: wsu
            type: string
      Timestamp:
        xml:
          namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd
          prefix: wsu
        type: object
        properties:
          Created:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd
              prefix: wsu
            type: string
          Expires:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd
              prefix: wsu
            type: string
          Id:
            xml:
              namespace: http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd
              prefix: wsu
              attribute: true
            type: string
  getBalanceInput:
    type: object
    properties:
      Envelope:
        xml:
          prefix: soapenv
          namespace: http://schemas.xmlsoap.org/soap/envelope/
        type: object
        properties:
          Header:
            $ref: '#/definitions/getBalanceHeader'
          Body:
            type: object
            properties:
              getBalance:
                $ref: '#/definitions/getBalance_tns'
            required:
            - getBalance
        required:
        - Body
    required:
    - Envelope
    example: >2-

      <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
       <soapenv:Header>
        <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
         <wsse:UsernameToken>
          <wsse:Username>string</wsse:Username>
          <wsse:Password>string</wsse:Password>
          <wsse:Nonce EncodingType="string">string</wsse:Nonce>
          <wsu:Created>string</wsu:Created>
         </wsse:UsernameToken>
         <wsu:Timestamp wsu:Id="string">
          <wsu:Created>string</wsu:Created>
          <wsu:Expires>string</wsu:Expires>
         </wsu:Timestamp>
        </wsse:Security>
       </soapenv:Header>
       <soapenv:Body>
        <tns:getBalance xmlns:tns="http://bankA.sample.ibm.com/"><!-- mandatory -->
         <arg0><!-- mandatory -->3</arg0>
        </tns:getBalance>
       </soapenv:Body>
      </soapenv:Envelope>
  getBalanceHeader:
    type: object
    properties:
      Security:
        $ref: '#/definitions/Security'
  getBalanceOutput:
    type: object
    properties:
      Envelope:
        xml:
          prefix: soapenv
          namespace: http://schemas.xmlsoap.org/soap/envelope/
        type: object
        properties:
          Body:
            type: object
            properties:
              getBalanceResponse:
                $ref: '#/definitions/getBalanceResponse_tns'
        required:
        - Body
    required:
    - Envelope
    example: >2-

      <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
       <soapenv:Body>
        <tns:getBalanceResponse xmlns:tns="http://bankA.sample.ibm.com/">
         <return><!-- mandatory -->3</return>
        </tns:getBalanceResponse>
       </soapenv:Body>
      </soapenv:Envelope>
  getBalance_tns:
    xml:
      namespace: http://bankA.sample.ibm.com/
      prefix: tns
    type: object
    properties:
      arg0:
        type: integer
        format: int32
        xml:
          namespace: ''
    required:
    - arg0
    example: >2-

      <tns:getBalance xmlns:tns="http://bankA.sample.ibm.com/">
       <arg0><!-- mandatory -->3</arg0>
      </tns:getBalance>
  getBalanceResponse_tns:
    xml:
      namespace: http://bankA.sample.ibm.com/
      prefix: tns
    type: object
    properties:
      return:
        type: integer
        format: int32
        xml:
          namespace: ''
    required:
    - return
    example: >2-

      <tns:getBalanceResponse xmlns:tns="http://bankA.sample.ibm.com/">
       <return><!-- mandatory -->3</return>
      </tns:getBalanceResponse>

It is adding a ">2-" to certain lines, eg fifth to last line and all non-boolean values lost the double-quotes. The product ingesting the yaml file requires the double quotes as part of the string value it reads. Having trouble using the SerializerBuilder to account for this and I have no clue why I am getting that ">2-". When I outout the value for that field though there is a new-line before the value, so wondering if it is some kind of new-line character. It has no effect on the future deserialization of the yaml, but am concerned it will have issues down the line with the intake of the yaml by the receiving application.

@aaubry
Copy link
Owner

aaubry commented Nov 23, 2019

If you could properly enclose your code blocks between code fences (three backtick characters), it would be easier to read your question.
Regarding the actual question, the YAML specification specifies different styles for scalars, and the serializer attempts to select the best one for each case. When quotes are removed, it is because they were not needed in the first place. The >2- in some strings is a block scalar header. It indicates how to interpret that string. The > denotes a "folded" scalar, which means that long lines have been broken into smaller ones to fit into the preferred text width. The 2 tells how many spaces are used to indent the content, when that cannot be inferred from the first line. The - indicates that the last newline of the scalar should be ignored.

@aaubry
Copy link
Owner

aaubry commented Nov 23, 2019

Please refer to #428 if you want to force quotes on strings.

@samsmithnz
Copy link

I'm also seeing the >2- in some strings, as well as >+ and >-. For me, it tends to happen when I'm trying to embed powershell in YAML, (GitHub Actions), for example:

- name: Run Selenium smoke tests on website
  run: |
    Write-Host "Line 1"
    Write-Host "Line 2"
  shell: powershell

will produce:

- name: Run Selenium smoke tests on website
  run: >2-|
    Write-Host "Line 1"
    Write-Host "Line 2"
  shell: powershell

@DevOpsAzurance
Copy link
Author

Please refer to #428 if you want to force quotes on strings.

Would this ForceQuotedStringValuesEventEmitter also work in deserializing a swagger yaml and preserving the quotes to later serialize it again?

@aaubry
Copy link
Owner

aaubry commented Dec 8, 2019

@samsmithnz the >2- indicator is technically correct, although unnecessary in this case. The logic that is deciding on which indicators are needed comes from the libyaml port and I haven't really looked into it. It is probably time to do so.

@jobzombi ForceQuotedStringValuesEventEmitter does not preserve the quotes. Instead it forces them. Therefore it should work regardless of the source of the data.

@aaubry aaubry closed this as completed Dec 8, 2019
@aaubry aaubry reopened this Dec 8, 2019
@aaubry
Copy link
Owner

aaubry commented Dec 8, 2019

Sorry I didn't mean to close this issue, but the GitHub UI on mobile makes it really easy to click by mistake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants