Examples of API market transactions in testmis?

[matthewkennedy5]

Hello,

I am a Market Participant in ERCOT and am trying to get set up with using the API to automate market transactions. I was reading through the documentation for the Market Transaction Service here but am still confused on how to get started with using the API. Are there examples showing API requests for making market transactions to testmis?

Thanks!
Matthew

[bbrandaw-ercot]

Matthew,

The first step is to reach our to your Client Services representative at ERCOT. He or she will have all of the information needed to get started. If you are a Java shop we have a sample client that was developed using Springboot. It's pretty old, but the security implementation has not changed so it would still be helpful. You can find that here: https://github.com/ERCOTCP/ews-client.

Hope this helps!

-Brian

[matthewkennedy5]

I have managed to get a MarketInfo request working in Python using Zeep but am struggling to get a MarketTransaction request working. I think I am making a mistake in the structure of my payload. Here is my code:

header = {
    "Verb": "create",
    "Noun": "BidSet",
    "ReplayDetection": {
        "Nonce": generate_nonce(),
        "Created": created_timestamp,
    },
    "Revision": "1",
    "Source": "my_source",
    "UserID": "my_user_id",
}

payload = {
    "Document": {
        "BidSet": {
            "tradingDate": "2007-01-04",
            "EnergyOnlyOffer": [
                {
                    "startTime": "2007-12-17T00:00:00-06:00",
                    "endTime": "2007-12-18T00:00:00-06:00",
                    "externalId": "myExternalID45473",
                    "bidID": "myTestID",
                    "EnergyOfferCurve": [
                        {
                            "startTime": "2007-12-17T00:00:00-06:00",
                            "endTime": "2007-12-18T00:00:00-06:00",
                            "point": [
                                {"mw": 1, "price": 1.00},
                            ],
                        }
                    ],
                }
            ],
        }
    }
}

with client.settings(raw_response=True):
    response = client.service.MarketTransactions(Header=header, Payload=payload)
    print(response)
    print(response.content)

The error message says Error while validating the Market Transactions request received from the Market Participant - No Bids or Product Types found under the BidSet in the request payload. It seems that it's not recognizing the EnergyOnlyOffer in the payload. I've tried different possibilities but none has worked so far. Just wondering if anyone has had success submitting BidSets using python. Thanks!

[tylernisonoff]

@matthewkennedy5 I've been having trouble getting even a MarketInfo request working. Do you have any sample code in terms of configuring the zeep client correctly for the testmis endpoint? I'm assuming I need to do something particular to the Signature.

I'd be happy to help try to get an EnergyOnlyOffer payload working if I can get the MarketInfo one working!

[matthewkennedy5]

I was able to get EnergyOnlyOffer working:
(with some help from this stackoverflow answer: https://stackoverflow.com/questions/73738245/python-xmlsec-verificationerror-signature-is-invalid-error-even-after-receivi/73739097#73739097). I think for MarketInfo you might not need the Payload section.

from datetime import timedelta, datetime, UTC
import random
from requests import Session
from zeep import Client, Settings
from zeep.transports import Transport
import zeep.wsse.utils as utils
from zeep.wsse.signature import BinarySignature
from lxml import etree


session = Session()

# Use openssl to convert your ERCOT .pfx certificate to 2 files: private.pem and public.pem for your private key and public key

# Combine together private.pem and public.pem with "cat private.pem public.pem > combined.pem"
session.cert = f"combined.pem"
transport = Transport(session=session)
settings = Settings(
    strict=False,
    forbid_dtd=False,
    forbid_entities=False,
    forbid_external=False,
    xml_huge_tree=True,
)


class BinarySignatureTimestamp(BinarySignature):
    def apply(self, envelope, headers):
        security = utils.get_security_header(envelope)

        created = datetime.now(UTC)
        expired = created + timedelta(seconds=1 * 60)

        timestamp = utils.WSU("Timestamp")
        timestamp.append(
            utils.WSU("Created", created.replace(microsecond=0).isoformat()[:-6] + "Z")
        )
        timestamp.append(
            utils.WSU("Expires", expired.replace(microsecond=0).isoformat()[:-6] + "Z")
        )

        security.append(timestamp)

        super().apply(envelope, headers)
        return envelope, headers

    def verify(self, envelope):
        return envelope



client = Client(
    # I downloaded the xsd and wsdl files from 
    # https://github.com/ERCOTCP/ews-client/tree/master/ews-client/src/main/schemas
    # This file contains the API URL, determining whether to send the request to testmisapi or misapi
    "Nodal.wsdl",
    transport=transport,
    settings=settings,
    wsse=BinarySignatureTimestamp(
        "private.pem",
        "public.pem",
    ),
)


def generate_nonce(length=15):
    """Generate pseudorandom number."""
    return "".join([str(random.randint(0, 9)) for i in range(length)])

created_timestamp = datetime.now(UTC).replace(microsecond=0).isoformat()[:-6] + "Z"

header = {
    "Verb": "create",
    "Noun": "BidSet",
    "ReplayDetection": {
        "Nonce": generate_nonce(),
        "Created": created_timestamp,
    },
    "Revision": "1",
    "Source": <QSE_NAME>,
    "UserID": <USER_ID>,
}


bid_set = etree.fromstring(
    f"""
<BidSet xmlns="http://www.ercot.com/schema/2007-06/nodal/ews" 
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        xsi:schemaLocation="http://www.ercot.com/schema/2007-06/nodal/ews ErcotTransactions.xsd">
    <tradingDate>2024-07-24</tradingDate>
           <EnergyOnlyOffer>
            <startTime>2024-07-24T00:00:00-05:00</startTime>
            <endTime>2024-07-24T01:00:00-05:00</endTime>
            <marketType>DAM</marketType>
            <expirationTime>2024-07-25T00:00:00-05:00</expirationTime>
            <sp>HB_HUBAVG</sp>
            <bidID>qenef</bidID>
            <EnergyOfferCurve>
                <startTime>2024-07-24T00:00:00-05:00</startTime>
                <endTime>2024-07-24T01:00:00-05:00</endTime>
                <curveStyle>VARIABLE</curveStyle>
                <CurveData>
                    <xvalue>1</xvalue>
                    <y1value>100</y1value>
                </CurveData>
                <multiHourBlock>false</multiHourBlock>
            </EnergyOfferCurve>
        </EnergyOnlyOffer>
</BidSet>
"""
)

payload = {
    "_value1": [bid_set]
}


# Make the request
with client.settings(raw_response=False):
    response = client.service.MarketTransactions(Header=header, Payload=payload)

[nealbgit]

@matthewkennedy5 - In testing your code above, I'm getting this error:
socket.gaierror: [Errno 11001] getaddrinfo failed

Any idea why?

[maxrausch]

how do you specify the API in the WSDL?

[mmanishh]

Hey Guys, Did you figure this out ? We are stuck on same and ERCOT doesn't seem to be helping!

[bbrandaw-ercot]

@mmanishh In looking at it, I think request 1 has an extra tag:

For request 2, it looks like there are some other extra tags:

The Annotated SOAP message in Appendix D of the spec is helpful since its shows the full SOAP message rather than just the RequestMessage section.

Also, I think that your ASOffer structure is invalid:

You're specifying an asType that is On Line non Spin but under the ASPriceCurve section you're using the OffLine Non Spin element. Don't think your validation has reached that point yet so you might not have seen an error for that one.

This is covered in the AS Offers section of the Developer Portal: https://developer.ercot.com/applications/ews/Market%20Transaction%20Messages/Ancillary%20Service%20Offer%20%28ASO%29/

Let me know if this helps.

Thanks,

-Brian

[jamesviotas]

Thanks to @matthewkennedy5, I took your python code and managed to get it working on our (myself and mmannishh's) side in python. One small thing from your code, inside the payload object, we used "_value_1" and that worked for us.

We have a node based implementation (with the "soap" node lib), here's an abridged view what we got working in the end. Thanks to all for the pointers. A more modern quick start guide would definitely be useful for other devs I think.

` const nodalClient = await createNodalClient(path.resolve(__dirname, '../ews/wsdls/Nodal.wsdl'), options);
const soapClient = nodalClient.MarketTransactions;

    const header: Header = {
        Verb: 'create',
        Noun: 'BidSet', 
        ReplayDetection: {
            Nonce: getRandomInt().toString(), 
            Created: new Date().toISOString()
        },
        Revision: '1', 
        Source: qseName,
        UserID: apiUserID, 
        MessageID: getRandomInt().toString(), 
    }

    const payloadObject = {
        BidSet: {
            '@xmlns': 'http://www.ercot.com/schema/2007-06/nodal/ews',
            '@xmlns:xsi': 'http://www.w3.org/2001/XMLSchema-instance',
            '@xsi:schemaLocation': 'http://www.ercot.com/schema/2007-06/nodal/ews ErcotTransactions.xsd',
            tradingDate: '2024-08-19',
            ASOffer: {
                expirationTime: '2024-08-19T00:00:00-05:00',
                resource: resourceName,
                asType: asType,
                ASPriceCurve: {
                    startTime: '2024-08-16T00:00:00-05:00',
                    endTime: '2024-08-16T00:00:00-05:00',
                    OnLineReserves: {
                        xvalue: 1000,
                        REGUP: 6.5,
                        RRSPF: 6.5,
                        RRSFF: 6.5,
                        RRSUF: 6.5,
                        ONNS: 6.5,
                        ECRS: 6.5,
                        block: 'VARIABLE'
                    },
                }
            }
        }
    };

    const xmlDocument = create(payloadObject);
    const xmlString = xmlDocument.end({ headless: 'true' })

    const parser = new DOMParser();
    const xmlDoc = parser.parseFromString(xmlString, 'text/xml');
    const serializer = new XMLSerializer();
    const serializedXML = serializer.serializeToString(xmlDoc);

    const payload = {
        $xml: serializedXML
    };

    const requestMessage = {
        Header: header,
        Payload: payload
    };

    // Using WSSecurityCert
    const privateKey = fs.readFileSync(path.resolve(__dirname, '../certs/private.pem'));
    const publicKey = fs.readFileSync(path.resolve(__dirname, '../certs/public.pem'));
    const password = ''

    const wsSecurityCert = new WSSecurityCert(privateKey, publicKey, password);

    nodalClient.setSecurity(wsSecurityCert);

    const marketTransaction = await nodalClient.MarketTransactionsAsync(requestMessage);`

[nealbgit]

@jamesviotas - care to share a complete sample, with imports, etc.? Would love to get a working example.

[pzemljic-git]

@jamesviotas could you please share your createNodalClient. I keep getting this error with node-soap:

InvalidSecurityToken: SECU1045: Authentication identity is malformed. Reason: Malformed WS-Security binary token: "Input block size is not 4"

[maxrausch]

Is there any way to use zeep to create the payload instead of doing a raw XML for the bidset?

payload = {
"_value1": [bid_set]
}

I tried creating a zeep bidset object but can not figure out how to correctly construct the payload.