Transaction Signing Issue with Custom Variant Definition

[JPig]

When signing and pushing a transaction with a custom defined VariantDefinition, the node often rejects the transaction with the following message ( 2 examples )

Internal Service Error: Provided keys, permissions, and delays do not satisfy declared authorizations: transaction declares authority '{"actor":"redacted","permission":"active"}', but does not have signatures for it under a provided delay of 0 ms, provided permissions [], provided keys ["EOS7BLMKJW2458NoCKL9FMGSHHYb8qhDvJPVExCSUphyrwuFpD8FF"], and a delay max limit of 3888000000 ms

Internal Service Error: Provided keys, permissions, and delays do not satisfy declared authorizations: transaction declares authority '{"actor":"redacted","permission":"active"}', but does not have signatures for it under a provided delay of 0 ms, provided permissions [], provided keys ["EOS8i16Gok8G15DBUbz3r5gphp5vQpJoGMU5pqXPH54sHJWppZtg3"], and a delay max limit of 3888000000 ms

It appears as if the signature of the transaction is either wrong or different 9 times out of 10, but sometimes it isn't and the transaction will go through fine. the action being sent is an atomicassets mintasset action.

Notes:

• One thing i have observed is the more AtomicAttribute's in the AttributeMap the more likely it is to fail with the error.
• The chain id is correct as it fills tx options from the chain
• The correct private key is provided, as about 1 in 10 ( ish ) transactions actually make it through to the chain successfully
• The other thing to note is that the provided keys it logs out changes every time it produces the above error.
• This sympton doesn't occur when using "standard" actions such as an eosio.transfer action etc.

Relevant code for custom variant definition;

var AtomicAttributeVariant = eos.NewVariantDefinition([]eos.VariantType{
	{Name: "int8", Type: int8(0)},
	{Name: "int16", Type: int16(0)},
	{Name: "int32", Type: int32(0)},
	{Name: "int64", Type: int64(0)},
	{Name: "uint8", Type: uint8(0)},
	{Name: "uint16", Type: uint16(0)},
	{Name: "uint32", Type: uint32(0)},
	{Name: "uint64", Type: uint64(0)},
	{Name: "float", Type: float32(0)},
	{Name: "double", Type: float64(0)},
	{Name: "string", Type: ""},
	{Name: "INT8_VEC", Type: []int8{}},
	{Name: "INT16_VEC", Type: []int16{}},
	{Name: "INT32_VEC", Type: []int32{}},
	{Name: "INT64_VEC", Type: []int64{}},
	{Name: "UINT8_VEC", Type: []uint8{}},
	{Name: "UINT16_VEC", Type: []uint16{}},
	{Name: "UINT32_VEC", Type: []uint32{}},
	{Name: "UINT64_VEC", Type: []uint64{}},
	{Name: "FLOAT_VEC", Type: []float32{}},
	{Name: "DOUBLE_VEC", Type: []float64{}},
	{Name: "STRING_VEC", Type: []string{}},
})

type AttributeMap map[string]AtomicAttribute

type AtomicAttribute struct {
	eos.BaseVariant
}

func (a *AtomicAttribute) MarshalJSON() ([]byte, error) {
	return a.BaseVariant.MarshalJSON(AtomicAttributeVariant)
}

func (a *AtomicAttribute) UnmarshalJSON(data []byte) error {
	return a.BaseVariant.UnmarshalJSON(data, AtomicAttributeVariant)
}

func (a *AtomicAttribute) UnmarshalBinary(decoder *eos.Decoder) error {
	return a.BaseVariant.UnmarshalBinaryVariant(decoder, AtomicAttributeVariant)
}

func NewAtomicAttribute(typeId string, value interface{}) AtomicAttribute {
	return AtomicAttribute{
		BaseVariant: eos.BaseVariant{
			TypeID: AtomicAttributeVariant.TypeID(typeId),
			Impl:   value,
		},
	}
}

type mintAssetActionData struct {
	AuthorizedMinter eos.AccountName `json:"authorized_minter"`
	CollectionName   eos.Name        `json:"collection_name"`
	SchemaName       eos.Name        `json:"schema_name"`
	TemplateID       int32           `json:"template_id"`
	NewAssetOwner    eos.AccountName `json:"new_asset_owner"`
	ImmutableData    AttributeMap    `json:"immutable_data"`
	MutableData      AttributeMap    `json:"mutable_data"`
	TokensToBack     []eos.Asset     `json:"tokens_to_back"`
}

immutableData := AttributeMap{
      "name":         NewAtomicAttribute("string","Abc123"), 
}

mintActionData := eos.NewActionData(&mintAssetActionData{
        AuthorizedMinter: eos.AccountName("<redacted>"),
        CollectionName:   eos.Name("<redacted>"),
        SchemaName:       eos.Name("<redacted>"),
        TemplateID:       <redacted>,
        NewAssetOwner:    eos.AccountName("<redacted>"),
        ImmutableData:    immutableData,
        MutableData:      AttributeMap{},
        TokensToBack:     []eos.Asset{},
})

mintAction := &eos.Action{
        Account: eos.AccountName("atomicassets"),
        Name:    eos.ActionName("mintasset"),
        Authorization: []eos.PermissionLevel{
	        {
		        Actor:      eos.AccountName("redacted"),
		        Permission: eos.PermissionName("active"),
	        },
        },
        ActionData: mintActionData,
}

[maoueh]

Oufff that's a hard one. Would would be able to log the content of the SignedTransaction before the broadcast maybe? Do you think you could reproduce on a local chain by pumping transaction to it?

Seems weird that it's caused by using variant thing, specially if I recall right, signature verification is done on the payload we send to the server, so even if the encoding was incorrect, I think it would not lead to signature problem (although it's been a long time since I've work on those).

Having the full SignedTransaction will be required to investigate further I think. Running with DEBUG=true might also help identify some potential other problem.

While I don't think it's going to change anything to the problem, ensure your eos.NewVariantDefinition strictly respects order defined in the ABI.

[JPig]

Yeah the custom variant ordering follows exactly that of the one on-chain in the ABI.

Transaction with Error

Signed transaction:

{"expiration":"2022-11-30T00:05:58","ref_block_num":7429,"ref_block_prefix":1866304876,"max_net_usage_words":0,"max_cpu_usage_ms":0,"delay_sec":0,"context_free_actions":[],"actions":[{"account":"atomicassets","name":"mintasset","authorization":[{"actor":"testvariants","permission":"active"}],"data":"80f334ee9a9db1ca80f334ee9a9db1ca000000804d97b1cab0ba080080f334ee9a9db1ca06046e616d650a0b54657374696e672031323304757569640a2465343438643130372d616534312d346634342d613165622d38376130633162323739373204747970650a0548656c6c6f096174747269627574650a04436f6f6c0a6964656e7469666965720a1831413430363033303030303637384130303030313333373209736f6d657468696e670a0f54657374696e672054657374696e670000"}],"transaction_extensions":[],"signatures":["SIG_K1_K8sSc1BZzLBfa5T5WWeCFkQN4Di2ywSKCUhcjgXANJtNgD94hqUg1mxkyJDASe18Yi8eVgs1QMMaSPezqsdiH7GGoLu5DE"],"context_free_data":[]}

Push Transaction Payload:

{"signatures":["SIG_K1_K8sSc1BZzLBfa5T5WWeCFkQN4Di2ywSKCUhcjgXANJtNgD94hqUg1mxkyJDASe18Yi8eVgs1QMMaSPezqsdiH7GGoLu5DE"],"compression":"none","packed_context_free_data":"","packed_trx":"669e8663051d6c8d3d6f000000000180b3c2d8202769360000c80a6393a7930180f334ee9a9db1ca00000000a8ed3232c00180f334ee9a9db1ca80f334ee9a9db1ca000000804d97b1cab0ba080080f334ee9a9db1ca06046e616d650a0b54657374696e672031323304757569640a2465343438643130372d616534312d346634342d613165622d38376130633162323739373204747970650a0548656c6c6f096174747269627574650a04436f6f6c0a6964656e7469666965720a1831413430363033303030303637384130303030313333373209736f6d657468696e670a0f54657374696e672054657374696e67000000"}

---

Successful Transaction

• exact same data values as the failed transaction above

Signed Transaction:

{"expiration":"2022-11-30T00:08:58","ref_block_num":7789,"ref_block_prefix":1670264858,"max_net_usage_words":0,"max_cpu_usage_ms":0,"delay_sec":0,"context_free_actions":[],"actions":[{"account":"atomicassets","name":"mintasset","authorization":[{"actor":"testvariants","permission":"active"}],"data":"80f334ee9a9db1ca80f334ee9a9db1ca000000804d97b1cab0ba080080f334ee9a9db1ca06096174747269627574650a04436f6f6c0a6964656e7469666965720a1831413430363033303030303637384130303030313333373209736f6d657468696e670a0f54657374696e672054657374696e67046e616d650a0b54657374696e672031323304757569640a2465343438643130372d616534312d346634342d613165622d38376130633162323739373204747970650a0548656c6c6f0000"}],"transaction_extensions":[],"signatures":["SIG_K1_KXqWCeeEJrFCHdCWiYFfcRHmT2CCMmSU9FozpQcmznfXGojJRuXgGkAGQhr7Kiy8MsE91X64E9jxzh5ifjvTb5ypxCZoB4"],"context_free_data":[]}

Push Transaction Payload:

{"signatures":["SIG_K1_KXqWCeeEJrFCHdCWiYFfcRHmT2CCMmSU9FozpQcmznfXGojJRuXgGkAGQhr7Kiy8MsE91X64E9jxzh5ifjvTb5ypxCZoB4"],"compression":"none","packed_context_free_data":"","packed_trx":"1a9f86636d1e1a388e63000000000180b3c2d8202769360000c80a6393a7930180f334ee9a9db1ca00000000a8ed3232c00180f334ee9a9db1ca80f334ee9a9db1ca000000804d97b1cab0ba080080f334ee9a9db1ca0604747970650a0548656c6c6f096174747269627574650a04436f6f6c0a6964656e7469666965720a1831413430363033303030303637384130303030313333373209736f6d657468696e670a0f54657374696e672054657374696e67046e616d650a0b54657374696e672031323304757569640a2465343438643130372d616534312d346634342d613165622d383761306331623237393732000000"}

---

Data used for mint asset

immutableData := AttributeMap{
      "name":       NewAtomicAttribute("string", "Testing 123"),
      "uuid":       NewAtomicAttribute("string", "e448d107-ae41-4f44-a1eb-87a0c1b27972"),
      "type":       NewAtomicAttribute("string", "Hello"),
      "attribute":  NewAtomicAttribute("string", "Cool"),
      "identifier": NewAtomicAttribute("string", "1A406030000678A000013372"),
      "something":  NewAtomicAttribute("string", "Testing Testing"),
}

Thanks for your help/assistance! 😃

[maoueh]

Can you provide also the EOS account that generated the signature.

[maoueh]

Can you also describe what is your set up for transaction signing.

[JPig]

Yeah so here is a minimal repro i made that produces the error described, only thing missing from it is the private key.

package main

import (
	"context"
	"encoding/json"
	"fmt"

	"github.com/eoscanada/eos-go"
)

var AtomicAttributeVariant = eos.NewVariantDefinition([]eos.VariantType{
	{Name: "int8", Type: int8(0)},
	{Name: "int16", Type: int16(0)},
	{Name: "int32", Type: int32(0)},
	{Name: "int64", Type: int64(0)},
	{Name: "uint8", Type: uint8(0)},
	{Name: "uint16", Type: uint16(0)},
	{Name: "uint32", Type: uint32(0)},
	{Name: "uint64", Type: uint64(0)},
	{Name: "float", Type: float32(0)},
	{Name: "double", Type: float64(0)},
	{Name: "string", Type: ""},
	{Name: "INT8_VEC", Type: []int8{}},
	{Name: "INT16_VEC", Type: []int16{}},
	{Name: "INT32_VEC", Type: []int32{}},
	{Name: "INT64_VEC", Type: []int64{}},
	{Name: "UINT8_VEC", Type: []uint8{}},
	{Name: "UINT16_VEC", Type: []uint16{}},
	{Name: "UINT32_VEC", Type: []uint32{}},
	{Name: "UINT64_VEC", Type: []uint64{}},
	{Name: "FLOAT_VEC", Type: []float32{}},
	{Name: "DOUBLE_VEC", Type: []float64{}},
	{Name: "STRING_VEC", Type: []string{}},
})

type AttributeMap map[string]AtomicAttribute

type AtomicAttribute struct {
	eos.BaseVariant
}

func (a *AtomicAttribute) MarshalJSON() ([]byte, error) {
	return a.BaseVariant.MarshalJSON(AtomicAttributeVariant)
}

func (a *AtomicAttribute) UnmarshalJSON(data []byte) error {
	return a.BaseVariant.UnmarshalJSON(data, AtomicAttributeVariant)
}

func (a *AtomicAttribute) UnmarshalBinary(decoder *eos.Decoder) error {
	return a.BaseVariant.UnmarshalBinaryVariant(decoder, AtomicAttributeVariant)
}

func NewAtomicAttribute(typeId string, value interface{}) AtomicAttribute {
	return AtomicAttribute{
		BaseVariant: eos.BaseVariant{
			TypeID: AtomicAttributeVariant.TypeID(typeId),
			Impl:   value,
		},
	}
}

type mintAssetActionData struct {
	AuthorizedMinter eos.AccountName `json:"authorized_minter"`
	CollectionName   eos.Name        `json:"collection_name"`
	SchemaName       eos.Name        `json:"schema_name"`
	TemplateID       int32           `json:"template_id"`
	NewAssetOwner    eos.AccountName `json:"new_asset_owner"`
	ImmutableData    AttributeMap    `json:"immutable_data"`
	MutableData      AttributeMap    `json:"mutable_data"`
	TokensToBack     []eos.Asset     `json:"tokens_to_back"`
}

var privateKey = ""
var nodeURL = "http://api.waxtest.alohaeos.com"

var templateID int32 = 572080

var minter eos.AccountName = "testvariants"
var wallet eos.AccountName = "testvariants"

var collectionName = eos.Name("testvariants")
var schemaName = eos.Name("testing")

func main() {
	api := eos.New(nodeURL)

	keyBag := &eos.KeyBag{}

	err := keyBag.ImportPrivateKey(context.Background(), privateKey)
	if err != nil {
		panic(fmt.Errorf("error importing private key: %w", err))
	}

	api.SetSigner(keyBag)

	api.Debug = true

	minter := eos.AccountName(minter)

	immutableData := AttributeMap{
		"name":       NewAtomicAttribute("string", "Testing 123"),
		"uuid":       NewAtomicAttribute("string", "e448d107-ae41-4f44-a1eb-87a0c1b27972"),
		"type":       NewAtomicAttribute("string", "Hello"),
		"attribute":  NewAtomicAttribute("string", "Cool"),
		"identifier": NewAtomicAttribute("string", "1A406030000678A000013372"),
		"something":  NewAtomicAttribute("string", "Testing Testing"),
	}

	mintActionData := eos.NewActionData(&mintAssetActionData{
		AuthorizedMinter: minter,
		CollectionName:   collectionName,
		SchemaName:       schemaName,
		TemplateID:       templateID,
		NewAssetOwner:    wallet,
		ImmutableData:    immutableData,
		MutableData:      AttributeMap{},
		TokensToBack:     []eos.Asset{},
	})

	mintAction := &eos.Action{
		Account: eos.AccountName("atomicassets"),
		Name:    eos.ActionName("mintasset"),
		Authorization: []eos.PermissionLevel{
			{
				Actor:      minter,
				Permission: eos.PermissionName("active"),
			},
		},
		ActionData: mintActionData,
	}

	actions := []*eos.Action{
		mintAction,
	}

	txOpts := &eos.TxOptions{}
	if err := txOpts.FillFromChain(context.Background(), api); err != nil {
		panic(fmt.Errorf("error filling tx opts: %w", err))
	}

	tx := eos.NewTransaction(actions, txOpts)

	signedTx, packedTx, err := api.SignTransaction(context.Background(), tx, txOpts.ChainID, eos.CompressionNone)
	if err != nil {
		panic(fmt.Errorf("error signing transaction: %w", err))
	}

	fmt.Println(signedTx)

	content, err := json.MarshalIndent(signedTx, "", "  ")
	if err != nil {
		panic(fmt.Errorf("json marshalling transaction: %w", err))
	}

	fmt.Println(string(content))
	fmt.Println()

	response, err := api.PushTransaction(context.Background(), packedTx)
	if err != nil {
		panic(fmt.Errorf("error pushing transaction: %w", err))
	}

	fmt.Println(response.TransactionID)
}

[maoueh]

Ok, I investigated and the problem comes from the:

type AttributeMap map[string]AtomicAttribute

Definition. The actual ABI defines those type as pair_string_attribute[] so ultimately it looks like [["key1", <value1>], ["key2", <value2>], ...].

The encoder was able to serialize map[string]any to the right format, however iterating through map has no guaranteed order. So, the serialization to packed data was done in one order, but the JSON marshalling sending the actual transaction data to the server was done in lexicographical order.

I've just add sorting of keys to lexicographical order also now when doing eos.MarshalBinary(tx) for map[K]V fields.

Please try the latest develop branch commit and report.

[JPig]

Ah awesome yeah that fixed it, works perfectly now every time.

[maoueh]

Note that if you need to specify the order in which record should be provided, AttributeMap should be turned into an array to control the order. Does not seems important for AtomicAssets cases where I imagine the attributes have no requirements on ordering.

Closing.