Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic SSA Validation #27

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7acf07c
Update Validate tests to assignment oriented form
umar456 Nov 18, 2015
d6aaa17
Fix isValid for functions with multiple parameters
umar456 Nov 22, 2015
8033cdb
Basic SSA validation.
umar456 Nov 23, 2015
da35d28
Update style to follow the Google C++ style guide
umar456 Nov 24, 2015
b0a5367
Remove color-diagnostics flag.
umar456 Nov 25, 2015
3949bf4
Bugfixes to SSA validation. Additional Tests
umar456 Nov 25, 2015
70b6cf0
Return SPV_ERROR_INVALID_ID for ID related errors
umar456 Nov 26, 2015
3017901
Tests for OpMemberDecorate; Minimize test cases
umar456 Nov 26, 2015
73bfa70
Simplify SSAPass Logic
umar456 Nov 26, 2015
9691d3c
Fix OpGroupDecorate SSAPass. Additional Unit Tests
umar456 Nov 27, 2015
fc71d20
Remove templates for forward declaration function
umar456 Nov 30, 2015
9f8db87
Documentation; Minor formatting fixes
umar456 Nov 30, 2015
92d6b47
Renamed validate file;Extracted common TestFixture
umar456 Nov 30, 2015
ad5add6
Additional unit tests for Device Enqueue ops
umar456 Dec 2, 2015
0fbc35c
Formatting; Style; Documentation; Unit Tests
umar456 Dec 3, 2015
2b61dbd
Update Fixtures to return results;Update unit test
umar456 Dec 4, 2015
be947ef
Fixed OpConstant order in SSA unit tests
umar456 Dec 4, 2015
89c4a21
Remove regex to avoid problems with gcc.
umar456 Dec 6, 2015
ee4c90a
Add OpName support in diag message and unit tests
umar456 Dec 7, 2015
71b49c4
Fix missing label in func;Phi tests;LoopMerge Fix
umar456 Dec 8, 2015
64190ec
Addressed feedback
umar456 Dec 9, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 158 additions & 20 deletions source/validate.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@
#include <string.h>

#include <vector>
#include <iterator>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sort these. Not a big deal, we can come back to this. The rest of the code is not fully compliant either.
For reference: https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now clang-format can do this automatically for you, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28

#include <set>

using std::vector;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The using declarations should come after all #includes. Otherwise you might mess up the type mappings seen by those other includes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28


#include "binary.h"
#include "diagnostic.h"
Expand Down Expand Up @@ -80,6 +84,7 @@ spv_result_t spvValidateOperandValue(const spv_operand_type_t type,
spv_diagnostic* pDiagnostic) {
switch (type) {
case SPV_OPERAND_TYPE_ID:
case SPV_OPERAND_TYPE_TYPE_ID:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's also: SPV_OPERAND_TYPE_MEMORY_SEMANTICS_ID, SPV_OPERAND_TYPE_SCOPE_ID.

Since you started working on this, I've reorganized the enumerants in spv_operand_type_t so these groupings are easier to see.

case SPV_OPERAND_TYPE_RESULT_ID: {
// NOTE: ID's are validated in SPV_VALIDATION_LEVEL_1, this is
// SPV_VALIDATION_LEVEL_0
Expand Down Expand Up @@ -166,14 +171,15 @@ spv_result_t spvValidateBasic(const spv_instruction_t* pInsts,
// of the parse.
spv_operand_type_t type = spvBinaryOperandInfo(
word, index, opcodeEntry, operandTable, &operandEntry);

if (SPV_OPERAND_TYPE_LITERAL_STRING == type) {
spvCheckReturn(spvValidateOperandsString(
words + index, wordCount - index, position, pDiagnostic));
// NOTE: String literals are always at the end of Opcodes
break;
} else if (SPV_OPERAND_TYPE_LITERAL_INTEGER == type) {
spvCheckReturn(spvValidateOperandsLiteral(
words + index, wordCount - index, 2, position, pDiagnostic));
// spvCheckReturn(spvValidateOperandsNumber(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you plan on re-enabling this code, then use:
#if 0
#endif
Around disabled code, and add a TODO(): to re-enable at some point

Otherwise just delete the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28

// words + index, wordCount - index, 2, position, pDiagnostic));
} else {
spvCheckReturn(spvValidateOperandValue(type, word, operandTable,
position, pDiagnostic));
Expand Down Expand Up @@ -261,6 +267,143 @@ spv_result_t spvValidateIDs(const spv_instruction_t* pInsts,
return SPV_SUCCESS;
}

// TODO(umar): Validate header
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A subtle thing: the Id bound should be validated also. But you can only do that after you've seen all the instructions in the module.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI. The binary parser validates the magic word, and the length of the header, but nothing else.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added notes da35d28

spv_result_t setHeader(void* user_data, spv_endianness_t endian, uint32_t magic,
uint32_t version, uint32_t generator, uint32_t id_bound,
uint32_t reserved) {
(void)user_data;
(void)endian;
(void)version;
(void)generator;
(void)id_bound;
(void)reserved;
return SPV_SUCCESS;
}

// TODO(umar): Move this class to another file
class validation_state_t {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class names in CamelCase

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28

const uint32_t* binary;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Names of class data members should end in "_".
Declarations in a struct/class should be public:, then protected:, then private:.
So these go at the end and are binary_, diagnostic_, and program_counter_.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the "binary" member is only used by the binary() method, which itself is never used. Remove it, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a relic which was based on an older version of spy_parsed_instruction_t struct which did not have the words pointer.

Fixed da35d28

spv_diagnostic* diagnostic;
const uint32_t* program_counter;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the program counter? What's it counting? I presume it's referencing someone else's storage?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From usage, seems this should be instruction_words_
?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a reference to the program counter register in a CPU. Its probably confusing in this context. I can change it to instruction_ptr_ or instruction_word_


std::set<uint32_t> declared_ids;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer std::unordered_set. Regular sets are usually implemented as trees of nodes which use up a lot of space and are slow since there's so much pointer chasing.
(Just use the default hash function.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, these are class data members, so they should go at the end.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28

std::set<uint32_t> forward_ids;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From usage, this isn't really forward_ids. It's unresolved_forward_ids. That is, it's the set of Ids for which we've seen a forward declaration but no definition. I think the name needs to be fixed, and we need a comment to describe it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Fixed da35d28


public:
validation_state_t(const uint32_t* binary, spv_diagnostic* diag)
: binary(binary), diagnostic(diag), program_counter(binary) {}

void set_instruction_words(const uint32_t * const words) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* alignment. Make sure to run clang-format on the file. :)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much as I dislike camelcase (and I really do), convention is to use camelcase for method names.
Style guide says method names start with a captial. The rest of the SPIR-V Tools hasn't caught up to that yet.

program_counter = words;
}

spv_result_t declare_id(uint32_t id) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All methods should have a description before their declaration. E.g. for this one
// Records the given Id as having been declared.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, aren't these definitions, not declarations?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Fixed da35d28

if (declared_ids.find(id) == std::end(declared_ids)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd usually write declared_ids.end(). Seems more direct. ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

declared_ids.insert(id);
} else {
return diag(SPV_ERROR_INVALID_ID)
<< "ID cannot be assigned multiple times";
}
return SPV_SUCCESS;
}

spv_result_t forward_declare_id(uint32_t id) {
forward_ids.insert(id);
return SPV_SUCCESS;
}

spv_result_t remove_if_fwd_declared(uint32_t id) {
forward_ids.erase(id);
return SPV_SUCCESS;
}

size_t forward_declared_id_count() { return forward_ids.size(); }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const function.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unresolved_forward_id_count(), or num_unresolved_forward_ids()


bool check_declared_id(uint32_t id) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const function.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"check" is ambiguous. Prefer "isDefinedId"

return declared_ids.find(id) != std::end(declared_ids);
}

const uint32_t* get_binary() const { return binary; }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used? Remove it, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28


const uint32_t* get_instruction_ptr() { return program_counter; }

libspirv::DiagnosticStream diag(spv_result_t error_code) {
return libspirv::DiagnosticStream({0, 0, (uint64_t)(program_counter - binary)},
diagnostic, error_code);
}
};

spv_result_t SSAPass(validation_state_t& _, bool can_have_forward_declared_ids,
const spv_parsed_instruction_t* inst) {
const uint32_t* inst_ptr = _.get_instruction_ptr();

for (int i = 0; i < inst->num_operands; i++) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and by extension, i should probably be unsigned.

spv_parsed_operand_t operand = inst->operands[i];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const variables.

spv_operand_type_t type = operand.type;
const uint32_t* operand_ptr = inst_ptr + operand.offset;

auto ret = SPV_ERROR_INTERNAL;
switch (type) {
case SPV_OPERAND_TYPE_RESULT_ID: {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for the braces since you're not defining or declaring variables. But you can keep if you like.

_.remove_if_fwd_declared(*operand_ptr);
ret = _.declare_id(*operand_ptr);
break;
}
case SPV_OPERAND_TYPE_TYPE_ID:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two other ID kinds you have to check: SPV_OPERAND_TYPE_MEMORY_SEMANTICS_ID, and SPV_OPERAND_TYPE_SCOPE_ID.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28

case SPV_OPERAND_TYPE_ID: {
if (can_have_forward_declared_ids) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For OpBranchConditional, this is not quite right. The condition Id must have been defined by now. It's the true branch and false branch that could be forward references.

The general point is that whether an Id can be forward declared is a function of both opcode and argument index.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3949bf4

if (_.check_declared_id(*operand_ptr)) {
ret = SPV_SUCCESS;
} else {
ret = _.forward_declare_id(*operand_ptr);
}
} else {
if (_.check_declared_id(*operand_ptr)) {
ret = SPV_SUCCESS;
} else {
ret = _.diag(SPV_ERROR_INVALID_ID) << "ID " << *operand_ptr
<< " has not been declared";
}
}
break;
}
default:
ret = SPV_SUCCESS;
break;
}
if (SPV_SUCCESS != ret) {
return ret;
}
}
return SPV_SUCCESS;
}

spv_result_t pushInstructions(void* user_data,
const spv_parsed_instruction_t* const inst) {
validation_state_t& _ = *(reinterpret_cast<validation_state_t*>(user_data));
_.set_instruction_words(inst->words);

const uint32_t* inst_ptr = _.get_instruction_ptr();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, inst_ptr is just inst->words. No need to go through a method to get it out again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed da35d28


vector<SpvOp_> instructions_with_forward_ids = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SpvOp instead of SpvOp_?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's cleaner and likely faster to use a separate method bool InstructionAllowsForwardIdReferences(SpvOp) that just has a switch statement with the positive cases, a default that breaks out, and a default return of false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a placeholder. I have updated this to return a functor based off of the opcode and return true or false based on the index of the operand. See 3949bf4

3949bf4#diff-b8a71d60452cadda1e5f190db6789d2dR411

SpvOpExecutionMode, SpvOpEntryPoint, SpvOpPhi, SpvOpFunctionCall,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also:
OpBranch, OpBranchConditional, (See section 2.5)
OpTypeForwardPointer
Any instruction with an "Invoke" parameter, since the Invoke parameter is an OpFunction. As 2.4 says "This allows for recursion and early declaration of entry points." So this list includes: OpEnqueueKernel, OpGetKernelNDRangeSubGroupCount, OpGetKernelNDRangeMaxSubGroupSize, OpGetKernelWorkGroupSize, OpGetKernelPreferredWorkGroupSizeMultiple

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpSwitch is another branching instruction. But its selector Id must be defined (can't be a forward reference). Its default Id and case label Ids can be forward references.

SpvOpName, SpvOpMemberName, SpvOpDecorate, SpvOpMemberDecorate,
SpvOpGroupDecorate};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing SpvOpGroupMemberDecorate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed 3949bf4


bool can_have_forward_declared_ids = false;
for (auto& instruction : instructions_with_forward_ids) {
can_have_forward_declared_ids |= inst->opcode == instruction;
}

auto err = SSAPass(_, can_have_forward_declared_ids, inst);

if (err != SPV_SUCCESS) {
return err;
}

return SPV_SUCCESS;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe L398 - L404 can be combined into one instruction return SSAPass(...)?

}

spv_result_t spvValidate(const spv_const_context context,
const spv_const_binary binary, const uint32_t options,
spv_diagnostic* pDiagnostic) {
Expand All @@ -279,6 +422,19 @@ spv_result_t spvValidate(const spv_const_context context,
return SPV_ERROR_INVALID_BINARY;
}

validation_state_t vstate(binary->code, pDiagnostic);
auto err = spvBinaryParse(context, &vstate, binary->code, binary->wordCount, setHeader,
pushInstructions, pDiagnostic);

if (vstate.forward_declared_id_count() > 0) {
// TODO(umar): print undefined IDs
return vstate.diag(SPV_ERROR_INVALID_ID)
<< "Some forward referenced IDs have not be defined. \n";
}
if (err) {
return err;
}

// NOTE: Copy each instruction for easier processing
std::vector<spv_instruction_t> instructions;
uint64_t index = SPV_INDEX_INSTRUCTION;
Expand All @@ -293,19 +449,6 @@ spv_result_t spvValidate(const spv_const_context context,
index += wordCount;
}

if (spvIsInBitfield(SPV_VALIDATE_BASIC_BIT, options)) {
position.index = SPV_INDEX_INSTRUCTION;
// TODO: Imcomplete implementation
spvCheckReturn(spvValidateBasic(
instructions.data(), instructions.size(), context->opcode_table,
context->operand_table, &position, pDiagnostic));
}

if (spvIsInBitfield(SPV_VALIDATE_LAYOUT_BIT, options)) {
position.index = SPV_INDEX_INSTRUCTION;
// TODO: spvBinaryValidateLayout
}

if (spvIsInBitfield(SPV_VALIDATE_ID_BIT, options)) {
position.index = SPV_INDEX_INSTRUCTION;
spvCheckReturn(
Expand All @@ -314,10 +457,5 @@ spv_result_t spvValidate(const spv_const_context context,
context->ext_inst_table, &position, pDiagnostic));
}

if (spvIsInBitfield(SPV_VALIDATE_RULES_BIT, options)) {
position.index = SPV_INDEX_INSTRUCTION;
// TODO: Specified validation rules...
}

return SPV_SUCCESS;
}
7 changes: 3 additions & 4 deletions source/validate_id.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1216,12 +1216,11 @@ bool idUsage::isValid<SpvOpFunctionParameter>(const spv_instruction_t* inst,
<< inst->words[resultTypeIndex]
<< "' is not defined.";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI. The spvValidateIDs function in validate.cpp needs to be reimplemented around the spvBinaryParse and spv_parsed_instruction_t because the technique of using spvBinaryOperandInfo function is not expressive enough to describe all the types of instructions, and the fact that they can be variable length. For example OpSwitch has variable number of operands, and some of them are literal numbers, and others are label Ids.
I think the entire validate_id.cpp file can be shunk to about 100 lines of code by using a data-driven approach around the spv_parsed_instruction_t.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was my thought. I didn't want to break all of the ValidateID unit tests in this commit. That is something I have been prototyping in another branch.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

return false);
auto function = inst - 1;
auto function = inst;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems kind of pointless. Just go ahead and rename the parameter inst to function if you like.

// NOTE: Find OpFunction & ensure OpFunctionParameter is not out of place.
size_t paramIndex = 0;
while (firstInst != function) {
spvCheck(SpvOpFunction != function->opcode &&
SpvOpFunctionParameter != function->opcode,
while (firstInst != --function) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is dangerous. A malicious input will cause this code to read out of bounds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed using assertion in da35d28

spvCheck(SpvOpFunction != function->opcode && SpvOpFunctionParameter != function->opcode,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, I'd suggest to remove spvCheck. It's just an if-statement defined as a macro...

DIAG(0) << "OpFunctionParameter is not preceded by OpFunction or "
"OpFunctionParameter sequence.";
return false);
Expand Down
Loading