What is the codegen really done at the Executorch flow?

[kris-himax]

Hi,

Although I study the https://pytorch.org/executorch/stable/concepts.html#codegen about codegen part, I do not understand very well about this part.

Above the concepts map, after I export the model.pte file which is the binary file.
Can I directly select the kernel op to run the model with Executorch Runtime library ?

And there is another branch of model.pte file which do the codegen to gen the Kernel Registration Library. I do not understand very well about this part.

My question is that if I can run with model.pte file with kernel op run time library, why need to codegen again?
Or what is the codegen output at real flow? Is it a c code about the graph of the model with ops and the weight?

[kimishpatel]

@larryliu0820

[larryliu0820]

On a high level codegen does 2 things and we decide to combine them into a single layer:

1. Use the operator information from models to do selective build.
2. Generate adapter logic to translate void(EValue**) call to void(Tensor, Tensor, ...) call. We call this "unboxing" and we generate code to do unboxing.

For 1 @lucylq can explain more on how selective build works.

For 2, the alternative is to do a C++ template-based unboxing like what TVM and PyTorch JIT is doing. It could be beneficial in terms of keeping everything in C++, making it easier for user to use. However it is also harder to connect with selective build. In the future we may be able to support both solutions.

[lucylq]

Can I directly select the kernel op to run the model with Executorch Runtime library ?
My question is that if I can run with model.pte file with kernel op run time library, why need to codegen again?

Codegen is used to generate the kernel registration library which connects the op names referenced in the model with the kernel implementation.

Re the diagram, the arrow between model.pte and codegen indicates selective build. Codegen can consume a model.pte file and select only the ops used by the model. So yes, you can directly select specific ops to include and link with the ExecuTorch runtime (which helps to reduce binary size).

Keep in mind that selective build is optional (we can build with all the portable kernels, etc. as well). Without it, codegen still occurs to glue together the ExecuTorch runtime and kernel library.

Or what is the codegen output at real flow? Is it a c code about the graph of the model with ops and the weight?

The codegen output is a set of C++ bindings (various .h, .cpp files) that connect the kernels to the ExecuTorch runtime.

[kris-himax]

Hi,
Thanks for reply.
In my option, can I think the codegen at the Executorch is different from TVM codegen?

According to your reply, the codegen at the Executorch seems like that it generate binding or linking to the ExecuTorch runtime and kernel library. It seems that it is not like the codegen of TVM which input model.pte file and output the c code with weight scalar and the op about the model. The codegen at the Executorch just output .c and .h file which do the linking to the two library.
for example (executorch-cmake-out/kernels/portable/RegisterCodegenUnboxedKernelsEverything.cpp and executorch-cmake-out/kernels/portable/NativeFunctions.h):

Kernel(
        "aten::_softmax.out",
        [](torch::executor::KernelRuntimeContext & context, EValue** stack) {
            EValue& self = *stack[0];
    	EValue& dim = *stack[1];
    	EValue& half_to_float = *stack[2];
    	EValue& out = *stack[3];
    	const torch::executor::Tensor & self_base = self.to<torch::executor::Tensor>();
    	int64_t dim_base = dim.to<int64_t>();
    	bool half_to_float_base = half_to_float.to<bool>();
    	torch::executor::Tensor & out_base = out.to<torch::executor::Tensor>();
    
            internal::EventTracerProfileScope event_tracer_scope(context.internal_event_tracer(), "native_call__softmax.out");
            EXECUTORCH_SCOPE_PROF("native_call__softmax.out");
            torch::executor::native::softmax_out(context, self_base, dim_base, half_to_float_base, out_base);
    
            
        }
    ),
    
    Kernel(
        "aten::convolution.out",
        [](torch::executor::KernelRuntimeContext & context, EValue** stack) {
            EValue& input = *stack[0];
    	EValue& weight = *stack[1];
    	EValue& bias = *stack[2];
    	EValue& stride = *stack[3];
    	EValue& padding = *stack[4];
    	EValue& dilation = *stack[5];
    	EValue& transposed = *stack[6];
    	EValue& output_padding = *stack[7];
    	EValue& groups = *stack[8];
    	EValue& out = *stack[9];
    	const torch::executor::Tensor & input_base = input.to<torch::executor::Tensor>();
    	const torch::executor::Tensor & weight_base = weight.to<torch::executor::Tensor>();
    	
    	    torch::executor::optional<torch::executor::Tensor> bias_opt_out = bias.toOptional<torch::executor::Tensor>();
    	            
    	
    	    torch::executor::ArrayRef<int64_t> stride_list_out = stride.toIntList();
    	                
    	
    	    torch::executor::ArrayRef<int64_t> padding_list_out = padding.toIntList();
    	                
    	
    	    torch::executor::ArrayRef<int64_t> dilation_list_out = dilation.toIntList();
    	                
    	bool transposed_base = transposed.to<bool>();
    	
    	    torch::executor::ArrayRef<int64_t> output_padding_list_out = output_padding.toIntList();
    	                
    	int64_t groups_base = groups.to<int64_t>();
    	torch::executor::Tensor & out_base = out.to<torch::executor::Tensor>();
    
            internal::EventTracerProfileScope event_tracer_scope(context.internal_event_tracer(), "native_call_convolution.out");
            EXECUTORCH_SCOPE_PROF("native_call_convolution.out");
            torch::executor::native::convolution_out(context, input_base, weight_base, bias_opt_out, stride_list_out, padding_list_out, dilation_list_out, transposed_base, output_padding_list_out, groups_base, out_base);
    
            
        }
    ), // Generated kernels

If I want to do the inference by the Executorch, I still should input the model.pte file. Am I correct?

Another question, where can I see I should select which op to build?
Is it the op list under the link?
https://github.com/pytorch/pytorch/blob/265d6aac0b71b917d6e36c5dd65c22f61644b715/test/expect/HasDecompTest.test_aten_core_operators.expect
If it is the link, there are many types for each ops what do they mean or which one should I choose?
for example,

aten::abs
aten::abs.out
aten::abs_
aten::bitwise_and.Scalar
aten::bitwise_and.Scalar_Tensor
aten::bitwise_and.Scalar_Tensor_out
aten::bitwise_and.Scalar_out
aten::bitwise_and.Tensor
aten::bitwise_and.Tensor_out
aten::bitwise_and_.Scalar
aten::bitwise_and_.Tensor

Another question is at runner.cpp. How could I decide the size of method_allocator_pool?
Could I imagine it like the tensor arena space at the tensorflow lite for microcontroller?

   torch::executor::MemoryAllocator method_allocator{
        torch::executor::MemoryAllocator(sizeof(method_allocator_pool), method_allocator_pool)};

At runner.cpp what is the meaning about program->num_methods() ?

torch::executor::runtime_init();

    auto loader = torch::executor::util::BufferDataLoader(model_pte, sizeof(model_pte));
    ET_LOG(Info, "Model PTE file loaded. Size: %lu bytes.", sizeof(model_pte));
    Result<torch::executor::Program> program = torch::executor::Program::load(&loader);
    if(!program.ok()) {
       ET_LOG(Info,"Program loading failed @ 0x%p: 0x%" PRIx32, model_pte, program.error());
    }

    ET_LOG(Info,"Model buffer loaded, has %lu methods", program->num_methods());

[lucylq]

In my option, can I think the codegen at the Executorch is different from TVM codegen?

Yeah, ExecuTorch codegen generates code that connects the runtime and kernel library. I'm not too familiar with TVM codegen, @larryliu0820 can probably speak more on this.

If I want to do the inference by the Executorch, I still should input the model.pte file. Am I correct?

You should input the model.pte file to codegen if you want selective build. The sample runtime, executor_runner, includes all the portable kernels; you can try run your model with executor_runner as a first step. By the way, there are some selective build examples here. These show how to include specific ops into the executorch_generated_lib.

Another question, where can I see I should select which op to build?

You can check the functions.yaml files for operators, eg. portable ops, optimized ops.

Another question is at runner.cpp. How could I decide the size of method_allocator_pool?

The amount of memory depends on the loaded method and the runtime code - see method allocator.

At runner.cpp what is the meaning about program->num_methods() ?

ExecuTorch methods are the executable equivalent of an nn.Module Python method. If an eager mode nn.Module contains one forward() method, that would compile into an ExecuTorch method and num_methods() would equal 1.

[larryliu0820]

We don't support codegen kernels yet. There's inductor on PT2 that is able to generate C++/triton kernels for ops, but that's more suitable to run in server environment with GPU support. We may be able to leverage that in the future, but currently we will ask the users to provide kernels.

there are many types for each ops what do they mean or which one should I choose?

We've defined a core ATen Op set that contains all the ops may possibly appear in the exported graph. Once we lower exported graph to ExecuTorch, we will convert these operators to their out-variant, e.g., add.Tensor -> add.out. Therefore, at runtime we only see these "out" operators.