Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to generate correct import using grpcio-tools #9575

Closed
GrapeBaBa opened this issue Feb 3, 2017 · 14 comments

Comments

@GrapeBaBa
Copy link

commented Feb 3, 2017

We have several proto files and some import others. When we use grpcio-tools to generate python file, the import statement in python file not correct.
For example, generated below

from common import common_pb2 as common_dot_common__pb2

What I expect

from hfc.protos.common import common_pb2 as common_dot_common__pb2
import "google/protobuf/timestamp.proto";

option go_package = "github.com/hyperledger/fabric/protos/common";

package common;

// These status codes are intended to resemble selected HTTP status codes
enum Status {
    UNKNOWN = 0;
    SUCCESS = 200;
    BAD_REQUEST = 400;
    FORBIDDEN = 403;
    NOT_FOUND = 404;
    REQUEST_ENTITY_TOO_LARGE = 413;
    INTERNAL_SERVER_ERROR = 500;
    SERVICE_UNAVAILABLE = 503;
}

enum HeaderType {
    MESSAGE = 0;                   // Used for messages which are signed but opaque
    CONFIGURATION_TRANSACTION = 1; // Used for messages which reconfigure the chain
    CONFIGURATION_ITEM = 2;        // Used inside of the the reconfiguration message for signing over ConfigurationItems
    ENDORSER_TRANSACTION = 3;      // Used by the SDK to submit endorser based transactions
    ORDERER_TRANSACTION = 4;       // Used internally by the orderer for management
    DELIVER_SEEK_INFO = 5;         // Used as the type for Envelope messages submitted to instruct the Deliver API to seek
}

// This enum enlist indexes of the block metadata array
enum BlockMetadataIndex {
    SIGNATURES = 0;             // Block metadata array position for block signatures
    LAST_CONFIGURATION = 1;     // Block metadata array poistion to store last configuration block sequence number
    TRANSACTIONS_FILTER = 2;    // Block metadata array poistion to store serialized bit array filter of invalid transactions
}

// LastConfiguration is the encoded value for the Metadata message which is encoded in the LAST_CONFIGURATION block metadata index
message LastConfiguration {
	uint64 index  = 1;
}

// Metadata is a common structure to be used to encode block metadata
message Metadata {
    bytes value = 1;
    repeated MetadataSignature signatures = 2;
}

message MetadataSignature {
    bytes signatureHeader = 1; // An encoded SignatureHeader
    bytes signature = 2;       // The signature over the concatenation of the Metadata value bytes, signatureHeader, and block header
}

message Header {
    ChainHeader chainHeader = 1;
    SignatureHeader signatureHeader = 2;
}

// Header is a generic replay prevention and identity message to include in a signed payload
message ChainHeader {
    int32 type = 1; // Header types 0-10000 are reserved and defined by HeaderType

    // Version indicates message protocol version
    int32 version = 2;

    // Timestamp is the local time when the message was created
    // by the sender
    google.protobuf.Timestamp timestamp = 3;

    // Identifier of the chain this message is bound for
    string chainID = 4;

    // An unique identifier that is used end-to-end.
    //  -  set by higher layers such as end user or SDK
    //  -  passed to the endorser (which will check for uniqueness)
    //  -  as the header is passed along unchanged, it will be
    //     be retrieved by the committer (uniqueness check here as well)
    //  -  to be stored in the ledger
    string txID = 5;

    // The epoch in which this header was generated, where epoch is defined based on block height
    // Epoch in which the response has been generated. This field identifies a
    // logical window of time. A proposal response is accepted by a peer only if
    // two conditions hold:
    // 1. the epoch specified in the message is the current epoch
    // 2. this message has been only seen once during this epoch (i.e. it hasn't
    //    been replayed)
    uint64 epoch = 6;

    // Extension that may be attached based on the header type
    bytes extension = 7;
}

message SignatureHeader {
    // Creator of the message, specified as a certificate chain
    bytes creator = 1;

    // Arbitrary number that may only be used once. Can be used to detect replay attacks.
    bytes nonce = 2;
}

// Payload is the message contents (and header to allow for signing)
message Payload {

    // Header is included to provide identity and prevent replay
    Header header = 1;

    // Data, the encoding of which is defined by the type in the header
    bytes data = 2;
}

// Envelope wraps a Payload with a signature so that the message may be authenticated
message Envelope {
    // A marshaled Payload
    bytes payload = 1;

    // A signature by the creator specified in the Payload header
    bytes signature = 2;
}

// This is finalized block structure to be shared among the orderer and peer
// Note that the BlockHeader chains to the previous BlockHeader, and the BlockData hash is embedded
// in the BlockHeader.  This makes it natural and obvious that the Data is included in the hash, but
// the Metadata is not.
message Block {
    BlockHeader Header = 1;
    BlockData Data = 2;
    BlockMetadata Metadata = 3;
}

message BlockHeader {
    uint64 Number = 1; // The position in the blockchain
    bytes PreviousHash = 2; // The hash of the previous block header
    bytes DataHash = 3; // The hash of the BlockData, by MerkleTree
}

message BlockData {
    repeated bytes Data = 1;
}

message BlockMetadata {
    repeated bytes Metadata = 1;
}
syntax = "proto3";

import "common/common.proto";
import "common/chain-config.proto";

option go_package = "github.com/hyperledger/fabric/protos/common";

package common;

// ConfigurationEnvelope is designed to contain _all_ configuration for a chain with no dependency
// on previous configuration transactions.
//
// It is generated with the following scheme:
//   1. Retrieve the existing configuration
//   2. Note the highest configuration sequence number, store it and increment it by one
//   3. Modify desired ConfigurationItems, setting each LastModified to the stored and incremented sequence number
//     a) Note that the ConfigurationItem has a ChainHeader header attached to it, who's type is set to CONFIGURATION_ITEM
//   4. Update SignedConfigurationItem with appropriate signatures over the modified ConfigurationItem
//     a) Each signature is of type ConfigurationSignature
//     b) The ConfigurationSignature signature is over the concatenation of signatureHeader and the ConfigurationItem bytes (which includes a ChainHeader)
//   5. Submit new Configuration for ordering in Envelope signed by submitter
//     a) The Envelope Payload has data set to the marshaled ConfigurationEnvelope
//     b) The Envelope Payload has a header of type Header.Type.CONFIGURATION_TRANSACTION
//
// The configuration manager will verify:
//   1. All configuration items and the envelope refer to the correct chain
//   2. Some configuration item has been added or modified
//   3. No existing configuration item has been ommitted
//   4. All configuration changes have a LastModification of one more than the last configuration's highest LastModification number
//   5. All configuration changes satisfy the corresponding modification policy
message ConfigurationEnvelope {
    repeated SignedConfigurationItem Items = 1;
}

// This message may change slightly depending on the finalization of signature schemes for transactions
message SignedConfigurationItem {
    bytes ConfigurationItem = 1;
    repeated ConfigurationSignature Signatures = 2;
}

message ConfigurationItem {
    enum ConfigurationType {
        Policy = 0;   // Implies that the Value is a marshaled Policy message, and may be referred to by Key as a ModificationPolicy
        Chain = 1;    // Marshaled format for this type is yet to be determined
        Orderer = 2;  // Marshaled format for this type is yet to be determined
        Peer = 3;   // Marshaled format for this type is yet to be determined
    }
    ChainHeader Header = 1;  // The header which ties this configuration to a particular chain
    ConfigurationType Type = 2;     // The type of configuration this is.
    uint64 LastModified = 3;        // The Sequence number in the ConfigurationEnvelope this item was last modified
    string ModificationPolicy = 4;  // What policy to check before allowing modification
    string Key = 5;                 // A unique ID, unique scoped by Type, to reference the value by
    bytes Value = 6;                // The byte representation of this configuration, usually a marshaled message
}

message ConfigurationSignature {
    bytes signatureHeader = 1; // A marshaled SignatureHeader
    bytes signature = 2;       // Signature over the concatenation of configurationItem bytes and signatureHeader bytes
}

// Policy expresses a policy which the orderer can evaluate, because there has been some desire expressed to support
// multiple policy engines, this is typed as a oneof for now
message Policy {
    enum PolicyType {
        UNKNOWN = 0; // Reserved to check for proper initialization
        SIGNATURE = 1;
        MSP = 2;
    }
    int32 type = 1; // For outside implementors, consider the first 1000 types reserved, otherwise one of PolicyType
    bytes policy = 2;
}

// SignaturePolicyEnvelope wraps a SignaturePolicy and includes a version for future enhancements
message SignaturePolicyEnvelope {
    int32 Version = 1;
    SignaturePolicy Policy = 2;
    repeated MSPPrincipal Identities = 3;
}

// SignaturePolicy is a recursive message structure which defines a featherweight DSL for describing
// policies which are more complicated than 'exactly this signature'.  The NOutOf operator is sufficent
// to express AND as well as OR, as well as of course N out of the following M policies
// SignedBy implies that the signature is from a valid certificate which is signed by the trusted
// authority specified in the bytes.  This will be the certificate itself for a self-signed certificate
// and will be the CA for more traditional certificates
message SignaturePolicy {
    message NOutOf {
        int32 N = 1;
        repeated SignaturePolicy Policies = 2;
    }
    oneof Type {
        int32 SignedBy = 1;
        NOutOf From = 2;
    }
}
@nathanielmanistaatgoogle

This comment has been minimized.

Copy link
Member

commented Feb 7, 2017

That's a lot of code - how much can you simplify it (and how short can you make it) while still demonstrating the problem?

Does the problem described in this issue have any relationship with the problem described in issue 9450?

@GrapeBaBa

This comment has been minimized.

Copy link
Author

commented Feb 8, 2017

@nathanielmanistaatgoogle I already fixed it. I change the package from 'common' to 'hfc.protos.common' and import 'common/' to "hfc/protos/common/". However I wonder if python protoc plugin can support feature like 'java_package' and 'go_package''

@ewang

This comment has been minimized.

Copy link

commented Feb 14, 2017

Running into a similar issue as well. Seems like the python grpc import statements generated is not python3 compatible. It needs to either be relative with a '.' or the absolute import path.

@nathanielmanistaatgoogle

This comment has been minimized.

Copy link
Member

commented Feb 16, 2017

@haberman and @xfxyjwf: are you able to answer "However I wonder if python protoc plugin can support feature like 'java_package' and 'go_package'"?

@dgn

This comment has been minimized.

Copy link

commented Apr 13, 2017

I think I'm hitting the same problem here, I'll try to make the description a bit shorter. We're generating python modules using the following command:
python3 -m grpc_tools.protoc -I some/folder/proto --python_out=some/folder --grpc_python_out=some/folder some/folder/proto/*.proto

That then generates the expected files, but within the _pb2_grpc.py files we get an import statement such as:
import someservice_pb2 as someservice__pb2
which works in python2 but not python3. In python3 I would expect either
from . import someservice_pb2 as someservice__pb2
or
import some.folder.someservice_pb2 as someservice__pb2

Currently I see no fix except manually editing generated files...

@dgn

This comment has been minimized.

Copy link

commented Apr 13, 2017

Nevermind, I found the solution. I will quickly explain it in case anyone else is looking at this.

I now moved my proto files into a directory called proto/some/folder/

Then using the following command, the module names and imports are correct and work as expected in python3:

python3 -m grpc_tools.protoc -I proto --python_out=. --grpc_python_out=. proto/some/folder/*.proto

Note that the combination of including the root directory (proto/) and directing the output to --python_out=. achieves the desired results:

Generated files end up in some/folder/, also imports are as expected:

import some.folder.someservice_pb2 as some_dot_folder_dot_someservice__pb2

@jclinto1

This comment has been minimized.

Copy link

commented Oct 18, 2017

Yes this worked, so in essence you have to put your proto files in a sub directory and then adjust the protoc command to point to the other paths to be at the parent, then the module is created as expected. I did find I had to change some imports based on the gPRC examples.

@eHattori

This comment has been minimized.

Copy link

commented Feb 20, 2018

Hello, I pass by this problem with python 3 and need edit the generated python files , this is my structure:

--- my_app
---- gen_py
---- protos
--Makefile

the command in the Makefile that execute it's:

python -m grpc_tools.protoc -I ./myapp/protos --python_out=./myapp/gen_py --grpc_python_out=./myapp/gen_py ./myapp/protos/*.proto

Someone can help me
tks

@raulAtNines

This comment has been minimized.

Copy link

commented Feb 20, 2018

I'm also running into this issue, and the solution by @dgn and @jclinto1 does not work for me.

With the folder structure:

/root
  /protos
  /src/python/stubs

Running

python -m grpc_tools.protoc -I ./protos/ --python_out=. --grpc_python_out=. ./protos/controller.proto

from root, does not generate python 3 proper module syntax.
import controller_pb2 as controller__pb2

@dgn This is similar to your original question. I do not fully understand what you changed that worked. Do you see any differences in this set up with respect to your proposed solution?

Thanks!

@raulAtNines

This comment has been minimized.

Copy link

commented Mar 16, 2018

Ok, I figured out that the restriction here is that imports are generated based on the proto hierarchy, not the output hierarchy. Does it make sense to change the generated import syntax to match the output folder structure instead @ctiller ?

@nathanielmanistaatgoogle

This comment has been minimized.

Copy link
Member

commented Mar 29, 2018

@raulAtNines: what makes sense is for us (gRPC Python) to do whatever Protocol Buffers Python does, and to accomplish that sameness with some code-sharing. Unfortunately that will take a bit of refactoring that has been started but has been back-burnered for the last year or so.

@raulAtNines

This comment has been minimized.

Copy link

commented Mar 29, 2018

I see, I didn't realize the import generation was part of PBP.

@nathanielmanistaatgoogle

This comment has been minimized.

Copy link
Member

commented Apr 2, 2018

@raulAtNines: it isn't today the case that the import generation is fully integrated with Protocol Buffers Python, but it's something we'd like to be the case in the future.

@tvalentyn

This comment has been minimized.

Copy link

commented Sep 5, 2018

See protocolbuffers/protobuf#1491 for a discussion of this issue.

hemarkus added a commit to QMSTR/pyqmstr that referenced this issue Feb 6, 2019

Fix python3 imports
Adding ugly hack to fix import statements to work using a python3
interpreter.

Find the sad story here: grpc/grpc#9575

@lock lock bot locked as resolved and limited conversation to collaborators Feb 7, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.