XML syntax error with Bitbucket private repo

[jtblin]

I have some generated code for grpc.io using the following command protoc -I ./apipb apipb/*.proto --go_out=plugins=grpc:apipb.

When I run glide up from another repo which imports this repo I get the following error:

[INFO] Fetching bitbucket.org/jtblin/some-repoi/apipb into /Users/jtblin/src/go/src/bitbucket.org/jtblin/some-other-repo/vendor
[ERROR] Error looking for bitbucket.org/jtblin/some-repo/apipb: XML syntax error on line 7: expected attribute name in element

I am not sure exactly which generated code is creating the problem.

Example of protobuf definition:

syntax = "proto3";
package api;

// AppService is an App service
service AppService {
    rpc CreateApp (App) returns (AppCreateResponse) {}
    rpc GetApp (App) returns (App) {}
}

// App represents an app
message App {
    string  name = 1;
    string  owner = 2;
}

// AppCreateResponse represents an app creation response
message AppCreateResponse {
    string status = 1;
}

The generated code is like this

// Code generated by protoc-gen-go.
// source: app.proto
// DO NOT EDIT!

/*
Package api is a generated protocol buffer package.

It is generated from these files:
    app.proto
    auth.proto
    deployment.proto

It has these top-level messages:
    App
    AppCreateResponse
    Credentials
    AuthResponse
    DeploymentRequest
    DeploymentJob
    DeploymentResponse
*/
package api

import proto "github.com/golang/protobuf/proto"
import fmt "fmt"
import math "math"

import (
    context "golang.org/x/net/context"
    grpc "google.golang.org/grpc"
)

// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf

// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
const _ = proto.ProtoPackageIsVersion1

// App represents an app
type App struct {
    Name  string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"`
    Owner string `protobuf:"bytes,2,opt,name=owner" json:"owner,omitempty"`
}

func (m *App) Reset()                    { *m = App{} }
func (m *App) String() string            { return proto.CompactTextString(m) }
func (*App) ProtoMessage()               {}
func (*App) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0} }

// AppCreateResponse represents an app creation response
type AppCreateResponse struct {
    Status string `protobuf:"bytes,1,opt,name=status" json:"status,omitempty"`
}

func (m *AppCreateResponse) Reset()                    { *m = AppCreateResponse{} }
func (m *AppCreateResponse) String() string            { return proto.CompactTextString(m) }
func (*AppCreateResponse) ProtoMessage()               {}
func (*AppCreateResponse) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{1} }

func init() {
    proto.RegisterType((*App)(nil), "api.App")
    proto.RegisterType((*AppCreateResponse)(nil), "api.AppCreateResponse")
}

// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn

// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
const _ = grpc.SupportPackageIsVersion1

// Client API for AppService service

type AppServiceClient interface {
    CreateApp(ctx context.Context, in *App, opts ...grpc.CallOption) (*AppCreateResponse, error)
    GetApp(ctx context.Context, in *App, opts ...grpc.CallOption) (*App, error)
}

type appServiceClient struct {
    cc *grpc.ClientConn
}

func NewAppServiceClient(cc *grpc.ClientConn) AppServiceClient {
    return &appServiceClient{cc}
}

func (c *appServiceClient) CreateApp(ctx context.Context, in *App, opts ...grpc.CallOption) (*AppCreateResponse, error) {
    out := new(AppCreateResponse)
    err := grpc.Invoke(ctx, "/api.AppService/CreateApp", in, out, c.cc, opts...)
    if err != nil {
        return nil, err
    }
    return out, nil
}

func (c *appServiceClient) GetApp(ctx context.Context, in *App, opts ...grpc.CallOption) (*App, error) {
    out := new(App)
    err := grpc.Invoke(ctx, "/api.AppService/GetApp", in, out, c.cc, opts...)
    if err != nil {
        return nil, err
    }
    return out, nil
}

// Server API for AppService service

type AppServiceServer interface {
    CreateApp(context.Context, *App) (*AppCreateResponse, error)
    GetApp(context.Context, *App) (*App, error)
}

func RegisterAppServiceServer(s *grpc.Server, srv AppServiceServer) {
    s.RegisterService(&_AppService_serviceDesc, srv)
}

func _AppService_CreateApp_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error) (interface{}, error) {
    in := new(App)
    if err := dec(in); err != nil {
        return nil, err
    }
    out, err := srv.(AppServiceServer).CreateApp(ctx, in)
    if err != nil {
        return nil, err
    }
    return out, nil
}

func _AppService_GetApp_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error) (interface{}, error) {
    in := new(App)
    if err := dec(in); err != nil {
        return nil, err
    }
    out, err := srv.(AppServiceServer).GetApp(ctx, in)
    if err != nil {
        return nil, err
    }
    return out, nil
}

var _AppService_serviceDesc = grpc.ServiceDesc{
    ServiceName: "api.AppService",
    HandlerType: (*AppServiceServer)(nil),
    Methods: []grpc.MethodDesc{
        {
            MethodName: "CreateApp",
            Handler:    _AppService_CreateApp_Handler,
        },
        {
            MethodName: "GetApp",
            Handler:    _AppService_GetApp_Handler,
        },
    },
    Streams: []grpc.StreamDesc{},
}

var fileDescriptor0 = []byte{
    // 166 bytes of a gzipped FileDescriptorProto
    0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0xe2, 0xe2, 0x4c, 0x2c, 0x28, 0xd0,
    0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x4e, 0x2c, 0xc8, 0x54, 0xd2, 0xe7, 0x62, 0x76, 0x2c,
    0x28, 0x10, 0x12, 0xe2, 0x62, 0xc9, 0x4b, 0xcc, 0x4d, 0x95, 0x60, 0x54, 0x60, 0xd4, 0xe0, 0x0c,
    0x02, 0xb3, 0x85, 0x44, 0xb8, 0x58, 0xf3, 0xcb, 0xf3, 0x52, 0x8b, 0x24, 0x98, 0xc0, 0x82, 0x10,
    0x8e, 0x92, 0x36, 0x97, 0x20, 0x50, 0x83, 0x73, 0x51, 0x6a, 0x62, 0x49, 0x6a, 0x50, 0x6a, 0x71,
    0x41, 0x7e, 0x5e, 0x71, 0xaa, 0x90, 0x18, 0x17, 0x5b, 0x71, 0x49, 0x62, 0x49, 0x69, 0x31, 0xd4,
    0x00, 0x28, 0xcf, 0x28, 0x96, 0x8b, 0x0b, 0xa8, 0x38, 0x38, 0xb5, 0xa8, 0x2c, 0x33, 0x39, 0x55,
    0x48, 0x9f, 0x8b, 0x13, 0xa2, 0x0f, 0x64, 0x23, 0x87, 0x1e, 0xd0, 0x7a, 0x3d, 0x20, 0x4b, 0x4a,
    0x0c, 0xc6, 0x42, 0x35, 0x54, 0x89, 0x41, 0x48, 0x8e, 0x8b, 0xcd, 0x3d, 0xb5, 0x04, 0x55, 0x35,
    0x9c, 0xa5, 0xc4, 0x90, 0xc4, 0x06, 0xf6, 0x88, 0x31, 0x20, 0x00, 0x00, 0xff, 0xff, 0xcb, 0xac,
    0xc9, 0x13, 0xd5, 0x00, 0x00, 0x00,
}

Maybe it's the byte data that creates this issue?

It totally breaks glide and I am not able to generate the glide.lock file.

Thanks for your help

[sdboyer]

hmm...XML complaints. I think this actually might not have anything to do with protobuf. You're on bitbucket - are you using git, or hg?

@mattfarina may have a better immediate idea, but this is the first thing that jumps out at me, because hg logic parses commit information using hg's XML output style. If you are using hg, could you paste in the contents of hg log -r <rev> --style=xml from the bitbucket.org/jtblin/some-repoi/apipb, substituting in whatever the revision in question being used is for <rev?

If you're not using hg, then i'm not immediately sure what could be the cause here :)

[jtblin]

No I use git not hg. I don't have any commit yet as the repo is not pushed yet (developing locally), could that be the issue?

[sdboyer]

That might do it. Looks like XML is also used in doing remote discovery/detection. Does the remote repository exist, but no commits are yet pushed to it (so, empty repo), or does it simply not exist at all?

[mattfarina]

@jtblin might bitbucket.org/jtblin/some-repo/apipb be a private repo? Or, is the repo path correct?

I was just able to reproduce this error when the endpoint on bitbucket.org is an inaccessible or nonexistent repo. If this is a private repo can you updated the glide.yaml entry to look like:

- package: bitbucket.org/jtblin/some-repo
  repo: git@bitbucket.org:jtblin/some-repo
  subpackage:
  - apipb

This will tell git to use your credentials to access the repo.

If this is the case we may be able to catch this bitbucket error and provide better context.

[jtblin]

I pushed some commits and I'm still getting the same error so it doesn't look like it's the issue. Given the error message specifically mentions the code generated from protobuf, it does seem like this is what causing the issue. There are 2 types of files in this directory .proto and .go, not sure which one is causing the issue. Is there any advise to troubleshoot further?

[mattfarina]

@jtblin I get this error when there's an access issue to the repo. When the package is set to bitbucket.org/jtblin/some-repo and no repo is set it will try to clone as https://bitbucket.org/jtblin/some-repo. If this is a private repo, even if you can otherwise push to it, Glide may have issues pulling. That's why I suggested setting the repo to the ssh style syntax that would cause Glide to use your ssh keys for auth.

This error shouldn't be picked up by protobuf.

When you get the error:

[ERROR] Error looking for bitbucket.org/jtblin/some-repo/apipb: XML syntax error on line 7: expected attribute name in element

I believe it's saying the repo bitbucket.org/jtblin/some-repo is inaccessible. It's not able to fetch bitbucket.org/jtblin/some-repo. This is something we need a better message on.

[mattfarina]

I just ran a couple more tests. On bitbucket.org when the repo is private and no repo property is set in the glide.yaml I get an error of XML syntax error on line 9: expected attribute name in element. When the repo doesn't exist on bitbucket.org I consistently get the error XML syntax error on line 7: expected attribute name in element.

I've not tracked this yet but am able to reliable reproduce these errors.

I'm going to see if I can get to the bottom of these errors (as I didn't expect this error at all) and provide a better message.

You can get more information running glide --debug up. This will provide far more output that can pinpoint where the issue is happening.

[jtblin]

Thank you @mattfarina, sorry I missed your previous answer. Yes it is a private repo but it is set as ssh in the local git config.

Adding repo: git@bitbucket.org:jtblin/some-repo (with or without .git at the end) in the yaml file yields a different error:

[ERROR] Update failed for bitbucket.org/jtblin/some-repo: Cannot detect VCS
[ERROR] Failed to do initial checkout of config: Cannot detect VCS

If I do a glide get bitbucket.org:jtblin/some-repo it is able to fetch the packages imported by this repo but it fails to import the repo itself, always with the same error.

[ERROR] Update failed for bitbucket.org/jtblin/some-repo: XML syntax error on line 9: expected attribute name in element

The import is for the correct path and my IDE is able to see the imported package so I can confirm that everything is correct from this perspective.

I tried deleting all the glide files and started the entire process but same issue. The repo is added in the glide.lock file btw, but it doesn't make it to the vendor directory.

I tried with another bitbucket private repo which doesn't have protobuf and I am getting the same error so protobuf is not the issue indeed. It seems it just doesn't work with bitbucket private repos.

[mattfarina]

What version of Glide are you using and how did you install it? "I Think" that Cannot detect VCS error was recently fixed. I'll run some tests today.

[jtblin]

I have 0.10.1 installed via brew.

[mattfarina]

@jtblin now that I know how you got to this I'll look into fixing this.

[mattfarina]

This should not be fixed on the tip of master with the use of the latest vcs package.

[vishr]

With installation from HEAD, xml parsing error is gone but I still get Cannot detect VCS. It is a private bitbucket repo.

[mattfarina]

@vishr I didn't note it in the release notes but this should now be improved. If there's a not found or access denied it should tell you.

[JoseFMP]

Over here still happening.