Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process project.toml for app source metadata #1231

Merged
merged 25 commits into from
Aug 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
3c12be1
Parsed project descriptor
Haimantika Jul 7, 2021
a35b2c2
Added project metadata file
Haimantika Jul 15, 2021
b30806a
Removed syntax errors
Haimantika Jul 17, 2021
d732d44
Changed the metadata.go and container_ops_test.go file according to t…
Haimantika Jul 17, 2021
c7a379c
Added struct in lifecycle_executor.go and some minute changes on the …
Haimantika Jul 17, 2021
8c91798
Fix compiler errors
jromero Jul 17, 2021
c189f4b
Formatted
Haimantika Jul 19, 2021
593a70e
Changed path and wrote acceptance test
Haimantika Jul 20, 2021
912a83c
Testing for errors
Haimantika Jul 24, 2021
6201391
Debugging
Haimantika Jul 24, 2021
e72eef4
Fixing errors
Haimantika Jul 24, 2021
a83e8bc
Fixing error pt 2
Haimantika Jul 24, 2021
496c40d
Corrected printf to sprintf
Haimantika Jul 24, 2021
79a16be
Fixed sprintf and whitespace issue
Haimantika Jul 25, 2021
e955e6a
New changes with tests working locally
Haimantika Jul 27, 2021
9eed9b3
Update acceptance/acceptance_test.go
Haimantika Jul 30, 2021
7f390eb
Applied suggested changes
Haimantika Jul 30, 2021
bbcb9e2
Solved the conflict
Haimantika Jul 30, 2021
2dff56c
Fixed formatting
Haimantika Jul 30, 2021
9046f76
Corrected the json in acceptance_test.go
Haimantika Aug 2, 2021
7897d8e
Tried fixing the registry error
Haimantika Aug 2, 2021
5f0113a
Changes made in acceptance test
Haimantika Aug 2, 2021
210c064
Parathesis issue fixing
Haimantika Aug 2, 2021
3d6b7c1
Re-do changes
Haimantika Aug 2, 2021
946e2e6
Paranthesis error fixed
Haimantika Aug 2, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions acceptance/acceptance_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -901,6 +901,9 @@ func testAcceptance(
t.Log("sets the run image metadata")
assertImage.HasLabelWithData(repoName, "io.buildpacks.lifecycle.metadata", fmt.Sprintf(`"stack":{"runImage":{"image":"%s","mirrors":["%s"]}}}`, runImage, runImageMirror))

t.Log("sets the source metadata")
assertImage.HasLabelWithData(repoName, "io.buildpacks.project.metadata", (`{"source":{"type":"project","version":{"declared":"1.0.2"},"metadata":{"url":"https://github.com/buildpacks/pack"}}}`))

t.Log("registry is empty")
assertImage.NotExistsInRegistry(repo)

Expand Down
4 changes: 4 additions & 0 deletions acceptance/testdata/mock_app/project.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[project]
version = "1.0.2"
source-url = "https://github.com/buildpacks/pack"

26 changes: 18 additions & 8 deletions build.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import (
"github.com/pkg/errors"
ignore "github.com/sabhiram/go-gitignore"

"github.com/buildpacks/lifecycle/platform"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import Groups

Imports should be grouped into the following:

  1. Standard library
  2. External libraries
  3. Local packages


"github.com/buildpacks/pack/config"
"github.com/buildpacks/pack/internal/blob"
"github.com/buildpacks/pack/internal/build"
Expand Down Expand Up @@ -290,22 +292,31 @@ func (c *Client) Build(ctx context.Context, opts BuildOptions) error {
return err
}

version := opts.ProjectDescriptor.Project.Version
sourceURL := opts.ProjectDescriptor.Project.SourceURL
runImageName, err = pname.TranslateRegistry(runImageName, c.registryMirrors, c.logger)
if err != nil {
return err
}

lifecycleOpts := build.LifecycleOptions{
AppPath: appPath,
Image: imageRef,
Builder: ephemeralBuilder,
RunImage: runImageName,
AppPath: appPath,
Image: imageRef,
Builder: ephemeralBuilder,
LifecycleImage: ephemeralBuilder.Name(),
RunImage: runImageName,
ProjectMetadata: platform.ProjectMetadata{Source: &platform.ProjectSource{
Type: "project",
Version: map[string]interface{}{"declared": version},
Metadata: map[string]interface{}{"url": sourceURL},
}},
ProjectPath: "",
ClearCache: opts.ClearCache,
Publish: opts.Publish,
DockerHost: opts.DockerHost,
UseCreator: false,
TrustBuilder: opts.TrustBuilder,
LifecycleImage: ephemeralBuilder.Name(),
UseCreator: false,
DockerHost: opts.DockerHost,
CacheImage: opts.CacheImage,
HTTPProxy: proxyConfig.HTTPProxy,
HTTPSProxy: proxyConfig.HTTPSProxy,
NoProxy: proxyConfig.NoProxy,
Expand All @@ -314,7 +325,6 @@ func (c *Client) Build(ctx context.Context, opts BuildOptions) error {
Volumes: processedVolumes,
DefaultProcessType: opts.DefaultProcessType,
FileFilter: fileFilter,
CacheImage: opts.CacheImage,
Workspace: opts.Workspace,
GID: opts.GroupID,
}
Expand Down
9 changes: 9 additions & 0 deletions buildpack.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
go:
targets:
- ./cmd/pack

build:
# The go.build.flags property allows you to override the default build
# flags when compiling your program.
flags:
- -ldflags="-s -w -X 'github.com/buildpacks/pack.Version=#{PACK_VERSION}#'"
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ require (
github.com/sabhiram/go-gitignore v0.0.0-20201211074657-223ce5d391b0
github.com/sclevine/spec v1.4.0
github.com/sergi/go-diff v1.1.0 // indirect
github.com/sirupsen/logrus v1.7.0 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spf13/cobra v1.2.1
github.com/willf/bitset v1.1.11 // indirect
github.com/xanzy/ssh-agent v0.3.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -467,8 +467,8 @@ github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPx
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/sirupsen/logrus v1.7.0 h1:ShrD1U9pZB12TX0cVy0DtePoCH97K8EtX+mg7ZARUtM=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/sirupsen/logrus v1.8.1 h1:dJKuHgqk1NNQlqoA6BTlM1Wf9DOH3NBjQyu0h9+AZZE=
github.com/sirupsen/logrus v1.8.1/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v1.0.0/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM=
Expand Down
31 changes: 31 additions & 0 deletions internal/build/container_ops.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ import (

"github.com/buildpacks/pack/internal/paths"

"github.com/buildpacks/lifecycle/platform"

"github.com/buildpacks/pack/internal/builder"
"github.com/buildpacks/pack/internal/container"
"github.com/buildpacks/pack/pkg/archive"
Expand Down Expand Up @@ -138,6 +140,35 @@ func findMount(info types.ContainerJSON, dst string) (types.MountPoint, error) {
return types.MountPoint{}, fmt.Errorf("no matching mount found for %s", dst)
}

//WriteProjectMetadata
func WriteProjectMetadata(p string, metadata platform.ProjectMetadata, os string) ContainerOperation {
return func(ctrClient client.CommonAPIClient, ctx context.Context, containerID string, stdout, stderr io.Writer) error {
buf := &bytes.Buffer{}
err := toml.NewEncoder(buf).Encode(metadata)
if err != nil {
return errors.Wrap(err, "marshaling project metadata")
}

tarBuilder := archive.TarBuilder{}

tarPath := p
if os == "windows" {
tarPath = paths.WindowsToSlash(p)
}

tarBuilder.AddFile(tarPath, 0755, archive.NormalizedDateTime, buf.Bytes())
reader := tarBuilder.Reader(archive.DefaultTarWriterFactory())
defer reader.Close()

if os == "windows" {
dirName := paths.WindowsDir(p)
return copyDirWindows(ctx, ctrClient, containerID, reader, dirName, stdout, stderr)
}

return ctrClient.CopyToContainer(ctx, containerID, "/", reader, types.CopyToContainerOptions{})
}
}

// WriteStackToml writes a `stack.toml` based on the StackMetadata provided to the destination path.
func WriteStackToml(dstPath string, stack builder.StackMetadata, os string) ContainerOperation {
return func(ctrClient client.CommonAPIClient, ctx context.Context, containerID string, stdout, stderr io.Writer) error {
Expand Down
94 changes: 94 additions & 0 deletions internal/build/container_ops_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import (
"github.com/sclevine/spec"
"github.com/sclevine/spec/report"

"github.com/buildpacks/lifecycle/platform"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See "Import Groups"


"github.com/buildpacks/pack/internal/build"
"github.com/buildpacks/pack/internal/builder"
"github.com/buildpacks/pack/internal/container"
Expand Down Expand Up @@ -330,6 +332,98 @@ drwsrwsrwt 2 123 456 (.*) some-vol
})
})

when("#WriteProjectMetadata", func() {
it("writes file", func() {
containerDir := "/layers-vol"
p := "/layers-vol/project-metadata.toml"
if osType == "windows" {
containerDir = `c:\layers-vol`
p = `c:\layers-vol\project-metadata.toml`
}

ctrCmd := []string{"ls", "-al", "/layers-vol/project-metadata.toml"}
if osType == "windows" {
ctrCmd = []string{"cmd", "/c", `dir /q /n c:\layers-vol\project-metadata.toml`}
}
ctx := context.Background()
ctr, err := createContainer(ctx, imageName, containerDir, osType, ctrCmd...)
h.AssertNil(t, err)
defer cleanupContainer(ctx, ctr.ID)

writeOp := build.WriteProjectMetadata(p, platform.ProjectMetadata{
Source: &platform.ProjectSource{
Type: "project",
Version: map[string]interface{}{
"declared": "1.0.2",
},
Metadata: map[string]interface{}{
"url": "https://github.com/buildpacks/pack",
},
},
}, osType)

var outBuf, errBuf bytes.Buffer
err = writeOp(ctrClient, ctx, ctr.ID, &outBuf, &errBuf)
h.AssertNil(t, err)

err = container.Run(ctx, ctrClient, ctr.ID, &outBuf, &errBuf)
h.AssertNil(t, err)

h.AssertEq(t, errBuf.String(), "")
if osType == "windows" {
h.AssertContains(t, outBuf.String(), `01/01/1980 12:00 AM 137 ... project-metadata.toml`)
} else {
h.AssertContains(t, outBuf.String(), `-rwxr-xr-x 1 root root 137 Jan 1 1980 /layers-vol/project-metadata.toml`)
}
})

it("has expected contents", func() {
containerDir := "/layers-vol"
p := "/layers-vol/project-metadata.toml"
if osType == "windows" {
containerDir = `c:\layers-vol`
p = `c:\layers-vol\project-metadata.toml`
}

ctrCmd := []string{"cat", "/layers-vol/project-metadata.toml"}
if osType == "windows" {
ctrCmd = []string{"cmd", "/c", `type c:\layers-vol\project-metadata.toml`}
}

ctx := context.Background()
ctr, err := createContainer(ctx, imageName, containerDir, osType, ctrCmd...)
h.AssertNil(t, err)
defer cleanupContainer(ctx, ctr.ID)

writeOp := build.WriteProjectMetadata(p, platform.ProjectMetadata{
Source: &platform.ProjectSource{
Type: "project",
Version: map[string]interface{}{
"declared": "1.0.2",
},
Metadata: map[string]interface{}{
"url": "https://github.com/buildpacks/pack",
},
},
}, osType)

var outBuf, errBuf bytes.Buffer
err = writeOp(ctrClient, ctx, ctr.ID, &outBuf, &errBuf)
h.AssertNil(t, err)

err = container.Run(ctx, ctrClient, ctr.ID, &outBuf, &errBuf)
h.AssertEq(t, errBuf.String(), "")
h.AssertNil(t, err)

h.AssertContains(t, outBuf.String(), `[source]
type = "project"
[source.version]
declared = "1.0.2"
[source.metadata]
url = "https://github.com/buildpacks/pack"
`)
})
})
when("#EnsureVolumeAccess", func() {
it("changes owner of volume", func() {
h.SkipIf(t, osType != "windows", "no-op for linux")
Expand Down
2 changes: 2 additions & 0 deletions internal/build/lifecycle_execution.go
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ func (l *LifecycleExecution) Create(ctx context.Context, publish bool, dockerHos
WithArgs(repoName),
WithNetwork(networkMode),
cacheOpts,
WithContainerOperations(WriteProjectMetadata(l.mountPaths.projectPath(), l.opts.ProjectMetadata, l.os)),
WithContainerOperations(CopyDir(l.opts.AppPath, l.mountPaths.appDir(), l.opts.Builder.UID(), l.opts.Builder.GID(), l.os, true, l.opts.FileFilter)),
}

Expand Down Expand Up @@ -438,6 +439,7 @@ func (l *LifecycleExecution) newExport(repoName, runImage string, publish bool,
WithNetwork(networkMode),
cacheOpt,
WithContainerOperations(WriteStackToml(l.mountPaths.stackPath(), l.opts.Builder.Stack(), l.os)),
WithContainerOperations(WriteProjectMetadata(l.mountPaths.projectPath(), l.opts.ProjectMetadata, l.os)),
}

if publish {
Expand Down
6 changes: 4 additions & 2 deletions internal/build/lifecycle_execution_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1775,8 +1775,9 @@ func testLifecycleExecution(t *testing.T, when spec.G, it spec.S) {
configProvider := fakePhaseFactory.NewCalledWithProvider[lastCallIndex]
h.AssertSliceContains(t, configProvider.HostConfig().Binds, expectedBinds...)

h.AssertEq(t, len(configProvider.ContainerOps()), 1)
h.AssertEq(t, len(configProvider.ContainerOps()), 2)
h.AssertFunctionName(t, configProvider.ContainerOps()[0], "WriteStackToml")
h.AssertFunctionName(t, configProvider.ContainerOps()[1], "WriteProjectMetadata")
})

it("configures the phase with default process type", func() {
Expand Down Expand Up @@ -2009,8 +2010,9 @@ func testLifecycleExecution(t *testing.T, when spec.G, it spec.S) {
configProvider := fakePhaseFactory.NewCalledWithProvider[lastCallIndex]
h.AssertSliceContains(t, configProvider.HostConfig().Binds, expectedBinds...)

h.AssertEq(t, len(configProvider.ContainerOps()), 1)
h.AssertEq(t, len(configProvider.ContainerOps()), 2)
h.AssertFunctionName(t, configProvider.ContainerOps()[0], "WriteStackToml")
h.AssertFunctionName(t, configProvider.ContainerOps()[1], "WriteProjectMetadata")
})

it("configures the phase with default process type", func() {
Expand Down
3 changes: 3 additions & 0 deletions internal/build/lifecycle_executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (

"github.com/buildpacks/imgutil"
"github.com/buildpacks/lifecycle/api"
"github.com/buildpacks/lifecycle/platform"
"github.com/docker/docker/client"
"github.com/google/go-containerregistry/pkg/name"

Expand Down Expand Up @@ -56,6 +57,8 @@ type LifecycleOptions struct {
Builder Builder
LifecycleImage string
RunImage string
ProjectMetadata platform.ProjectMetadata
ProjectPath string
ClearCache bool
Publish bool
TrustBuilder bool
Expand Down
4 changes: 4 additions & 0 deletions internal/build/mount_paths.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ func (m mountPaths) stackPath() string {
return m.join(m.layersDir(), "stack.toml")
}

func (m mountPaths) projectPath() string {
return m.join(m.layersDir(), "project-metadata.toml")
}

func (m mountPaths) appDirName() string {
return m.workspace
}
Expand Down
11 changes: 11 additions & 0 deletions internal/builder/metadata.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ type Metadata struct {
Stack StackMetadata `json:"stack"`
Lifecycle LifecycleMetadata `json:"lifecycle"`
CreatedBy CreatorMetadata `json:"createdBy"`
Project ProjectMetadata `json:"source,omitempty"`
}

type CreatorMetadata struct {
Expand All @@ -30,6 +31,16 @@ type StackMetadata struct {
RunImage RunImageMetadata `json:"runImage" toml:"run-image"`
}

type ProjectMetadata struct {
Source ProjectSource `json:"source,omitempty" toml:"source"`
}

type ProjectSource struct {
Type string `toml:"type" json:"type,omitempty"`
Version map[string]interface{} `toml:"version" json:"version,omitempty"`
Metadata map[string]interface{} `toml:"metadata" json:"metadata,omitempty"`
}

type RunImageMetadata struct {
Image string `json:"image" toml:"image"`
Mirrors []string `json:"mirrors" toml:"mirrors"`
Expand Down
4 changes: 4 additions & 0 deletions project.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[project]
version = "1.0.2"
source-url = "https://github.com/buildpacks/pack"

8 changes: 6 additions & 2 deletions project/project.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@ import (
"path/filepath"

"github.com/BurntSushi/toml"

"github.com/pkg/errors"

//"github.com/buildpacks/lifecycle/platform"
"github.com/buildpacks/pack/internal/dist"
)

Expand Down Expand Up @@ -37,8 +39,10 @@ type Build struct {
}

type Project struct {
Name string `toml:"name"`
Licenses []dist.License `toml:"licenses"`
Name string `toml:"name"`
Version string `toml:"version"`
SourceURL string `toml:"source-url"`
Licenses []dist.License `toml:"licenses"`
}

type Descriptor struct {
Expand Down
Loading