Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kfctl support relative path #4331

Merged
merged 6 commits into from
Oct 18, 2019
Merged

kfctl support relative path #4331

merged 6 commits into from
Oct 18, 2019

Conversation

lluunn
Copy link
Contributor

@lluunn lluunn commented Oct 17, 2019

use neturl.Parse instead of ParseRequestURI because the latter only works with abs path.

https://golang.org/pkg/net/url/#ParseRequestURI

/cc @jlewi @richardsliu

kubeflow/kfctl#49


This change is Reviewable


log.Infof("Downloading %v to %v", configFile, appFile)
err = gogetter.GetFile(appFile, configFile)
// Check if configFile is a remote file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we refactor the code to make this easier to test?

One thought would be to define a function isRemoteFile and then we could add a unittest to cover the cases

  • remote URI
  • relative path
  • absolute path.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@lluunn
Copy link
Contributor Author

lluunn commented Oct 17, 2019

test error:

NFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py|92| Importing path kubeflow.kubeflow.ci.kfctl_e2e_workflow
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| Traceback (most recent call last):
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| "__main__", fname, loader, pkg_name)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| exec code in run_globals
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/src/kubeflow/testing/py/kubeflow/testing/e2e_tool.py", line 154, in <module>
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| fire.Fire(E2EToolMain)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/usr/local/lib/python2.7/dist-packages/fire/core.py", line 138, in Fire
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| component_trace = _Fire(component, args, parsed_flag_args, context, name)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/usr/local/lib/python2.7/dist-packages/fire/core.py", line 471, in _Fire
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| target=component.__name__)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/usr/local/lib/python2.7/dist-packages/fire/core.py", line 675, in _CallAndUpdateTrace
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| component = fn(*varargs, **kwargs)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/src/kubeflow/testing/py/kubeflow/testing/e2e_tool.py", line 29, in show
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| workflow = run_e2e_workflow.py_func_import(py_func, kwargs)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 95, in py_func_import
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| return met(**kwargs)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| File "/src/kubeflow/kubeflow/py/kubeflow/kubeflow/ci/kfctl_e2e_workflow.py", line 624, in create_workflow
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| builder = Builder(**kwargs)
INFO|2019-10-17T23:39:40|/src/kubeflow/testing/py/kubeflow/testing/util.py|69| TypeError: __init__() got an unexpected keyword argument 'test_target_name'
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 521, in <module>
    final_result = main()
  File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 511, in main
    return run(args, file_handler)
  File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 364, in run
    util.run(command, env=env)
  File "/src/kubeflow/testing/py/kubeflow/testing/util.py", line 85, in run
    " ".join(command), process.returncode), "\n".join(output))
subprocess.CalledProcessError: Command 'cmd: python -m kubeflow.testing.e2e_tool show kubeflow.kubeflow.ci.kfctl_e2e_workflow.create_workflow --test-target-name=kfctl-go-iap-istio --namespace=kubeflow-test-infra --use-basic-auth=False --name=kubeflow-presubmit-kfctl-go-iap-istio-4331-fe6d3cf-6304-f6df --config-path=https://raw.githubusercontent.com/kubeflow/manifests/master/kfdef/kfctl_gcp_iap.yaml --output=/tmp/tmpfF29yV exited with code 1' returned non-zero exit status 1

related to kubeflow/testing#490?

@lluunn
Copy link
Contributor Author

lluunn commented Oct 17, 2019

/retest

1 similar comment
@lluunn
Copy link
Contributor Author

lluunn commented Oct 18, 2019

/retest

}

// If the config file is a remote URI, check to see if the current directory
// is empty because we will be generating the KfApp there.
cwd := ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why initialie to "" as opposed to getCwd()?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the current directory or is it a bool indicating whether its empty?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Should we clean this code up a bit? isCwdEmpty is returning the current directory but only if its empty.
That's a bit convulated. Should we change isCwdEmpty to isDirEmpty(cwd) and return a bool

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Code: int(kfapis.INVALID_ARGUMENT),
Message: fmt.Sprintf("Error parsing file path: %v", err),
}
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: move the code out of the else block since code in the if statement will never reach it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

filePath: "http://github.com",
isRemote: true,
},
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the test case "abc.txt"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

@jlewi jlewi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 4 files reviewed, 5 unresolved discussions (waiting on @jlewi, @lluunn, and @richardsliu)


bootstrap/pkg/kfapp/coordinator/coordinator.go, line 352 at r2 (raw file):

	if isRemoteFile {
		cwd, err = os.Getwd()
		if err != nil {

I think we can replace this if/else block and just do

c.kfDef.Spec.Appdir= path.Dir(appFile)

At this point appFile should be the path to a local File because

  1. Either cfgFile is a local file and we set appFile=localFile
  2. We copied the remote file to the current directory and appFile is the local path it was copied to.

@lluunn
Copy link
Contributor Author

lluunn commented Oct 18, 2019

/hold
testing locally

@lluunn
Copy link
Contributor Author

lluunn commented Oct 18, 2019

/hold cancel

@jlewi
Copy link
Contributor

jlewi commented Oct 18, 2019

/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 5c551fe into kubeflow:master Oct 18, 2019
saffaalvi pushed a commit to StatCan/kubeflow that referenced this pull request Feb 11, 2021
* support relative path

* refactor util, add unit test

* address review

* fix

* fix
saffaalvi pushed a commit to StatCan/kubeflow that referenced this pull request Feb 12, 2021
* support relative path

* refactor util, add unit test

* address review

* fix

* fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants