Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly convert unix time into meta time #480

Merged
merged 3 commits into from
Aug 16, 2022
Merged

Conversation

pingsutw
Copy link
Contributor

Why are these changes needed?

Failed to convert Unix time to meta time.

2022-08-16T10:29:43.043Z	DEBUG	controllers.RayJob	RayJob information	{"RayJob": "rayjob-sample", "jobInfo": {"status":"SUCCEEDED","entrypoint":"python /home/ray/samples/sample_code.py","message":"Job finished successfully.","start_time":1660645691607,"end_time":1660645710418}, "rayJobInstance": "PENDING"}
2022-08-16T10:29:43.043Z	INFO	controllers.RayJob	Update status from PENDING to SUCCEEDED	{"rayjob": "rayjob-sample-z7j5q"}
2022-08-16T10:29:43.043Z	INFO	controllers.RayJob	UpdateState	{"oldJobStatus": "PENDING", "newJobStatus": "SUCCEEDED", "oldJobDeploymentStatus": "Running", "newJobDeploymentStatus": "Running"}
2022-08-16T10:29:43.057Z	ERROR	controller.rayjob	Reconciler error	{"reconciler group": "ray.io", "reconciler kind": "RayJob", "name": "rayjob-sample", "namespace": "default", "error": "combined error: <nil> RayJob.ray.io \"rayjob-sample\" is invalid: [status.startTime: Invalid value: \"54593-10-07T05:53:27Z\": status.startTime in body must be of type date-time: \"54593-10-07T05:53:27Z\", status.endTime: Invalid value: \"54593-10-07T11:06:58Z\": status.endTime in body must be of type date-time: \"54593-10-07T11:06:58Z\"]", "errorVerbose": "combined error: <nil> RayJob.ray.io \"rayjob-sample\" is invalid: [status.startTime: Invalid value: \"54593-10-07T05:53:27Z\": status.startTime in body must be of type date-time: \"54593-10-07T05:53:27Z\", status.endTime: Invalid value: \"54593-10-07T11:06:58Z\": status.endTime in body must be of type date-time: \"54593-10-07T11:06:58Z\"]\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayJobReconciler).updateState\n\t/workspace/controllers/ray/rayjob_controller.go:311\ngithub.com/ray-project/kuberay/ray-operator/controllers/ray.(*RayJobReconciler).Reconcile\n\t/workspace/controllers/ray/rayjob_controller.go:181\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.1/pkg/internal/controller/controller.go:227

We should divide Unix time by 1000 because Ray job info returns the start_time in ms format.
https://docs.ray.io/en/latest/cluster/jobs-package-ref.html#jobinfo

Related issue number

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Copy link
Collaborator

@Jeffwan Jeffwan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the improvement!

@Jeffwan
Copy link
Collaborator

Jeffwan commented Aug 16, 2022

@pingsutw Do you rebase remote/master to your HEAD. the last test case doesn't go through, It's definitely not the issue of your code. I am just curious if there's any release blocker problem.

@pingsutw
Copy link
Contributor Author

Merged master to this branch. Hope the tests will pass.

@Jeffwan Jeffwan added this to the v0.3.0 release milestone Aug 16, 2022
@Jeffwan Jeffwan merged commit 094209b into ray-project:master Aug 16, 2022
Jeffwan pushed a commit to Jeffwan/kuberay that referenced this pull request Aug 16, 2022
* Add timeout to the query parameters

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Properly convert unix time into meta time

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>
Jeffwan added a commit that referenced this pull request Aug 17, 2022
* Fix panic in cleanupInvalidVolumeMounts  (#481)

* fix

* assert

* refine

* Properly convert unix time into meta time (#480)

* Add timeout to the query parameters

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Properly convert unix time into meta time

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: WanXing Wang <wangwanxing@bytedance.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023
* Add timeout to the query parameters

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Properly convert unix time into meta time

Signed-off-by: Kevin Su <pingsutw@apache.org>

Signed-off-by: Kevin Su <pingsutw@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants