Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: mlflow-e2e job taking too long to complete #174

Closed
flaviodsr opened this issue Jul 12, 2021 · 2 comments · Fixed by #282
Closed

CI: mlflow-e2e job taking too long to complete #174

flaviodsr opened this issue Jul 12, 2021 · 2 comments · Fixed by #282
Labels
area/ci good first issue Good for newcomers help wanted Extra attention is needed
Projects
FuseML Project Board

Comments

@flaviodsr
Copy link
Member

Since #171 the mlflow-e2e job is taking too long to complete.

This behaviour can only be reproduced when running the workflow on github actions hosts as the CI does and it seems to be related to the change in the kaniko task to push using --insecure.
@stefannica
Copy link
Member

More info taken from slack...

with --skip-tls-verify:

INFO[0154] Pushing image to registry.fuseml-registry/mlflow-builder/mlflow-wines:main 
INFO[0184] Pushed image to 1 destinations

took 2.3m to build and 30s to push

with --insecure:

INFO[0322] Pushing image to registry.fuseml-registry/mlflow-builder/mlflow-wines:main 
INFO[1118] Pushed image to 1 destinations

took ~5min to build and 13 minutes to push

@stefannica stefannica added this to Backlog in FuseML Project Board via automation Jul 13, 2021
@stefannica stefannica added good first issue Good for newcomers help wanted Extra attention is needed labels Jul 13, 2021
flaviodsr added a commit to flaviodsr/fuseml that referenced this issue Oct 22, 2021
@flaviodsr
fix: Map https port on registry (trow)
6c33c27 
When pushing an image kaniko performs a live check against the registry,
this check is first done through the https port and if there is no reply
it waits until the request times out then procceeds with http. This is
the case even when using the `--insecure` flag, resulting in
fuseml#174

To work around that, this commit maps the 443 port to the registry
container so when kaniko performs the live check the container can reply
with connection refused as there is no proccess listening on 443.

Fixes: fuseml#174
flaviodsr added a commit to flaviodsr/fuseml that referenced this issue Oct 22, 2021
@flaviodsr
fix: Map https port on registry (trow)
db6e1d4 
When pushing an image, kaniko performs a live check against the registry,
this check is first done through the https port and if there is no reply
it waits until the request times out then proceeds with http. This is
the case even when using the `--insecure` flag, resulting in
fuseml#174

To work around that, this commit maps the 443 port to the registry
container so when kaniko performs the live check the container can reply
with connection refused as there is no proccess listening on 443.

Fixes: fuseml#174
@flaviodsr
Copy link
Member Author

When pushing an image, kaniko performs a live check against the registry,
this check is first done through the https port and if there is no reply
it waits until the request times out then proceeds with http. This is
the case even when using the --insecure flag, resulting in this issue.

@flaviodsr flaviodsr mentioned this issue Oct 22, 2021
Merged
FuseML Project Board automation moved this from Backlog to Done Oct 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/ci good first issue Good for newcomers help wanted Extra attention is needed
Projects
FuseML Project Board
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Map https port on registry (trow) flaviodsr/fuseml
2 participants
@stefannica @flaviodsr