New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raw block device using persistent volume source #351

Open
erinboyd opened this Issue Jul 12, 2017 · 47 comments

Comments

@erinboyd

erinboyd commented Jul 12, 2017

Feature Description

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Jul 12, 2017

Contributor

@erinboyd: Reiterating the mentions to trigger a notification:
@kubernetes/sig-storage-feature-requests, @kubernetes/sig-apps-feature-requests.

In response to this:

Feature Description

  • One-line feature description (can be used as a release note): Make local raw block devices (non-networked) available for consumption via a Persistent Volume Source
  • Primary contact (assignee): @erinboyd, @screeley44, @mtanino
  • Responsible SIGs: @kubernetes/sig-storage-feature-requests @kubernetes/sig-apps-feature-requests
  • Design proposal link (community repo):
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
    @msau42
    @jsafrane
  • Approver (likely from SIG/area to which feature belongs):
    @saad-ali
  • Feature target (which target equals to which milestone):
  • Alpha release target (x.y): 1.8
  • Beta release target (x.y): 1.9
  • Stable release target (x.y): 1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Contributor

k8s-ci-robot commented Jul 12, 2017

@erinboyd: Reiterating the mentions to trigger a notification:
@kubernetes/sig-storage-feature-requests, @kubernetes/sig-apps-feature-requests.

In response to this:

Feature Description

  • One-line feature description (can be used as a release note): Make local raw block devices (non-networked) available for consumption via a Persistent Volume Source
  • Primary contact (assignee): @erinboyd, @screeley44, @mtanino
  • Responsible SIGs: @kubernetes/sig-storage-feature-requests @kubernetes/sig-apps-feature-requests
  • Design proposal link (community repo):
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
    @msau42
    @jsafrane
  • Approver (likely from SIG/area to which feature belongs):
    @saad-ali
  • Feature target (which target equals to which milestone):
  • Alpha release target (x.y): 1.8
  • Beta release target (x.y): 1.9
  • Stable release target (x.y): 1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@saad-ali saad-ali added this to the 1.8 milestone Jul 12, 2017

@mtanino

This comment has been minimized.

Show comment
Hide comment
@mtanino

mtanino Jul 25, 2017

Member

@idvoretskyi Could you assign me to this feature too?

Member

mtanino commented Jul 25, 2017

@idvoretskyi Could you assign me to this feature too?

@msau42

This comment has been minimized.

Show comment
Hide comment
@msau42

msau42 Jul 25, 2017

Member

Hi Erin, the feature you're working on is not specific to local storage right? There is work specific to the local storage plugin that I am coordinating. Just want to make sure that it doesn't conflict with the work you're doing.

Member

msau42 commented Jul 25, 2017

Hi Erin, the feature you're working on is not specific to local storage right? There is work specific to the local storage plugin that I am coordinating. Just want to make sure that it doesn't conflict with the work you're doing.

@idvoretskyi

This comment has been minimized.

Show comment
Hide comment
@idvoretskyi

idvoretskyi Jul 26, 2017

Member

@mtanino it seems to be not possible before you'll join Kubernetes organization - https://github.com/kubernetes/community/blob/master/community-membership.md. Sorry, It's a GitHub restriction.

Member

idvoretskyi commented Jul 26, 2017

@mtanino it seems to be not possible before you'll join Kubernetes organization - https://github.com/kubernetes/community/blob/master/community-membership.md. Sorry, It's a GitHub restriction.

@mtanino

This comment has been minimized.

Show comment
Hide comment
@mtanino

mtanino Jul 26, 2017

Member

@idvoretskyi Thank you for the explanation. I'll try to send membership request.

Member

mtanino commented Jul 26, 2017

@idvoretskyi Thank you for the explanation. I'll try to send membership request.

@ddysher

This comment has been minimized.

Show comment
Hide comment
@ddysher

ddysher Jul 28, 2017

@erinboyd I have the same question as @msau42, is this specific to local block or this is about block support in general?

ddysher commented Jul 28, 2017

@erinboyd I have the same question as @msau42, is this specific to local block or this is about block support in general?

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Jul 31, 2017

Member

@erinboyd I just saw this. What is required from SIG-Apps?

Member

erictune commented Jul 31, 2017

@erinboyd I just saw this. What is required from SIG-Apps?

@danpaik

This comment has been minimized.

Show comment
Hide comment
@danpaik

danpaik Aug 22, 2017

for sig-apps is it mostly testing w/ databases + raw block storage to see if we see the intended benefits as well as testing security?

danpaik commented Aug 22, 2017

for sig-apps is it mostly testing w/ databases + raw block storage to see if we see the intended benefits as well as testing security?

@mtanino

This comment has been minimized.

Show comment
Hide comment
@mtanino

mtanino Aug 22, 2017

Member

@danpaik @erictune
I agree on above.

@erinboyd
Anything else?

Member

mtanino commented Aug 22, 2017

@danpaik @erictune
I agree on above.

@erinboyd
Anything else?

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Aug 22, 2017

Member

I'm sure SIG-Apps has the necessary experience to advise on how to adapt existing database examples/charts to use the new volume type. Looking at MySQL, it looks like it won't be trivial: you need a separate step to initialize the block device, vs serving from it.

I agree that raw block access might pose some security issues, but I'm not sure SIG-Apps has the necessary expertise to evaluate security.

Member

erictune commented Aug 22, 2017

I'm sure SIG-Apps has the necessary experience to advise on how to adapt existing database examples/charts to use the new volume type. Looking at MySQL, it looks like it won't be trivial: you need a separate step to initialize the block device, vs serving from it.

I agree that raw block access might pose some security issues, but I'm not sure SIG-Apps has the necessary expertise to evaluate security.

@kow3ns

This comment has been minimized.

Show comment
Hide comment
@kow3ns

kow3ns Aug 23, 2017

Member

@etune I don't think it will be a big win for most OSS storage applications, but it will offer a migration path for users with custom or legacy applications to move their workloads onto Kuberentes.

  1. None of Postgres, Cassandra, MongoDB, Couchbase, CouchDB, Riak, ElasticSearch, Solr, Kafka, Redis,or HDFS, or HBase support the use raw block devices.
  2. Of the RDBMS systems that do (MySQL or Maria with InnoDB, Oracle, and DB2), I've only seen interest in MySQL and Maria.
  3. Of the DFSes that use it, they are all meant to run with local raw block devices. The IOPS caps for remote storage might not be practical for the most prevalent use cases.

@danpaik For InnoDB what is the optimization we hope to achieve?

  1. Are we just trying to remove the filesystem overhead?
  2. Are we trying to disable double buffering? If so, we need to map the configured page size to the disk sector size. Note that, for ext4 (data=journaled) you can probably disable double buffering if the InnoDB page size is 4k without using a raw block device.

I don't think having a helm chart that disables double buffering will necessarily be safe (or portable) as there is an underlying dependency on the page size representing an atomically writable disk sector. The sector size will be cloud provider specific, and it is currently not surfaced in any way. We should just use manifests for the purposes of testing InnoDB in this configuration, and I would propose using Sysbench for an evaluation of the measured performance.

Member

kow3ns commented Aug 23, 2017

@etune I don't think it will be a big win for most OSS storage applications, but it will offer a migration path for users with custom or legacy applications to move their workloads onto Kuberentes.

  1. None of Postgres, Cassandra, MongoDB, Couchbase, CouchDB, Riak, ElasticSearch, Solr, Kafka, Redis,or HDFS, or HBase support the use raw block devices.
  2. Of the RDBMS systems that do (MySQL or Maria with InnoDB, Oracle, and DB2), I've only seen interest in MySQL and Maria.
  3. Of the DFSes that use it, they are all meant to run with local raw block devices. The IOPS caps for remote storage might not be practical for the most prevalent use cases.

@danpaik For InnoDB what is the optimization we hope to achieve?

  1. Are we just trying to remove the filesystem overhead?
  2. Are we trying to disable double buffering? If so, we need to map the configured page size to the disk sector size. Note that, for ext4 (data=journaled) you can probably disable double buffering if the InnoDB page size is 4k without using a raw block device.

I don't think having a helm chart that disables double buffering will necessarily be safe (or portable) as there is an underlying dependency on the page size representing an atomically writable disk sector. The sector size will be cloud provider specific, and it is currently not surfaced in any way. We should just use manifests for the purposes of testing InnoDB in this configuration, and I would propose using Sysbench for an evaluation of the measured performance.

@erinboyd erinboyd changed the title from Local raw block device using persistent volume source to Raw block device using persistent volume source Aug 24, 2017

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Sep 12, 2017

Member

This did not make it in to 1.8, design was complete. Implementation will be punted to 1.8

Member

saad-ali commented Sep 12, 2017

This did not make it in to 1.8, design was complete. Implementation will be punted to 1.8

@idvoretskyi

This comment has been minimized.

Show comment
Hide comment
@idvoretskyi

idvoretskyi Sep 13, 2017

Member

@saad-ali to 1.8 or 1.9? If 1.9, let's update the milestone.

Member

idvoretskyi commented Sep 13, 2017

@saad-ali to 1.8 or 1.9? If 1.9, let's update the milestone.

@childsb childsb modified the milestones: 1.8, 1.9 Sep 13, 2017

@childsb

This comment has been minimized.

Show comment
Hide comment
@childsb

childsb Sep 13, 2017

Member

Moved to 1.9

Member

childsb commented Sep 13, 2017

Moved to 1.9

@msau42

This comment has been minimized.

Show comment
Hide comment
@msau42

msau42 Jun 21, 2018

Member

E2E tests automatically running in CI should definitely be a blocker for beta.

Member

msau42 commented Jun 21, 2018

E2E tests automatically running in CI should definitely be a blocker for beta.

@mkimuram

This comment has been minimized.

Show comment
Hide comment
@mkimuram

mkimuram Jun 21, 2018

Thank you for your comment. I will keep working on it. @msau42

Also, I would appreciate it if you could share a list of blockers for beta here, in order to make it easier for us to work together for this feature. (Or should each issue be added directory to 1.12 tracking spreadsheet?) @wongma7

mkimuram commented Jun 21, 2018

Thank you for your comment. I will keep working on it. @msau42

Also, I would appreciate it if you could share a list of blockers for beta here, in order to make it easier for us to work together for this feature. (Or should each issue be added directory to 1.12 tracking spreadsheet?) @wongma7

@wongma7

This comment has been minimized.

Show comment
Hide comment
@wongma7

wongma7 Jun 21, 2018

@mkimuram I'm not aware of any blockers for beta but yeah let's list them here, imo the tracking spreadsheet is long enough already :). Please, if anybody is aware of other blockers for beta, especially any changes we need to make to the API, let me know, as I'm picking this task up with, tbh, minimal knowledge about how the alpha phase has gone

wongma7 commented Jun 21, 2018

@mkimuram I'm not aware of any blockers for beta but yeah let's list them here, imo the tracking spreadsheet is long enough already :). Please, if anybody is aware of other blockers for beta, especially any changes we need to make to the API, let me know, as I'm picking this task up with, tbh, minimal knowledge about how the alpha phase has gone

@mkimuram

This comment has been minimized.

Show comment
Hide comment
@mkimuram

mkimuram commented Jun 21, 2018

@wongma7 Thank you!

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Jul 17, 2018

Member

@erinboyd @screeley44 @kubernetes/sig-storage-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

Member

justaugustus commented Jul 17, 2018

@erinboyd @screeley44 @kubernetes/sig-storage-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Jul 31, 2018

Member

@erinboyd @screeley44 --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

Member

justaugustus commented Jul 31, 2018

@erinboyd @screeley44 --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

@childsb

This comment has been minimized.

Show comment
Hide comment
@childsb

childsb Jul 31, 2018

Member

@justaugustus we are planning on moving this from alpha to beta in 1.12, what are you asking to update?

Member

childsb commented Jul 31, 2018

@justaugustus we are planning on moving this from alpha to beta in 1.12, what are you asking to update?

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Jul 31, 2018

Member

@childsb -- just confirming. I hadn't heard anything regarding this since the 1.11 release cycle.
I've added it to the 1.12 tracking sheet. :)

Member

justaugustus commented Jul 31, 2018

@childsb -- just confirming. I hadn't heard anything regarding this since the 1.11 release cycle.
I've added it to the 1.12 tracking sheet. :)

@zparnold

This comment has been minimized.

Show comment
Hide comment
@zparnold

zparnold Aug 20, 2018

Member

Hey there! @erinboyd I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

Member

zparnold commented Aug 20, 2018

Hey there! @erinboyd I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

@jimangel

This comment has been minimized.

Show comment
Hide comment
@jimangel

jimangel Aug 27, 2018

Member

@screeley44 @mtanino Bump for docs ☝️

Member

jimangel commented Aug 27, 2018

@screeley44 @mtanino Bump for docs ☝️

@msau42

This comment has been minimized.

Show comment
Hide comment
@msau42

msau42 Aug 27, 2018

Member

New owners
/assign @childsb @wongma7

Member

msau42 commented Aug 27, 2018

New owners
/assign @childsb @wongma7

@wongma7

This comment has been minimized.

Show comment
Hide comment
@wongma7

wongma7 Aug 27, 2018

@zparnold , placeholder/WIP PR created here: kubernetes/website#10100 .

wongma7 commented Aug 27, 2018

@zparnold , placeholder/WIP PR created here: kubernetes/website#10100 .

@zparnold

This comment has been minimized.

Show comment
Hide comment
@zparnold

zparnold Sep 1, 2018

Member

Thank you!

Member

zparnold commented Sep 1, 2018

Thank you!

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Sep 5, 2018

Member

@childsb @wongma7 --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

Member

justaugustus commented Sep 5, 2018

@childsb @wongma7 --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Sep 11, 2018

Member

@childsb @wongma7 @erinboyd @screeley44 -- Removing this from the milestone as we haven't heard anything back from you.
/milestone clear

Member

justaugustus commented Sep 11, 2018

@childsb @wongma7 @erinboyd @screeley44 -- Removing this from the milestone as we haven't heard anything back from you.
/milestone clear

@k8s-ci-robot k8s-ci-robot removed this from the v1.12 milestone Sep 11, 2018

@kacole2

This comment has been minimized.

Show comment
Hide comment
@kacole2

kacole2 Oct 8, 2018

Contributor

Hi @childsb @wongma7
This enhancement has been tracked before and slated for 1.13 graduation to stable according to the post, however, it's not on the milestone and I'm unsure if this made beta for 1.12. Any plans for this to graduate stages in Kubernetes 1.13? This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet.

Thanks!

Contributor

kacole2 commented Oct 8, 2018

Hi @childsb @wongma7
This enhancement has been tracked before and slated for 1.13 graduation to stable according to the post, however, it's not on the milestone and I'm unsure if this made beta for 1.12. Any plans for this to graduate stages in Kubernetes 1.13? This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet.

Thanks!

@childsb childsb added this to the v1.13 milestone Oct 8, 2018

@childsb

This comment has been minimized.

Show comment
Hide comment
@childsb

childsb Oct 8, 2018

Member

@kacole2 this missed 1.12 and is planned for 1.13.. i added the 1.13 milestone

Member

childsb commented Oct 8, 2018

@kacole2 this missed 1.12 and is planned for 1.13.. i added the 1.13 milestone

@kacole2 kacole2 added the tracked/yes label Oct 8, 2018

@AishSundar

This comment has been minimized.

Show comment
Hide comment
@AishSundar

AishSundar Oct 17, 2018

@childsb what work if left in k/k for this feature to go to beta in 1.13? Is there a list of pending PRs (code, tests and docs) that we can track on our end?

AishSundar commented Oct 17, 2018

@childsb what work if left in k/k for this feature to go to beta in 1.13? Is there a list of pending PRs (code, tests and docs) that we can track on our end?

@AishSundar

This comment has been minimized.

Show comment
Hide comment
@AishSundar

AishSundar Oct 17, 2018

@kacole2 this feature is marked as stable in the spreadsheet, while its actually going to Beta in 1.13

AishSundar commented Oct 17, 2018

@kacole2 this feature is marked as stable in the spreadsheet, while its actually going to Beta in 1.13

@wongma7

This comment has been minimized.

Show comment
Hide comment
@wongma7

wongma7 Oct 17, 2018

@AishSundar the k/k PR for code and tests is here kubernetes/kubernetes#65829 (needs API review). docs have not been started

wongma7 commented Oct 17, 2018

@AishSundar the k/k PR for code and tests is here kubernetes/kubernetes#65829 (needs API review). docs have not been started

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment