Skip to content
This repository has been archived by the owner on May 14, 2024. It is now read-only.

[DEPR]: Deployment of Blockstore as an independent service #296

Closed
kdmccormick opened this issue Oct 27, 2023 · 10 comments
Closed

[DEPR]: Deployment of Blockstore as an independent service #296

kdmccormick opened this issue Oct 27, 2023 · 10 comments
Assignees
@feanil @kdmccormick
Labels
depr Proposal for deprecation & removal per OEP-21 duplicate This issue or pull request already exists elsewhere

Comments

@kdmccormick
Copy link
Contributor

kdmccormick commented Oct 27, 2023 •
edited

2024 UPDATE: This was Accepted, but is also superseded by:

which will remove all of Blockstore and all references to it.

Proposal Date

2023-10-27

Target Ticket Acceptance Date

2023-11-10

Earliest Open edX Named Release Without This Functionality

Redwood

Rationale

It was decided three years ago that Blockstore, an authoring-oriented storage backend, makes more sense as an installable app than it does as an independently-deployable micro-service.

Since then, it has been both possible to deploy it as a service and install as an application, but this dual-capability comes at the high cost of layers of wrapper APIs, mostly-dead caching code, test suites that are 80% skipped, and complicated deployment instructions leading to superfluous tooling code,

Removal

#### Removal Tasks
- [x] In edx-platform: [remove the use_blockstore_app_api toggle](https://docs.openedx.org/projects/edx-platform/en/latest/references/featuretoggles.html#featuretoggle-blockstore.use_blocksto.re_app_api), keeping the "true" codepaths
- [x] In edx-platform: Remove content libraries V2 indexing and related settings.
- [x] In edx-platform: Remove `@skips` on content_libraries tests, which currently rely on Blockstore running as a micro-service. Make sure tests all pass. If some are seriously broken, put `@skip` back in and create follow-up tickets.
- [x] In edx-platform: Remove [blockstore methods wrapper](https://github.com/openedx/edx-platform/blob/master/openedx/core/lib/blockstore_api/methods.py).
- [ ] In edx-platform: Remove the [blockstore_api tests](https://github.com/openedx/edx-platform/tree/master/openedx/core/lib/blockstore_api/tests). If any mixins there are still needed for different tests, move them elsewhere. For example, they could be moved to the content_libraries app, or maybe a test
- [ ] In edx-platform: Remove the [blockstore_api wrapper module](https://github.com/openedx/edx-platform/blob/master/openedx/core/lib/blockstore_api/__init__.py). Replace imports to it with direct imports to the underlying blockstore functions.
- [ ] In edx-platform: In the [content_libraries app](https://github.com/openedx/edx-platform/blob/master/openedx/core/djangoapps/content_libraries)'s tests, there are a few classes [like this one](https://github.com/openedx/edx-platform/blob/aaea6e5/openedx/core/djangoapps/content_libraries/tests/test_content_libraries.py#L1213-L1219) which needlessly delegate out to a mixin. This is leftover from when blockstore was both an app & service. Collapse each mixin's body into its test class.
- [ ] In edx-platform: Move [blockstore db_routers](https://github.com/openedx/edx-platform/blob/master/openedx/core/lib/blockstore_api/db_routers.py) so that the blockstore_api folder can be deleted. Suggested new location: openedx/core/lib/blockstore_db_routers.py
- [ ] ~In blockstore: Update [openedx.yaml](https://github.com/openedx/blockstore~/blob/aca35cafed182b0b28197fdd9427d00bc20742a9/openedx.yaml#L12) so it isn't tagged in release.~
- [ ] ~In blockstore: Clean up the [readme](https://github.com/openedx/blockstore/tree/master#blockstore).~
- [ ] ~In blockstore: Remove all files, Makefile targets, and dotfile lines relating to of devstack, docker, and docker-compose .~
- [ ] ~In blockstore: Remove [settings files](https://github.com/openedx/blockstore/tree/master/blockstore/settings).~
- [ ] ~In blockstore: Remove [wsgi config](https://github.com/openedx/blockstore/blob/master/blockstore/wsgi.py).~
- [ ] ~In blockstore: Look through [base requirements](https://github.com/openedx/blockstore/blob/master/requirements/base.in) and remove any that was there just because it was a micro-service.~
- [ ] ~In blockstore: Update [this catalog-info comment](https://github.com/openedx/blockstore/blob/aca35cafed182b0b28197fdd9427d00bc20742a9/catalog-info.yaml#L17).~
- [ ] ~In blockstore: Do a final sweep of the code and docs to make sure there's nothing left there that's specific to running Blockstore as a service.~
- [ ] In frontend-app-library-authoring: Remove the part of the readme that tells you to run blockstore ("Now set up blockstore. Blockstore...").
- [ ] In edx-platform and tutor: Set configuration defaults so that blockstore (the app) works out-of-the-box using filesystem storage.
- [ ] In openedx-tutor-plugins: remove any parts of the tutor-contrib-blockstore-minio plugin that are now redundant.
- [ ] In openedx-tutor-plugins: remove the tutor-contrib-blockstore-filesystem plugin.

Replacement

Blockstore will continue to exist and be actively maintained as an installable Django app.

Deprecation

No plans to mark deprecation.

Migration

We will not provide any migration instructions because we are not aware of anyone using Blockstore as a micro-service.

If we are wrong and you do run Blockstore as a micro-service, please reach out and we can help you figure out how to merge your Blockstore database into your LMS/CMS database.

Additional Info

No response
@github-actions github-actions bot added the depr Proposal for deprecation & removal per OEP-21 label Oct 27, 2023
@kdmccormick kdmccormick self-assigned this Oct 30, 2023
@kdmccormick kdmccormick mentioned this issue Nov 1, 2023
Open
@kdmccormick
Copy link
Contributor Author

This has been accepted. The removal will happen during the current phase or next phase of the Content Libraries V2 project.

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue Nov 21, 2023
@kdmccormick
feat!: assume & remove BLOCKSTORE_USE_BLOCKSTORE_APP_API
a2d6c5c 
TODO add details

Ref: openedx-unsupported/blockstore#296
@kdmccormick kdmccormick mentioned this issue Nov 21, 2023
Merged
@kdmccormick
Copy link
Contributor Author

First of several: openedx/edx-platform#33765

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue Nov 27, 2023
@kdmccormick
feat!: assume & remove BLOCKSTORE_USE_BLOCKSTORE_APP_API
c5850d8 
TODO add details

Ref: openedx-unsupported/blockstore#296
kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue Dec 6, 2023
@kdmccormick
feat!: assume & remove BLOCKSTORE_USE_BLOCKSTORE_APP_API
424c0c9 
TODO add details

Ref: openedx-unsupported/blockstore#296
kdmccormick added a commit to openedx/edx-platform that referenced this issue Dec 6, 2023
@kdmccormick
feat!: assume & remove BLOCKSTORE_USE_BLOCKSTORE_APP_API (#33765)
27803f5 
Originally, Blockstore was an independent micro-service, accessed via a REST API.
Then, we changed Blockstore so it could be installed as an in-process Django app.

To support both modes, there existed a blockstore_api wrapper library in edx-platform,
with toggles controlling whether the wrapper called out to the micro-service's REST API versus the
Django app's Python API. Now that the micro-service Blockstore implementation is deprecated,
though, this wrapper library and toggles are just unnecessary complexity.

As a first step towards cleanup, we:

* remove several toggles and settings (details below);
* remove the blocokstore_api wrapper methods which called the REST API and
  marshalled them back into Python objects; and
* remove all test cases which relied on the Blockstore micro-service (and were skippped in CI).

In the future, we will remove the content libraries indexer, 
clean up the remaining bits of blockstore_api, and flatten out all
the Blockstore-related test class hierarchies which are no longer nceessary.

BREAKING CHANGE:
* These Django settings are removed:
  * BLOCKSTORE_PUBLIC_URL_ROOT
  * BLOCKSTORE_API_URL
  * BLOCKSTORE_API_AUTH_TOKEN
  * BLOCKSTORE_USE_BLOCKSTORE_APP_API
* The blockstore.use_blockstore_app_api Waffle switch is removed.
* edx-platform will act as it did when the DJango setting BLOCKSTORE_USE_BLOCKSTORE_APP_API
  or the Waffle switch blockstore.use_blockstore_app_api were enabled. That is, any running Blockstore
  micro-service instance will be ignored, and the Blockstore package which is installed into edx-platform
  will be used instead.

Ref: openedx-unsupported/blockstore#296
@connorhaugh
Copy link
Contributor

We should also move the django admin tables for the blockstore items to be viewable in the edx-platform django admin.

@kdmccormick
Copy link
Contributor Author

@connorhaugh I have this in my CMS admin (local tutor):

image

@kdmccormick
Copy link
Contributor Author

Indexing was removed: openedx/edx-platform#33888

@feanil feanil self-assigned this Dec 14, 2023
fazledyn-or pushed a commit to fazledyn-or/edx-platform that referenced this issue Dec 18, 2023
@kdmccormick @fazledyn-or
feat!: assume & remove BLOCKSTORE_USE_BLOCKSTORE_APP_API (openedx#33765)
6cdf63a 
Originally, Blockstore was an independent micro-service, accessed via a REST API.
Then, we changed Blockstore so it could be installed as an in-process Django app.

To support both modes, there existed a blockstore_api wrapper library in edx-platform,
with toggles controlling whether the wrapper called out to the micro-service's REST API versus the
Django app's Python API. Now that the micro-service Blockstore implementation is deprecated,
though, this wrapper library and toggles are just unnecessary complexity.

As a first step towards cleanup, we:

* remove several toggles and settings (details below);
* remove the blocokstore_api wrapper methods which called the REST API and
  marshalled them back into Python objects; and
* remove all test cases which relied on the Blockstore micro-service (and were skippped in CI).

In the future, we will remove the content libraries indexer, 
clean up the remaining bits of blockstore_api, and flatten out all
the Blockstore-related test class hierarchies which are no longer nceessary.

BREAKING CHANGE:
* These Django settings are removed:
  * BLOCKSTORE_PUBLIC_URL_ROOT
  * BLOCKSTORE_API_URL
  * BLOCKSTORE_API_AUTH_TOKEN
  * BLOCKSTORE_USE_BLOCKSTORE_APP_API
* The blockstore.use_blockstore_app_api Waffle switch is removed.
* edx-platform will act as it did when the DJango setting BLOCKSTORE_USE_BLOCKSTORE_APP_API
  or the Waffle switch blockstore.use_blockstore_app_api were enabled. That is, any running Blockstore
  micro-service instance will be ignored, and the Blockstore package which is installed into edx-platform
  will be used instead.

Ref: openedx-unsupported/blockstore#296
@kdmccormick
Copy link
Contributor Author

@feanil It actually looks like we will be deprecating Blockstore entirely in favor of Learning Core. I will write that up soon and see if there is any dissent.

In the meantime, I recommend that you folks stick with the parts of this ticket that are outside Blockstore repo, as the Blockstore repo itself will probably just be archived. I went ahead and crossed out the Blockstore-repo tasks.

@feanil
Copy link
Contributor

feanil commented Dec 19, 2023

Sounds good, thanks @kdmccormick. Is it worth it to update the tutor plugin or should we just leave that alone as that will get removed as well?

@kdmccormick
Copy link
Contributor Author

@feanil I think they're still worth doing because it'll remove some developer complexity as we port V2 libraries from Blockstore to Learning Core in the coming months, but also no big deal if you're not able to get to them. The edx-platform and frontend-app-library-authoring cleanup is the only stuff that I'd say is really important.

@kdmccormick kdmccormick added the duplicate This issue or pull request already exists elsewhere label Feb 1, 2024
@kdmccormick
Copy link
Contributor Author

2024 UPDATE: This was Accepted, but is also superseded by:

which will remove all of Blockstore and all references to it.

@kdmccormick kdmccormick closed this as not planned Won't fix, can't repro, duplicate, stale Feb 1, 2024
@kdmccormick kdmccormick mentioned this issue Feb 15, 2024
Open
@kdmccormick
Copy link
Contributor Author

Superseded by openedx/public-engineering#238

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@feanil feanil

@kdmccormick kdmccormick

Labels
depr Proposal for deprecation & removal per OEP-21 duplicate This issue or pull request already exists elsewhere
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@feanil @kdmccormick @connorhaugh