-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement IndexedDB. #6963
Comments
I like to work on it. |
cc @nox Note that this is a pretty large project. To start, you'll need to pick a backend and write Rust bindings for it. We were thinking SQLite4 or a LevelDB fork, probably the former. |
@farodin91 We don't support subdirectories in dom/ yet; I don't think it's worth blocking on that. |
@jdm Technically we just need to use reexports and it will work fine. |
@Manishearth SQLite 4 suffers from a complete lack of bindings and my days are unfortunately made of 24 hours like everyone. :( |
@Manishearth I can start with building an rust api for SQLite 4. |
That sounds valuable :) |
I like to write a binding in rust for SQLite. Any prefer which lib to use for it? |
There is only one SQLite 4 😄 |
This is my first time to create an rust binding. Is there a small example for it |
https://doc.rust-lang.org/book/ffi.html is a good start. |
Please note that SQLite resides in a Fossil repository. I had started writing bindings for them, but it's very primitive for now. I can push that somewhere this week if you want. |
I create an repo for this. I can start to work on it this week. |
@nox Any update on pushing the in my repo? |
Sorry things were a bit hectic for me this week. |
No problem |
I start work on it https://github.com/farodin91/sqlite-rs. But currently my build script doesn't work to build sqlite4 on linux. I have no idea. |
Any update to pushing in my repo? |
@nox Any updates? |
I'm picking this up. Will start by writing SQLite 4 Rust bindings. |
It seems there is no commit in SQLite 4 since 2015-08-15 and never had an official release, doesn't levelDB seems more mature (already used for indexeddb in Chrome) ? |
That might just be because they intend SQLite 4 to be stable. SQLite is under active development, see https://www.sqlite.org/src/timeline . That might be SQLite 3, but I think the codebase is mostly shared. |
I'm going to use SQLite3. For database storage, we have two ways:
For indices, we also have two ways, the old Gecko way and a very fancy way that I would like to experiment with:
Some interesting links: |
This would also require a reimplementation of the serialised structured clone format described here (or something similar to it that we can store and introspect directly from SQLite). |
We can't use |
There are various things necessary on the Servo side, at least these three things:
|
Blocks #13942 |
hi @nox, do you know if the issue has advanced any further? Thanks! |
It has not. |
I am looking for a new task, and I thought this one was interessting. I have a few questions before I am comfortable claiming this issue. As I understand it, IndexedDB should be implemented using a WAL to favor performance. This decision was taken in Gecko, and is seen in Chrome/IE. See change here. Despite the new 3.0 specification states it to be durable. Since SQLite4 never got released, and SQLite3 not being the obvious choice, perhaps RocksDB is a valid option? It is wildly used, and seems to fit indexeddb and servo quite well, i.e. fast and concurrent; embedded KV store. It also has great rust bindings through rust-rocksdb used by e.g. TiDB. This would make it a lot easier to get a working IndexedDB. Thoughts? |
Seems like it's worth a shot! |
@highfive: assign me |
Hey @rasviitanen! Thanks for your interest in working on this issue. It's now assigned to you! |
Just to provide some context on SQLite and other storage choices as they're being used in Gecko/Firefox: Re: SQLite
Re: other options:
Re: selfish concerns on my part ;)
|
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Basic IndexedDB Functionality <!-- Please describe your changes on the following line: --> This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. --- <!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: --> - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963 <!-- Either: --> - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ... <!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.--> <!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
(Just filing as a placeholder.)
The text was updated successfully, but these errors were encountered: