Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#10955] Add documentation for Datastore emulator and Solr setup #10958

Merged
merged 19 commits into from
Jun 6, 2021
Merged

[#10955] Add documentation for Datastore emulator and Solr setup #10958

merged 19 commits into from
Jun 6, 2021

Conversation

Derek-Hardy
Copy link
Contributor

@Derek-Hardy Derek-Hardy commented Feb 13, 2021 •
edited by jianhandev

Part of #10955

Task List:

@Derek-Hardy Derek-Hardy added the s.ToReview The PR is waiting for review(s) label Feb 13, 2021
jianhandev
jianhandev reviewed Mar 1, 2021
View reviewed changes
docs/datastore-emulator-setup.md Outdated
Comment on lines 15 to 19
1. Sometimes the recommended emulator setup in [wiki](https://github.com/objectify/objectify/wiki/Setup#initialising-the-objectifyservice-to-work-with-emulator-applies-to-v6) might not work with `Exiting due to exception: java.io.IOException: Failed to bind`.

**Solution:**
In that case, try out some other localhost ports such as `gcloud beta emulators datastore start --host-port=localhost:8482`.
Also change `DatastoreOptions.setHost()` parameter in `src/main/java/teammates/storage/api/OfyHelper.java` accordingly.
Copy link
Contributor

@jianhandev jianhandev Mar 1, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error is due to the emulator not shutting down properly, which leaves a process running in port 8484. Since we have fixed the port number in build.template.properties to port 8484, I guess we should change this part?

To kill the dangling process in port 8484 we can find its process id with the command lsof -i tcp:8484, and kill it with kill -9 <PID>, and then re-run gcloud beta emulators datastore start --host-port=localhost:8484.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jianhandev Right, you can go ahead and co-author on this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, will add in some other details also and seek your review afterwards.

@Derek-Hardy Derek-Hardy closed this Mar 1, 2021
@Derek-Hardy Derek-Hardy reopened this Mar 1, 2021
wkurniawan07
wkurniawan07 requested changes Mar 1, 2021
View reviewed changes
docs/datastore-emulator-setup.md Outdated Show resolved Hide resolved
docs/datastore-emulator-setup.md Outdated Show resolved Hide resolved
@Derek-Hardy Derek-Hardy added s.Ongoing The PR is being worked on by the author(s) and removed s.ToReview The PR is waiting for review(s) labels Mar 3, 2021
@Derek-Hardy Derek-Hardy added s.ToReview The PR is waiting for review(s) and removed s.Ongoing The PR is being worked on by the author(s) labels Mar 8, 2021
madanalogy
madanalogy reviewed Mar 11, 2021
View reviewed changes
docs/development.md Outdated

### Steps to local setup
**1. Installing the Datastore Emulator**
After you have installed the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) which contains the `gcloud` tool, the datastore emulator can be installed by running the command:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if relevant to this PR, but just wondering if we could have these run as gradle tasks? I remember devs previously would have to setup google cloud sdk for deployment but @wkurniawan07 did some magic with gradle and that requirement was removed. Now it's being brought back so just checking if alternatives have been considered

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the magic from App Engine plugin, not anywhere near anything done by me. Datastore emulator is however an entirely different product, so it looks like Google Cloud SDK becomes a necessity once again.

docs/troubleshooting-guide.md Outdated Show resolved Hide resolved
madanalogy
madanalogy approved these changes Mar 12, 2021
View reviewed changes
Copy link
Contributor

@madanalogy madanalogy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@madanalogy madanalogy added s.FinalReview The PR is ready for final review and removed s.ToReview The PR is waiting for review(s) labels Mar 12, 2021
@jianhandev
Copy link
Contributor

wkurniawan07@9d23b45

I fetched this branch and tried running ./gradlew runDatastoreEmulator, it says BUILD SUCCESSFUL but the emulator is not actually running?

@wkurniawan07
Copy link
Member

@jianhandev what makes you claim that the emulator is not running? Just because the build terminates doesn't mean there is no Datastore running.

@jianhandev
Copy link
Contributor

jianhandev commented Apr 14, 2021 •
edited

@jianhandev what makes you claim that the emulator is not running?

Maybe I missed out something here, but after running the gradlew command I didn't see any process running in port 8484. The log shows that the emulator might not have been started:

SEVERE: DatastoreException caught by WebApiServlet: 
com.google.cloud.datastore.DatastoreException: I/O error

Are there any preliminary steps we should take before running the gradlew command?

@jianhandev jianhandev changed the title [#10955] Add Datastore Emulator setup documentation [#10955] Documentation Objectify and Search API Migration Apr 15, 2021
@wkurniawan07 wkurniawan07 changed the title [#10955] Documentation Objectify and Search API Migration [#10955] Documentation Datastore emulator and Solr setup Apr 15, 2021
@wkurniawan07 wkurniawan07 changed the title [#10955] Documentation Datastore emulator and Solr setup [#10955] Add documentation for Datastore emulator and Solr setup Apr 16, 2021
@wkurniawan07
Copy link
Member

Are there any preliminary steps we should take before running the gradlew command?

Other than having the build.properties file ready, not really.
I have proof that the setup works: https://github.com/wkurniawan07/repo/runs/2352281867?check_suite_focus=true. I did modify the code slightly (wkurniawan07@bdb5823), but I really don't think those are the make-or-break factor.

Also, does your component tests pass? It is the exact same setup, other than the store-on-disk setup (which could be the make-or-break factor).

@Derek-Hardy
Copy link
Contributor Author

I tried out this setup. Component tests are passing.

@wkurniawan07
Copy link
Member

wkurniawan07 commented Apr 17, 2021 •
edited

Component tests are passing.

Unfortunately this doesn't count as confirmation, because component tests are using its own setup from the beginning. What I said previously was if the setup didn't work there's a likelihood that the component tests setup didn't work either, but getting the component tests to pass also doesn't necessarily mean that the setup will work.

@Derek-Hardy
Copy link
Contributor Author

Performed E2E tests locally, they pass as well.

@wkurniawan07 wkurniawan07 force-pushed the teammatesv8 branch 2 times, most recently from 744d365 to 92790b9 Compare May 3, 2021 10:43
@rrtheonlyone
Copy link
Contributor

rrtheonlyone commented May 29, 2021 •
edited by jianhandev

@wkurniawan07 @Derek-Hardy @jianhandev what is the status of this PR?

@jianhandev
Copy link
Contributor

@wkurniawan07 @Derek-Hardy @jianhandev what is the status of this PR?

There are still some pending changes to be made, I think we agreed to leave this PR open and update it along the way.

@wkurniawan07
Copy link
Member

@jianhandev I took the opportunity to reorganize the information and reduce verbosity. See if they are understandable enough for newcomers?

jianhandev
jianhandev reviewed Jun 5, 2021
View reviewed changes
Copy link
Contributor

@jianhandev jianhandev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes looks good to me, just a small comment to clarify things a bit:

docs/development.md Show resolved Hide resolved
@wkurniawan07 wkurniawan07 merged commit 5f1d2f5 into TEAMMATES:teammatesv8 Jun 6, 2021
wkurniawan07 pushed a commit that referenced this pull request Jun 6, 2021
@Derek-Hardy @wkurniawan07
[#10955] Add documentation for Datastore emulator and Solr setup (#10958
07e2151 
)
wkurniawan07 pushed a commit that referenced this pull request Jun 6, 2021
@Derek-Hardy @wkurniawan07
[#10955] Add documentation for Datastore emulator and Solr setup (#10958
b23e56e 
)
@wkurniawan07 wkurniawan07 added this to the V8.0.0-beta.0 milestone Jul 5, 2021
daongochieu2810 pushed a commit to daongochieu2810/teammates that referenced this pull request Jul 5, 2021
@Derek-Hardy @daongochieu2810
[TEAMMATES#10955] Add documentation for Datastore emulator and Solr s…
cf6d170 
…etup (TEAMMATES#10958)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wkurniawan07 wkurniawan07 wkurniawan07 requested changes

@jianhandev jianhandev jianhandev left review comments

@madanalogy madanalogy madanalogy approved these changes

Assignees

@wkurniawan07 wkurniawan07

@madanalogy madanalogy

@jianhandev jianhandev

Labels
s.Ongoing The PR is being worked on by the author(s)
Projects
None yet
Milestone
V8.0.0-beta.0
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@Derek-Hardy @wkurniawan07 @jianhandev @rrtheonlyone @madanalogy