GSoc 2016: Admin console and Elasticsearch Integration (WIP) #125

Open
wants to merge 90 commits into
from

Projects

None yet

6 participants

@rnikhil275
Contributor

No description provided.

Nikhil R and others added some commits May 27, 2016
Nikhil R feat(admin): Basic code for admin module
Add admin controller.
Add fields in conf file
eceed3a
Nikhil R Just checks for password for now 715e29e
@rnikhil275 rnikhil275 Merge pull request #1 from sailorproject/master
Update from master
ce0f765
@rnikhil275 rnikhil275 Merge pull request #2 from rnikhil275/master
update from master
ee01d57
Nikhil R (not working) add autogen and conf 3b6852c
Nikhil R (admin centre) Use access module
The access module is now used for logging in to the system
08ed53e
Nikhil R use access module 17b9324
Nikhil R (not working) use session open function directly ba96069
Nikhil R TEST 39e2512
Nikhil R (working with apache) autogen and hide the text box
autogen functions are protected, and the password input box disappears
when you login.
a43716b
Nikhil R logout button 1435aaa
Nikhil R logout button working e09db73
Nikhil R remove random text 47498b1
Nikhil R logout link working.
removed a button and set up a separate route
b5ad116
Nikhil R add link to go back to admin centre from autogen bdb1d7b
Nikhil R add else statement b28a562
Nikhil R add conf.lua dbd2173
Nikhil R edit rockspec eacf6ae
Nikhil R add conf function to admin module b1d1a73
Nikhil R parameters print to console 4054602
Nikhil R (working) change password through the conf editor d4aa925
Nikhil R (working) file write working. 45a76ec
Nikhil R (working) change admin password with quotes b8a0be6
Nikhil R (working) make fields with default values for confetti fd3c0bf
Nikhil R views for the config editor 912cbc4
@rnikhil275 rnikhil275 Merge pull request #3 from rnikhil275/devel
Devel branch merge
030d976
Nikhil R use tables instead of multiple file:writes 3e4b63d
Nikhil R (working) config editor 16319ff
@rnikhil275 rnikhil275 Merge pull request #4 from rnikhil275/devel
Pull from devel branch
9cbb468
Nikhil R feat(admin): Add mail/db settings to the config editor
The fronted for the config editor to edit the settings is ready. DB
settings are not separated into dev/prod which will be done while
re-organizing the code. The necessary backend processing will come in
the next commit :)
8bc5781
Nikhil R feat(admin) : add other settings to configedit
Now, config editor can be used to edit sailor, mail server, database,
luaatclient settings
e9aa764
Nikhil R Fix minor error
add a missing table.insert
36ac439
Nikhil R edit(admin): Add a success message when conf is changed 9181986
Nikhil R edit(conf): Make admin centre and autogen default to false 3e86023
Nikhil R test(admin): Sample test for admin console.
This test file is still work in progress.
ae08e32
Nikhil R refactor(admin): Re-organize admin module
Login working. Gets redirected to admin/dashboard. Next step is making
the dashboard. Removed functions from sailor.lua too. WIP
9bdbd2f
Nikhil R feat(admin): Views for config editor
Now the views are generated programatically
d457589
Nikhil R feat(admin): Controller for config editor
The writing into conf file is still done on individual basis. Looping
is confusing due to presence of separate strings in between.
b7927d4
Nikhil R fix(admin): Change mailer settings instead of db
I reverted the previous change and instead changed the mailer settings.
The tests pass.
f7d20be
Nikhil R refactor(admin): Check for enable_admin and remove functions from sai…
…lor.lua
38646fd
Nikhil R edit(admin): Add checks for empty password d4a4909
Nikhil R feat(admin): Add logout button 48f5e38
Nikhil R feat(admin): Add autogen functions to dashboard
Remove unnecessary function form sailor.lua. I left the rest of the
functions in autogen.lua as it is.
6d9206b
Nikhil R Remove unnecessary files 873fb9e
Nikhil R tests(admin): Tests for autogen
Add files to dev-app for testing, modify previous tests for autogen
5005b4c
Nikhil R add autogen back in the rock spec c911e0f
Nikhil R Remove autogen.lua test file. 74b8c18
Nikhil R BAD COMMIT: Deliberately change admin.lua to make tests pass.
25fce2d
Nikhil R add require statement to make travis pass
0f8f830
Nikhil R bugfix(admin): remove autogen related post variables
6ae526a
Nikhil R bugfix(admin): Add missing fields in conf file and fix typo in admin …
…controller
8b4f9fd
Nikhil R feat(admin): Highlight messages
ac476ca
@rnikhil275 rnikhil275 Merge pull request #5 from sailorproject/master
Updated Cookie for xavante
b6466fb
Nikhil R fix(admin): Add an extra function to access module to check username
Using the above mentioned function in the admin controller also to
check whether the user “admin” is actually logged in and not any other
user.
ae555a2
Nikhil R fix(admin): Add support for multiple db environments b0d3a13
Nikhil R test(admin): Help wanted, cannot figure out why admin test fails e02a32a
@rnikhil275 rnikhil275 Merge pull request #6 from sailorproject/master
fix(access): Reset settings before setting them
4b17e90
Nikhil R fix(admin test), test(access), docs(testing)
Fix the admin test, add a new test for the function added in the access
module, add a line to the testing docs to indicate that package.path
has to set in bootstrap.lua also
dda91d5
Nikhil R feat(elasticsearch): Add search module to search through elasticsearch
Sample in dev-app also added
37fbde5
@coveralls
coveralls commented Jul 28, 2016 edited

Coverage Status

Coverage decreased (-0.1%) to 82.66% when pulling 37fbde5 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@rnikhil275 rnikhil275 changed the title from Elasticsearch Integration into Sailor to Work in Progress: Elasticsearch Integration into Sailor Jul 28, 2016
Nikhil R feat(elasticsearch): Add scaffolding functions
77e78b7
@coveralls
coveralls commented Jul 29, 2016 edited

Coverage Status

Coverage decreased (-0.1%) to 82.66% when pulling 77e78b7 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@Etiene Etiene commented on an outdated diff Aug 1, 2016
src/sailor/db/elasticsearch.lua
@@ -0,0 +1,132 @@
+-------------------------------------------------------------------------------------------
+-- elasticsearch.lua: DB module for connecting, querying and serching through elasticsearch
+-- This file is a part of Sailor project
+-- Copyright (c) 2016 Etiene Dalcol <dalcol@etiene.net>
@Etiene
Etiene Aug 1, 2016 Member

you can put your own identification on this file because it was created by you :)

@Etiene
Etiene Aug 5, 2016 Member

Please address this

@Etiene Etiene commented on an outdated diff Aug 1, 2016
src/sailor/db/elasticsearch.lua
+function elastic.getinfo()
+ local data, err = client:info()
+ if data==nil then
+ return err
+ else
+ return data
+
+ end
+end
+
+function elastic.index(indexq, typeq, idq, body)
+ local data, err = client:index{
+ index = indexq,
+ type = typeq,
+ id = idq,
+ body = body
@Etiene
Etiene Aug 1, 2016 Member

please correct indentation here

@Etiene Etiene and 2 others commented on an outdated diff Aug 1, 2016
src/sailor/db/elasticsearch.lua
+
+ end
+end
+
+function elastic.get(indexq, typeq, idq)
+ local data, err = client:get{
+ index = indexq,
+ type = typeq,
+ id = idq
+}
+ if data==nil then
+ return err
+ else
+ return data
+ end
+end
@Etiene
Etiene Aug 1, 2016 Member

Why did you decide to make these wrappers to change how the values are returned?

@rnikhil275
rnikhil275 Aug 3, 2016 Contributor

These wrappers are used from inside the elastic_model.lua( This is the model file for ES). I use these functions inside the model when an object is instantiated. Is this approach fine ?

@pmusa
pmusa Aug 7, 2016

It does not make sense to me as well. is it the way we do with DBs? If not, I would suggest removing the same thing as the request does and let the user handle that in the app. Sometimes the response code is very important to chose what to do next and the current wrapper is doing nothing.

By the way, it is very hard to read the code with different indentation and spaces in each commit. You should decide in one way of writing code and stick to it. Sometimes there is space between '=' sometimes not.

@Etiene Etiene commented on an outdated diff Aug 1, 2016
src/sailor/db/elasticsearch.lua
+ else
+ return data
+
+ end
+end
+
+function elastic.searchbody(indexq, typeq, body)
+ local data, err = client:search{
+ index = indexq,
+ type = typeq,
+ body = {
+ query = {
+ match = body
+ }
+ }
+}
@Etiene
Etiene Aug 1, 2016 Member

indentation

@Etiene Etiene commented on an outdated diff Aug 1, 2016
src/sailor/db/elasticsearch.lua
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
@Etiene
Etiene Aug 1, 2016 Member

lots of blank lines

Nikhil R added some commits Aug 2, 2016
Nikhil R feat(elasticsearch): Take host parameters from the conf file d8c22c1
Nikhil R feat(elastic_model): Add a layer of abstraction for elasticsearch
2eedd9a
@coveralls
coveralls commented Aug 2, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling 2eedd9a on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@Etiene Etiene commented on an outdated diff Aug 2, 2016
src/sailor/db/elasticsearch.lua
@@ -101,6 +111,24 @@ function elastic.delete(indexq, typeq, idq)
end
+function elastic.update(indexq, typeq, idq, body)
+ local data, err = client:update{
+ index = indexq,
+ type = typeq,
+ id = idq,
+ body = {
+ doc = body
+ }
+ }
@Etiene
Etiene Aug 2, 2016 Member

Indentation

@rnikhil275
Contributor

Sorry about the weird indentation. I was writing this while I was on my way back to college. Would format it properly from now.

Nikhil R feat(Emodel): You can add parameters directly now
cf9d344
@coveralls
coveralls commented Aug 3, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling cf9d344 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R fest(ESmodel): search method
Remove unnecessary functions, add search method to the ES model class,
fix some functions
be9ae1a
@coveralls
coveralls commented Aug 3, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling be9ae1a on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@DhavalKapil DhavalKapil and 1 other commented on an outdated diff Aug 3, 2016
src/sailor/db/elasticsearch.lua
+-- This file is a part of Sailor project
+-- Copyright (c) 2016 Nikhil. R <rnikhil96@outlook.com>
+-- License: MIT
+-- http://sailorproject.org
+-------------------------------------------------------------------------------------------
+
+
+
+local elasticsearch = require "elasticsearch"
+local main_conf = require "conf.conf"
+local elastic_conf = main_conf.db[main_conf.sailor.search_database]
+
+local elastic = {}
+
+ local client = elasticsearch.client{
+ host={
@DhavalKapil
DhavalKapil Aug 3, 2016

It is hosts instead of host. Also, you can provide an array of hosts instead of a single host.

Refer to the documentation for details.

@rnikhil275
rnikhil275 Aug 4, 2016 Contributor

a77172f
With that, you can add multiple hosts now

Nikhil R feat(Emodel): Add support for multiple hosts
a77172f
@coveralls
coveralls commented Aug 4, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling a77172f on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R feat(conf): Add ES parameters also
67e54ef
@coveralls
coveralls commented Aug 4, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling 67e54ef on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@pmusa

Not sure why we are using elastic, instead of elasticsearch. Elastic is the company that creates different products and Elasticsearch is one product, which is the only one we are using. Having elastic as the name can be confusing and, later on, if there is something called elastic it will be a problem. I suggest you use elasticsearch or es.

pmusa replied Aug 7, 2016

Also, I would call it search_engine instead of search_database, as that is what the literature use.

@pmusa pmusa commented on an outdated diff Aug 7, 2016
src/sailor/db/elastic_model.lua
@@ -0,0 +1,50 @@
+local elastic = require "sailor.db.elasticsearch"
+local Emodel = {}
@pmusa
pmusa Aug 7, 2016

if this is the elasticsearch model, we usually use ES instead of E. Also when you have big name variables you usually use lower case with underscore. So maybe es_model would be a better naming here.

@pmusa pmusa commented on an outdated diff Aug 7, 2016
src/sailor/db/elastic_model.lua
+ return elastic.index(Etype, id , ...)
+ else
+ return elastic.index(Etype, id, parameters)
+ end
+ end
+
+ -- Get a document using ID
+ self.get = function(id)
+ return elastic.get(Etype, id)
+ end
+ -- Delete a document using ID
+ self.delete = function(id)
+ return elastic.delete(Etype,id)
+ end
+ self.search = function(word)
+ return elastic.search(Etype, word)
@pmusa
pmusa Aug 7, 2016

I guess that we can search for than one single word. We usually call it query in the search world. But if query conflicts with the notion of queries in relational databases, you can call them terms (it is still not correct as people can have something like +fox -quick).

@pmusa pmusa commented on an outdated diff Aug 7, 2016
src/sailor/db/elasticsearch.lua
+ local client = elasticsearch.client{
+ hosts=elastic_conf.hosts,
+ params = elastic_conf.params
+}
+
+function elastic.getinfo()
+ local data, err = client:info()
+ if data==nil then
+ return err
+ else
+ return data
+
+ end
+end
+
+function elastic.index(typeq, idq, body)
@pmusa
pmusa Aug 7, 2016

instead of body, we should call it document or doc

Nikhil R added some commits Aug 8, 2016
Nikhil R style(elasaticsearch): Make changes to naming of certain functions an…
…d tables
889f59b
Nikhil R feat(elasticsearch): Return status code also 7b43fd0
Nikhil R feat(elasticsearch): Support for parameters to be passed 3a32199
Nikhil R feat(elasticsearch): Improve functions
Add optional index support, parameter support, merge search function
f92d2e5
Nikhil R feat(configEditor): Add support for editing search server settings 57f90b6
Nikhil R doc(Elasticsearch): Add some basic comments
39da2bc
@rnikhil275
Contributor

I have corrected a lot of above mentioned things with the last few commits. Please review them too :)

@coveralls
coveralls commented Aug 9, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling 39da2bc on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@pmusa
pmusa commented on 7b43fd0 Aug 9, 2016

Sorry, but I still do not understand why you are doing the nil treatment here. If I am a user and I am calling es.get(), how am I supposed to know if there was an error or not? Is it handled by the Sailor framework?

Maybe @Etiene can jump in but I was thinking about:

function es.getinfo()
    return client:info()
end

If that wrapper seems useless to you, think that the above example is the simplest case, but in some others you are changing the way you pass parameters, and you give the Sailor project the ability to change the internals at time with the same abstraction to users.

Contributor

It does return the error when data is nil. data is usually set when the function call succeeds. When it succeeds, data is returned along with status code. Otherwise the error is returned. I am not able to figure out what you are referring to

Contributor

The wrappers are basically used to

  1. Write less code in the elastic_model.lua file
  2. To use elasticsearch functions easily. (There are lot of optional parameters).

Sorry, is es.getinfo() used by Sailor internally or by Sailor user?

Contributor

It can be used by the user. There is no separate wrapper now and everything is done in es_model.lua itself.
Also, I removed that function. I had used it initially to check connection to ES

Sorry if I caused confusion, but I am trying to understand the layers here. Are the functions from this file supposed to be called by Sailor internally or by the user?

Contributor

Those functions(which are not there now) were used internally by sailor. I thought of abstracting them to modify parameters before using elasticsearch-lua. Now, I just do that in the es_model file. I hope this clears it up :)

@pmusa

I guess it should be type and id without the q.

@pmusa
pmusa commented on 3a32199 Aug 9, 2016

Adding that huge list of parameters should definitely not be the solution. Checking with @DhavalKapil how it should be done.

Contributor

Yeah. Even I was a little skeptical about this. I will certainly talk to him regarding this

Hey, I would suggest a different method for passing params.
Consider this piece of code:

params = params or {}
params.index = ...
params.type = ...
local data, err = client:update(params)

Basically add index, body, type to params and then send it to the client function. The client internally will return an error if it finds any error in the parameter.

Also as elasticsearch versions will update, the list of allowed parameters will update. It would be tedious to update the present solution everytime.

Contributor
rnikhil275 replied Aug 10, 2016 edited

This should work when I have all parameters in one table. I have index, type, id , doc etc from one table and a separate table for optional params like routing, timeout etc.

Is it possible to do something like this ? :
local data, err = client:update{index, type, id, other_params = params}

Contributor

Actually, I will refactor it so that it implements everything in one table. It seems to be much more simpler and I can do away with the wrapper also. Thanks :)

This should work when I have all parameters in one table. I have index, type, id , doc etc from one table and a separate table for optional params like routing, timeout etc.

All you have to do is:

params.index = elastic_conf.index
params.type = type

I don't think that using a single table would be the solution because every request(index, update, etc) requires different parameters.

Is it possible to do something like this ? :
local data, err = client:update{index, type, id, other_params = params}

No, all parameters are passed directly. This is in accordance with the convention used by other official clients.

Just to clear the confusion, params above refers to the optional params passed(as in your code).

The other parameters such as index, type, etc. are limited and can be set within params before sending it to the client.

Contributor

Yes. I meant the parameters to be the single table. I will make a commit implementing the above and I think it would be clearer then

@pmusa

Etype -> es_type

@pmusa

I am confused with the index being part of the arg, and es_type being part of ob. Can you explain to me what is the flow? Or just point me to a simple example on how to use it?

Contributor

Okay. So basically there is a model file inside models/ . This file is must for using ES inside Sailor. In this file, you mention the type and attributes. There is one model file for each type. This file is imported and then used inside elastic_model.lua. "ob" is just the variable name holding the imported file and then its keys are accessed.

Contributor

You mentioned in your mail to allow setting indexes in the conf file but also to let the user override it if necessary. So I made index as an optional parameter in every function. In elastic_model.lua every function has variable named parameters. "arg" stores all the parameters passed as a table

Ok, if we have a model for documents from ES, it makes sense. However, the "model unit" should be index+type, as we can have the same type in different indices and they could potentially be different. Sorry I did not get it before.

Contributor

Sure, I will also add setting "index" in the model file along with the cluster setting in the conf

Nikhil R added some commits Aug 9, 2016
Nikhil R remove typo 66dc1f8
Nikhil R Elasticsearch: Add example and fix table names
8ae1cd1
@rnikhil275
Contributor

I changed the table names, implemented an example and then made a commit.

@coveralls
coveralls commented Aug 9, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling 8ae1cd1 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls
coveralls commented Aug 9, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling 8ae1cd1 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R bugfix(elasticsearch): Add support for elasticsearch in config editor
Missed to commit this change before
fd2b207
@coveralls
coveralls commented Aug 9, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling fd2b207 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R refactor(elasticsearch)
Remove wrappers, Implement index in model, change name of ES model
file,
f5fc9b5
@coveralls
coveralls commented Aug 10, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling f5fc9b5 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R bugfix(elasticsearch): remove one hardcoded parameter
2ce5404
@coveralls
coveralls commented Aug 10, 2016 edited

Coverage Status

Coverage decreased (-0.2%) to 82.622% when pulling 2ce5404 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@DhavalKapil

Looks good to me.

@pmusa

Why is id 1 here?

Contributor

I removed it in the next commit. I accidentally committed it when I was trouble shooting it. I am sorry.

@Etiene Etiene and 1 other commented on an outdated diff Aug 14, 2016
docs/manual_ES_model.md
@@ -0,0 +1,74 @@
+##Reference Manual
+### Elastic Search Model
+
+This module makes it possible to store, retrieve and search data from <a href = "https://www.elastic.co/downloads/elasticsearch">Elasticsearch </a>. It uses the <a href = "github.com/dhavalkapil/elasticsearch-lua">elasticsearch-lua</a> client for Lua. Refer the client <a href="http://elasticsearch-lua.readthedocs.io/en/latest/">here</a> to know the parameters which can be sent through the model functions if any. Return responses are also same as the one offered by this client.
+
+After importing "es_model" we proceed to create a new instance of the model. There should be a file inside models/ which contains the keys and types defined for the particular model.
+
+ local es_model = require "sailor.es_model"
+ local contact = contact.new("test")
@Etiene
Etiene Aug 14, 2016 Member

where is contact.new coming from?

@rnikhil275
rnikhil275 Aug 15, 2016 Contributor

I addressed it and also added a commit. Please check 2c8be47

@Etiene Etiene commented on the diff Aug 14, 2016
docs/tutorial_testing.md
@@ -51,7 +51,7 @@ Let's suppose we have a User model and we want to unit test some of the User met
The fixture is a regular Lua script that will be required. This for multiple possibilities, such as making loops for inserting many entries, as long as in the end you return a table containing samples of your model.
-After creating your fixture, you must add this on the bottom of your bootstrap file so they are loaded before running your tests.
+After creating your fixture, you must add this on the bottom of your bootstrap file so they are loaded before running your tests. Also don't forget to set the package.path to the sailor directory you want to test.
@Etiene Etiene and 1 other commented on an outdated diff Aug 14, 2016
test/dev-app/models/test.lua
@@ -0,0 +1,19 @@
+local val = require "valua"
@Etiene
Etiene Aug 14, 2016 Member

is this file being used anywhere?

@rnikhil275
rnikhil275 Aug 15, 2016 Contributor

Yes. This is the model file for ES documents. You have to create an instant of this model and then use it's methods for accessing ES. It has the "type" and "attributes" for the model.

If you talking about the "valua" import, I will add support for checking attributes in the model similar to "model.lua".

@Etiene
Etiene Aug 17, 2016 Member

if it's a model file for ES documents, why is it called test.lua?

@rnikhil275
rnikhil275 Aug 25, 2016 Contributor

The name of the model is "test". The file name should also be the same.

@Etiene Etiene commented on the diff Aug 14, 2016
test/dev-app/views/test/elastic.lp
@@ -0,0 +1,48 @@
+<?lua local form = require "sailor.form" ?>
@Etiene
Etiene Aug 14, 2016 Member

please use 2 spaces for indentation in this file, it will become much better to read

@rnikhil275
rnikhil275 Aug 15, 2016 Contributor

I edited it. I convert a nested table into a string. Check 2c8be47

@Etiene Etiene and 1 other commented on an outdated diff Aug 14, 2016
test/dev-app/views/test/elastic.lp
@@ -0,0 +1,48 @@
+<?lua local form = require "sailor.form" ?>
+<form method="post">
+ <h4>Search box</h4>
+ <input type="text" id="search" name="search" />
+ <input type="submit" class="btn btn-primary btn-md"/>
+ </form>
+ <?lua
+ function table.val_to_str ( v )
@Etiene
Etiene Aug 14, 2016 Member

could you explain what you are trying to do here?

@rnikhil275
rnikhil275 Aug 15, 2016 Contributor

The elasticsearch-lua client gives out a nested table. I convert this table into a string so that I can print it on the screen using page:write().

In the beginning of the file, I also create an input box for entering search terms. The result is printed on the screen.

This file is meant to be an example for testing ES models.

@Etiene
Etiene Aug 17, 2016 Member

Please prefer adding logic like this on the controller file, not the view.

@rnikhil275
rnikhil275 Aug 25, 2016 Contributor

Alright. I will move the table manipulation to the controller.

@Etiene Etiene commented on the diff Aug 14, 2016
docs/manual_ES_model.md
@@ -0,0 +1,74 @@
+##Reference Manual
+### Elastic Search Model
+
+This module makes it possible to store, retrieve and search data from <a href = "https://www.elastic.co/downloads/elasticsearch">Elasticsearch </a>. It uses the <a href = "github.com/dhavalkapil/elasticsearch-lua">elasticsearch-lua</a> client for Lua. Refer the client <a href="http://elasticsearch-lua.readthedocs.io/en/latest/">here</a> to know the parameters which can be sent through the model functions if any. Return responses are also same as the one offered by this client.
@Etiene
Etiene Aug 14, 2016 Member

Please include in the docs how to install the necessary files

@rnikhil275
rnikhil275 Aug 15, 2016 Contributor

I added some instructions. Please check 2c8be47

Nikhil R docs(ES integration), refactor(ES views).
Add two space indentation, add more docs,
2c8be47
@coveralls
coveralls commented Aug 15, 2016 edited

Coverage Status

Coverage increased (+0.3%) to 83.068% when pulling 2c8be47 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@rnikhil275
Contributor

@DhavalKapil . Can you please point me toward the endpoint I have to access to update the cluster name ? 'client.cluster.putSettings' takes only 'flat_settings' as parameter. Which method is used for the cluster update api ?

I also mailed you regarding this :)

Nikhil R docs, test, feat in ES integration for bulkIndex
Add docs, tests for bulkIndex function.
9ab556f
@coveralls

Coverage Status

Coverage decreased (-1.2%) to 81.563% when pulling 9ab556f on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls
coveralls commented Aug 15, 2016 edited

Coverage Status

Coverage increased (+0.3%) to 83.08% when pulling 9ab556f on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R (tests), (docs) for ES integration: Add client:mget() function
35a3c6f
@coveralls
coveralls commented Aug 15, 2016 edited

Coverage Status

Coverage increased (+0.3%) to 83.093% when pulling 35a3c6f on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.3%) to 83.093% when pulling 35a3c6f on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R tests(ES integration): Add more functional tests
2779b0b
@coveralls
coveralls commented Aug 15, 2016 edited

Coverage Status

Coverage increased (+0.5%) to 83.321% when pulling 2779b0b on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

Nikhil R feat(elasticsearch integration): Add support for using namespaces fro…
…m elasticsearch-lua
386ee24
@coveralls
coveralls commented Aug 15, 2016 edited

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 386ee24 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 386ee24 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 386ee24 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 386ee24 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@DhavalKapil

@rnikhil275 Cluster name is not dynamically updateable. So the only way to update it is by editing elasticsearch.yml before starting the elasticsearch server.

Otherwise, to update dynamic settings, use client.cluster.putSettings() endpoint. The settings which you have to update is passed in the body. Refer to this for complete documentation.

A sample example:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "discovery.zen.minimum_master_nodes" : 2
    }
}'

boils down to the following lua code:

local res, err = client.cluster.putSettings(
    body = {
        persistent = {
            ["discovery.zen.minimum_master_nodes"] = 2
        }
    }
}

I didn't get any email from you. My email address is: me@dhavalkapil.com

@rnikhil275
Contributor

I was referring to the same example. Anyway, I added support for accessing the "namespaces".
I wrongly assumed that "flat_settings" was the only parameter from reading the source file.

Also, I did mail you. It's there in my sent items. I mailed from rnikhil96@outlook.com

Nikhil R docs(admin centre): Add docs, edit functions
Remove validation check because it’s a search database.
63dbc44
@coveralls
coveralls commented Aug 16, 2016 edited

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 63dbc44 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 63dbc44 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 63dbc44 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls
coveralls commented Aug 16, 2016 edited

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling 63dbc44 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@rnikhil275 rnikhil275 changed the title from Work in Progress: Elasticsearch Integration into Sailor to GSoc 2016: Admin console and Elasticsearch Integration (WIP) Aug 23, 2016
Nikhil R refactor(ES): Move logic to controller
e454897
@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.5%) to 83.333% when pulling e454897 on rnikhil275:elasticsearch into a8e470b on sailorproject:master.

@CriztianiX

https://luarocks.org/modules/dhavalkapil/elasticsearch will block the worker running over openresty/nginx.
You can see more in
DhavalKapil/elasticsearch-lua#11

@DhavalKapil
DhavalKapil commented Sep 8, 2016 edited

@CriztianiX there have been some confusion regarding nginx, some other sources suggest no error. Anyways I'm working on a new release for elasticsearch. Your fix shall be incorporated with minor edits :) I'm not sure whether someone has already tested elasticsearch + nginx concerning sailor or not?

@rnikhil275 rnikhil275 Merge pull request #8 from sailorproject/master
pull from master
c96ee68
@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@coveralls

Coverage Status

Coverage increased (+0.6%) to 83.221% when pulling c96ee68 on rnikhil275:elasticsearch into 9349bd5 on sailorproject:master.

@@ -42,6 +43,7 @@ install:
- luarocks install luacov
- luarocks install luacov-coveralls
- luarocks make rockspecs/sailor-current-1.rockspec
+ - luarocks install --server=http://luarocks.org/dev elasticsearch
@DhavalKapil
DhavalKapil Jan 10, 2017

You can now use:

luarocks install elasticsearch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment