resolve conflicts

.github/workflows/nodejs.yml

@@ -0,0 +1,31 @@
+name: Node CI
+
+on: [push, pull_request]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        node-version: [10.x, 12.x]
+
+    steps:
+      - uses: actions/checkout@v1
+      - name: Use Node.js ${{ matrix.node-version }}
+        uses: actions/setup-node@v1
+        with:
+          node-version: ${{ matrix.node-version }}
+      - name: yarn lint
+        run: |
+          yarn
+          yarn lint
+        env:
+          CI: true
+      - name: yarn build
+        run: |
+          yarn build
+        env:
+          CI: true
+

.gitignore

@@ -11,7 +11,10 @@ src/config.json
 config/local.json
 config/certs/*.pem
 var/magento2-sample-data/
+var/*
 .migrate
 *.iml
 /docker/elasticsearch/data/
 !/docker/elasticsearch/data/.gitkeep
+/docker/elasticsearch7/data/
+!/docker/elasticsearch7/data/.gitkeep

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [1.11.0-rc.1] - 2019.10.03
 
 ### Added
+- Experimental Elastic 7 support - @pkarw (#342)
 - Output cache support with tagging and cache invalidate requests forwarding - @pkarw @resubaka (https://github.com/DivanteLtd/vue-storefront/issues/3367, #333)
 - Constant for Mailchimp subscription status - @KonstantinSoelch (#294)
 - mage2vs import now has optional `--generate-unique-url-keys` parameter which defaults to `false` to enable/disable the url key generation with name and id for categories - @rain2o (#232)
@@ -22,8 +23,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added jest support - @resubaka (#321)
 - Added caching factory and action factory for the image endpoint. This gives the possibility to use other services for caching or image processing - @resubaka (#317, #315)
 - Added support for tax calculation where the values from customer_tax_class_ids is used - @resubaka (#307)
+- The `db` context object - passed to every api endpoint now has two usefull methods: `getElasticClient` and `getRedisClient` for accesing the data stores - @pkarw (#328)
+- The `lib/utils` got two new methods `getStoreCode(req: Express.Request)` and `getStoreView(code: string)` for getting the current multistore context from `vue-storefront` frontend requests - @pkarw
+
+### Removed
+- The `scripts/seo.js` tool has been removed, the legacy `migrations` scripts have been removed, the unused legacy extensions (`gls-parcelshop-dk`, `postnord-parcelshop-dk`) - @pkarw (#342)
 
 ### Fixed
+- The way Elastic and Redis clients have been fixed and code duplication removed across the app - @pkarw (#327)
 - The `product.price_*` fields have been normalized with the backward compatibility support (see `config.tax.deprecatedPriceFieldsSupport` which is by default true) - @pkarw (#289)
 - The `product.final_price` field is now being taken into product price calcualtion. Moreover, we've added the `config.tax.finalPriceIncludesTax` - which is set to `true` by default. All the `price`, `original_price` and `special_price` fields are calculated accordingly. It was required as Magento2 uses `final_price` to set the catalog pricing rules after-prices - @pkarw (#289)
 - Force ES connections to use protocol config option - @cewald (#303, #304)

README.md

@@ -65,6 +65,68 @@ A [Kibana](https://www.elastic.co/products/kibana) service is available to explo
 
 At first access it will ask to specify an index pattern, insert `vue_storefront*`
 
+**Note:** Kibana is not provided with the Elastic 7 docker file. Please install it on your own when needed or check the [`es-head`](https://chrome.google.com/webstore/detail/elasticsearch-head/ffmkiejjmecolpfloofpjologoblkegm) Chrome plugin.
+
+## Elastic 7 support
+
+By default, Vue Storefront API docker files and config are based on Elastic 5.6. We plan to change the default Elastic version to 7 with the 1.11 stable release. As for now, the [Elastic 7 support](https://github.com/DivanteLtd/vue-storefront-api/pull/342) is marked as **experimental**. 
+
+**How to use it?**
+
+In order to use the new Elastic 7 please make sure your `vue-storefront-api` has been checked out from the [PR#342](https://github.com/DivanteLtd/vue-storefront-api/pull/342) branch.
+To start the Elastic 7 docker service please use the `docker-compose.elastic7.yml` file provided:
+
+```bash
+docker-compose -f docker-compose.elastic7.yml up
+```
+
+Then, please do change the `config/local.json` to start using the new Elastic API. Key properties in the `elasticsearch` section are: `indexTypes` and `apiVersion` (to be set to 7.1). If you're using the multistore configuration please make sure you adjusted the `storeViews.*.elasticsearch` section as well - per each separate store.
+
+```json
+  "elasticsearch": {
+    "host": "localhost",
+    "port": 9200,
+    "protocol": "http",
+    "min_score": 0.01,
+    "indices": [
+      "vue_storefront_catalog",
+      "vue_storefront_catalog_de",
+      "vue_storefront_catalog_it"
+    ],
+    "indexTypes": [
+      "product",
+      "category",
+      "cms_block",
+      "cms_page",
+      "attribute",
+      "taxrule",
+      "review"
+    ],
+    "apiVersion": "7.1"
+  }
+```
+
+Starting from [Elasitc 6 and 7](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html) we can have **just single** document type per single index. Vue Storefront used to have `product`, `category` ... types defined in the `vue_storefront_catalog`.
+
+From now on, we're using the separate indexes per each entity type. The convention is: `${indexName}_${entityType}`. If your' **logical index name** is `vue_storefront_catalog` then it will be mapped to the **physical indexes** of: `vue_storefront_catalog_product`, `vue_storefront_catalog_category` ...
+
+To take the advantage of this new logical->physical index distinction we've provided new Elastic tools: `db7`, `migrate7`, `dump7`, `restore7` tools. They can be used exactly the same way [like the old tools](https://docs.vuestorefront.io/guide/data/database-tool.html) were. The only distinction is that they work on separate indexes.
+
+**Create new index**
+
+Before restoreing or importing data you might want to create a new Elastic index with the proper data types/schema applied. You can run just the `yarn db7 new` command in order to do that.
+
+**Restore the data**
+
+After you ran the docker file and have Elastic7 up and running you might want to:
+
+a) restore the demo data by running `yarn restore7` and `yarn restore7_it; yarn restore7_de` for default multistores. The data is restored from `var/catalog_product.json`, `var/catalog_category.json` and so on...
+
+b) import the data from Magento to proper physical indexes. To do so, currently you can do this only with [mage2vuestorefront](https://github.com/DivanteLtd/mage2vuestorefront/pull/96).
+
+**Note:** After 1.11 stable release (around November, 2019) we'll **replace** the standard tools: `db`, `migrate`, `dump`, `restore` with the Elastic 7 equivalents and it will become the default.
+
+
 ## API access
 Catalog API calls are compliant with ElasticSearch (it works like a filtering proxy to ES). More on ES queries: [ElasticSearch queries tutorial](http://okfnlabs.org/blog/2013/07/01/elasticsearch-query-tutorial.html)
 

config/default.json

@@ -30,7 +30,8 @@
     "indexTypes": [
       "product",
       "category",
-      "cms",
+      "cms_block",
+      "cms_page",
       "attribute",
       "taxrule",
       "review"

config/elastic.schema.attribute.json

@@ -9,8 +9,7 @@
         "options": {
             "properties": {
                 "value": {
-                    "type": "text",
-                    "index": true
+                    "type": "text"
                 }
             }
         }

config/elastic.schema.category.json

@@ -6,12 +6,13 @@
        "is_active": {"type": "boolean"},
        "product_count": {"type": "integer"},
        "parent_id": {"type": "integer"},
-       "created_at": { 
-           "type": "date",           
+       "position": {"type": "integer"},
+       "created_at": {
+           "type": "date",
            "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
         },
-       "updated_at": { 
-           "type": "date",           
+       "updated_at": {
+           "type": "date",
            "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
         }
     }

config/elastic.schema.product.json

@@ -7,7 +7,7 @@
        "size": {"type": "integer"},
        "size_options": {"type": "integer"},
        "price": {"type": "float"},
-       "has_options": {"type": "boolean"},            
+       "has_options": {"type": "keyword"},            
        "special_price": {"type": "float"},
        "color": {"type": "integer"},
        "color_options": {"type": "integer"},
@@ -46,7 +46,7 @@
            "properties": {
                "url_key": {"type": "keyword"},
                "sku": {"type": "keyword"},
-               "has_options": {"type": "boolean"},
+               "has_options": {"type": "keyword"},
                "price": {"type": "float"},
                "special_price": {"type": "float"}
             }

docker-compose.elastic7.yml

@@ -0,0 +1,24 @@
+version: '3.0'
+services:
+  es7:
+    container_name: es7
+    build: docker/elasticsearch7/
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1    
+    volumes:
+      - ./docker/elasticsearch7/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
+    ports:
+      - '9200:9200'
+      - '9300:9300'
+    environment:
+      - discovery.type=single-node
+      - cluster.name=docker-cluster
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xmx512m -Xms512m"
+
+  redis:
+    image: 'redis:4-alpine'
+    ports:
+      - '6379:6379'

docker-compose.yml

@@ -3,13 +3,20 @@ services:
   es1:
     container_name: elasticsearch
     build: docker/elasticsearch/
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1    
     volumes:
       - ./docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
     ports:
       - '9200:9200'
       - '9300:9300'
     environment:
-      ES_JAVA_OPTS: "-Xmx512m -Xms512m"
+      - discovery.type=single-node
+      - cluster.name=docker-cluster
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xmx512m -Xms512m"
 
   kibana:
     build: docker/kibana/

docker/elasticsearch7/Dockerfile

@@ -0,0 +1,2 @@
+FROM docker.elastic.co/elasticsearch/elasticsearch:7.3.2
+

docker/elasticsearch7/config/elasticsearch.yml

@@ -0,0 +1,16 @@
+---
+## Default Elasticsearch configuration from elasticsearch-docker.
+## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
+#
+cluster.name: "docker-cluster"
+network.host: 0.0.0.0
+
+# minimum_master_nodes need to be explicitly set when bound on a public IP
+# set to 1 to allow single node clusters
+# Details: https://github.com/elastic/elasticsearch/pull/17288
+discovery.zen.minimum_master_nodes: 1
+
+## Use single node discovery in order to disable production mode and avoid bootstrap checks
+## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
+#
+discovery.type: single-node

migrations/.common.js

@@ -1,24 +1,9 @@
 
-let config = require('config')
-let kue = require('kue')
-let queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }))
-
-let es = require('elasticsearch')
-const esConfig = {
-  host: {
-    host: config.elasticsearch.host,
-    port: config.elasticsearch.port,
-    protocol: config.elasticsearch.protocol
-  },
-  log: 'debug',
-  apiVersion: config.elasticsearch.apiVersion,
-  requestTimeout: 1000 * 60 * 60,
-  keepAlive: false
-}
-if (config.elasticsearch.user) {
-  esConfig.httpAuth = config.elasticsearch.user + ':' + config.elasticsearch.password
-}
-let client = new es.Client(esConfig)
+const config = require('config')
+const kue = require('kue')
+const queue = kue.createQueue(Object.assign(config.kue, { redis: config.redis }))
+const es = require('../src/lib/elastic')
+const client = es.getClient(config)
 
 exports.db = client
 exports.queue = queue