CHANGELOG.md

CHANGELOG

v1.11.28 (2024-09-30)

Chore

• chore(deps): update all non-major dependencies (1a1af87)

Fix

• fix(deps): update all non-major dependencies (cd0d706)

v1.11.27 (2024-07-31)

Fix

• fix(deps): update all non-major dependencies (c6c347d)

v1.11.26 (2024-07-10)

Fix

• fix(spider): update pattern for get_product_quantity (34660e0)

• fix(spider): update coldstorage spider due to change in site html structure (6509cac)

• fix(spider): update hopshop spider due to change in site html structure (42f1c30)

v1.11.25 (2024-05-31)

Fix

• fix(deps): update all non-major dependencies (350bf81)

v1.11.24 (2024-04-30)

Fix

• fix(deps): update all non-major dependencies (a7e1318)

v1.11.23 (2024-03-31)

Fix

• fix(deps): update all non-major dependencies (4612a86)

v1.11.22 (2024-02-29)

Fix

• fix(deps): update all non-major dependencies (f6085c6)

v1.11.21 (2024-02-05)

Chore

• chore: update pre-commit hooks versions (d7b5b12)

Fix

• fix(spider): use localized cookies when sending requests (a1c50cb)

v1.11.20 (2024-01-31)

Fix

• fix(deps): update all non-major dependencies (ea35f0b)

v1.11.19 (2024-01-28)

Chore

• chore(deps): update dependency flake8 to v7 (1ccc5de)

Fix

• fix(dockerfile): revert ARG changes (0764e61)

Unknown

• Merge pull request #162 from ngshiheng/renovate/flake8-7.x

chore(deps): update dependency flake8 to v7 (aa872d2)

v1.11.18 (2024-01-28)

Chore

• chore: bump python version to ^3.10 (775a62f)

• chore(deps): update dependency mypy to v1 (79c3ee4)

• chore(deps): update abatilo/actions-poetry action to v3 (7b1d851)

Fix

• fix(deps): update all non-major dependencies (093e5a4)

Unknown

• Merge pull request #158 from ngshiheng/renovate/all-minor-patch

fix(deps): update all non-major dependencies (f61a214)

• Merge pull request #145 from ngshiheng/renovate/mypy-1.x

chore(deps): update dependency mypy to v1 (479f864)

• Merge pull request #161 from ngshiheng/renovate/abatilo-actions-poetry-3.x

chore(deps): update abatilo/actions-poetry action to v3 (ed8f6bb)

v1.11.17 (2024-01-28)

Chore

• chore(deps): update actions/setup-python action to v5 (12bde47)

Fix

• fix(spider): update craftbeersg due to site html layout change (b9eaf08)

Unknown

• Merge pull request #160 from ngshiheng/fix-craftbeersg-spider

fix(spider): update craftbeersg due to site html layout change (8328121)

• Merge pull request #159 from ngshiheng/renovate/actions-setup-python-5.x

chore(deps): update actions/setup-python action to v5 (9634979)

v1.11.16 (2023-12-21)

Chore

• chore(deps): update dependency pylint to v3 (071cbd3)

Fix

• fix: attr-defined error (e554b31)

• fix(deps): update all non-major dependencies (abdde6c)

Unknown

• Merge pull request #157 from ngshiheng/renovate/all-minor-patch

fix(deps): update all non-major dependencies (ad03b2f)

• Merge pull request #156 from ngshiheng/renovate/pylint-3.x

chore(deps): update dependency pylint to v3 (6cea93f)

v1.11.15 (2023-10-31)

Chore

• chore(deps): update actions/checkout action to v4 (47c3127)

Fix

• fix(deps): update all non-major dependencies (063d96c)

Unknown

• Merge pull request #154 from ngshiheng/renovate/actions-checkout-4.x

chore(deps): update actions/checkout action to v4 (169a7ee)

v1.11.14 (2023-09-30)

Fix

• fix(deps): update all non-major dependencies (7c26430)

v1.11.13 (2023-08-31)

Fix

• fix(deps): update all non-major dependencies (b9d8fa5)

v1.11.12 (2023-07-31)

Fix

• fix(deps): update all non-major dependencies (f031116)

v1.11.11 (2023-06-26)

Fix

• fix(deps): update all non-major dependencies (9d2c966)

v1.11.10 (2023-05-31)

Fix

• fix(deps): update all non-major dependencies (a7115e1)

v1.11.9 (2023-04-30)

Fix

• fix(deps): update all non-major dependencies (30bbed8)

v1.11.8 (2023-04-17)

Fix

• fix: update domain name (633cc70)

v1.11.7 (2023-03-31)

Chore

• chore(deps): update all non-major dependencies (672a325)

Fix

• fix(deps): update all non-major dependencies (9cb0f35)

v1.11.6 (2023-02-07)

Chore

• chore(deps): update dependency pre-commit to v3 (c299328)

Ci

• ci: update poeetry version to 1.3.2 and remove paths (8ecd762)

• ci: update python-semantic-releas to master (706b9b1)

Fix

• fix(ci): typo in cd steps (5c19562)

• fix(deps): update all non-major dependencies (2fa23c4)

Style

• style: update log colors (2f31d89)

Unknown

• Merge pull request #140 from ngshiheng/renovate/pre-commit-3.x

chore(deps): update dependency pre-commit to v3 (4a3ebf9)

v1.11.5 (2023-01-29)

Chore

• chore: remove docker inspect running state check (368175a)

Fix

• fix(spider): handle page not found error and invalid price (f40ca3b)

v1.11.4 (2023-01-29)

Chore

• chore(docker): bump POETRY_VERSION (8d292a9)

• chore(renovate): group all non major updates (fc92cd9)

• chore: update pre-commit isort (0f60365)

Fix

• fix(spider): handle cass where product_info does not have abv (7b4b722)

• fix(spider): skip craftbeersg item without quantity (6d1161c)

Refactor

• refactor: update docstrings, logs, and other misc. things (a3c24c3)

v1.11.3 (2023-01-28)

Fix

• fix(deps): update dependency sentry-sdk to v1.14.0 (07bc1a5)

v1.11.2 (2023-01-28)

Chore

• chore(deps): update dependency pre-commit to v2.21.0 (a6337aa)

• chore(deps): update dependency mypy to v0.991 (967dd30)

• chore(deps): update dependency pylint to v2.15.10 (d2a220a)

Fix

• fix(deps): update dependency ipython to v8.9.0 (a4efa4e)

v1.11.1 (2023-01-28)

Fix

• fix(deps): update dependency alembic to v1.9.2 (7fad6b4)

Unknown

• Merge pull request #137 from ngshiheng/renovate/alembic-1.x

fix(deps): update dependency alembic to v1.9.2 (08c60a5)

v1.11.0 (2023-01-28)

Feature

• feat: add craftbeersg spider (f9badd0)

Style

• style: update comments and spacing (6dfb862)

Unknown

• Merge pull request #142 from ngshiheng/add-craftbeersg-spider

feat: add craftbeersg spider (7f7d0ed)

v1.10.13 (2023-01-28)

Chore

• chore(deps): pin dependency mypy to 0.910 (ad0e004)

• chore: remove setuptools (9f891ce)

• chore: update Makefile to add new target to setup postgres in Docker (7ab3e6f)

• chore(deps): update dependency autopep8 to v2.0.1 (37ca00d)

Documentation

• docs: move some content to DEVELOPMENT.md (e1bd085)

• docs: move sections into different files (e267f1d)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.46 (c31a78f)

Unknown

• Merge pull request #134 from ngshiheng/renovate/sqlalchemy-1.x

fix(deps): update dependency sqlalchemy to v1.4.46 (af58e15)

• Merge pull request #131 from ngshiheng/renovate/pin-dependencies

chore(deps): pin dependency mypy to 0.910 (4ecc2ef)

• BREAKING CHANGE: thirsty site html layout change (5804ddd)

• BREAKING CHANGE: removed giant and shopee spiders (2b687b7)

v1.10.12 (2022-12-21)

Fix

• fix(spider): increase download delay for shopee (46d154a)

v1.10.11 (2022-12-18)

Fix

• fix(spider): handle captcha challenge (2bc54e7)

v1.10.10 (2022-12-16)

Chore

• chore(deps): update dependency flake8 to v6 (b515441)

• chore(deps): update dependency autopep8 to v2 (418728d)

Fix

• fix(spider): api change (ca66838)

Unknown

• Merge pull request #130 from ngshiheng/renovate/flake8-6.x

chore(deps): update dependency flake8 to v6 (b375e9d)

• Merge pull request #123 from ngshiheng/renovate/autopep8-2.x

chore(deps): update dependency autopep8 to v2 (9b03570)

v1.10.9 (2022-12-06)

Fix

• fix(deps): update dependency sentry-sdk to v1.11.1 (237bc08)

v1.10.8 (2022-12-05)

Fix

• fix(deps): update dependency scrapy to v2.7.1 (5f708e6)

v1.10.7 (2022-12-05)

Chore

• chore(deps): update dependency pylint to v2.15.8 (17ea8f0)

• chore(deps): update dependency autopep8 to v1.7.0 (1436a21)

Fix

• fix(deps): update dependency ipython to v8.7.0 (4ee24b4)

v1.10.6 (2022-12-05)

Fix

• fix: poetry tries to remove setuptools and then complains it can't find it (88b2a81)

v1.10.5 (2022-12-02)

Chore

• chore: run poetry update (157619e)

Ci

• ci: remove heroku deploy (01fd51e)

Fix

• fix(locators): beerforce spider theme change (cafeba6)

Unknown

• Merge pull request #126 from ngshiheng/ci-remove-heroku

ci: remove heroku deploy (eb05522)

v1.10.4 (2022-11-23)

Fix

• fix(deps): update dependency colorlog to v6.7.0 (a9fb005)

Unknown

• Merge pull request #120 from ngshiheng/renovate/colorlog-6.x

fix(deps): update dependency colorlog to v6.7.0 (0e5757a)

v1.10.3 (2022-11-23)

Fix

• fix(deps): regenerate poetry.lock file (313789f)

• fix(deps): pin dependencies (94ef414)

Unknown

• Merge pull request #116 from ngshiheng/renovate/pin-dependencies

fix(deps): pin dependencies (62773e1)

v1.10.2 (2022-11-23)

Chore

• chore: add setuptools (973f3c5)

• chore: regenerate poetry.lock (9813dfd)

Ci

• ci: downgrade poetry-version to 1.1.13 (0b9af92)

Fix

• fix: install setuptool (9e94e5d)

Unknown

• Merge pull request #125 from ngshiheng/fix-setuptool-error

chore: add setuptools (169ab08)

• Merge pull request #124 from ngshiheng/fix-setuptool-error

fix: install setuptool (9e5b4de)

v1.10.1 (2022-11-17)

Ci

• ci: add workflow_dispatch to cd (d4574aa)

• ci: remove cache poetry (8d65fb6)

• ci: cache poetry dependencies (18772a1)

• ci: upgrade poetry version (14b17d8)

Fix

• fix(spider): handle case where product is returned as null from fairprice api (700980b)

v1.10.0 (2022-11-07)

Feature

• feat: add item deduplication pipeline (424a3ce)

Fix

• fix(spider): skip merchandise page for beerforce (1036068)

Refactor

• refactor(spider): lazada spider to always use proxy (a20f815)

v1.9.0 (2022-11-07)

Feature

• feat: add proxy middleware (fa0b3ec)

v1.8.0 (2022-11-05)

Feature

• feat: use scrapeops for job monitoring (113444e)

Refactor

• refactor: makefile to be self-documenting (abfbbdd)

Unknown

• Merge pull request #115 from ngshiheng/scrapeops-scrapy-integration

feat: scrapeops scrapy integration (96ff70c)

v1.7.0 (2022-11-05)

Documentation

• docs: add section to delete products via scrapy shell (9bd7750)

Feature

• feat: add FiltersPipeline to filter unwanted products (5ffbdcc)

Refactor

• refactor: move renovate.json to root dir (4f984b6)

• refactor: item pipelines (4e8343d)

v1.6.21 (2022-11-05)

Fix

• fix(spider): lazada spider url param path (7f84e11)

v1.6.20 (2022-11-05)

Fix

• fix(spider): shopee spider regression due to api requirement change (7e0fa59)

v1.6.19 (2022-11-03)

Fix

• fix(spider): remove rate limiting flag & fetch beer style from filterItems (5831825)

• fix: missing trailing comma (a2e4f30)

v1.6.18 (2022-11-02)

Fix

• fix: downgrade to python3.9 (e8949d1)

v1.6.17 (2022-10-31)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.42 (07cad9b)

v1.6.16 (2022-10-31)

Chore

• chore: run poetry install (6891d1f)

Fix

• fix(deps): update dependency psycopg2-binary to v2.9.5 (d869198)

v1.6.15 (2022-10-01)

Fix

• fix(deps): update dependency sentry-sdk to v1.9.9 (677826f)

v1.6.14 (2022-09-30)

Chore

• chore: improve Makefile (7134b23)

• chore(deps): update abatilo/actions-poetry action to v2.1.6 (ea1d7d6)

• chore(deps): update dependency flake8 to v5.0.4 (fc36d43)

Fix

• fix(deps): update dependency scrapy to v2.6.3 (935ba9c)

Refactor

• refactor: improve Dockerfile (d334992)

• refactor: improve Makefile (f8a3556)

v1.6.13 (2022-08-23)

Chore

• chore(spider): update mainstream beer brand list (e35dc7f)

Fix

• fix(spider): update lazada spider endpoint (077cd48)

Unknown

• Merge pull request #108 from ngshiheng/update-lazada-spider

fix(spider): update lazada spider endpoint (47bd287)

v1.6.12 (2022-08-12)

Chore

• chore(renovate): update schedule to monthly (117ce16)

• chore(deps): update dependency flake8 to v5 (088b91c)

Fix

• fix(spider): skip items if brand is None for giant.sg spider (29d3368)

Unknown

• Merge pull request #107 from ngshiheng/renovate/flake8-5.x

chore(deps): update dependency flake8 to v5 (fb76fc9)

v1.6.11 (2022-07-30)

Fix

• fix(deps): update dependency sentry-sdk to v1.9.0 (daf2f21)

v1.6.10 (2022-07-25)

Chore

• chore: add FUNDING.yml (f86fdb5)

Fix

• fix(deps): update dependency scrapy to v2.6.2 (b3271e7)

v1.6.9 (2022-07-21)

Chore

• chore(deps): update dependency pylint to v2.14.5 (5c896de)

Fix

• fix(deps): update dependency sentry-sdk to v1.8.0 (dd3716c)

v1.6.8 (2022-07-16)

Fix

• fix(deps): update dependency sentry-sdk to v1.7.2 (fed5a2f)

v1.6.7 (2022-07-13)

Fix

• fix(deps): update dependency sentry-sdk to v1.7.1 (212864d)

v1.6.6 (2022-07-13)

Fix

• fix(deps): update dependency alembic to v1.8.1 (bfe5b90)

v1.6.5 (2022-07-11)

Chore

• chore(deps): update dependency pre-commit to v2.20.0 (f1c0885)

• chore(deps): update abatilo/actions-poetry action to v2.1.5 (683174d)

• chore(deps): update dependency pylint to v2.14.4 (5ab3169)

Fix

• fix(deps): update dependency sentry-sdk to v1.7.0 (0ed1c1d)

Refactor

• refactor(spider): move greatbeerexperiment xpaths to locators/ (230b7bc)

Style

• style: update docstring (fc1ecfe)

Test

• test(parser): add test_parse_name (c875fa2)

v1.6.4 (2022-06-24)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.39 (4abd32c)

v1.6.3 (2022-06-24)

Chore

• chore(deps): update dependency pylint to v2.14.3 (413a1b5)

• chore(deps): update dependency pylint to v2.14.2 (c82845e)

• chore: update pre-commit versions (0c443a5)

• chore(deps): update actions/setup-python action to v4 (3247f60)

• chore(deps): update dependency pylint to v2.14.1 (eb1a40e)

• chore: update style & flake8 exclude list (606db23)

Ci

• ci: only run cd for changes in *.py files and poetry.lock (9f0389c)

Documentation

• docs: add features (3685743)

• docs: add badges (dc5304e)

• docs: move example sql queries to .sql files (7636768)

Fix

• fix(deps): update dependency sentry-sdk to v1.6.0 (973d0aa)

Unknown

• Merge pull request #91 from ngshiheng/renovate/actions-setup-python-4.x

chore(deps): update actions/setup-python action to v4 (148eb8c)

v1.6.2 (2022-06-11)

Chore

• chore(spider): update contract (e00cc80)

Fix

• fix(spider): skip personalised items for troublebrewing spider (38aea1f)

Refactor

• refactor(locator): update imports (e1bf084)

v1.6.1 (2022-06-08)

Fix

• fix: missing trailing comma which causes unintended concat (e89fead)

v1.6.0 (2022-06-07)

Chore

• chore: run poetry update (17e910f)

• chore: add Makefile (b05e944)

• chore(deps): update dependency pylint to v2.14.0 (2fe78de)

Feature

• feat: skip non-beer items (45bab87)

v1.5.3 (2022-06-01)

Fix

• fix(deps): update dependency alembic to v1.8.0 (a7d2bd2)

v1.5.2 (2022-06-01)

Chore

• chore: ignore python-semantic-release (9394acb)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.37 (5c6d125)

v1.5.1 (2022-05-31)

Chore

• chore: update autpmergeType to pr (af33b9a)

• chore(deps): update relekang/python-semantic-release action to v7.29.0 (87c4340)

Ci

• ci: revert relekang/python-semantic-release action to v7.28.1 (783648c)

• ci: update release job to use relekang/python-semantic-release@master (0c9e037)

Fix

• fix(deps): update dependency ipython to v8.4.0 (e62ffe8)

v1.5.0 (2022-05-19)

Chore

• chore: bump POETRY_VERSION to 1.1.13 (112feec)

• chore(deps): update dependency pylint to v2.13.9 (736d6dd)

• chore: run poetry update (7a8636c)

• chore(deps): update dependency pre-commit to v2.19.0 (0e4c062)

• chore(deps): update dependency pylint to v2.13.8 (4cefa4d)

• chore: move bash scripts to folder (2a748ab)

Ci

• ci: pin relekang/python-semantic-release@v7.28.1 due to #442 (302a357)

• ci: avoid running the same test job twice (81e483b)

• ci: make release and deploy require test (d2575d0)

Feature

• feat: skip more mainstream beer brands (b9039c5)

Fix

• fix(deps): update dependency sentry-sdk to v1.5.12 (ccb340c)

• fix(deps): update dependency ipython to v8.3.0 (a9dc049)

• fix(deps): update dependency sentry-sdk to v1.5.11 (75c9ab6)

Refactor

• refactor(spider): add typing for parse* methods (76391b7)

• refactor(spider): move hopshop spider XPath to use locator class (675b1ac)

Style

• style(ci): reformat yml (ab85ee5)

• style(spider): remove redundant xpath (b86233f)

• style(spider): group locators according to items.py (c4c3869)

v1.4.1 (2022-05-02)

Fix

• fix(spider): invalid xpath expression error (b3676b4)

v1.4.0 (2022-05-01)

Feature

• feat(command): allow prune.sh to specify stale_days (8a0b4bb)

v1.3.1 (2022-05-01)

Fix

• fix(command): move optparse to argparse (b0175bb)

v1.3.0 (2022-05-01)

Chore

• chore: update vscode settings (de31764)

Documentation

• docs: add sentry details (c66f5b0)

Feature

• feat: skip mainstream beer brands (5d90e2b)

• feat: log skipped items (3ecf173)

Refactor

• refactor: log and skip mainstream beers (ee1ce31)

• refactor: move const to its own file (6fc6964)

Style

• style: ordering and line break (1dd31af)

Unknown

• Merge pull request #85 from ngshiheng/improve-filter-and-log

feat: improve filter and log (3ef1269)

v1.2.1 (2022-04-30)

Chore

• chore: update branch name to main (7c6143a)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.36 (3cfb9c6)

v1.2.0 (2022-04-30)

Chore

• chore: update LICENSE year (4c0e66e)

Ci

• ci: disable ci on PR (0771c43)

• ci: add auto deploy to Heroku (c9f955a)

Feature

• feat: collect volume and quantity for giant.sg spider (60146b7)

• feat: add giant.sg spider (e76b4e2)

Refactor

• refactor: move coldstorage spider XPath to use locator class (0276702)

Unknown

• Merge pull request #84 from ngshiheng/add-giant-spider

feat: add giant.sg spider (17364d7)

v1.1.10 (2022-04-30)

Chore

• chore(deps): update dependency pylint to v2.13.7 (b980c46)

Documentation

• docs: add disclaimer (db7aede)

Fix

• fix: api query param currentPage to be 0-indexed (b127e13)

v1.1.9 (2022-04-17)

Chore

• chore(deps): update dependency pre-commit to v2.18.1 (222c417)

Fix

• fix(deps): update dependency sentry-sdk to v1.5.10 (2c55e7b)

v1.1.8 (2022-04-12)

Fix

• fix(deps): update dependency sentry-sdk to v1.5.9 (0d2c520)

v1.1.7 (2022-04-10)

Chore

• chore(deps): update dependency pylint to v2.13.5 (58c9498)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.35 (8ec5708)

v1.1.6 (2022-04-06)

Chore

• chore(deps): update dependency pylint to v2.13.4 (9db05f6)

Fix

• fix(deps): update dependency scrapy to v2.6.1 (7c4c677)

v1.1.5 (2022-03-29)

Fix

• fix(deps): update dependency ipython to v8.2.0 (1214d35)

v1.1.4 (2022-03-26)

Fix

• fix(deps): update dependency ipython to v8.1.1 (14ff8b6)

v1.1.3 (2022-03-23)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.32 (6265f7b)

v1.1.2 (2022-03-22)

Fix

• fix(deps): update dependency sentry-sdk to v1.5.8 (fe7237e)

v1.1.1 (2022-03-19)

Chore

• chore: run poetry lock (2afe045)

Documentation

• docs: update semantic commit url (6f2976e)

Fix

• fix(deps): update dependency alembic to v1.7.7 (85f326c)

Style

• style: update order and formatting (cdf6f45)

v1.1.0 (2022-03-17)

Chore

• chore: update renovate schedule to daily (b979bc0)

• chore(deps): update actions/checkout action to v3 (e63a0c2)

• chore: run poetry update (931107a)

• chore(deps): update actions/checkout action to v3 (0aa3b7a)

• chore(deps): update actions/setup-python action to v3 (bae2973)

• chore: retain stale data for 60 days instead of 14 days (aab0211)

• chore(deps): update dependency pre-commit to v2.17.0 (d03b33a)

• chore: remove needs (afa71ab)

• chore: update branches to production (6ed3e07)

• chore: only release on pushes for main branch (3d3667b)

• chore: update script to prune last 14 days of data (446babb)

• chore: update timezone and schedule (4d5295d)

• chore: update timezone and schedule (8657393)

• chore(deps): update dependency autopep8 to v1.6.0 (e2795c8)

• chore(deps): update dependency flake8 to v4 (aa8e5c3)

• chore: add schedule for typical non-office hours (957bc9b)

Ci

• ci: bump poetry version & add python 3.10 to matrix (f8a84ad)

• ci: use relekang/python-semantic-release@master (6b46022)

Documentation

• docs: remove github page (3b2bf5a)

Feature

• feat: use semanticCommits (17abcdb)

Fix

• fix(deps): update dependency sqlalchemy to v1.4.31 (4e7a24d)

• fix(deps): update dependency sentry-sdk to v1.5.4 (d2420a3)

• fix(deps): update dependency ipython to v8.0.1 (4b2757d)

• fix: html layout change (3becc92)

• fix(deps): update dependency colorlog to v6 (c843e60)

• fix(deps): update dependency ipython to v8 (8551978)

Refactor

• refactor: make lazada spider use scraper api proxy (3a55501)

• refactor: log request headers & ip_address (8afa195)

• refactor: add download delay (7af13ce)

• refactor: add ROBOTSTXT_OBEY (0ea189a)

• refactor: rework redmart spider to use lazada api; exclude more mainstream beer brands (b67611c)

• refactor: remove craftbeergsg spider (4334018)

• refactor: use XPATH with page object model (POM) (327f6d2)

Style

• style: add line break (fc70651)

Unknown

• Merge pull request #83 from ngshiheng/renovate/actions-checkout-3.x

chore(deps): update actions/checkout action to v3 (e094e95)

• Merge pull request #82 from ngshiheng/update-ci

ci: add python 3.10 into matrix & bump poetry version (706b4fd)

• Merge pull request #81 from ngshiheng/rework-redmart-spider

refactor: rework redmart spider to use lazada api; exclude more mainstream beer brands (3931504)

• Merge pull request #79 from ngshiheng/renovate/actions-checkout-3.x

chore(deps): update actions/checkout action to v3 (7f9b11a)

• Merge pull request #78 from ngshiheng/renovate/actions-setup-python-3.x

chore(deps): update actions/setup-python action to v3 (ea6d15a)

• Update CNAME (7f69548)

• Merge branch 'production' of github.com:ngshiheng/burplist into production (205c733)

• Merge pull request #74 from ngshiheng/fix-beerforce-spider

fix: html layout change (3ec61eb)

• Merge pull request #71 from ngshiheng/renovate/colorlog-6.x

fix(deps): update dependency colorlog to v6 (55eddb3)

• Merge pull request #68 from ngshiheng/renovate/autopep8-1.x

chore(deps): update dependency autopep8 to v1.6.0 (c32192d)

• Merge pull request #70 from ngshiheng/renovate/flake8-4.x

chore(deps): update dependency flake8 to v4 (dab1fb5)

• Merge pull request #72 from ngshiheng/renovate/ipython-8.x

fix(deps): update dependency ipython to v8 (0926ff4)

v1.0.4 (2022-01-14)

Chore

• chore: create PR for major update (d3c0b49)

• chore: change config file to json5 (dea6b5b)

• chore(deps): update python docker tag to v3.10 (ae7d810)

• chore: add automerge for non-major update (33535e3)

• chore: use pin rangeStrategy (a3a3b50)

• chore(deps): update abatilo/actions-poetry action to v2.1.4 (6384089)

• chore(deps): add renovate.json (098e6f1)

• chore: run poetry update to update lxml (d50f0e4)

Fix

• fix: remove baseBranches (0ff705e)

• fix(deps): pin dependencies (d7c295c)

• fix: spider API requires new query param (a9894cf)

Unknown

• Merge pull request #73 from ngshiheng/add-renovate-bot

chore: create PR for major update (87da684)

• Merge pull request #67 from ngshiheng/add-renovate-bot

chore: add automerge for non-major update (209f28d)

• Merge pull request #66 from ngshiheng/renovate/pin-dependencies

fix(deps): pin dependencies (dc1b921)

• Merge pull request #65 from ngshiheng/add-renovate-bot

chore: use pin rangeStrategy (b075b19)

• Merge pull request #62 from ngshiheng/renovate/abatilo-actions-poetry-2.x

chore(deps): update abatilo/actions-poetry action to v2.1.4 (ed96e30)

• Merge pull request #61 from ngshiheng/renovate/configure

Configure Renovate (9dec6ca)

• Merge pull request #59 from ngshiheng/fix-fairprice-spider

fix: spider API requires new query param (142f5c5)

v1.0.3 (2021-12-07)

Chore

• chore: add unittest to pre-commit hook (c3299b4)

Documentation

• docs: add adsense code (b8afcbb)

• docs: update default view to gallery (96f3c14)

• docs: update README with more detailed information (47a6ca7)

• docs: update favicon and add Google Analytics tag (b4931a2)

Fix

• fix: missing text() (205fb28)

• fix: wrong import (d459be2)

• fix: inconsistent API response where sometimes it returns quantity as a (e86814e)

Refactor

• refactor: use contains (1512397)

• refactor: use XPATH with page object model (POM) (07a45e3)

• refactor: add docker volume for pdb and move command to Dockerfile (1b61db4)

Style

• style: minor nitpicks (081d144)

Test

• test: add basic test cases for parse_quantity (c2ec5ca)

Unknown

• Update CNAME (503d439)

• Merge pull request #58 from ngshiheng/fix-spiders

fix: inconsistent API response where sometimes it returns quantity as a (1ae6410)

• Merge pull request #57 from ngshiheng/refactor-docker

Refactor docker (20278b9)

v1.0.2 (2021-12-03)

Documentation

• docs: add favicon (eb857ce)

• docs: add html title and meta description (6323aa8)

• docs: add index.html (16fa26d)

Fix

• fix: bug where all products have the same image_url (b4677e4)

• fix: filter products with the name containing 'gift' (4b1230b)

Refactor

• refactor: Make stale_days >= 7 instead of > 0 to prevent accidental prune (b70b0e5)

Unknown

• Merge branch 'production' of github.com:ngshiheng/burplist into production (d8020d4)

• Create CNAME (32cc9c0)

• Merge pull request #56 from ngshiheng/ignore-gift-cards

Filter products with the name containing 'gift' (11c5b5e)

v1.0.1 (2021-11-13)

Build

• build: Add docker-compose support (5d39ece)

Chore

• chore: run poetry update (610db6d)

Documentation

• docs: add PR template (22273a0)

Fix

• fix: update next page xpath (16d8012)

• fix: site html layout change (2501f9c)

Unknown

• Merge pull request #55 from ngshiheng/update-site-spider-layout-change

fix: site html layout change (8018c49)

• Merge pull request #54 from ngshiheng/dockerize-app

build:Add docker-compose support (1f467f5)

• Merge pull request #53 from ngshiheng/dockerize-app

Dockerize app (605722e)

• Add multi-stage build Dockerfile (3b77124)

• Run poetry update (a63d762)

• Raise exception if spider doesn't find any existing product (f1fde08)

• Fix xpath and query param (8ae68d2)

• Update README (6fdf610)

• Update README and add MIT license (ef7d1c9)

• Minor refactoring (2169d1b)

• Clean up docs and minor style changes with poetry update (c6460ed)

• Clean up docs and minor style changes (aa25e66)

• Handle case where description only has 2 attributes instead of 3 (e598809)

• Move SENTRY_DSN to environment var (cdd3545)

• Packages update (0a764f6)

• Handle case where description is not deliminated by | (1e15d1d)

• Packages update (b1181d5)

• Add doc description for quantize_price (5563352)

v1.0.0 (2021-10-05)

Unknown

• Merge pull request #52 from ngshiheng/update-currency-type

feat: Update price to numeric type (20856ab)

• Remove str conversion at spider level and compare prices in Decimal (b7a54a2)

• Update price column to be numeric (37fe87b)

• Make alembic autogenerate detects column type change (1013e9d)

• Regular packages update (7983fb7)

• Update lock file (bf2bf22)

• Merge pull request #51 from ngshiheng/auto-add-semantic-release

ci: Add auto semantic release (67dd155)

• Add release job to GA workflow (38e431b)

• Update python version (a3c3f87)

• Packages update (e5fd6db)

• Add auto semantic release (25540ca)

• Update product description xpath to be more flexible (fc6de51)

• Remove unused setup (24b9fc7)

• Add typing (fb8869d)

• Refactor spiders to use ProductLoader (698a824)

• Use scoped session (0324a5e)

• Add test_prevent_remove_stale_products_from_deleting_all (7d1817d)

• Add proxy unit test (d644dab)

• Packages update (f2aeb5c)

• Merge pull request #50 from ngshiheng/scrape-image-urls

Add image_url to product table (d9fa49a)

• Parse image_url for troublebrewing (ac3991b)

• Parse image_url for thirsty (4463292)

• Parse image_url for shopee (a519a6b)

• Parse image_url for redmart (3312648)

• Parse image_url for ishopchangi (cb1c59e)

• Parse image_url for hopshop (25ba411)

• Parse image_url for greatbeerexperiment (4280638)

• Parse image_url for fairprice (6f24070)

• Parse image_url for craftbeersg (91de518)

• Parse image_url for coldstorage (b1e9a03)

• Parse image_url for beerforce (346a9c7)

• Parse image_url for alcoholdelivery (12eb2e5)

• Parse image_url for alcohaul (412bbd1)

• Add image_url to product table (f73edc0)

• Update SQL query for price list (a3cad3e)

• Update github action (f8a3812)

• Merge pull request #49 from ngshiheng/add-mypy

Add mypy and type annotations (2274e28)

• Add mypy and type annotations (93b043d)

• Packages update (4f4caa3)

• Update craftbeersg spider to perform nested crawl (406cfc7)

• Fix incorrect price error (d4f653b)

• Add temporary hotfix to craftbeersg due to page change (609d4e1)

• Remove check json (53aa265)

• Run poetry lock (210b964)

• Update python version for lock file (bbe2afc)

• Update python version in GitHub action (5e10f98)

• Specify python version (f0ec8bf)

• Update packages and troublebrewing spider (cf4124c)

• Fix bug where default raw_name is not returned because remove_spaces may be whitespace (68147b3)

• Update documentation (3d5451e)

• Add str.strip to Item MapCompose (5454dd1)

• Return raw_name when parse_name returns an empty str (80923d6)

• Update spider specific utility functions to be staticmethods (e5dc360)

• Avoid use of generic exceptions (964517d)

• Update shopee spider (9048da6)

• Enable auto-migration upon Heroku release (f8310e0)

• Temporary remove Procfile to run alembic stamp head first (9d688c9)

• Set sqlalchemy.url with DATABASE_CONNECTION_STRING (bb642df)

• Merge pull request #47 from ngshiheng/add-alembic-migration

Add alembic (07f0b14)

• Add Procfile to automate migration (6a4c7f8)

• Create zero state migration for existing database (600340d)

• Initialize alembic (53bbfdb)

• Add alembic (75359f2)

• Update install instruction (02a755c)

• Temporarily remove scrapy check (661b52f)

• Ignore DOWNLOADER_MIDDLEWARES on CI/CD (909ad6c)

• Fix spider contract issue scrapy/scrapy#3222 (6fb6b35)

• Specify an exact Python version (6ac17ee)

• Merge pull request #46 from ngshiheng/migrate-pipenv-to-poetry

Migrate from pipenv to poetry (3d3e40a)

• Update README (b977ebe)

• Update pythonPath (d579b12)

• Migrate from pipenv to poetry (f49e31f)

• Minor code refactoring based on pylint (36ff0a4)

• Polish scrapy contracts (6777f76)

• Polish scrapy contracts (18e7252)

• Add more contracts for existing spiders (cf076ce)

• Add contracts for existing spiders (c161387)

• Update precommit hooks to include scrapy check (b7ccfbf)

• Add precommit hooks (6c1ee10)

• Handle case where listItems do not exist (b13397b)

• Fix where could be returned as or by the API (fbb3363)

• Fix bug where parsers always return None (0e3b1b9)

• Remove the use of commit method when using session context manager (befcd78)

• Use session with context manager (52ed2cb)

• Rename error to exception (1454e2c)

• Move db_connect and create_table to database.utils (3893115)

• Fix bug where exception is not raised when caught (3bb56d1)

• Update Pipfile (7545038)

• Add util functions to get distinct beer brands and styles (450634f)

• Add bash script to prune stale products (70f3029)

• Add scrapy prune custom management command (c87552e)

• Perform some minor code refactoring and update packages (12af5e5)

• Move middlewares into folder (1cd0b74)

• Move models.py to database folder (acd5860)

• Filter out merchandise (49e0cdc)

• Add greatbeerexperiment spider (61ca0c5)

• Fix minor bug (b74b1ee)

• Sanitize POPULAR_BRANDS list (89d7e06)

• Parse brand by product name (14ca2cc)

• Skip out of stock products for alcoholdelivery (4124c5c)

• Fix bug where parse_quantity is called twice in alcohaul (15fab94)

• Add alcohaul spider (84940ad)

• Add a function to delete stale products (59842e3)

• Refactor quantity parser (2532bde)

• Merge pull request #45 from ngshiheng/add-redmart-spider

Revive redmart spider (333d7f1)

• Ignore mainstream beer brands (63f1330)

• Update name parser (1a87793)

• Revive redmart spider (7cdceb9)

• Merge pull request #44 from ngshiheng/add-shopee-spider

Revive shopee spider (f846ac6)

• Add to-do (d42445f)

• Revive shopee spider (fe312fd)

• Merge pull request #41 from ngshiheng/major-refactoring

Major refactoring (ef21f40)

• 🧹 Clean up codebase and add some notes (ea76c70)

• Update hopshop spider with new database model (f9100ac)

• Update craftbeersg spider with new database model (b92ad57)

• Update troublebrewing spider (06a3286)

• Update ishopchangi spider (bd7e95d)

• Add style parser (889806e)

• Update SQL example (2c23f80)

• Remove lazada and shopee spiders for now (d916005)

• Make thirsty spider start crawling from 'browse by beer style page' in order to get beer style (50117ea)

• Use follow_all (ecaf325)

• Update thirsty spider with the latest database model (d74ab1d)

• Update ExistingProductPricePipeline to update product info (808f797)

• Update beerforce spider (71807e7)

• Add fallback when some ABV are displayed only as int (b36c89f)

• Make column nullable (bbcd3b5)

• Remove old tests and brewerkz spider (66543e0)

• Update to support new database models (98a54ed)

• Add more parsers (f689c3d)

• Update database models (00b0da6)

• Remove remdart spider and scrape only craft beer data for some sites (d8025c5)

• Update vendor to platform (7ddcb43)

• Update regex (ff5ebd0)

• Merge pull request #40 from ngshiheng/add-shopee-spider

Add shopee spider (926a7f6)

• Add shopee spider (62b3f97)

• Update reference link (a7f0c18)

• Remove duplicated prices before commiting to DB (cfdefdf)

• Refactor ishopchangi to use regex groups (a480fc0)

• Refactor brewerkz spider to use get_product_name_quantity (d0e7f76)

• Use regex groups (1cea872)

• Add comments about additional product information that could be scraped (bb919d0)

• Make sure product url always starts with https (e4b1d1a)

• Merge pull request #38 from ngshiheng/add-lazada-spider

Add lazada spider (24c8489)

• Remove headers because it is automatically populated (0cc1bca)

• Disable HTTPCACHE for redmart due to failing retries (a210462)

• Fix case where using custom_settings completely left out global DOWNLOADER_MIDDLEWARES (dc62336)

• Make use of productInfo from redmart's API (dcef1cc)

• Add DelayedRequestsMiddleware to add delay in between retries (1be9a51)

• Add get_retry_request to retry requests (2a0bfad)

• Refactoring (9720886)

• Update redmart spider to scrape from a single URL first (8a30fd9)

• Add redmart spider (79bf5a7)

• Add long DOWNLOAD_DELAY for lazada spider (963763b)

• Use middleware to automatically rotate User-Agent in request headers (33609de)

• Add scrapy-fake-useragent (e5c6f6d)

• Make lazada spider use rotating proxy (c011dc2)

• Add lazada spider (9b37aee)

• Add useful SQL queries (a6b5653)

• Merge pull request #36 from ngshiheng/add-ishopchangi-spider

Add ishopchangi spider (02358c0)

• Add test for ishopchangi and fix trailing comma name (441e7e5)

• Add ishopchangi spider (7fbdc12)

• Add index to product name (e4328c9)

• Merge pull request #33 from ngshiheng/add-troublebrewing-spider

Add troublebrewing spider (78e8e73)

• Temporarily disable statsmailer (1fd4a46)

• Add pylint and fix pylint warnings (dc42aab)

• Dependencies update (42d0226)

• Add troublebrewing spider (7d6f426)

• Update README (6bdaa3b)

• Add Sentry warning whenever and have different length (1ca19a3)

• Merge pull request #32 from ngshiheng/add-hopshop-spider

Add hopshop spider (14b16a8)

• Add hopshop spider (e0ca36c)

• Merge pull request #30 from ngshiheng/fix-last-price-bug

Fix CardinalityViolation bug with Product.last_price (76c0e6f)

• Update settings to only send email in production (67e4610)

• Fix CardinalityViolation bug with Product.last_price (069c17c)

• Merge pull request #29 from ngshiheng/add-email-notification

Use StatsMailer extension to send Scrapy info (76c9d0c)

• Use StatsMailer extension to send Scrapy info (2ef73f9)

• Update README instructions (9692be9)

• Uncomment scrapy commands (73c0240)

• Add task.sh for Heroku Scheduler to run spiders (b70f22a)

• Set ROBOTSTXT_OBEY False for thirsty spider (acabfae)

• Fix bug where we constantly run into UniqueViolation because fairprice API returns duplicated item (f983ec4)

• Merge pull request #27 from ngshiheng/update-database-model

Update database model (066122c)

• Fix brewerkz spider bug where it attemps to scrape duplicated data on side panel (a56adee)

• Add UniqueConstraint for Product quantity and url (dd7dd11)

• Merge pull request #23 from ngshiheng/bug-fixes

Fix issue where duplicated products are created when there is a change in price (66d36ed)

• Fix issue where duplicated products are created when there is a change in price (b9aa520)

• Prettier README (720b321)

• Add gif to README (0c4219a)

• Read database name from environment variable (eae84c7)

• Merge pull request #13 from ngshiheng/zyte-deployment

Update Sentry integration and code refactoring (6622534)

• Remove sample output (825ae4e)

• Add additional test case for trailing 's' (58b9a7b)

• Add basic brewerkz spider (7d494d6)

• Use bulk_insert_mappings for even faster performance (8901ac3)

• Allow Sentry transaction and use SqlalchemyIntegration (0c3095f)

• Update README (a81ae4a)

• Update documentations (69ee0b5)

• Merge pull request #11 from ngshiheng/code-refactoring

Code refactoring and improvements (04d7bab)

• Use bulk operation to insert new products and prices to DB (67c0e6f)

• Ignore .scrapy/ generated by httpcache (2808722)

• Enable AUTOTHROTTLE_ENABLED and HTTPCACHE_ENABLED (5d90aab)

• Fix timeout error issue where SQLAlchemy hits QueuePool limit due to unclosed session (a7f8f9f)

• Add column_property to query for last_price and set Sentry environment (5f5e96c)

• Add ipython (597a0da)

• Merge pull request #10 from ngshiheng/color-logging

Color logging (b931c2e)

• Fix bug where discounted product price is not extracted (ee178d5)

• Fix bug where condition check should be 'is True' not 'is None' (b1c44a9)

• Update sample output of all spiders (21d8eef)

• Add unit tests (3bda2e1)

• Set Sentry to use sentry-sdk (16a5f15)

• Add colourized log (07db31a)

• Merge pull request #8 from ngshiheng/add-sentry

Add sentry and price parser (17853b1)

• Add NewProductPricePipeline and DuplicatePricePipeline (c4808b8)

• Update model and add pipeline to add item into database (53a63ec)

• Add SQLAlchemy and some basic pipelines (bfcfbda)

• Use price_parser and fix coldstorage bug where price element has different class name (3f9c3b0)

• Add Sentry (62980c9)

• Merge pull request #7 from ngshiheng/add-pipelines

Update coldstorage spider to scrape next pages (e0388f2)

• Update coldstorage spider to scrape next pages (77fa0a0)

• Merge pull request #3 from ngshiheng/add-beerforce-spider

Add basic spider to scrape from beerforce.sg (f345ba8)

• Clear flake8 errors (80bc7ba)

• Update alcoholdelivery sample output (bd3af22)

• Scrape product quantity from thirsty (720503a)

• Scrape product quantity from fairprice (95e5a59)

• Sanitize craftbeersg beer name and get product quantity (c0e98c9)

• Remove Unit Enum, change unit to quantity (f81765e)

• Move example output folder (8da40ee)

• Fix bug where beerforce spider did not recursively follow the link to next page (c398c72)

• Update scrapy ItemLoader to parse price correctly (4e227ce)

• Filter out kegs (73b89ca)

• Add basic spider for alcoholdelivery.com (f148bb3)

• Add basic spider to scrape from beerforce.sg (e4e45aa)

• Merge pull request #2 from ngshiheng/scaper-api-proxy

Use Scaper API as a proxy management measure (c9f6d3a)

• Use Scaper API as a proxy management measure (482cf70)

• Merge pull request #1 from ngshiheng/add-craftbeersg-spider

Add basic spiders for all main sites (5f153c1)

• Add flake8 and scrapy check to github action (434dd2a)

• Add basic spider to scrape from thirsty.com (2265749)

• Scrape fairprice using their API (f4a142b)

• Add basic spider to scrape from fairprice.com (ab9b996)

• Add scrapy-splash for selecting dynamically loaded content (b9fe2da)

• Add basic spider to scrape from coldstorage.com (c816152)

• Add basic spider to scrape from craftbeersg.com (6aabec3)

• Initialize scrapy project (eade821)

• Initialize project (42e44e5)