Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nx migration, refactor main package as core, publish auxiliary packages instagram & linkedin #92

Merged
merged 45 commits into from
Feb 14, 2021

Conversation

mrWh1te
Copy link
Owner

@mrWh1te mrWh1te commented Jan 22, 2021

Description

The main webpack.config.js file builds the main Botmation npm package.

Goal is to make it dynamic so it can build other npm packages, for starters, the botmation-instagram package.

Solution

Replace developing the build script(s) with Nx.

Botmation v3.0 published as @botmation/core v1.0 & Instagram specific published as @botmation/instagram v1.0

Related

Botmation npm

Checklist

  • Instagram npm package barrel file for all instagram stuff
  • Webpack build dynamic flag (CLI ?)
    • Webpack can dynamically build dist packages
      • instagram
      • core
    • Readme specific to the Botmation-Instagram package (botactions, helpers, etc)
    • dynamic package.json for the npm package (core or instagram)
    • custom tsconfig for instagram
  • Nx repo migration
    • core tests passing nx test core
    • core, instagram & linkedin packages
      • fix tslib importHelper issue
      • publish working v1.0 (drafted with working v0.x)
      • polish readme's
  • Document Prepping docs for new packages release botmation-docs#25
    • build
    • publish process
      • package.json tweaks
    • testing
      • run npm run localtestsite in tandem with nx test core as the local HTTP server is required for some e2e tests

Documentation

Update the Instagram documentation page to include a link to the npm package

Currently referring to core as the main botmation package in the build.

Extra

@codecov
Copy link

codecov bot commented Jan 22, 2021

Codecov Report

Merging #92 (9d87908) into master (2d1a1f5) will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##            master       #92   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           26        32    +6     
  Lines          612       637   +25     
  Branches       164       166    +2     
=========================================
+ Hits           612       637   +25     
Impacted Files Coverage Δ
libs/core/src/lib/actions/abort.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/assembly-lines.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/console.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/cookies.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/errors.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/inject.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/input.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/pipe.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/scrapers.ts 100.00% <ø> (ø)
libs/core/src/lib/actions/utilities.ts 100.00% <ø> (ø)
... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d1a1f5...9d87908. Read the comment docs.

- consolidating with main webpack.config.js that creates the main
  botmation package in /dist
- referring to 'botmation' npm package as 'core'
@mrWh1te
Copy link
Owner Author

mrWh1te commented Jan 23, 2021

It's been a few weeks since I've last written code for this project. I mistakenly created this branch off master instead of dev.

I might delete the dev branch as a tmp fix, but it's only deviated by 3 commits so not a huge deal, but if I'm the only one coding on this project now, there isn't much value in maintaining both branches

food for thought

edit: maybe going forward, delete dev and in the future, when it's worth the cost, do a dev branch except name it something more useful like next-release

- tested locally, had to upgrade types/puppetter
- builds to different dist sub directories
- core is working in local test
- instagram isnt building a types file wip
@mrWh1te
Copy link
Owner Author

mrWh1te commented Jan 28, 2021

need to pull in changes from #93

@mrWh1te
Copy link
Owner Author

mrWh1te commented Jan 31, 2021

exploring Nx mono-repo for lib publishing

thus far, locally, got a nx repo setup with botmation v3.0 building and working (the dist build package installs locally in a separate repo and works fine with doc's installation guide)

setup an org on npm @botmation

intention is to publish botmation 3.0 as @botmation/core

then publish site specific packages as @botmation/instagram etc

@mrWh1te
Copy link
Owner Author

mrWh1te commented Jan 31, 2021

core & instagram packages builds work, dists' codes look good

however in a local test, there is a Module not found error from the instagram package when referencing the peerDependency @botmation/core

in a local test with just the @botmation/core package it works

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 3, 2021

got both packages building in Nx environment and published on npm v0.0.1 @botmation/core and @botmation/instagram

in a local environment, the published packages are missing library tslib ?? but once installed locally, work fine

may switch Botmation over to Nx monorepo

edit:
@botmation/core v0.0.1 is botmation v3.0.0 just built differently (smaller package size)
@botmation/instagram is a set of instagram focused BotAction's, helpers, selectors, constants, etc that work with @botmation/core package

edit2:
main botmation package has shrunk from 450 kB to 173 kB

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 9, 2021

added tests from main repo into the new local nx repo wip
2021-02-08-193153_1920x1200_scrot

edit: see navigation.spec.ts puppeteer working with const puppeteer = require('puppeteer'); instead of import ref

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 9, 2021

linkedin package code setup; punting testing/setting up the dist/npm package

now libs are core, instagram, linkedin, and test

test lib has shareable test files & mock server HTML files

unit test files are now coupled with their files in the respective libs. since all tests for botmation focuses on core, all tests have been moved into that lib, inside /actions, /helpers, /etc -- this is a diversion from the current repo structure that separated tests from the lib code

- libs: core (was botmation v3), instagram, linkedin, test
  - core & instagram can be published but need tslib added
  - linkedin punted for publishing
- nx test core runs all pre-existing tests for environment but requires
  an additional script running `npm run localtestsite`
  - to be integrated with jest in future dev
@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 10, 2021

impressed with how easy it was to migrate botmation to Nx

excited for this upgrade

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 10, 2021

existing CI for automated testing will break and be adjusted in subsequent PR

refraining from scope creep as this PR's focus is publishing an instagram package (now including separate core package under an organization)

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 10, 2021

need to restore GIT files (ie changelog) and other CI config files (ie travis.yml)

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 10, 2021

could publish as minor releases v0.1.x and reserve major v1 for what was "v4"

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 13, 2021

in the future, can consolidate the botmation-docs repo with this one as an app

- remove /assets from example build
- test/server moved into /assets & localtestsite script updated
- mocks & localtestsite (selectors + urls) moved into lib core/mocks
  - with new barrel
- core tests passing
- with mocks included in core lib, no more circular dependency so mock
  can be strongly typed without duplicating code
- example of using puppeteer-cluster
   - works with current Puppeteer v7 (at least in this basic test)
- concurrently npm scripts to run tests
- text for CI logs
- json for code coverage collecting
@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 14, 2021

issue #94 has been resolved; TravisCI passing is no longer a false positive

@mrWh1te
Copy link
Owner Author

mrWh1te commented Feb 14, 2021

current dep-graph:
botmation-nx

circles = libraries
squares = apps i.e. bots

@mrWh1te mrWh1te changed the title botmation-instagram npm package Nx migration, refactor main package as core, publish auxiliary packages instagram & linkedin Feb 14, 2021
- polished readme's
- polished notes
- would like to add e2e tests for auxiliary packages
- @botmation/core, the only peer dependency of an auxiliary botmation
package is the major version anchor for all @Botmation packages
@sonarcloud
Copy link

sonarcloud bot commented Feb 14, 2021

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@mrWh1te mrWh1te merged commit 7fc3706 into master Feb 14, 2021
@mrWh1te mrWh1te deleted the ml-instagram-package branch February 14, 2021 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant