Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
4f781da
feat: upgrade backend image to v0.0.4
amhsirak Dec 4, 2024
1a59255
chore: v0.0.5 maxun core
amhsirak Dec 5, 2024
7180305
chore: use v0.0.5 maxun-core
amhsirak Dec 5, 2024
b1e2c30
chore: use v0.0.5 maxun-core
amhsirak Dec 5, 2024
939e7a7
chore: maxun v0.4
amhsirak Dec 5, 2024
1fc7ddc
chore: merge pagination hotfix
amhsirak Dec 5, 2024
236f3ed
chore: merge pagination hotfix
amhsirak Dec 5, 2024
5ed8e8a
fix: use window.location.origin instead of base url
amhsirak Dec 5, 2024
5985bc1
fix: use window.location.origin instead of base url
amhsirak Dec 5, 2024
429ddaa
chore: lint
amhsirak Dec 5, 2024
079863d
feat: add earliest selectors logic for page state
RohitR311 Dec 5, 2024
9649137
fix: add on load emit urlChanged
RohitR311 Dec 5, 2024
95b2c50
chore: proper spacing
amhsirak Dec 5, 2024
4550718
chore: lint
amhsirak Dec 5, 2024
a6f4d04
feat: show maxun version
amhsirak Dec 5, 2024
bbf9699
chore: lint
amhsirak Dec 5, 2024
8a0f2b6
feat: format version chip
amhsirak Dec 5, 2024
76255c2
chore: remove unused chip import
amhsirak Dec 5, 2024
0d66331
fix: add on load socket emit
RohitR311 Dec 6, 2024
bffe838
feat: add earliest selectors from workflow
RohitR311 Dec 6, 2024
0ee50e1
feat: add bottom up workflow traversal
RohitR311 Dec 6, 2024
d6be268
feat: add check to match action url and return
RohitR311 Dec 7, 2024
db37c72
fix: add goto frame navigation
RohitR311 Dec 7, 2024
a8e8c1d
fix: rm about:blank url check for action
RohitR311 Dec 7, 2024
342fd79
feat: add bottom up get selectors logic
RohitR311 Dec 7, 2024
ef571c4
feat: traverse dom tree for parent element selection
amhsirak Dec 7, 2024
9f24e00
feat: !return null
amhsirak Dec 7, 2024
8c4c0b7
feat: handle selector generation if no parent element
amhsirak Dec 7, 2024
be6d8ab
feat: add selectors in bottom up order
RohitR311 Dec 8, 2024
5259e3e
feat: add on flag logic for InterpretRecording
RohitR311 Dec 8, 2024
aec65d1
feat: add flag generation logic
RohitR311 Dec 8, 2024
0a81292
feat: add flag generation logic
RohitR311 Dec 8, 2024
45f0c81
feat: add flag generation logic
RohitR311 Dec 8, 2024
a30211d
fix: add frame navigation logic in place of load
RohitR311 Dec 8, 2024
2ea6438
feat: youtube icon
amhsirak Dec 8, 2024
e2cb4c7
feat: youtube link
amhsirak Dec 8, 2024
99d6210
chore: lint
amhsirak Dec 8, 2024
6975705
feat: x link
amhsirak Dec 8, 2024
7186540
feat: increase width of menu
amhsirak Dec 8, 2024
a8ea055
feat: move discord icon to menu
amhsirak Dec 8, 2024
8d6b962
feat: remove discord icon button
amhsirak Dec 8, 2024
96bd270
feat: add ref to yt x links
amhsirak Dec 8, 2024
f8f1d92
wip: upgrade maxun button
amhsirak Dec 8, 2024
07c1404
refactor: rename menu & tab update functions
amhsirak Dec 8, 2024
37ed5fa
chore: lint
amhsirak Dec 8, 2024
929bd91
feat: store package.json version in currentVersion
amhsirak Dec 8, 2024
3d619ad
feat: margin right to 30px
amhsirak Dec 8, 2024
6a2222e
feat: pass event prop to handle update tab change
amhsirak Dec 8, 2024
06fdfbc
chore: lint
amhsirak Dec 8, 2024
b30be4b
feat: add setup
amhsirak Dec 8, 2024
58aedac
wip: updates ui
amhsirak Dec 8, 2024
ebd866b
feat: add comments for commands
amhsirak Dec 8, 2024
0c66e86
feat: format code blocks
amhsirak Dec 8, 2024
e82863a
chore: lint
amhsirak Dec 8, 2024
f7eccd4
feat: box border radius
amhsirak Dec 8, 2024
885120c
feat: add changelog link
amhsirak Dec 8, 2024
2a7cb08
chore: lint
amhsirak Dec 8, 2024
5fa2deb
chore: -rm unused import
amhsirak Dec 8, 2024
c8b95bd
chore: -rm v
amhsirak Dec 8, 2024
5bda5fc
feat: rename to upgrade maxun
amhsirak Dec 8, 2024
6492405
feat: use update icon
amhsirak Dec 8, 2024
30a425d
feat: match upgrade icon style to rest of navbar elementa
amhsirak Dec 8, 2024
e5c045f
chore: lint
amhsirak Dec 8, 2024
fc7655b
feat: increase margin right
amhsirak Dec 8, 2024
8034dd0
feat: show snackbar to upgrade maxum
amhsirak Dec 8, 2024
e190941
chore: lint
amhsirak Dec 8, 2024
6126227
feat: snackbar ui
amhsirak Dec 8, 2024
dad4a4a
feat: snackbar ui
amhsirak Dec 8, 2024
cbe05c2
feat: snackbar ui
amhsirak Dec 8, 2024
d443503
feat: proper rect and element info
amhsirak Dec 9, 2024
560f0ea
fix: a tags
amhsirak Dec 9, 2024
4da462f
fix: a tags
amhsirak Dec 9, 2024
3dfe911
feat: inject cookie remover script
RohitR311 Dec 9, 2024
549a0d3
chore(deps): install idcac-playwright
amhsirak Dec 9, 2024
117dddc
feat: inject cookie remover script
RohitR311 Dec 9, 2024
308c527
Merge branch 'rect-improve' of https://github.com/getmaxun/maxun into…
RohitR311 Dec 9, 2024
386e7c9
feat: add programmatic click event for clickNext
RohitR311 Dec 9, 2024
7cd621b
Merge pull request #228 from getmaxun/list-url
amhsirak Dec 9, 2024
763bc82
Merge pull request #233 from getmaxun/maxun-v
amhsirak Dec 9, 2024
863ad23
Merge pull request #229 from getmaxun/login-scrape
amhsirak Dec 9, 2024
9023680
Merge branch 'develop' into rect-improve
RohitR311 Dec 9, 2024
d6e822e
Merge pull request #235 from getmaxun/rect-improve
amhsirak Dec 9, 2024
b84e918
fix: inject cookie script on page load
RohitR311 Dec 9, 2024
4f8b1d0
Merge branch 'master' into develop
amhsirak Dec 9, 2024
f561ef7
feat: apply conditional visual containment
amhsirak Dec 9, 2024
14c9626
Merge pull request #238 from getmaxun/cap-list-select
amhsirak Dec 9, 2024
668a670
fix: include visual containment for capture list selection
amhsirak Dec 10, 2024
142c90a
fix: handle context destroyed, frame navigation URL
RohitR311 Dec 10, 2024
c7af54e
feat: add fallback mechanism for click and waitForLoadState action
RohitR311 Dec 10, 2024
1383193
Merge pull request #241 from getmaxun/cap-list-select
amhsirak Dec 10, 2024
59cee8e
Merge pull request #243 from getmaxun/navigation-fix
amhsirak Dec 10, 2024
fd16aff
feat: emit setGetList:false socket event in pagination mode
amhsirak Dec 10, 2024
ebb6e3a
Merge pull request #244 from getmaxun/socket-list-initial
amhsirak Dec 10, 2024
c994072
feat: pass listSelector getRect and getElementInfo
amhsirak Dec 10, 2024
81bbba4
feat: condtionally handle getRect & getelementInfo
amhsirak Dec 10, 2024
a34e865
chore: lint
amhsirak Dec 10, 2024
8533ea5
chore: remove unused imprts
amhsirak Dec 10, 2024
0b1b243
chore: remove todo
amhsirak Dec 10, 2024
fea0c03
chore: cleanup
amhsirak Dec 10, 2024
b72baca
docs: re-add jsdoc
amhsirak Dec 10, 2024
d6e4b88
chore: remove console logs
amhsirak Dec 10, 2024
2135f2e
chore: use node:22-slim as base
amhsirak Dec 11, 2024
08f8684
chore: use BACKEND_PORT
amhsirak Dec 11, 2024
a2b8776
chore: use BACKEND_PORT
amhsirak Dec 11, 2024
b93712b
Merge pull request #246 from getmaxun/socket-list-initial
amhsirak Dec 11, 2024
53e4292
Merge pull request #247 from getmaxun/docker-arm
amhsirak Dec 11, 2024
c2ce3f3
feat: explicitly fetch easylist url
amhsirak Dec 11, 2024
44880cf
feat: explicitly fetch easylist url
amhsirak Dec 11, 2024
a37414f
Merge pull request #248 from getmaxun/docker-arm
amhsirak Dec 11, 2024
53cb428
chore: core v0.0.6
amhsirak Dec 11, 2024
6323048
chore: use core v0.0.6
amhsirak Dec 11, 2024
ab2c32c
chore: use core v0.0.6
amhsirak Dec 11, 2024
337ad21
feat: update description
amhsirak Dec 11, 2024
2025d09
feat: rename to website to api
amhsirak Dec 11, 2024
319f9fc
feat: ensure swagger is accessible with or without build
amhsirak Dec 11, 2024
f3c5969
Merge pull request #249 from getmaxun/swagger-fix
amhsirak Dec 11, 2024
6219837
chore: 0.0.6 BE img, 0.0.3 FE img
amhsirak Dec 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ COPY vite.config.js ./
COPY tsconfig.json ./

# Expose the frontend port
EXPOSE 5173
EXPOSE ${FRONTEND_PORT:-5173}

# Start the frontend using the client script
CMD ["npm", "run", "client", "--", "--host"]
4 changes: 2 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ services:
#build:
#context: .
#dockerfile: server/Dockerfile
image: getmaxun/maxun-backend:v0.0.5
image: getmaxun/maxun-backend:v0.0.6
ports:
- "${BACKEND_PORT:-8080}:${BACKEND_PORT:-8080}"
env_file: .env
Expand Down Expand Up @@ -72,7 +72,7 @@ services:
#build:
#context: .
#dockerfile: Dockerfile
image: getmaxun/maxun-frontend:v0.0.2
image: getmaxun/maxun-frontend:v0.0.3
ports:
- "${FRONTEND_PORT:-5173}:${FRONTEND_PORT:-5173}"
env_file: .env
Expand Down
2 changes: 1 addition & 1 deletion maxun-core/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "maxun-core",
"version": "0.0.4",
"version": "0.0.6",
"description": "Core package for Maxun, responsible for data extraction",
"main": "build/index.js",
"typings": "build/index.d.ts",
Expand Down
10 changes: 5 additions & 5 deletions maxun-core/src/browserSide/scraper.js
Original file line number Diff line number Diff line change
Expand Up @@ -283,13 +283,13 @@ function scrapableHeuristics(maxCountPerPage = 50, minArea = 20000, scrolls = 3,
} else if (attribute === 'innerHTML') {
record[label] = fieldElement.innerHTML.trim();
} else if (attribute === 'src') {
// Handle relative 'src' URLs
const src = fieldElement.getAttribute('src');
record[label] = src ? new URL(src, baseUrl).href : null;
// Handle relative 'src' URLs
const src = fieldElement.getAttribute('src');
record[label] = src ? new URL(src, window.location.origin).href : null;
Comment on lines +286 to +288
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Enhance URL handling with additional checks

The URL construction should handle invalid URLs and include additional security checks.

-              const src = fieldElement.getAttribute('src');
-              record[label] = src ? new URL(src, window.location.origin).href : null;
+              const src = fieldElement.getAttribute('src');
+              try {
+                record[label] = src ? new URL(src, window.location.origin).href : null;
+                // Validate URL protocol
+                if (record[label] && !['http:', 'https:'].includes(new URL(record[label]).protocol)) {
+                  record[label] = null;
+                }
+              } catch (error) {
+                console.error(`Invalid URL: ${src}`);
+                record[label] = null;
+              }

Apply similar changes to the href handling:

-              const href = fieldElement.getAttribute('href');
-              record[label] = href ? new URL(href, window.location.origin).href : null;
+              const href = fieldElement.getAttribute('href');
+              try {
+                record[label] = href ? new URL(href, window.location.origin).href : null;
+                // Validate URL protocol
+                if (record[label] && !['http:', 'https:'].includes(new URL(record[label]).protocol)) {
+                  record[label] = null;
+                }
+              } catch (error) {
+                console.error(`Invalid URL: ${href}`);
+                record[label] = null;
+              }

Also applies to: 292-292

} else if (attribute === 'href') {
// Handle relative 'href' URLs
const href = fieldElement.getAttribute('href');
record[label] = href ? new URL(href, baseUrl).href : null;
record[label] = href ? new URL(href, window.location.origin).href : null;
} else {
record[label] = fieldElement.getAttribute(attribute);
}
Expand Down Expand Up @@ -346,5 +346,5 @@ function scrapableHeuristics(maxCountPerPage = 50, minArea = 20000, scrolls = 3,

return results;
};

})(window);
Loading