Can no longer search code without being logged in.

[koepnick]

Select Topic Area

Product Feedback

Body

This is revolting and an anathema to the open source movement. A movement, I might add, Microsoft is abusing here.

We're told that this is for security... But what possible point is there when I can simply clone the repository and use more dedicated tools for proper searching and analysis?

So what possible reason is there?! Do you NOT have enough of our data? Is it not enough to monetize every bowel movement, you now feel the need to track which individual lines of code I'm browsing?

I was on an older machine and needed to search for something in OUR OWN REPOSITORY and couldn't. I actually want people to be able to search our codebase.

So what did I have to do? I tried logging in. Didn't have my password manager nearby. So I had to grab my phone. Oh! Now I need to 2FA. So back to my office to grab my Yubi key. The old laptop doesn't have USB-C ports? Well now I'm SOL.

Not only is this change unncecessary, it's downright hostile towards your own customers. Ambitiously hostile!

Obviously we're too far into the Github ecosystem to be able to easily change providers to one that even gives a pretext of user privacy or responsiveness. So kudos for that.

But I'm done with the web interface. If you're not going to even bother hiding the fact that you see us as a source of data and a resource to be extracted, I choose to not give you that information.

This is the final straw. I will no longer be creating new projects on GitHub. I want them to be useful to the public.

Anybody reading this needs to realize something: Every time you create something new here, future audiences will only be able to search your code after bending the knee to Microsoft.

Please, GItHub maintainers! Ignore the marketing jackals and the middle managers. Fight back! This is ethically indefensible and needs to be reversed.

[piperun]

Or find better alternatives that don't do this shady crap.

[winterqt]

This has been happening since before the Microsoft acquisition, as far as I remember. It is certainly not a new limitation either way.

[martinwoodward]

Sorry for the inconvenience @koepnick - while searching across all repos has required being logged in for a long time, when we enhanced the search capabilities earlier in the 2023 we had to extend this to repos as well (see https://github.blog/changelog/2023-06-07-code-search-now-requires-login/).

This is primarily to ensure we can support the load for developers on GitHub and help protect the servers from being overwhelmed by anonymous requests from bots etc.

[rooiratel]

@martinwoodward nice of you to mark your own answer as the "accepted answer". Congratulations.

I also see you have : "I solemnly swear I am up to no good" as your status. That sounds on brand for Microsoft, and this whole clown show.

[neuroradiology]

Google can search the entire Internet, hosted on other people's computers, in an instant, with no throttling, and no logging in.

But Microsoft can't manage to allow anonymous unthrottled searching on a miniscule FRACTION of the web on their OWN SERVERS?

If your technical reasoning is true, it's an indictment of Microsoft's technical capabilities.

[Athari]

@neuroradiology Google requires solving ridiculous CAPTCHAs even for logged in users, if their IP happens to be shared, or if the prompt contains an operator like "site:", or for myriad of other random reasons — multiple times per session. I wouldn't consider it something worth praising.

[teor2345]

This answer also provides some useful context, much more than the one at the top of this thread:
https://github.com/orgs/community/discussions/77046#discussioncomment-7683240

[gingerbeardman]

What a thoroughly disappointing corporate decision.

[revskill10]

Is this technical issue, or political issue ? Rails can scale right ?

[gsisko]

Pretty clear they want to track what devs are looking at.

[ism]

It's naive to think that login requirement will stop any dedicated bot operator.

[nukeop]

No need to stop dedicated bot operators, if it stops 99% of bot traffic by stopping casual bot operators.

[climbit]

@jorendorff
I would be interested in what a bot is actually doing.

• Does it have a good reason for searching? Like you say yourself: "codesearch"
• What is the abuse?
• Or is it just crawling for specific content?

Additionally you stated: "reduces abusive bot activity by something like 90%".

• It would be interesting how you get this number, because obviously github is not able to detect all bots
  • otherwise you would not need this log in solution
  • and you state it yourself: "false positives"
• I would be interested in how much it reduces the rest of the search, which github calls not abusive.
  For example:
  • I will not log in to just search something
  • Also lets say there is any interested human, which is not in the open source community, but maybe it gets more and more convinced. But now it searches something, but will be forced to log in. This must be very frustrating and makes github responsible, that the open source community will shrink. This is in contradiction with the github front page slogan: "The largest open source community in the world".

Every open source code hosted here is not githubs code, but the communities code. And this code makes the platform interesting. If this community would not exist, this platform would have never gained the current popularity.
So in the end what github is saying is, you can host your code here, thanks for your added value, but you may not search it without tracking.
Now, with this decision github restricts the freedom of the open source community and it might be meaningful to think about if github really want to support open source or if this is just a slogan.

[TWiStErRob]

[genuine question, idk, but it feels like it's not a thing.]

Where in the open source world is it clearly documented that a code hosting service has to provide an extremely powerful computationally intensive (the indexing) search for free for everyone?

Examples:

• Sourceforge has no search.
• Google Code has no search (granted it's in archive mode); this hints there was search: old screenshot, unclear if it required login.
• GitLab requires login for all-search, and allows per-repo search. But, I tried on with 4 tiny files and I got 30 seconds of spinner, and no results.
• BitBucket Search redirects to login, but if you can find a repo hosted there, there's search without login.

[climbit]

Well, where is it documented that:

• companies should track there users in every detail
• abuse private data for ridiculous reasons like better target ads (honestly I dont know any reason why private data should be used in any way without permission)
• buy other companies for ridiculous much money and destroy a whole eco system.
• search for every bit how they can tie users to the platform (lock in effect)
• don't care for it if data gets lost and rather hiding it

Best things are not documented but might anyhow worth to do it. If anyone have been at an insider tip of lonely planet...

And the enumerated platforms above have at least in common, that they dont state something like "The largest open source community in the world" in the eye catcher on the front page.

[nukeop]

Where do you see targeted ads on Github?

[wiblaa]

Microsoft is closing the door on open source. Who woulda thought that could happen. Oh right.

[fwuffyboi]

where did he say that?

[nukeop]

In the post above yours.

[fwuffyboi]

can you state exactly where? maybe even quote it?

[nukeop]

No.

[fwuffyboi]

wdym "no"?

[crvdgc]

It's a constant source of nuisance when browsing in private/incognito mode. Surely there are other ways to limit bots, like adding a rate limit and if that's reached, then requiring CAPTCHA or login?

[pr3y]

You can look at https://grep.app/, which allows you to search for repositories without logging in, but even so, this Github policy is bad and will probably get worse over time

[rlove]

I doubt this was a move against open-source. As an Enterprise Customer we needed better search which required this level of security. Granted it could have been made optional but given that AI models of our code are being used for major new Co-Pilot features security had to be heavily considered.

[rooiratel]

Nobody is complaining that they can't search your private repos without a login. This is about opensource code that is publicly available, which people can still clone each repo and do a local text search without a login. This "solution" is just Microsoft being either assholes or incompetent, their only two skills.

[nukeop]

Or maybe they don't want to spend an order of magnitude more on infrastructure just to serve chinese botnets.

[rooiratel]

So the Chinese botnets are going to clone the repos and do a local search instead. That will really save bandwidth instead of just returning the result of a search query.

[nukeop]

Exactly, so that they're not wasting compute and are doing it on their own and it's no longer a denial of service vector. Now you're getting it.

[mbomb007]

Plus the cloned data would become stale pretty quickly

[orliesaurus]

Since this is relevant to the larger topic, I would like to add an anecdote, as a logged in user using search extensively:

• I have been trying to look up results for a specific file - i.e. contributing.md across all open repos
• I have been using this search params: path:contributing.md
  but I only get 5 pages of results.

I wonder why that is and if there's a way to retrieve more than 5 pages?

[chidiwilliams]

It's a current limitation of search and not just your specific query. There's a discussion here: https://github.com/orgs/community/discussions/9868

[demostanis]

https://codeberg.org

[climbit]

nice hint :)

[Zezesheng]

GitHub 的反抓爬理由其实能说得过去，不过这也可能是 GitHub 和微软用来防止某些竞争对手抓取数据来训练 AI 模型，毕竟现在各大公司都在想方设法的限制数据抓取避免数据被人白嫖。
但是，这并不能成为不开源的理由，可以使用对真实用户更友好的办法不是吗？

[Zezesheng]

The anti crawling reasons of GitHub can actually be justified, but it may also be that GitHub and Microsoft use it to prevent certain competitors from crawling data to train AI models, after all, major companies are now trying to limit data crawling to avoid data being wasted.

However, this cannot be a reason for not being open-source, can we use methods that are more user-friendly for real users?

[fwcd]

This makes sense for global search, but not for repo-wide search. As mentioned above, crawlers can easily clone repos anonymously. So why restrict anonymous repo-wide search?

[nukeop]

Because it costs a lot of compute, and cloning doesn't.

[qgustavor]

If code search costs a lot of compute, then, one, it seems to be badly implemented, or, two, if there is some fancy tech that justifies it costing a lot to compute, which tech is this? Do users really need this fancy tech in their every day searches?

Sure, from my experience old GitHub search was bad, IIRC it was something akin to Elasticsearch: you could search keywords, but not code snippets, so it was quite limited. I remember having issues searching code in huge repositories because of that, I had to manually check multiple search results to find a specific file I was looking for when debugging some issue. New search seems to have fixed that.

But, is it too costly to make it a logged-in only feature? If so, it still seems badly implemented to me: they could use the simpler algorithm to handle simple queries (and make it available for logged-off users) and the current one for complex queries (and make it available for logged-in users). Then make it clear "Search results are limited when logged off to save server resources, sign in to get better results."

Then, is it worth the cost supporting two algorithms? If you don't want to support third party GitHub search websites, that's great, if you don't mind those, then keep it as it is, some people will sign in to search, others will give up searching, others will just look for third party alternatives. By giving some results, even if they are not as good as the modern algorithm, IMHO less people would give up searching or look for third party alternatives and would login to check better results. Well, can you AB test that? I don't think so, how you would differentiate people looking for third party solutions from those giving up searching?

[mbomb007]

@qgustavor That's why I wonder if AI will be integrated into GitHub search, because of rate-limiting and cost of AI requests. (And also because every company seems to think it's the coolest thing ever, without seeing all its flaws or caring what users think.)

[nukeop]

Or perhaps it's not so trivial, and you're just not qualified to talk about how to implement it when faced with teams of some of the best engineers in the world trying to tackle this problem with a custom implementation. They wrote a lot about it here: https://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/

[GreyXor]

GitHub, which prides itself on being the world's largest open source community, is owned by Microsoft, a for-profit company. Interestingly, Microsoft's commitment to open source has always been limited, starting with GitHub itself, which is not an open source platform. The situation seems somewhat hypocritical, but given that this is Microsoft, perhaps there's no need to be surprised.

[GrahamSH-LLK]

It wasn't open source before Microsoft bought it.

[frnmst]

I find this feature kind of annoying as well. My current solution to find something quickly, if limited to a single repo is:

1. cd /dev/shm (don't use up precious disk read/writes)
2. git clone https://github.com/user/repo
3. cd repo
4. grep -r "pattern" .
5. cd ..
6. rm -rf repo

If you want to search for a pattern all over GitHub you can use a normal search engine, with something like: "pattern" site:github.com. I don't need to search through all GitHub frequently, but I don't think using an external search engine is as powerful as using GitHub's built-in one. Maybe some code is not indexed. Moreover, this system only works for public repos.

This is primarily to ensure we can support the load for developers on GitHub and help protect the servers from being overwhelmed by anonymous requests from bots etc.

This seems a mitigation to me. What about if someone creates n GitHub bot accounts using n different emails and remains well below API limits for each bot? I think the bot manager would win. It would be much simpler to disable code search altogether (!) Problem solved.

As mentioned in other comments there are other GIT forges (Gitlab (Framagit), Gitea, Forgejo (Codeberg), Gogs, etc...). I think some of these allow you to enable code search if you self-host. Apparently Codeberg and Framagit don't have code search enabled at this moment. These two Codeberg issues seem also interesting.

Personal advice

Not directly related to code search but more broad: my personal advice is to treat GitHub as one of the mirrors of your projects. GIT is by nature decentralized (Distributed Version Control System (DVCS)). If you use GitHub (or any other) as your only code forge you get a single point of failure. You can do something like this:

git remote add github https://github.com/user/repo.git
git remote add codeberg https://codeberg.org/user/repo.git
git remote add framagit https://framagit.org/user/repo.git
git remote add private https://example.com/user/repo.git

git add ...
git commit ...

git push github
git push codeberg
git push framagit
git push private

[joekiller]

Enforcing a login for advanced features makes it easier to stop abuse. Unfortunately it does come at some expense of the user convenience. I could imagine another reason for requiring logins is that it makes the audit log more accurate.

The op could have cloned the repo and grepped if they really needed to. They could've cloned and committed and pushed later. They could've just made another login. They could've been having a bad day and decided to make the post.

I could've not posted on this but here I am lol. If you need search via cli, you can issue an API key for yourself. I dunno, just sounds like it was a bad experience at that moment. Things change, sometimes they're not universally considered "the best". Somehow I doubt this is M$ trying to monitize. More likely they're responding to scaling issues as they stated. Indexing all the code and making that universally accessible eventually is a scaling issue and clobbering the least effort bots with a login need seems to be low effort/high reward.

[Simarilius-uk]

add 1s after the github before the .com. Opens the repo in an online vscode, search is better than the github one anyway.

[TWiStErRob]

Can you give a concrete example please?

[gingerbeardman]

compare

• https://github.com/libretro/px68k-libretro/
• https://github1s.com/libretro/px68k-libretro/

[HansSchulze]

Or simply a grep.github.com for simple searches?

[devmuhib009]

Excellent trick. Hope I will use it often.